When I first started working with product experiments, statistical terms like p-values felt more complicated than they needed to be.

I’d see statements like, “The experiment has a p-value of 0.03,” and the immediate question was:

“So, did the experiment work?”

Not necessarily.

Understanding p-values isn’t about becoming a statistician. For Product Managers, it’s about knowing what the number can tell you, what it can’t, and how much confidence you should have in an experiment’s result.

What Is a P-Value?

In simple terms, a p-value helps you understand how surprising your observed result would be if there were actually no difference between the groups.

Imagine you’re testing a new onboarding flow.

• Control conversion: 20%
• New flow conversion: 23%

You observe a 3 percentage-point improvement.

But there is always some randomness in an experiment. The question is whether this difference is large enough, relative to the noise in the data, to make us question the idea that there is actually no difference.

That’s where the p-value comes in.

A smaller p-value means the observed result would be less likely under the assumption that there is no real difference.

The Common 0.05 Threshold

You will often hear Product Managers say:

“The result is statistically significant because p < 0.05.”

A p-value below 0.05 is commonly used as a threshold for statistical significance.

For example:

p = 0.03

This means that, under the assumptions of the statistical test and assuming no true difference, results at least this extreme would occur about 3% of the time.

It does not mean there is a 97% probability that your hypothesis is correct.

That’s one of the most common misunderstandings.

Statistical Significance Doesn’t Mean Product Significance

Suppose an experiment increases conversion from 20.00% to 20.15%, with a p-value of 0.01.

Statistically significant?

Potentially, yes.

Meaningful for the business?

Maybe not.

If the improvement is only 0.15 percentage points, the engineering effort, operational complexity, or customer impact might not justify shipping it.

This is why Product Managers should look at effect size alongside statistical significance.

A statistically significant result can still be practically insignificant.

What If the P-Value Is High?

Now imagine your experiment produces:

p = 0.42

You shouldn’t conclude:

“The new experience doesn’t work.”

A high p-value means you don’t have strong enough evidence to reject the null hypothesis under the chosen statistical framework.

There could be no meaningful effect.

Or the experiment might not have had enough participants to detect the effect.

Or the actual effect might simply be smaller than expected.

The correct conclusion is usually closer to:

“We don’t have sufficient evidence to conclude that the variants differ.”

That distinction matters.

Don’t Keep Checking Until You Get a Good P-Value

One temptation is to launch an experiment, check the results every few hours, and stop as soon as p < 0.05.

This can increase the chance of finding a statistically significant result simply because you kept looking until something appeared significant.

Similarly, repeatedly testing different metrics or user segments until one produces a good result can create misleading findings.

Good experimentation requires deciding your methodology and success criteria before looking at the final results.

P-Values Are Not the Whole Experiment

A p-value is one piece of evidence.

As a PM, you should also ask:

• How large is the effect?
• Is the result directionally consistent?
• How large was the sample?
• Was the experiment properly randomized?
• Were there important segments behaving differently?
• Did the experiment affect guardrail metrics?
• Is the result meaningful to customers?
• Is the improvement worth the cost of implementation?

The p-value shouldn’t make the product decision for you.

It should inform the decision.

Final Thought

P-values can look intimidating because they’re presented as statistical concepts.

But the Product Manager’s job isn’t to memorize formulas.

It’s to understand the evidence well enough to avoid making false conclusions.

A p-value can help answer:

“Could this result reasonably be explained by randomness?”

It cannot answer:

“Should we definitely ship this?”

That’s still a product decision.

And that distinction is exactly why statistical literacy matters for Product Managers.