One of the first things I learned about experimentation was to wait for statistical significance.
If an A/B test didn’t reach the magic threshold, the answer seemed simple:
“The experiment didn’t work.”
But product decisions rarely feel that simple.
Sometimes an experiment shows a promising improvement but doesn’t have enough data yet. Sometimes the result is technically significant but too small to matter. And sometimes we have strong evidence that one option is better, even if we’re not 100% certain.
That’s where Bayesian experimentation changed the way I think about product decisions.
It doesn’t ask, “Did we prove this hypothesis?”
It asks a more practical question:
“Given what we’ve observed so far, what should we believe and what should we do next?”
What Is Bayesian Experimentation?
At a high level, Bayesian experimentation is a way of updating our beliefs as new evidence becomes available.
Before an experiment starts, we have an assumption about what might happen. That’s our prior belief.
We then run the experiment and collect data.
The results update our belief, giving us a posterior belief.
In simple terms:
Prior belief + New evidence = Updated belief
This sounds straightforward, but it creates a very different way of thinking about experiments.
We’re not trying to reach an absolute “yes” or “no.”
We’re estimating how likely different outcomes are based on the evidence we have.
A Product Example
Imagine we’re testing a new onboarding flow.
Our existing activation rate is around 30%.
We believe the new flow could increase activation.
After running the experiment, the new version shows a 33% activation rate.
A traditional approach might ask:
“Is the result statistically significant?”
If the answer is no, we might conclude that there’s no meaningful difference.
A Bayesian approach asks something slightly different:
“What is the probability that the new onboarding flow is better than the existing one?”
Maybe the analysis suggests there’s an 85% probability that the new version performs better.
That’s useful information.
It doesn’t mean we should automatically launch it.
But it gives us a much clearer picture of what the evidence currently suggests.
Bayesian Thinking Fits Product Decisions
This is probably what I like most about Bayesian experimentation.
Product decisions rarely happen in perfectly controlled environments.
We don’t always have unlimited traffic.
We don’t always have months to run an experiment.
And we rarely need absolute certainty.
Instead, we’re constantly making decisions with incomplete information.
Should we continue the experiment?
Should we launch to more users?
Should we abandon the idea?
Should we run another test?
Bayesian thinking gives us a framework for making those decisions while explicitly acknowledging uncertainty.
You Can Incorporate What You Already Know
Another interesting aspect of Bayesian analysis is the use of prior knowledge.
Suppose we’ve run five experiments around onboarding before.
We already know certain patterns about how users behave.
That knowledge doesn’t have to be ignored when starting the next experiment.
It can inform our initial assumptions.
This is particularly useful for mature products where teams have accumulated years of customer and product data.
We’re not starting from zero every time.
But Bayesian Doesn’t Mean “Anything Goes”
There’s a common misunderstanding that Bayesian experimentation lets Product Managers look at results whenever they want and stop an experiment as soon as they like the numbers.
That’s not the point.
Good experimentation still requires discipline.
You need a clear hypothesis, defined metrics, appropriate sample sizes, and a thoughtful decision framework.
Bayesian methods don’t eliminate statistical rigor.
They provide a different way of interpreting evidence.
Probability Isn’t the Same as Certainty
This distinction is important.
If your analysis says there’s a 90% probability that Version B is better, that doesn’t mean Version B is definitely better.
There’s still uncertainty.
But now you can make that uncertainty explicit.
For a low-risk change, 90% confidence might be enough to move forward.
For a high-risk pricing change affecting millions in revenue, you may want much stronger evidence.
The right decision depends on the cost of being wrong.
Bayesian vs Frequentist Thinking
I don’t think Product Managers need to treat this as a battle between two statistical philosophies.
Frequentist approaches can be extremely useful.
Bayesian approaches can be extremely useful too.
The important thing is understanding what question you’re trying to answer.
If you need to determine whether an observed effect is unlikely under a specific statistical assumption, frequentist methods can help.
If you want to reason about the probability of different outcomes given the evidence you’ve collected, Bayesian methods can be very intuitive.
The Real Value Is Better Decision-Making
For me, Bayesian experimentation isn’t really about using a more sophisticated statistical technique.
It’s about changing the conversation.
Instead of:
“Did we win the experiment?”
We start asking:
“What did we learn?”
“How strong is the evidence?”
“What is the risk of acting now?”
“What would we need to learn before making a bigger decision?”
Those are much more useful product questions.
Final Thought
Experimentation isn’t about eliminating uncertainty.
That’s impossible.
It’s about reducing uncertainty enough to make a better decision.
Bayesian experimentation provides a practical way to do that by continuously updating what we believe as new evidence arrives.
And perhaps that’s the biggest lesson I’ve taken from it:
Good Product Managers don’t wait for perfect certainty. They learn how to make better decisions with the evidence available.
Leave a Reply