Significant, but Below the MDE
You sized the test for a minimum detectable effect, ran it to its planned duration, and the result came back significant — but the observed lift is smaller than the MDE. It’s tempting to call the win untrustworthy. That comparison is the mistake, and roughly half of true-on-the-MDE wins land there by design.
The minimum detectable effect (MDE) is the smallest lift we can reliably detect, where “reliably” is set by the power parameter. An MDE of $50 at 80% power means that if the true effect is exactly $50, the test has an 80% chance of returning a statistically significant result. So when we design the experiment, we work out how long it must run to detect an effect we care about at the target power, run it to that planned duration, and look at the result once.
The tricky part comes once that planned duration is reached and the results are significant. It is tempting to compare the observed lift to the MDE and, if the observed lift comes out smaller, to conclude the result can’t be trusted. That comparison is the mistake.
Why it happens
The reason is that the observed lift is not the true
effect — it is one noisy estimate, the true effect plus random sampling
error. Because that error is symmetric, the observed lift falls below
the MDE in half of all tests even when the true effect sits
exactly on the MDE. And the observed lift needed to clear the
significance bar — the cutoff line in the figure, the
minimum observed difference (MOD) — is only
zα / (zα + zβ) × MDE,
about 0.70×MDE at 80% power in a two-sided test. So a whole band of
observed lifts sits below the MDE yet is fully significant.
Run the simulation below. Each bar pools thousands of hypothetical fixed-sample tests, one sample per test. Set the target power the test was designed for, then drag the true effect (as a % of the MDE) and watch how many significant wins — the amber bars — land beneath the MDE line.
How to read it: each bar pools thousands of simulated fixed-sample tests, one “sample” per test. The x-axis is the observed lift; the dashed lines mark zero, the significance cutoff (the MOD), and the MDE. Set the target power the test was designed for, then drag the true effect (as a % of the MDE) to watch where observed results fall.
What to take away
The takeaway: a significant result whose observed lift is smaller than the MDE is the normal, expected case — not a sign the test is broken or underpowered. The MDE’s job is to plan the experiment; once the planned duration is reached, it should not be compared to any effect observed in the data. Significance answers “is there an effect?”; whether the effect is big enough is a separate, practical-significance question. The simulation above makes the point concrete: each sample is one hypothetical fixed-sample test, and the counters show how many of your significant wins land beneath the MDE.
References
- Zhou, J., Lu, J., & Shallah, A. (2023). All about sample-size calculations for A/B testing: novel extensions and practical guide. CIKM ’23. arXiv:2305.16459.
- Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: the pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. PDF.