The Eligibility Trap in A/B Testing

When you run an A/B test, randomization is what makes the comparison fair: split users evenly between control and treatment, and the two groups start out statistically identical. Any difference you measure afterward can then be attributed to the change you made. But this guarantee is fragile. It quietly breaks when the two groups are not logged into the analysis on equal terms — and one of the most common ways that happens is an eligibility check applied to only one side of the experiment.

Picture a feature that should only affect premium users. The natural-seeming implementation puts an is_premium_user check in front of the treatment code, and logs an exposure event whenever a treatment user passes it. Control, meanwhile, has no new feature to gate, so it logs exposure for everyone. The result is an asymmetry that's easy to miss: control's logged population is all users, while treatment's logged population is premium users only. Randomization split the traffic 50/50 upstream, but by the time you look at who actually shows up in each group, the split is gone.

This corrupts the analysis in two ways. First, it produces a sample-ratio mismatch — control ends up with noticeably more logged users than treatment, even though assignment was even. That imbalance is a red flag that the experiment can't be trusted as-is. Second, and more dangerously, the two groups are no longer comparable populations. Premium users typically spend more, engage more, and behave differently in almost every metric. So treatment's average is computed over a richer slice of users than control's, and the gap between them mixes the real effect of your feature with the difference between “everyone” and “premium-only.” A feature that does nothing at all can look like a big win, or a real improvement can be hidden — and collecting more data only sharpens your estimate of the wrong number.

The fix is to make the eligibility check symmetric: evaluate it first, before logging exposure, and apply it identically in both arms. Control should run the same is_premium_user check and log an exposure only when it passes — even though control's experience doesn't change. Both groups then contain exactly the same kind of user, the sample ratio returns to its designed balance, and the metric comparison becomes a true apples-to-apples measurement of your feature's effect. The guiding principle is simple: whatever decides who enters the analysis must be applied the same way to control and treatment, and it must depend only on facts that exist before the experiment touches the user.