Don’t Condition on a Collider

You A/B-test a checkout with an upsell step (treatment) against the plain checkout (control), splitting visitors at random. You expect a tradeoff: the extra step adds friction, so conversion dips, but the people who do buy add the upsell, so revenue per buyer rises. The bet is that the per-buyer gain outweighs the conversion loss. To check, you do the natural thing — compare revenue per buyer across the two arms. That comparison conditions on “purchased,” a collider, and it doesn’t just measure your upsell: it quietly changes who is in each group.

Why it breaks the analysis

A shopper’s basket intent (B) drives both whether they buy and how much they spend. The upsell fires two arrows at once: it genuinely lifts each buyer’s basket (a real positive effect on order value) and it cuts conversion (friction, a negative effect on “purchased”) — and the friction scares off the small-basket, price-sensitive shoppers first. So the buyers it drives away are the low spenders, which means the buyers left in the treatment arm are a richer crowd than control’s. Condition on “purchased” — restrict to buyers — and you open a path that randomization had severed (T → S ← B, where S is the collider). The observed gap stops being the upsell’s effect alone:

revenue-per-buyer(upsell | bought) − revenue-per-buyer(plain | bought)
    =  real upsell effect  +  ( basket(upsell | bought) − basket(plain | bought) )
                              └──────────────── selection bias ───────────────┘

In the demo the real per-buyer lift is +$7, but the gap among purchasers reads +$14 — the extra ~$7 is pure selection bias from the low spenders bailing out. Meanwhile the metric you actually care about, revenue per visitor (which counts the non-buyers as $0 and is collider-free), has slipped below zero. Worse, this is bias, not noise: more traffic just tightens the confidence interval around the wrong number.

Drag the slider below to set the upsell’s friction — the share of would-be buyers it scares off. At 0% it adds no friction, so the revenue-per-buyer gap is the honest +$7. Push it up and conversion falls, the small baskets drop out, the per-buyer gap inflates, and revenue per visitor goes red.