How to measure a website change with a holdout group

• One conversion you can count for every visit, such as a sign-up, a demo request or an order
• A visitor id that survives between visits, for example a first-party cookie or a local storage value
• A way to show the original or the changed version depending on that id
• Enough traffic to reach at least 100 sessions in each group and 10 conversions overall, and usually far more

1. Pick one primary conversionChoose the single action the change is meant to produce and write it down before the test starts. Conversion rate is sessions that completed that action divided by all sessions.You can watch other numbers. Only the primary one decides whether the change worked, because the more metrics you allow to decide, the more likely one of them moves by chance.
2. Set the holdout share and the run length up frontA 10% holdout exposes most visitors to the change and takes longer to read, since the smaller group collects conversions slowly. An even split reaches an answer fastest.Fix the end date now. Estimate how many weeks it takes the holdout to collect a few hundred conversions, round up to whole weeks so weekdays and weekends are both covered, and commit to it.
3. Assign by visitor and keep it stickyHash the visitor id to a number between 0 and 1 and put everyone below the holdout share in the holdout. The same id always gives the same number, so a person gets the same site on every page and every return visit without you storing the group anywhere.Include a salt for the test in the hash. A new salt reshuffles everyone, which is what you want for the next test and never during this one.

   // FNV-1a hash of "salt:visitorId", mapped to [0, 1)
   function bucket(key) {
     let h = 0x811c9dc5;
     for (let i = 0; i < key.length; i++) {
       h ^= key.charCodeAt(i);
       h = Math.imul(h, 0x01000193);
     }
     return (h >>> 0) / 4294967296;
   }
   
   const HOLDOUT = 0.1;
   const group = bucket('pricing-test-1:' + visitorId) < HOLDOUT ? 'holdout' : 'changed';

   Copy

4. Count every session in its assigned groupRecord the group with each session and each conversion. A visitor assigned to the changed site counts in that group even if they bounced before the changed section loaded or never scrolled to it. Statisticians call this intent to treat.It feels like it dilutes the result. It is also the only comparison where the two groups differ by chance alone.
5. Work out relative lift and its intervalOn the end date, divide the changed group’s conversion rate by the holdout’s. Subtract 1 and you have relative lift. Put a 95% confidence interval around it using the log of the ratio, as below; a two-proportion z-test on the same counts gives the matching p-value.Report the interval alongside the lift. If it spans zero, the test has not shown an effect in either direction.

   changed:  9,000 sessions, 360 conversions  -> 4.0%
   holdout:  1,000 sessions,  32 conversions  -> 3.2%
   
   ratio          = 4.0 / 3.2 = 1.25        -> lift +25%
   SE of ln ratio = sqrt(1/360 - 1/9000 + 1/32 - 1/1000) = 0.181
   95% interval   = exp(ln 1.25 +/- 1.96 * 0.181) = 0.88 to 1.78
                  = lift between -12% and +78%
   
   z-test p-value = 0.22. Not conclusive: 32 holdout conversions is too few.

   Copy

• Group sizes match the split you set. With a 10% holdout and 10,000 sessions, expect close to 1,000 in the holdout. A clear gap is a sample ratio mismatch, a symptom of a data problem, and the result should not be trusted until you find the cause.
• Load the site twice with the same visitor id and confirm you land in the same group both times.
• Force a holdout id and confirm the page is the untouched original, with no flash of the changed version.

Peeking

Checking daily and stopping the first time the result crosses 95% produces false wins, because random swings cross that line sooner or later. Treat anything before the end date as directional and decide on the date you set.

Comparing only the people who saw the change

Whether someone scrolls to the changed section depends on how engaged they are, so that subgroup already converts better than average. Compare whole assigned groups. If you want a closer look at one section, compare it with holdout visitors who reached the same section, and treat that as a hint.

Changing the split mid-test

Moving from 90/10 to 50/50 shifts some visitors between groups, and their earlier sessions sit in the wrong one. Start a new measurement period with a new salt and a new start date, and keep the two periods apart.

Too few conversions

A big lift on a handful of conversions is noise. Hold off on any lift figure until each group has 100 sessions and the test has 10 conversions, and expect to need hundreds per group before the interval is narrow enough to act on.

Checked on 2026-11-11 against Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (KDD 2019), Google Search Central: Minimize A/B testing impact in Google Search.