An A/B Test Needs a Stopping Rule Before the Dashboard Starts Looking Convincing

A new checkout design moves ahead on the experiment dashboard. The team wants to stop before the advantage disappears. Yesterday, the same team said it would keep collecting data because the result was inconclusive. That asymmetry is a clue: the outcome is beginning to choose the amount of evidence the team will collect.

The issue is not whether people are allowed to look at data. Teams need to notice broken tracking, allocation problems and customer harm. The statistical issue is whether the inferential method remains valid under the rule used to stop and declare a winner.

This evergreen guide, checked 30 October 2026, explains the decision with an original conversion example. It draws on Johari and colleagues' research on always-valid A/B inference, which addresses the mismatch between conventional fixed-sample analysis and outcome-driven continuous monitoring. No experiment result or p-value below is presented as a real business finding.

Define the decision before choosing the display

Start with a question the experiment can answer: for eligible users assigned to a design, how does a specified conversion outcome change over a specified follow-up period? Record the randomization unit, eligibility rules, primary outcome and analysis population.

Then define what would justify an action. A tiny conversion change might not cover implementation costs; a larger change might arrive with more payment failures. A winning label should therefore connect to effect size, uncertainty and operational consequences.

The American Statistical Association's 2016 statement cautions against making business or scientific decisions solely from a threshold crossing. Statistical significance does not measure practical importance, and a p-value is not the probability that the hypothesis is true.

Fixed-horizon analysis needs the planned horizon

In a fixed-horizon design, the team plans the analysis for a specified amount of information, commonly using a sample-size calculation tied to assumptions about the outcome and the effect worth detecting. It also needs a calendar plan that respects outcome maturation and business cycles.

Running the planned test once at its intended analysis point is different from repeatedly applying the same fixed-sample threshold and stopping at the first favorable result. The latter adds a selection rule that the ordinary one-look calculation did not account for.

The Johari paper develops methods designed for monitoring, rather than treating this mismatch as a reason never to inspect an experiment. Its relevance is the choice of statistical procedure, not a guarantee that all sequential tests are faster or that every dashboard implements them correctly.

Audit an original changing-leader example

Suppose an original fictional test is planned to observe 10,000 eligible visitors per arm, with one fully observed conversion opportunity per visitor. The following counts are cumulative and invented. They describe a possible arithmetic path, not evidence that either design caused a change.

The remaining 9,000 control visitors therefore contribute 900 conversions, while the remaining new-design visitors contribute 780. Both rows are consistent with nonnegative additions to the same cumulative totals.

Stopping at the first checkpoint because the estimate looks appealing would tell a different descriptive story from observing the planned horizon. But this table alone does not calculate a p-value, determine statistical significance or demonstrate the repeated-testing error rate. Those conclusions require an appropriate model and analysis.

Sequential inference is a design, not a dashboard refresh rate

A sequential method explicitly accommodates the permitted monitoring and stopping behavior. Some designs use prespecified interim looks with adjusted boundaries. Others use procedures that remain valid across a broader set of stopping times. Their assumptions and outputs need to match the experiment.

Howard and colleagues' confidence-sequence research defines sequences of intervals with joint coverage across time under specified conditions. That differs from calculating an ordinary confidence interval each morning and assuming the entire sequence has the advertised one-time coverage.

An organization adopting sequential inference should document the actual implementation, outcome model, allocation assumptions, permitted decisions and treatment of multiple metrics. Calling an interval “live” or “Bayesian” does not, on its own, answer those design questions.

Separate safety monitoring from declaring superiority

A team should not continue exposing users to an obvious operational defect merely to finish an experiment. It can stop a broken or harmful rollout for operational reasons while accurately describing why it stopped.

The original policy distinction might be: stop immediately for a confirmed payment-processing failure, investigate a tracking outage, and assess conversion superiority only through the prespecified inferential procedure. Record these reasons separately.

If a test is stopped early for an unplanned reason, report the deviation. Do not silently relabel the last attractive checkpoint as the intended final result. Exploratory observations can inform a new test without being presented as a completed confirmatory experiment.

Include outcome maturity and assignment integrity

A visitor arriving shortly before a reporting cut-off may not yet have had the same opportunity to convert as someone assigned earlier. Define the outcome window and when a record becomes mature enough for analysis. Otherwise, apparent changes can reflect different follow-up periods.

Likewise, verify that an eligible user remains in the intended assigned group and that missing events or duplicate identifiers have not distorted the comparison. A sophisticated stopping rule cannot rescue the wrong numerator or denominator.

For a related analysis risk, see data leakage in machine learning: information crossing a supposedly protected evaluation boundary can make evidence look stronger than the process actually supports. Experiment design has its own boundaries around assignment, outcomes and decisions.

Write a reviewable experiment contract

Before launch, create a short record covering the primary outcome, minimum business-relevant effect, sample or sequential design, planned duration, outcome window, guardrails, exception rules and reporting responsibilities. Specify whether secondary outcomes are exploratory or part of an adjusted multiple-testing plan.

When reporting, show the effect estimate in meaningful units, the appropriate uncertainty measure and the actual stopping reason. Explain deviations, incomplete outcomes and exclusions. If the result is inconclusive, distinguish insufficient information from evidence of practical equivalence; those are different claims.

The useful habit is simple: choose how evidence will govern the decision before the emerging result starts governing the evidence. A well-designed sequential test can permit flexibility. An improvised stopping decision cannot acquire that validity merely because the dashboard looks persuasive.

Sources and evidence limits

• Johari, Koomen, Pekelis and Walsh, Always Valid Inference, published online 10 August 2021, Operations Research; abstract and bibliographic record reviewed.
• Howard and colleagues, Time-uniform confidence sequences, author manuscript associated with the April 2021 Annals of Statistics paper; abstract and publication record reviewed.
• American Statistical Association, P-value statement announcement, 7 March 2016; six principles reviewed.

The synthetic counts establish arithmetic only. This article does not supply a validated sequential-testing implementation, sample-size recommendation or calculation of the significance of its invented example.