Statistics and verdicts

SplitHub Cloud turns per-variant counts into a verdict. The same analysis answers the CLI, the MCP server, the dashboard, the reports and the emails, so every surface quotes the same numbers. A self-hosted endpoint returns the raw counts (visitors, conversions, rate, revenue per variant) with analysis: { available: false }; verdicts, stars and the rest of this page come with SplitHub Cloud.

Peeking-safe by design#

Agents and people check results all the time. A classic p-value read on every check and acted on at the first significant moment inflates false positives several times over. SplitHub decides with an always-valid sequential test (a mixture sequential probability ratio test): its p-value and confidence interval stay valid however often you look, so you can check any time and stop as soon as the verdict says so. Verdicts depend on sample size, not on days.

What is computed#

For each variant: unique exposed visitors, converted visitors, rate, revenue (when the goal has it), and the chance to be the best variant. For each non-control variant against the control:

Field	Meaning
absoluteLift	rate − control rate, as a fraction; multiply by 100 for percentage points
relativeLift	(rate − control rate) / control rate
sequential.pValue, sequential.ci	always-valid p-value and 95% interval of the absolute difference: what decisions use
relativeCi	the always-valid interval as a relative lift
significant	the always-valid p-value is below 0.05 after Holm correction across the comparisons
pValue, ci, zScore	the fixed-horizon two-proportion z-test, for reference only
probabilityToBeat	probability that the variant's true rate is above the control's (normal approximation)

Plus the sample-ratio-mismatch check: a chi-square test of the observed split against the configured weights, flagged when p < 0.001. When a targeting flow's conditions decide who reaches which variant, the expected split depends on who visits, so the check is skipped and verdict.notes says so.

The verdict#

status	When	What to do
no_data	no exposures yet	wait, or check the setup with splithub doctor
insufficient_sample	fewer than 100 visitors in a variant or fewer than 50 conversions in total, or the sequential test has not decided yet	keep running; the message says about how many visitors per variant will settle it
srm_detected	the split does not match the weights	the setup is broken (bots, caching, a variant that errors, a redirect loop); fix it, restart as a new experiment
winner_found	a variant beats the control, or the control beats a significantly worse variant	read winner and message, then finish the experiment
no_significant_difference	every interval rules out a change of 10% or more, or every variant reached the planned sample	ship the simpler variant, finish
guardrail_breached	a variant is significantly worse on a guardrail goal	do not ship it; find out what broke

"No difference" must be earned like a winner: the always-valid interval of every comparison sits within ±10% of the control rate (mde, the relative effect worth detecting), or every variant passed the sample where the test has 80% power for that effect.

message is one sentence written for a person or an agent to act on, for example: Stop: "bold" beats "control" (+18.3%, always-valid p=0.0120). Ship "bold". Agents are told in the skills to quote it verbatim and never to call a winner from raw rates. Next to it, voice is a friendlier line for people, and daysToVerdict estimates when to look again at the current traffic.

When an experiment has run 14 days (or reached its planned sample) with nothing significant and a verdict is more than 30 days off or every lift is under 3%, or when the gap left is so small that proving it would take months, voice.advice is move_on: the experiment has run its course. It never changes verdict.status.

minDays (opt-in, 0–30, default 0) holds a verdict until that many days of data are in; waiting messages mention days only when it is set.

Sample quality and stars#

Both are informational: they never change the verdict.

• verdict.sample.quality is small below the minimums, good once every variant has its share of goodSample (default 5,000 visitors in total, set per experiment with goodSample or create --good-sample), and ok in between.
• verdict.sample.stars (0–5) counts five checks in verdict.sample.checks: the minimums are met; the split matches the weights with 1,000+ visitors; the good sample is reached; the sample is big enough to spot a 10% change; it has run a full week. With a broken split the score is capped at 1.

Sample size#

Rule of thumb for a classic test at 95% confidence and 80% power: visitors per variant ≈ 16 · p(1−p) / (p·L)² with baseline rate p and relative lift L. Baseline 5% and lift 20% → about 7,600 per variant; baseline 2% and lift 10% → about 78,000. Being safe to check any time costs sample: SplitHub's sequential test needs about twice that (about 16,000 and 170,000 per variant for the same two cases). Ask before you start:

npx splithub plan --baseline 0.05 --mde 0.2 [--variants 3] [<id>]

It answers visitors per variant for SplitHub's test, and with an experiment id the days at that experiment's traffic. No account needed; the same answer is GET https://go.splithub.io/v1/plan?baseline=0.05&mde=0.2&variants=2.

Several variants#

With three or more variants every comparison is against the control, and significance is Holm-corrected across the comparisons (same guarantee as Bonferroni, more power). Pairwise comparisons between non-control variants are not reported; create a new experiment with the two finalists if you need that.

Secondary metrics and guardrails#

Goals in metrics are read with the same sequential test as the primary goal:

• secondary: two-sided, Holm-corrected across all secondary comparisons. Informational.
• guardrails: one-sided (harm only, in the direction you set with worse), Holm-corrected across all guardrail comparisons, judged once both sides have the minimum 100 visitors. A variant significantly worse on a guardrail is never the winner: the verdict becomes guardrail_breached and verdict.guardrailBreached[] lists the metric, the variant and the lift. When the decided winner is the control or is clean, the call stands and the harm is added to verdict.notes.

Each family is corrected on its own, so adding metrics never weakens the primary verdict.

Multivariate experiments#

An mvt experiment gets the verdict over its combinations, plus factors[]: for each factor, every level's visitors and conversions summed over the combinations that use it and compared with the factor's first level. Factor levels and combinations are Holm-corrected together, so reading both keeps the 5% false-positive rate. best names the level that beats the first one significantly with the largest lift.

Segments#

Breakdowns by country, device, browser, platform, app version, source, channel, landing page and more are for hypotheses, not verdicts: slicing after the fact finds noise. Each segment carries its own comparison and a small flag when it has too few visitors. A slice (results --from --to --filter) is marked as one and carries no advice; the decision stays with the whole experiment. If a segment looks interesting, run a targeted experiment on it.

Holdout#

With a holdout in the config, splithub results __holdout compares the held-out visitors with everyone who was enrolled, by the same test: the long-term effect of all your experiments together.