# Statistics and verdicts

SplitHub Cloud turns per-variant counts into a verdict. The same analysis answers the CLI, the MCP server, the dashboard, the reports and the emails, so every surface quotes the same numbers. A self-hosted endpoint returns the raw counts (visitors, conversions, rate, revenue per variant) with `analysis: { available: false }`; verdicts, stars and the rest of this page come with SplitHub Cloud.

## Peeking-safe by design

Agents and people check results all the time. A classic p-value read on every check and acted on at the first significant moment inflates false positives several times over. SplitHub decides with an **always-valid sequential test** (a mixture sequential probability ratio test): its p-value and confidence interval stay valid however often you look, so you can check any time and stop as soon as the verdict says so. Verdicts depend on sample size, not on days.

## What is computed

For each variant: unique exposed visitors, converted visitors, rate, revenue (when the goal has it), and the chance to be the best variant. For each non-control variant against the control:

| Field | Meaning |
|---|---|
| `absoluteLift` | rate − control rate, as a fraction; multiply by 100 for percentage points |
| `relativeLift` | (rate − control rate) / control rate |
| `sequential.pValue`, `sequential.ci` | always-valid p-value and 95% interval of the absolute difference: what decisions use |
| `relativeCi` | the always-valid interval as a relative lift |
| `significant` | the always-valid p-value is below 0.05 after Holm correction across the comparisons |
| `pValue`, `ci`, `zScore` | the fixed-horizon two-proportion z-test, for reference only |
| `probabilityToBeat` | probability that the variant's true rate is above the control's (normal approximation) |

Plus the sample-ratio-mismatch check: a chi-square test of the observed split against the configured weights, flagged when p < 0.001. When a targeting flow's conditions decide who reaches which variant, the expected split depends on who visits, so the check is skipped and `verdict.notes` says so.

## The verdict

| `status` | When | What to do |
|---|---|---|
| `no_data` | no exposures yet | wait, or check the setup with `splithub doctor` |
| `insufficient_sample` | fewer than 100 visitors in a variant or fewer than 50 conversions in total, or the sequential test has not decided yet | keep running; the message says about how many visitors per variant will settle it |
| `srm_detected` | the split does not match the weights | the setup is broken (bots, caching, a variant that errors, a redirect loop); fix it, restart as a new experiment |
| `winner_found` | a variant beats the control, or the control beats a significantly worse variant | read `winner` and `message`, then finish the experiment |
| `no_significant_difference` | every interval rules out a change of 10% or more, or every variant reached the planned sample | ship the simpler variant, finish |
| `guardrail_breached` | a variant is significantly worse on a guardrail goal | do not ship it; find out what broke |

"No difference" must be earned like a winner: the always-valid interval of every comparison sits within ±10% of the control rate (`mde`, the relative effect worth detecting), or every variant passed the sample where the test has 80% power for that effect.

`message` is one sentence written for a person or an agent to act on, for example: *Stop: "bold" beats "control" (+18.3%, always-valid p=0.0120). Ship "bold".* Agents are told in the skills to quote it verbatim and never to call a winner from raw rates. Next to it, `voice` is a friendlier line for people, and `daysToVerdict` estimates when to look again at the current traffic.

When an experiment has run 14 days (or reached its planned sample) with nothing significant and a verdict is more than 30 days off or every lift is under 3%, or when the gap left is so small that proving it would take months, `voice.advice` is `move_on`: the experiment has run its course. It never changes `verdict.status`.

`minDays` (opt-in, 0–30, default 0) holds a verdict until that many days of data are in; waiting messages mention days only when it is set.

## Sample quality and stars

Both are informational: they never change the verdict.

- `verdict.sample.quality` is `small` below the minimums, `good` once every variant has its share of `goodSample` (default 5,000 visitors in total, set per experiment with `goodSample` or `create --good-sample`), and `ok` in between.
- `verdict.sample.stars` (0–5) counts five checks in `verdict.sample.checks`: the minimums are met; the split matches the weights with 1,000+ visitors; the good sample is reached; the sample is big enough to spot a 10% change; it has run a full week. With a broken split the score is capped at 1.

## Sample size

Rule of thumb for a classic test at 95% confidence and 80% power: visitors per variant ≈ `16 · p(1−p) / (p·L)²` with baseline rate `p` and relative lift `L`. Baseline 5% and lift 20% → about 7,600 per variant; baseline 2% and lift 10% → about 78,000. Being safe to check any time costs sample: SplitHub's sequential test needs about twice that (about 16,000 and 170,000 per variant for the same two cases). Ask before you start:

```bash
npx splithub plan --baseline 0.05 --mde 0.2 [--variants 3] [<id>]
```

It answers visitors per variant for SplitHub's test, and with an experiment id the days at that experiment's traffic. No account needed; the same answer is `GET https://go.splithub.io/v1/plan?baseline=0.05&mde=0.2&variants=2`.

## Several variants

With three or more variants every comparison is against the control, and significance is Holm-corrected across the comparisons (same guarantee as Bonferroni, more power). Pairwise comparisons between non-control variants are not reported; create a new experiment with the two finalists if you need that.

## Secondary metrics and guardrails

Goals in `metrics` are read with the same sequential test as the primary goal:

- **secondary**: two-sided, Holm-corrected across all secondary comparisons. Informational.
- **guardrails**: one-sided (harm only, in the direction you set with `worse`), Holm-corrected across all guardrail comparisons, judged once both sides have the minimum 100 visitors. A variant significantly worse on a guardrail is never the winner: the verdict becomes `guardrail_breached` and `verdict.guardrailBreached[]` lists the metric, the variant and the lift. When the decided winner is the control or is clean, the call stands and the harm is added to `verdict.notes`.

Each family is corrected on its own, so adding metrics never weakens the primary verdict.

## Multivariate experiments

An `mvt` experiment gets the verdict over its combinations, plus `factors[]`: for each factor, every level's visitors and conversions summed over the combinations that use it and compared with the factor's first level. Factor levels and combinations are Holm-corrected together, so reading both keeps the 5% false-positive rate. `best` names the level that beats the first one significantly with the largest lift.

## Segments

Breakdowns by country, device, browser, platform, app version, source, channel, landing page and more are for hypotheses, not verdicts: slicing after the fact finds noise. Each segment carries its own comparison and a `small` flag when it has too few visitors. A slice (`results --from --to --filter`) is marked as one and carries no advice; the decision stays with the whole experiment. If a segment looks interesting, run a targeted experiment on it.

## Holdout

With a `holdout` in the config, `splithub results __holdout` compares the held-out visitors with everyone who was enrolled, by the same test: the long-term effect of all your experiments together.