Alerts, digest, AI reviews

Three ways to be told instead of checking, for experiments and analytics properties alike. All three live on the Alerts tab and the AI tab of a cloud project and need Pro.

Alert rules#

A rule is a scope, a condition with thresholds, and actions. Experiment rules watch one experiment or every running one; analytics rules watch one property. Rules are evaluated every hour and fire at most once per experiment per cooldown window.

On experiments#

Condition	Fires when
srm	the split differs from the configured weights (p < 0.001)
winner	the verdict is winner_found (optionally only once every variant has N visitors)
loser	a variant is at least X% worse than the control with enough sample
exposures	unique visitors on the experiment reach N
conversions	conversions reach N
duration	the experiment has run N days
no_events	no exposures for N hours while running

On properties#

Condition	Fires when
traffic_drop	visitors in the last 24 hours fell by X% or more against the same 24 hours a week before (default 50%; needs 20+ visitors a week before)
traffic_spike	visitors in the last 24 hours rose by X% or more against the same 24 hours a week before (default 50%; needs 20+ visitors now)
goal_silent	a goal of the property had no conversions in the last N hours (default 24) after converting the week before
event_count	an event, or any event of the property, reaches N, counted from when the rule was made; with repeat, at every multiple of N

On the account#

Condition	Fires when
quota	the project used X% of its monthly allowance
usage	this month's billed events cross another multiple of N (every 1,000, say)

Actions: send to channels, and for experiment rules also pause the experiment, finish it, or drop the losing variant (set its weight to 0). A rule can combine them: "SRM → pause and email", "winner → finish and email", "loser after 10k visitors → drop the loser". Analytics and account rules only notify. event_count and usage are milestones: each one fires once, whatever the cooldown.

Every project starts with sensible experiment rules created disabled; enabling one needs Pro. Test runs the rule as a dry run and lists what it would hit right now.

From the CLI:

npx splithub alerts                                   # rules, progress, recent alerts
npx splithub alerts add srm --pause
npx splithub alerts add loser --lift 10 --experiment checkout-steps --drop-loser
npx splithub alerts add traffic_drop --property sh-x1h78w --percent 50
npx splithub alerts add goal_silent --property sh-x1h78w --goal signup --hours 24
npx splithub alerts add event_count --property sh-x1h78w --event Signup --count 1000 --repeat
npx splithub alerts test <id>                         # what would fire now, nothing sent

Other flags: exposures --visitors N, conversions --conversions N, duration --days N, no_events --hours 24, quota --percent 80, usage --every 1000, winner|loser --min-visitors N, --experiment <key> for one experiment, and on any rule --name, --cooldown <hours>, --disabled, --finish. alerts enable|disable|delete <id> manage a rule. The MCP tool is splithub_alerts.

Channels#

Saved once, reused by rules and the digest: an email address, a Slack or Discord webhook, or a webhook URL of your own (JSON POST with the rule, the experiment or property, the message and the actions taken). Add them on the Alerts tab; pick them in a rule's "deliver to" multiselect. Test on a channel sends a test message now.

The digest#

A scheduled summary, separate from the rules: on chosen weekdays, weekly or monthly, at a chosen hour in the project's time zone. It has two parts:

• Experiments: running experiments with their verdicts, finished ones since the last digest, quota usage, and what to decide next.
• Analytics: per property, visitors, pageviews, top sources and pages, goal conversions and the notable events of the period, each against the period before (up to 10 properties; pick which ones or cover them all).

The team's notes of the period close it. Sent to the chosen channels; each member can also switch it off in their profile.

AI reviews#

An analyst's second opinion written by a language model from the experiment setup, aggregate results and available variant screenshots. Raw event rows and visitor ids are not supplied as statistical input:

• On start: a setup review, including available variant screenshots.
• On finish: when an experiment finishes (dashboard, CLI, API, cap, rule), a review is written and attached to the finish email.
• Progress checks: the daily job checks running experiments with at least 100 visitors. It writes a review when the verdict changes from the previous snapshot or a week has passed since the last review, subject to the automatic review cooldown.
• On demand: the AI tab's "Review now".

Reviews are stored with the experiment and readable from the dashboard, npx splithub review [<id>], the MCP tool splithub_review, and GET /v1/reviews[/<id>] with an API key. They are advice; the verdict from the statistics stays the source of truth, and the skills tell agents exactly that.

Webhook payload#

{
  "kind": "traffic_drop",
  "project": "project-id",
  "subject": "[SplitHub] My app: Traffic drop",
  "text": "Alert summary and dashboard link",
  "rule": "Traffic drop",
  "experiment": null,
  "property": "sh-x1h78w",
  "message": "Visitors to \"sh-x1h78w\" fell 62% in the last 24 hours: 410, against 1,080 the same day last week.",
  "actions": [],
  "link": "https://app.splithub.io/…"
}

Experiment alerts carry the experiment key in experiment and no property; verdict alerts add a voice line. Requests carry X-SplitHub-Event and X-SplitHub-Delivery headers, and X-SplitHub-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<body>"> when the channel has a signing secret.