Experiments

An experiment splits traffic between variants of a feature flag, attributes metric events to each variant, and tells you the probability that each variant is the best — with guardrails that refuse to offer a verdict on too little data.

Create and launch

An experiment moves through four states: draft → running → stopped → archived. Creating one also creates its feature flag, inactive — a draft serves no traffic. The default setup is two variants, control and test, split 50/50; edit variants and targeting on the flag while the experiment is still a draft.

Launching requires a primary metric — an experiment with nothing to measure can’t be launched — and is what activates the flag and stamps the start date. Metrics come in three shapes: conversion (did the person do it), count per user (how many times), and mean (average of a numeric property). Secondary metrics ride along for context.

Before launching, the sample-size helper estimates how many exposures per variant you need for a given baseline rate and minimum detectable effect — and, given your traffic, how many days that will take. Running an experiment your traffic can’t power is the most common way to waste two weeks.

How exposure works

A person enters the experiment at their first az.flag_called event whose flag response matches one of the experiment’s variant keys, inside the experiment’s start/end window. The SDKs emit that event automatically when your code reads the flag — see exposure on the flags page.

Two rules keep the numbers honest:

• A person observed in more than one variant is excluded outright — attributing them to either arm would contaminate both. The excluded count is reported in the results, not hidden.
• Metric events count only at or after the person’s first exposure. A purchase from before someone ever saw the test can’t credit the test.

Results attribute on the raw per-event person id; the canonical-person join that folds in late identify-merges is a known follow-up, not current behavior.

Read the results

The primary display is Bayesian: each variant gets a probability-to-be-best and a 95% credible interval. Bayesian results remain valid under continuous peeking — you can look at a running experiment every day without invalidating the statistics, which is how people actually behave. A frequentist p-value and confidence interval are shown next to them as a familiar reference point, not as the significance gate.

No verdict is offered until every variant has at least 100 exposures, and — for conversion metrics — at least 5 successes. A winner is called at 95% probability-to-be-best. Probability-to-be-best is estimated by Monte Carlo with a fixed seed, so the same inputs always produce the same numbers — results don’t jitter between refreshes.

Results are computed from compacted storage (cached for five minutes); the separate exposures counter merges the live ingest buffer, so a just-launched experiment shows traffic immediately even before results fill in.

Stop and roll out

Stopping an experiment stamps the end date but leaves the flag active — participants keep their assignments instead of being snapped back to control the moment you stop measuring. To roll out the winner, edit the flag: set the winning variant to 100% (or flip a boolean flag fully on). To roll back, turn the flag off.

Deleting an experiment keeps its flag, and deleting a flag that an experiment references is blocked — you can’t orphan a running test or silently un-ship a rollout by cleaning up the wrong object.

Reference

Guardrails

Guardrail	Value
Minimum exposures	100 per variant before any verdict
Minimum successes	5 per variant (conversion metrics)
Winner threshold	95% probability-to-be-best
Credible intervals	95%
Multi-variant persons	Excluded, count reported

Experiments are created, launched, and read entirely in the app; the only token-authenticated surface involved is /api/v1/decide, which serves the flag itself.