Glossary · Analytics

A/B Testing

A statistical method of comparing two versions of something to determine which performs better.

A/B testing (also called split testing) is a controlled experiment where two or more variants of a page, feature, or experience are shown to different user segments simultaneously to determine which variant performs better on a defined metric. A/B tests require: a hypothesis (what you expect to happen and why), a control group (existing version A), a treatment group (new version B), a primary metric (what you're measuring), statistical significance (confidence that the result isn't due to chance), and sufficient sample size. Advanced forms include multivariate testing (testing multiple changes simultaneously), multi-armed bandit (dynamically allocating more traffic to winning variants), and sequential testing (analyzing results as data accumulates).

Fig — every answer ships with the tables, rows and SQL behind it.