A/B Test

An A/B Test is a controlled experiment that randomly splits users into groups, shows each group a different version, and compares a chosen metric to decide which version performs better. Randomization is what makes the comparison meaningful, because it distributes unknown differences between people evenly across groups, so a gap in results can be attributed to the change rather than to who happened to see it. Running one properly requires committing in advance to a primary metric, a minimum detectable effect, and a sample size, then letting the test run to that size instead of stopping when the numbers look good. Teams use platforms such as Optimizely, LaunchDarkly, GrowthBook, and Statsig, or feature flags wired to their own analytics. Product changes, pricing pages, onboarding flows, and email subject lines are the usual subjects, and AI now helps by generating variants and summarizing results. The pitfall is peeking: checking repeatedly and stopping at the first significant reading inflates false positives badly. Small samples also produce large swings, and a test that reaches significance on a rare event may still be noise. Related techniques include multivariate testing and multi-armed bandits, which shift traffic toward winners while the test runs.