What is Control Group?
The baseline group in an experiment that receives the current experience, used for comparison against test variants.
The control group (also called the baseline or holdout) is the group of users in an experiment who receive the current, unchanged experience. Their behavior serves as the benchmark against which variant performance is measured.
Why Control Groups Matter
Without a control group, you cannot measure the impact of a change. The control provides the counterfactual: what would have happened without the change.
Control Group Best Practices
Random assignment: Users must be randomly assigned to control vs variant to avoid selection bias.
Sufficient size: The control needs enough users for statistical power.
Consistent experience: Control users should see the same experience throughout the test.
No contamination: Ensure control users are not exposed to variant changes.
Holdout Groups
Some teams maintain permanent holdout groups that never receive new features. This helps measure the cumulative impact of multiple changes over time.