Experimentation helps determine whether a proposed change causes a measurable difference. A/B testing compares two or more variants of an experience; multivariate testing evaluates combinations of multiple elements and generally needs more traffic to estimate interactions reliably.
A/B Testing #
Split eligible traffic between a control and a variant, ideally through random assignment. Change a clearly defined element or experience, such as a headline or form layout, and measure a predefined primary outcome.
Multivariate Testing #
Test combinations of elements—for example, headline A/B and image X/Y—to estimate individual effects and interactions. Because combinations multiply quickly, this approach usually requires substantially more traffic and careful design.
Design a Sound Experiment #
- Write a hypothesis that explains the observed problem and expected mechanism.
- Choose one primary metric and appropriate guardrails.
- Define eligible traffic, exclusions and the assignment unit.
- Estimate the sample size using baseline rate, minimum detectable effect, significance level and statistical power.
- Run long enough to cover normal cycles and reach the planned sample.
- Analyse results and document the decision, including uncertainty.
Common Mistakes #
Avoid peeking and stopping as soon as results look favourable, changing multiple things without a plan, testing on tiny samples, running overlapping experiments that interfere and treating a non-significant result as proof of no effect.
After the Test #
Check implementation, segment results only when planned or clearly exploratory, monitor guardrails and validate that the effect persists. A statistically detectable lift may still be too small to matter commercially.
Related Topics #
See Conversion Rate Optimization and CRO Tools and Reporting.
