Detecting sample ratio mismatch with sequential testing

Abstract

Online controlled experiments, or A/B tests, are the most reliable method for evaluating the impact of product changes and making data-driven business decisions. In A/B tests, unintended deviations from the designed group allocation ratio can occur. This phenomenon is called a sample ratio mismatch (SRM). The presence of SRM indicates an issue with the experiment and suggests that the results may be biased. Early detection of SRM is crucial because it allows biased experiments to be stopped quickly. In this report, we study the sequential methods for detecting SRM both theoretically and numerically. Specifically, we focus on group sequential methods based on the Pearson chi-squared statistic, as well as sequential methods with a Bayesian alternative. We compare these methods through numerical experiments, using both real and synthetic data