Statistical problems at computing online controlled experiments at scale

Abstract

Online controlled experiments, or A/B tests, are a reliable tool for data-driven decision making in industrial applications. We discuss several applied statistical problems related to the A/B testing at scale. The list of problems includes detecting inherent biases in samples, reducing metric variance to accelerate A/B tests, and improving statistical power by multivariate hypothesis testing. For each problem, we propose a practical solution and show the results in numerical experiments