Five ways to reduce variance in A/B testing

Marton Trencseni - Sun 19 September 2021 • Tagged with ab-testing, variance, stratification, cuped

I use toy Monte Carlo simulations to demonstrate 5 ways to reduce variance in A/B testing: increase sample size, move towards a more even split, reduce variance in the metric definition, stratification and CUPED.

Historic lift

Continue reading

Correlations, seasonality, lift and CUPED

Marton Trencseni - Sun 05 September 2021 • Tagged with ab-testing, cuped

In this final blog post about CUPED, I will address some questions about CUPED, such as, is correlation between "before" and "after" the same as seasonality?

Continue reading

A/A testing and false positives with CUPED

Marton Trencseni - Sun 15 August 2021 • Tagged with ab-testing, cuped

I use Monte Carlo simulations of A/A tests to demonstrate how Data Scientists can incorrectly skew lift and p-values if they pick-and-choose between reporting traditional and CUPED results after the experiment has concluded.

Historic lift

Continue reading

Reducing variance in conversion A/B testing with CUPED

Marton Trencseni - Sat 07 August 2021 • Tagged with ab-testing, cuped

I use Monte Carlo simulations of conversion A/B tests to demonstrate how CUPED reduces measurement variance in conversion experiments.

Historic lift

Continue reading

Reducing variance in A/B testing with CUPED

Marton Trencseni - Sat 31 July 2021 • Tagged with ab-testing, cuped

I use Monte Carlo simulations of A/B tests to demonstrate CUPED, a method to use historic "before" data to reduce the variance in the measurement of the treatment lift.

Historic lift

Continue reading

Beyond the Central Limit Theorem

Marton Trencseni - Thu 06 February 2020 • Tagged with data, ab testing, statistics

In the previous post, I talked about the importance of the Central Limit Theorem (CLT) to A/B testing. Here we will explore cases when we cannot rely on the CLT to hold.

Running mean for Cauchy distribution

Continue reading

A/B testing and the Central Limit Theorem

Marton Trencseni - Wed 05 February 2020 • Tagged with data, ab testing, statistics

When working with hypothesis testing, the desciptions of the statistical method often has normality assumptions. For example, the Wikipedia page for the z-test starts like this: "A Z-test is any statistical test for which the distribution of the test statistic under the null hypothesis can be approximated by a normal distribution". What does this mean? How do I know it’s a valid assumption for my data?

Normal distribution from uniform

Continue reading

A/B tests: Moving Fast vs Being Sure

Marton Trencseni - Mon 01 July 2019 • Tagged with ab-testing, fetchr

Most A/B testing tools default to α=0.05, meaning the expected false positive rate is 5%. In this post I explore the trade-offs between moving fast, ie. using higher α, versus being sure, ie. using lower α.

14. slide

Continue reading

Beautiful A/B testing

Marton Trencseni - Sun 05 June 2016 • Tagged with ab-testing, strata, statistics, data

I gave this talk at the O’Reilly Strata Conference London in 2016 June, mostly based on what I learned at Prezi from 2012-2016.

14. slide

Continue reading