Training a Pytorch Wasserstein MNIST GAN on Google Colab
Marton Trencseni - Wed 03 March 2021 • Tagged with ab-testing
I train a Pytorch Wasserstein MNIST GAN on Google Colab to beautiful MNIST digits.
Training a Pytorch Classic MNIST GAN on Google Colab
Marton Trencseni - Tue 02 March 2021 • Tagged with ab-testing
I train a Pytorch Classic MNIST GAN on Google Colab to generate MNIST digits.
Making statistics lie for the 2020 Presidential election
Marton Trencseni - Thu 17 December 2020 • Tagged with ab-testing
After the 2020 US presidential election, the Trump campaign filed over 50 lawsuits and attacked the integrity of the elections by claiming there was voter fraud. One of the last lawsuits was filed in the Supreme Court of the United States by the state of Texas. Here I look at the statistical claims made in this lawsuit that were supposed to show irregularities in the Georgia vote.
Comparing conversion at control and treatment sites
Marton Trencseni - Thu 03 December 2020 • Tagged with ab-testing
In real-life, non-digital situations, it's often not feasible to run true A/B tests. In such cases, we can compare before and after rollout conversions at a treatment site, while using a similar control site to measure and correct for seasonality. The post discusses how to compute increasingly correct p-values and bayesian probabilities in such scenarios.
Multi-armed bandits and false positives
Marton Trencseni - Fri 21 August 2020 • Tagged with ab-testing
I use Monte Carlo simulations to explore the false positive rate of Multi-armed bandits.
A/B testing and Multi-armed bandits
Marton Trencseni - Fri 07 August 2020 • Tagged with ab-testing
Multi-armed bandits minimize regret when performing A/B tests, trading off between exploration and exploitation. Monte Carlo simulations shows that less exploration yields less statistical significance.
Understanding Facebookâs Planout A/B testing framework
Marton Trencseni - Fri 22 May 2020 • Tagged with ab-testing
PlanOut is a framework for online field experiments. It was created by Facebook in 2014 to make it easy to run and iterate on sophisticated experiments in a statistically sound manner.
Validation checks for A/B tests
Marton Trencseni - Thu 16 April 2020 • Tagged with ab-testing
A/B tests go wrong all the time, even in sophisticated product teams. As this article shows, for a range of problems we can run automated validation checks to catch problems early, before they have too bad of an effect on customers or the business. These validation checks compare various statistical properties of the funnels A and B to catch likely problems. Large technology companies are running such validation checks automatically and continuously for their online experiments.
Running multiple A/B tests in parallel
Marton Trencseni - Mon 06 April 2020 • Tagged with ab-testing
I show using Monte Carlo simulations that randomizing user assignments into A/B test experiments makes it possible to run multiple A/B tests at once and measure accurate lifts on the same metric, assuming the experiments are independent.
A/B testing and the G-test
Marton Trencseni - Mon 23 March 2020 • Tagged with ab-testing
The G-test for conversion A/B tests is similar to the Chi-squared test. Monte-Carlo simulations show that the two are indistinguishable in practice.
