Unevenness at the edges

Marton Trencseni - Fri 30 October 2020 • Tagged with stats, data

Sometimes we look at the top performers in a field and see obviously uneven representations of groups (gender, ethnicity, etc). There a multitude of factors that can lead to it, such as unfair bias in access to opportunities. Here I will show one unintuitive mathematical effect that can contribute to such unevenness in the case of normal distributions.

Continue reading

Effective Data Visualization Part 3: Line charts and stacked area charts

Marton Trencseni - Tue 01 September 2020 • Tagged with charts, dashboards, data, visualization

Most charts should be line charts or stacked area chart, because they communicate valuable trend information and are easy to parse for the human eyes and brain.

Continue reading

Effective Data Visualization Part 2: Formatting numbers

Marton Trencseni - Sun 23 August 2020 • Tagged with charts, dashboards, data, visualization

Format numbers for human consumption. What is more readable, 1.539e+5 or 153,859? Showing numbers effectively on spreadsheets, charts, dashboards, reports is a basic ingredient for readability, like formatting code in programming.

Continue reading

Effective Data Visualization Part 1: Categorical data

Marton Trencseni - Sat 22 August 2020 • Tagged with charts, dashboards, data, visualization

Making clear, readable charts is part of the craftmanship minimum for any data related role. In part one, I look at how to present categorical data.

A pie chart

Continue reading

Multi-armed bandits and false positives

Marton Trencseni - Fri 21 August 2020 • Tagged with ab-testing

I use Monte Carlo simulations to explore the false positive rate of Multi-armed bandits.

Epsilon-greedy

Continue reading

A/B testing and Multi-armed bandits

Marton Trencseni - Fri 07 August 2020 • Tagged with ab-testing

Multi-armed bandits minimize regret when performing A/B tests, trading off between exploration and exploitation. Monte Carlo simulations shows that less exploration yields less statistical significance.

Epsilon-greedy

Continue reading

Understanding Facebook’s Planout A/B testing framework

Marton Trencseni - Fri 22 May 2020 • Tagged with ab-testing

PlanOut is a framework for online field experiments. It was created by Facebook in 2014 to make it easy to run and iterate on sophisticated experiments in a statistically sound manner.

Planout

Continue reading

Validation checks for A/B tests

Marton Trencseni - Thu 16 April 2020 • Tagged with ab-testing

A/B tests go wrong all the time, even in sophisticated product teams. As this article shows, for a range of problems we can run automated validation checks to catch problems early, before they have too bad of an effect on customers or the business. These validation checks compare various statistical properties of the funnels A and B to catch likely problems. Large technology companies are running such validation checks automatically and continuously for their online experiments.

Kolmogorov-Smirnov test

Continue reading

Running multiple A/B tests in parallel

Marton Trencseni - Mon 06 April 2020 • Tagged with ab-testing

I show using Monte Carlo simulations that randomizing user assignments into A/B test experiments makes it possible to run multiple A/B tests at once and measure accurate lifts on the same metric, assuming the experiments are independent.

Watts-Strogatz

Continue reading

Bayesian A/B conversion tests

Marton Trencseni - Tue 31 March 2020 • Tagged with bayesian, ab-testing

I compare probabilities from Bayesian A/B testing with Beta distributions to frequentist A/B tests using Monte Carlo simulations. Under a lot of circumstances, the bayesian probability of the action hypothesis being true and the frequentist p value are complementary.

Bayes vs z-test

Continue reading

A/B testing and the G-test

Marton Trencseni - Mon 23 March 2020 • Tagged with ab-testing

The G-test for conversion A/B tests is similar to the Chi-squared test. Monte-Carlo simulations show that the two are indistinguishable in practice.

G-test vs Chi-squared p differences

Continue reading

A/B testing and networks effects

Marton Trencseni - Sat 21 March 2020 • Tagged with ab-testing

I use Monte Carlo simulations to explore how A/B testing on Watts–Strogatz random graphs depends on the degree distribution of the social network.

Watts-Strogatz degree distribution

Continue reading

A/B testing on social networks

Marton Trencseni - Mon 09 March 2020 • Tagged with ab-testing

I use Monte Carlo simulations to show that experimentation on social networks is a beatiful statistical problem with unexpected nuances due to network effects.

Watts-Strogatz

Continue reading

Early stopping in A/B testing

Marton Trencseni - Thu 05 March 2020 • Tagged with ab-testing

Increased false positive rate due to early stopping is beautiful nuance of statistical testing. It is equivalent to running at an overall higher alpha. Data scientists need to be aware of this phenomenon so they can control it and keep their organizations honest about their experimental results.

Early stopping

Continue reading

A/B testing and Fisher's exact test

Marton Trencseni - Tue 03 March 2020 • Tagged with ab-testing

Fisher’s exact test directly computes the same p value as the Chi-squared test, so it does not rely on the Central Limit Theorem to hold.

Fisher's test, Fisher Monte Carlo and Chi-squared test p values

Continue reading

A/B testing and the Chi-squared test

Marton Trencseni - Fri 28 February 2020 • Tagged with ab-testing

In an ealier post, I wrote about A/B testing conversion data with the Z-test. The Chi-squared test is a more general test for conversion data, because it can work with multiple conversion events and multiple funnels being tested (A/B/C/D/..).

Chi-squared distribution

Continue reading

A/B testing and the t-test

Marton Trencseni - Sun 23 February 2020 • Tagged with ab-testing

The t-test is better than the z-test for timespent A/B tests, because it explicitly models the uncertainty of the variance due to sampling. Using Monte-Carlo simulations I show that around N=100, the t-test becomes the z-test.

Normal distribution vs t-distribution

Continue reading

A/B testing and the Z-test

Marton Trencseni - Sat 15 February 2020 • Tagged with ab-testing

I discuss the Z-test for A/B testing and show how to compute parameters such as sample size from first principles. I use Monte Carlo simulations to validate significance level and statistical power, and visualize parameter scaling behaviour.

Conversion difference vs N

Continue reading

Beyond the Central Limit Theorem

Marton Trencseni - Thu 06 February 2020 • Tagged with data, ab testing, statistics

In the previous post, I talked about the importance of the Central Limit Theorem (CLT) to A/B testing. Here we will explore cases when we cannot rely on the CLT to hold.

Running mean for Cauchy distribution

Continue reading

A/B testing and the Central Limit Theorem

Marton Trencseni - Wed 05 February 2020 • Tagged with data, ab testing, statistics

When working with hypothesis testing, the desciptions of the statistical method often has normality assumptions. For example, the Wikipedia page for the z-test starts like this: "A Z-test is any statistical test for which the distribution of the test statistic under the null hypothesis can be approximated by a normal distribution". What does this mean? How do I know it’s a valid assumption for my data?

Normal distribution from uniform

Continue reading