Why are Randomized Controlled Trials (RCTs, known as A/B testing in much of the industry) testing is widely regarded as the golden standard of causal inference? What else can a Data Scientist do if A/B testing is not possible, and why are those alternatives inferior to A/B testing?

This papers shows, using 15 experiments (for ads on Facebook) where a RCT was conducted, that common observational methods (run on the Facebook data, by ignoring the control group) severely mis-estimate the true treatment life (as measured by the RCT), often by a factor of 3x or more. This is true, even though Facebook has (i) very large sample sizes, and, (ii) very high quality data (per-user feature vector) about its users which are used in the observational methods. This should be a major red flag for Data Scientists working on common marketing measurements (such as marketing campaigns) using observational methods.

Continue reading

Many companies have some sort of "Culture Doc", a booklet or similar, which explains to new joiners what the company is about. I received Facebook's "Little Red Booklet" when I joined in 2016 February, and I was amazed how good it was. Recently I was researching other companies' Culture Docs, and found a version of Netflix's and Valve's online. It's interesting to compare and contrast what these different companies choose to put in their Culture Doc. Facebook's Culture Doc is very mission and execution oriented and serious, Netflix is analytical and HR-focused, and Valve's is a lighthearted explanation of how the company works.

Continue reading

The idea behind WeToddle came from the Baby Fanclub group we have on Messenger, which has most of our family in it. It turns out some ex-Facebook people had a similar idea in 2019, raised $3M, spent a 2 years on it, and then gave up because it didn’t go anywhere (presumably).

Continue reading