Bytepawn - Marton Trencseni

2024 Data Outlook

2024-02-11T00:00:00+01:00

Introduction

It is the beginning of the year — a good time to reflect on the previous year and make plans for the year ahead. This document is an attempt at that, focusing on the latter, since our 2023 performance reviews covered the previous year.

I invite you to the same for yourself — think about what happened to the industry in 2023, where it's going in 2024, and what does that mean to you? What do you want to learn, whether at the company or outside? What do you want to "import" into the company, whether into an existing project or as a new project? What does a good 2024, the year of AI, look like on your area, in your Business Unit? What do you want to do more of, and what do you want to do less of — both at work and outside of work.

2024 will be a great year for data.

It will be what we make of it.

It will be what you make of it.

Industry opportunities

Data Science and AI

I started studying at the Technical University in Budapest in 1999 for a Computer Science major. Around the third year we had to take the mandatory Artificial Intelligence course, taught by the polish-hungarian professor Dobrowiecki Tadeusz. We were using the dreadful Norvig Artificial Intelligence book, but even worse, the hungarian translation, by Tadeusz, himself not a native speaker.

I didn't get much out of this course, but I do remember some things:

Learning that neural networks are very limited: a neural network — even with multiple "hidden" layers — is essentially a linear transform, a large matrix that maps vectors into vectors
.. but, no linear map can implement even XOR — this is of course is true, but only if you don't have activation functions in there. Once you add activation functions, you introduce nonlinearity, and the network can "even" learn XOR 🙂
AI springs and winters: Tadeusz explained to us that optimism about the role of AI comes and then goes, with long periods of AI winters in between. "This time it's real!" has been said many times, but so far it always ended up being unreal.
The first AI winter was triggered by the 1969 book Perceptrons, written famously by Marvin Minsky, and using — among others — the above XOR argument to discourage work in AI.

In 2024, after ~50 years of research, we know that the great man (Minsky) was wrong. We now understand the importance of activation functions, stochastic gradient descent, batch learning, dropouts, different cost functions, regularization, we have invented transformer architectures, and so on. I pay \$20 per month for my OpenAI subscription and use ChatGPT on a weekly basis, and I can't wait for further commoditization, so the monthly price drops to \$5 and eventually \$0. The opening image was drawn by ChatGPT4 (after numerous re-prompts by yours truly, because it kept drawing male figures only). Even John Carmack (creator Wolfenstein, Doom, Quake, then CTO of Oculus) is working on AI these days. "This time it's real!"

The transformer neural network (and training) architectures driving Generative AI are interesting technologies, and I don't think we should (or can afford to) ignore them. In 2024, having a good grasp of how and why they work is part of the forward-thinking (and intellectual) Data Scientist's (or ML Engineer's, or, any data person's) area of interest. Also, there are hundreds of startups trying to commercialize these technologies (Word+GenAI, Powerpoint+GenAI, Jira+GenAI, IDE+GenAI, etc). We need to keep our eyes open for possible application areas within our company.

Personally, I have spent a fair amount of time in the past learning about and training neural networks (in the pre-generative era), in 2024 I plan to invest time into this area again, eg. play around with Llamas, play around with vector databases and langchain.

But it's not just the latest and greatest Generative AI that is exciting in 2024. The investment and excitement into (neural network based and/or generative) AI spills over into adjacent technical areas of traditional Data Science and Machine Learning (and Data Engineering), so we have new forecasting libraries, new vision models, better implementations in SKL and LGBM, and tons of technologies to try out on the MLOps and tooling front — literally 100s of tools and libraries.

Data Engineering and Platforms

This is one of the most exciting areas for us in 2024! When we built our organization's Data Strategy for 2024, our data platform and the data engineering work was the center piece of the plan!

There's so much to do, I don't have to sell it — this stuff sells itself. In 2024, we want to:

detach from "corporate IT" and build out own way-of-working (eg. with devboxes)
significantly improve our developer experience
replace Vertica with a managed DWH solution (such as Snowflake)
stop building N DWHs and just have N=1, ours
..and thus, ingest from a lot more source operational databases
move more of our ingestion to be real-time (vs batch)
build out a set of core datasets
build a new, thin ETL framework on top of Airflow
- .. and open source it!
significantly improve our pipeline and dataset monitoring and alerting
significantly improve the way we deploy, track and update models in production (MLOps)
sunset significant portions of the Data Science infra, as move to a unified platform

That's 10+ items, each of them non-trivial!

Some technologies that I want to look at in 2024:

DagsHub: GitHub for machine learning
OpenLlama: let's make our own LLM
DevBox: open source dev box management
Airflow 2.8 and beyond
- I think we're getting a lot of bang for our buck from Airflow
- I ❤️ Airflow, I've been using it since 2017 (!)
- .. but, I wonder what's next?
- Today we use Airflow for ML pipelines, but it seems inefficient
DataHub: we have already deployed it, we need more adoption \

AI products

Traditionally most of our product-work is adding Data Science to an existing product: e.g. we have our loyalty app, it displays offers [anyway], can we rank them better? We have our cinema app, it shows movie recommendations [anyway], can be rank them better? We send out marketing emails [anyway], can we personalize them better? To be clear, there is no problem with the [anyway], in the end it's impact that matters! Having said that, it's interesting to ponder whether the new AI technologies available in 2024 enable us to do new things, ship entirely new products or experiences [for the company]?

For example, I've been thinking that Amazon has Alexa, which among other functionalities, allows one to order products from Amazon by voice. It would certainly be easier to build something like Alexa today, given that voice recognition and intent recognition is more commoditized today.

Having said that, some of us have seen the difficulties in execution around previous AI apps, and we know it's hard to find engagement with Business Units over the Generative AI prototypes we built previously, so AI products are a very long shot, but still something we should think about.

Finance: an opportunity?

When I was getting my Physics degree at the Eotvos University in Budapest, I took a class on quantitative finance. The context is that in the US, around the 80s and 90, (some) physicists started to work in the finance industry, on Wall Street, as quants, building models [to find arbitrage opportunities]. In today's lingo they were "Data Scientists in Finance". At a high level, they were (and are) doing the same thing as we do as Data Scientists: trying to build forecasting and decision support models based off data to make more money, using mostly the same tools that we use (simple math, statistical tests, linear regression, decision trees, etc). This phenomenon became so widespread that one such physicist quant called Emanuel Derman even wrote a book about it titled My Life as a Quant, which I read as a University student. And we had a few such quant physicists in Budapest too, working at our local banks (and Morgan Stanley, who has an office in Budapest), so eventually they also taught a course at University. So ever since then I've been interested in understanding how quantitative finance works, and following this area from a distance. Somewhat related to this, I've also been reading a lot of investing related books in the last 10 years, many by or about Warren Buffet, James Simons, George Soros, Ray Dalio, etc. What does this have to do with our work as Data Scientists?

One thing I've been pondering is the success we have had working with the Financial Planning & Analysis (FP&A) teams. Over the last 12 to 18 months we proved that we (Data Scientists) are able to make significantly better (less biased), more objective forecasts than Finance teams. We are also able to better create what-if scenarios to model and understand the impacts of events such as pandemics, boycotts and wars. We are now not just forecasting revenues, but has also started to work on forecasting costs.

Over the winter break I was reading the excellent book In Pursuit of the Perfect Portfolio, about luminaries and their work in the Finance industry: Markowitz, Sharpe, Fama, Black, Scholes, etc. They invented modern portfolio theory with the idea of modeling risk vs return, covariance and diversification, CAPM and efficient fronter, efficient market hypothesis, Black-Scholes option pricing formula, etc. This is all stuff I've leared at University 20 years ago. When I learned about these, and most of the time when I read about these, it's in the context of trading: you are investing/trading, where do you put your (or your hedge fund's) money, how much do you diversify, etc. One thing that surprised me was that many of their original papers were addressing these topics in the context of corporate finance.

So I've began thinking: currently the role of data (and data science) is to improve execution when the company decides to do X. But what about helping to decide what X to do and not to do, ie. where to invest money, or not to invest, and how to balance this portfolio? In other words, given where and how we can invest money [+]:

what are the expected returns?
- what is the best way to approximate returns?
what are the risks and covariances?
- what is the best way to approximate risk and covariance?
in the language of CAPM, what is the efficient frontier for our company?

In practice this may work out to be very similar to the Monte Carlo / variance project that we worked on previously, where they supported decisions by creating a distribution of outcomes (vs point estimates), which reportedly had significant impact.

[+] This is the exercise we play every year Q4 called budgeting: we ask for money.

Personal Development

One of my 2023 OKRs for Data Science was "Build a credible career path for Data Scientists", where I scored 0. I want to change this 0 to a 1 in 2024, also for Data Engineers and Platform Engineers.

The fundamental problem is that we are a small organization, with 12 Data Scientists (6 Dubai, 6 Remote), and even less Data Engineers (2 Dubai, 2 Remote) and Platform Engineers (4 Dubai). We have very small population per function, and it's hard to establish statistical baselines/calibrations at each level, especially with most people working alone (esp. DS), and having to do be able to "swim" alone in the "stormy waters", irrespective of level. At tech companies, with 100+ populations per function, the people making the promotion decisions are of the same function, but ranked higher (the people deciding about the promotion of a Data Engineer are the TL/Data Engineering Manager, Director DE, VP DE, CTO, all technical folks. For us this is not the case — of the roughly 5-6 people who need to agree to do a promotion, most will be non-technical. So the reason for promotion needs to be something that non-technical people understand. Examples that seem to work:

Alice is a Data Scientist, but is also leading a team of 4 Data Engineers
Bob is a Data Scientist, we have a big PAIN [+] in area X, and Bob solved it
Alice led project X to completion and project X was highly visible to executives
Bob is critical to keep the lights on, otherwise PAIN
Alice is already managing 5 people of the same functions
Bob had X million AED impact this year

Realistically, we need at least 2 of these for a strong promotion case.

[+] PAIN needs to be something that non-technical management understands, like escalations

We need to create personalized plans for everybody that attempt to create these situations.
We will create the plan by understanding the strengths and weakness of everybody, and where they want to improve, along dimensions like in the spider chart above.
It's the person's job to remember this plan and work on it continuously!
We need to accept that there's an element of luck, in the sense that the environment also plays a role, and we don't control all of our environment.
We create our own luck, and, luck favors the prepared. Eg. if there's an opportunity to work on a high-impact, high visibility project, it's likely we will pick somebody who is already performing at a high level, because that's the right choice for the organizatioon. So even if you feel you're currently not in a lucky situation, you need to run at high performance, so when the opportunity comes, you're in a good position to leverage it.

Job security and layoffs

We are not planning layoffs — this has not been on my mind for many months. The reason I added this section is that — to my surprise — some tech companies started doing layoffs in the second week of January: Google, Twitch, Amazon, Discord, etc.

As I just said, we are not planning layoffs. Also, even that were the case, we're already a very barebones team, so I personally am not worried. Having said that, in 2024, like in every year, the best strategy to hedge the risk of job loss (for you, me and everybody) is to:

be a high performer
work on your visibility, so people know you're a high performer (!)
"be so good they can't ignore you", so in case it comes to it, you quickly get a new job
outside of work, invest time and energy into thinking about your personal finances and investments — the best job is the one you don't really need 🙂

The part in quotes is a reference to Cal Newport's excellent book So Good They Can't Ignore You (originally a quote from comedian Steve Martin), which I read many years ago, and I found to be true:

Georgetown University professor Cal Newport debunks the long-held belief that follow your passion is good advice, and sets out on a quest to discover the reality of how people end up loving their careers. Not only are pre-existing passions rare and have little to do with how most people end up loving their work, but a focus on passion over skill can be dangerous, leading to anxiety and chronic job hopping. Spending time with organic farmers, venture capitalists, screenwriters, freelance computer programmers, and others who admitted to deriving great satisfaction from their work, Newport uncovers the strategies they used and the pitfalls they avoided in developing their compelling careers. Cal reveals that matching your job to a pre-existing passion does not matter. Passion comes after you put in the hard work to become excellent at something valuable, not before.

In other words, look out for areas where you have an opportunity to excel, work hard, invest a lot of time to become a master in that area, and then enjoy the fruits of your work. For example, if you're working on forecasting projects, you should become the master of forecasting and financial data science, even if he had no a priori "passion" for this topic back when he started working on this 2 years ago. Then, once you're the undisputed master at the organization, then you'll enjoy it, and maybe it will feel like you had passion all along!

Books

In no particular order, here are some books relevant to 2024 (that I have read or have bought and plan to read):

Experimentation for Engineers
Reliable Machine Learning
Information Theory, Inference and Learning Algorithms
Making Databases Work
- collection of Stonebraker papers
- he's the father (along with Jim Gray) of much of the database technology we use today
- started Ingres, which then became Postres
The Staff Engineer's Path
- Pradeep is reading this
Patterns, Predictions, and Actions
- I've already read this, highly recommended
Financial Econometrics
- because I mentioned Markowitz and CAPM
Beautiful C++
- I've already this, highly recommended
- as a former C++ programmer, I'm always looking for ways to bring back C++ into my life 🙂
Machine Learning Design Patterns
The Little Book of Deep Learning (recommended by the legendary John Carmack)
Deep Learning: Foundations and Concepts
Understanding Deep Learning
The Science of Deep Learning
Efficient Linux at the Command Line
- Mohit is reading this
System Design Interview Volume 1 and Volume 2
Machine Learning System Design Interview
- I've read all 3, they are great
- One of the best ways you can learn how other companies do things is to see what they ask on interviews
- Here, the author has done this work and asked lots of people at big tech companies, and compiled it into these books!
CPython Internals
- I read this, I didn't get much out of it
- CPython mostly works as you'd expect it works if you know the basics about compilers and interpreters
So Good They Can't Ignore You
My Life as a Quant

Note: I definitely buy more books than I can read..

Personal reflections

In 2024 I plan to make some minor modifications to how I manage myself:

be more rigorous with keeping a daily log
write weekly updates
defend my time better
- I think I spent a lot of time in 2023 in pointless meetings, which ultimately didn't add any value for the company o rme
- eg. reduce time spent in 1v1s, but increase the effectiveness: I continue to strongly believe in 1v1s, but my hypothesis is that a lot of my hours spent in 1v1s in 2023 was not effective
- .. but, 1v1s are not the main culprit, it's just "too many pointless meetings"
write more
- eg. documents like this
in the last couple of months I've been thinking a lot about the productivity of people like Elon Musk — why is it that some people are 100x or 1000x much more productive than me?
- of course billionaires who own multiple companies and tell 1000s of people what to work on have significantly higher leverage, so there is no real mystery here 🙂
- there are lots of self-help books written, with advice like "wake up at 5am", "write lists", etc.
  - this is actually good advice, I do a lot of things like this (but far from perfect), and I think things like this can 2x your productivity
- the one thing I notice when reading stories about highly successful people is that they have the ability to be demanding (vs always just being nice)
  - a little bit like when I said in Q1 "everybody owes me a Generative AI prototype"
  - maybe I should be more like that
    - with myself and others
stretch goal: in terms of the way we run, we are essentially a remote team: how should our tooling look like, given that? eg. simple things like vacation tracking or who-is-in-the-office are still not solved

How to achieve high performance, and, ratings

2024-01-21T00:00:00+01:00

I wrote this document for my team members — Data Scientists and Data Engineers — to help them do a better on their annual performance reviews.

Introduction

The purpose of this brief note is to remind us what the key to getting good performance, and, ratings is. I write this because, after looking through 10+ reviews this week, I think it's a good idea — we have room for improvement.

The key to getting good performance ratings is to consciously invest time, energy and attention to all steps of the performance journey. The steps are, roughly:

Set performance goals and OKRs 📝
Throughout the year, work hard and focus on impact 📈
Visibility 👀
Fill out self-review and score OKRs 🖋️
Manager speaks to employee and rates employee
Calibration

I will concentrate on the first 4 below.

Set performance goals and OKRs

Setting strong, ambitious goals, so that you feel like you cannot hit all of them is key here. If you set yourself a few, easy goals, even if you hit them, your manager will say, with all honesty, "great job, this is baseline performance (Meets Expectation)". If you don't set strong, ambitious key results on a certain objective, your best chance of going beyond the baseline is probably to hope for an external piece of work "forcing" you to go beyond that, even though it was not part of the OKRs. But why would you do that? Plan for excellence, don't wait or hope for it to happen!

Most large organizations have generic objectives that don't change much year to year. A good rule of thumb is to have 5-10 Key Results (KRs) per Objective. Even for objectives like Company Values you can enumerate plans/ideas of how you will exceed! Make them ambitious: when you set them, you should feel like a bit overwhelmed, like "I'll be happy if I hit half of this". I feel like that every time we build team OKRs, and in the end we usually end up hitting 70-80%, because a year is actually a long time, we're all hard workers and often things are not as hard as they seem.

Personally I'm a big fan of making goals and OKRs. I also do it personally. Every year, in the first couple of weeks in January I build my annual goals (not OKRs) in a Google doc. Usually it ends up being 2-3 pages long. I group them under headings like Reading, Writing, Professional, Fitness, Relax, Money. You can check a sample here.

OKRs, or more broadly, goals, are not something we do because the company is asking us to. We'd do OKRs and goaling anyway. As you can see, although I don't follow the OKR structure specifically, I do something similar personally also, and have for a long time. The point of goaling is to think through what we want to do, to make a plan, and then write down what "good" looks like in terms of outcomes. An additional point about "goaling" is the dopamine hit that we get when we accomplish a key result or goal item.

We are data people and love metrics, so I won't sell SMART goaling goaling, but give a brief overview below. Here are the five parts of SMART:

Specific: Goals should be clear and specific, so you know exactly what you're aiming for. This involves answering the "W" questions: What do you want to accomplish? Why is this goal important? Who is involved? Where is it located? Which resources are needed?
Measurable: A goal must have criteria for measuring progress. This helps to stay on track and meet deadlines. Ask questions like: How much? How many? How will I know when it is accomplished?
Achievable: The goal should be realistic and attainable to be successful. This means you have to figure out how to accomplish it and whether it's possible, given the current constraints such as time and resources.
Relevant: The goal should matter to you and align with other relevant goals. It should address something that is the right time and fits with other efforts or needs. Ask: Does this seem worthwhile? Is this the right time? Does this match our other efforts/needs?
Time-Bound: Every goal needs a target date, so there's a deadline to focus on. This part of the SMART goal criteria helps to prevent everyday tasks from taking priority over longer-term goals. Ask: When can I achieve this goal? What can I do six months from now? What can I do six weeks from now? What can I do today?

In my experience, when making annual goals, most of the time it's not realistic to hit all 5 components of SMART, with Specific and Measurable being the trickiest. My approach is not to obsess too much over this, but to write something which is clear to you and your manager, and perhaps later on you can come back and improve the wording. I don't believe in hard-locked OKRs, minor refinements to definitions are fine!

Are you doing a good job on this? Most of us can improve:

have you thought through what you want to achieve this year?
- also on Value objectives?
do you have 5-10 KRs per Objective?
did you invest time and mental energy to make your goals as SMART as possible?
have you taken multiple stabs at goaling, you've slept over it between iterations?
have the OKRs made you stretch and improve or it is about doing the baseline?
bonus: have you built personal goals?

Work hard and focus on impact

This will be the shortest section.

We don't have a problem here.

Most of us work hard and accomplish good outcomes for our organization.

But remember: if you do a "good job" throughout the year, that's the baseline, so that will be "Meets Expectations". The expectation, towards all of us (also your manager), is that we do a good job. The baseline is not that we don't work, and if we do our job then we're already exceeding. The reason I point this out is that sometimes, on self-review, the person will essentially list out baseline activities of their job and then rate themselves "Exceeds" or "Exceptional".

The easiest way to not exceed is to forget about the performance journey and OKRs thoughout the year. I see this happening a lot, when people scramble in December to hit as many key results as possible. The smart thing to do is to create a system — with your manager — to continuously keep the OKRs in mind and work on them. For example, if your KR is to "fix 24 tech debt tickets", it's definitely easier to do 2 per month versus trying to do as many as possible in December!

Visibility

If a tree falls in a forest, and there’s no one around to hear it, does it make a sound?

The physics answer is, of course it does. But a company is a set of people, and people are not like physics! At any organization, you need to make sure that people know about all the good work you're doing! No exceptions.

You may think, "But why? I just want to do a good job, and I want to be recognized for it. Isn't it enough if you know? This is unfair, it wasn't on my JD."

The situation is simple: in any organization beyond a trivial size (the cut-off is somewhere around 100-150, the Dunbar number), there will be some sort of "quorum" or "committee" that will ultimately evaluate you. So, it's not just your manager that needs to know — you have to have social proof. You and your manager can write down the list of these 5-10 people! Your best bet at getting good outcomes based on your work is if these people:

know who you are
know what you did
think you did a great job that had a positive impact for the company

You may think, "But Marton, isn't it your job to make sure I get recognized by these people?". In a way, yes, it's my job. But imagine Alice and Bob, both did a great job, and Alice worked on her visibility, while Bob relied on his manager for it. However, Bob's manager is actually responsible for providing this visibility for 20+ people, and is not able to remember everything that happened with 20 people across 12 months. Who has better chances of getting recognized? If senior management is already aware about Alice and her work, they are already primed before they come to the calibration meeting.

There is all the reason in the world to work on your visibility, and no reasons against it. Working on your visibility is not "cheating". Also, your manager will personally be grateful to you, because they will have an easier time getting you good outcomes.

It's also worth noting, as I said, that this is not just our company. I can tell you that for example promotions at Facebook, which has a very different structure, but it was the same story at a high-level. To get promoted was very hard, you had to present a case to a Promotion Committee, and it was much more likely to get approved if the members knew who you were and knew what you did, so it was an "obvious case". It was hard, people worked very hard on improving their visibility.

Fill out self-review and score OKRs

This week, as I was doing everybody's reviews, my starting point was to read everybody's self-review. I was surprised that many people chose to not fill out the self-review at all, or not put much energy into it! In these cases I sent it back and asked the person to fill it out and put effort into it.

The self-review is essentially the last part of visibility. Not putting in the effort to present your case here is a huge wasted opportunity! As a manager, I can tell you how I do reviews: I open up the system, and see what you wrote (text), and then see if it supports your rating. Of course, I also try to remember all the things that happened in the year, the good things that you did. But in general, I'm assuming that if something is worth mentioning, you will mention it in the self-review — because it's your job to do so at this stage, and also your self-interest!

A good self-review looks like this:

it lists out all the good things you did this year — be concrete, not generic
it is specific about what you did (vs the team that you worked on)
- use "I" instead of "we"!
it is specific about the impact (eg. 10M USD)
- concrete numbers are better than estimates
- bottom-of-funnel numbers (eg. net profit) are better than top-of-funnel numbers (eg. income)
- numbers other than USD numbers are fine (eg. # of techtalks, # of interviews, # pageviews, etc)
it's fine to practice some salesmanship, but be concrete
this is the time (along with generic within-the-year visibility) to be bold
your text should support your rating — if you rate yourself "Exceeds" or "Exceptional" for an objective, don't put down bullet points and sentences that essentially say "This is my baseline job and I did it." Your comments need to support why you were beyond the baseline.
if you don't take OKRs and performance reviews seriously, don't expect your manager to take you seriously!

A note on calibrations

The way calibrations work is the following: Corporate HR sets a budget of a certain % of people that can be within each bucket (Meets, Exceeds, Exceptional). The input of calibration is all the people in business unit, with their scores as their manager set it.

Every year, there are way too many people in the Exceeds and Exceptional buckets, so Calibration is about discussing and figuring out "who to move down". So let's say there are too many people in Exceeds, then Alice would have to explain to the group why she rated Bob 3.8 Exceeds. All the other managers also explain, and by the end of the meeting (or meetings), the managers have to agree who is actually, upon hearing their manager’s explanation, not really Exceeds, but Meets Expectation. In a way, it's grading on a curve.

This is why visibility is so important. If I'm explaining why Bob is Exceeds, and nobody in the calibration meeting even knows who Bob is, it's much tougher to make that argument. For Exceptional, a lot of people better know who Bob is and what he did!

As I mentioned above, calibration is not per function or per department, it's across the business unit.

A note on promotions

I think everybody knows this, but I will say it just to be safe. There is never any guarantee for a given rating on promotion, even if we do a great job. For example during Covid, everybody got a default "Meets" rating, no increment and no bonus.

This applies also for promotions. In my experience, at our company, promotions are "needs"-based. So the most likely way to get promoted is if there's a slot that the company needs to fill. It has happened, but not so often, that people get promoted simply based on a performance plus level argument (Bob did great, he is currently at level L, but he has been consistently performing at level L+1).

Wordcount III: Beating the system `wc`

2023-09-29T00:00:00+02:00

Introduction

In the previous two articles I re-implemented wc is modern C++ as a toy exercise:

I had three reasons to write a third version.

First, my core architectural takeaways at the end of the second article was since wc is a fully specified tool that is not going to see new features introduced, a lot of the usual C++ abstractions are not neccesary. For example, the code doesn't need to follow the Open–closed principle and be open for extension, but closed for modification. So all the complexity introduced in the second version isn't good complexity.

Second, I also found that the second (and also first) implementation loses time compared to my system wc in the following for_each loop (same for different ways of iterating a C++ container):

unsigned process_block(
    const char* cp, unsigned n, stream_counter_vector_t& counters, bool eof) {
    long remaining = n;
    std::mbstate_t ps{};
    while (remaining > 0) {
        // ... code omitted
        std::for_each(counters.begin(), counters.end(),                    <-----
            [&](auto& c) { c->process_wchar(wc, num_bytes, error); });     <-----
        // ... code omitted
    }
    return remaining;
}

Third, I found that the system wc has a trick up its sleeve when doing line counting. Instead of processing each character and checking whether it's a \n, it uses the memchr() family of functions. memchr() takes a buffer of character and finds the first occurence of a character, in this case \n, and is much faster than doing the same in a C loop.

Simples classes and code

To keep this implementation simple I used a very simple base counter class, and derived all other implementation classes from this:

class counter_t {
    public:
    void reset() { count = 0; active = true; }
    bool active = false;
    long count = 0;
};

For example, the class counting bytes is now as simple as:

lass byte_counter_t : public counter_t {
    public:
    void process_file(const std::string fname) {
        struct stat st;
        stat(fname.c_str(), &st);
        count = st.st_size;
    }
    void process_wchar(const wchar_t, unsigned num_bytes, bool) {
        count += num_bytes;
    }
};

Since I realized extensibility is not required here, the class itself stores whether the count should happen or not. The active flag is flipped based on the command-line parameters, and checked in the internal loops. This means that the dynamic for loop is now replaced with compiled ifs:

void process_wchar(const wchar_t wc, unsigned num_bytes, bool error) {
    if (byte_counter.active)
        byte_counter.process_wchar(wc, num_bytes, error);
    if (char_counter.active)
        char_counter.process_wchar(wc, num_bytes, error);
    if (line_counter.active)
        line_counter.process_wchar(wc, num_bytes, error);
    if (longest_line_counter.active)
        longest_line_counter.process_wchar(wc, num_bytes, error);
    if (word_counter.active)
        word_counter.process_wchar(wc, num_bytes, error);
}

As I will show shortly, this yield much faste code. Obviously this is repeated code that goes against the DRY principle, but here it's permissible.

Lastly, the use of memchr() for in linecounter:

unsigned process_block_lines_only(
    const char* cp, unsigned n, bool eof) {
    void* cur = static_cast<void*>(const_cast<char*>(cp));
    while (cur != nullptr) {
        auto remaining = n - ((char*)cur - cp);
        cur = memchr(static_cast<void*>(cur), '\n', remaining);
        if (cur != nullptr) {
            line_counter.count++;
            cur++;
        }
    }
    return 0;
}

Headers

To further clean up the code, I factored the command-line parsing and tabulation code into a separate headers clargs.hpp and tabular.hpp. The complete code is up on Github.

Performance

With the above changes and optimizations, the code now runs about 10% faster than my system wc when running all 5 counters in file mode:

$ time wc -c -m -l -w -L playground/MNIST_data/t10k-images-idx3-ubyte.gz
   6121   34669  874589 1648877     402 playground/MNIST_data/t10k-images-idx3-ubyte.gz

real    0m0.101s
user    0m0.100s
sys     0m0.000s

$ time ./wcpp_v3 -c -m -l -w -L playground/MNIST_data/t10k-images-idx3-ubyte.gz
    6121   34669  874589 1648877     402 playground/MNIST_data/t10k-images-idx3-ubyte.gz

real    0m0.089s
user    0m0.088s
sys     0m0.000s

Results are similar when reading from stdin:

$ time cat playground/MNIST_data/t10k-images-idx3-ubyte.gz | wc -c -m -l -w -L
   6121   34669  874589 1648877     402

real    0m0.101s
user    0m0.104s
sys     0m0.000s

$ time cat playground/MNIST_data/t10k-images-idx3-ubyte.gz | ./wcpp_v3 -c -m -l -w -L
    6121   34669  874589 1648877     402

real    0m0.091s
user    0m0.080s
sys     0m0.008s

Conclusion

This was a fun toy exercise so far. One final optimization possibility would be to multi-threading when parsing multiple files.

Wordcount II: Introducing a cleaner C++ class hierarchy to `wc`

2023-09-23T00:00:00+02:00

Introduction

In the previous article, I wrote a modern C++ version of the Unix command-line utility wc. The basic architecture of the program was a collection of state machines, one for each type of count that the utility supports:

bytes
characters (Unicode or ASCII)
words
lines
the longest line

Each counter was implemented as a class that derives from and implements from class counter:

class counter {
    public:
    virtual bool has_optimization() const { return false; }
    virtual void optimized_count(const std::string fname) { }
    virtual void process_wchar(const wchar_t, unsigned, bool) = 0;
    unsigned long get_count() const { return count; }
    void add(unsigned long i) { count += i; }
    void clear() { count = 0; }
    protected:
    unsigned long count = 0;
};

The core state machine logic is implemented in process_wchar(). In the case of eg. byte counting this is trivial:

void process_wchar(const wchar_t, unsigned num_bytes, bool) {
    count += num_bytes;
}

In the case of words it's a real edge-triggered state machine:

void process_wchar(const wchar_t wc, unsigned, bool error) {
    if (error)
        return;
    bool whitespace = is_word_sep(wc);
    bool printable = iswprint(wc);
    if (in_word && whitespace) {
        in_word = false;
    } else if (!in_word && printable && !whitespace) {
        in_word = true;
        count++;
    }
}

The source code for the previous version is here.

Optimized counts

However, for some counts, there are more optimized ways to get the desired result. The simplest example, and the one I implemented in the previous post, is byte counting. If the user of wc just wants to know the byte count by typing wc -c or wc --bytes and the input is a file (not stdin), then we can just use fstat() and ask the file system for the size in bytes, and return it — no need to actually open the file and count the bytes.

This is what the has_optimization() and optimized_count() functions in class counter are for. But, I was unhappy with this, since it mixes multiple things in one class:

querying whether the counter supports optimized counting
performing the file-based optimized count
performing the character-wise state machine-based count

Also, even classes that don't support optimized counting will contain (empty) functions for this, derived from the base class.

Abstractions

To address this, I started to create separate classes to encapsulate separate concepts. In general there are counters, which can return an unsigned long count; in our case, a counter is either a file system based one (eg. fstat() for character counting), or a stream-based one (the state machines):

class counter {
    public:
    unsigned long get_count() const { return count; }
    protected:
    unsigned long count = 0;
};

class filesystem_counter : public counter {
    public:
    virtual void process_file(const std::string fname) = 0;
};

class stream_counter : public counter {
    public:
    virtual void process_wchar(const wchar_t, const unsigned, const bool) = 0;
};

With this structure, the fstat() case is nicely separated from other concerns:

class byte_filesystem_counter : public filesystem_counter {
    public:
    void process_file(const std::string fname) {
        struct stat st;
        stat(fname.c_str(), &st);
        count = st.st_size;
    }
};

Then there is the concept of deciding which one is supported for the given count, and retrieving the supported one (with sensible default implementation):

class counter_policy {
    public:
    virtual bool supports_file_based() const {
        return false;
    }
    virtual filesystem_counter_t new_filesystem_counter() const {
        return nullptr;
    }
    virtual stream_counter_t new_stream_counter() const = 0;
};

To implement, other class byte_filesystem_counter shows above, we need two more classes:

class byte_stream_counter : public stream_counter {
    public:
    void process_wchar(const wchar_t, const unsigned num_bytes, const bool) {
        count += num_bytes;
    }
};

class byte_counter_policy : public counter_policy {
    public:
    virtual bool supports_file_based() const {
        return true;
    }
    virtual filesystem_counter_t new_filesystem_counter() const {
        return std::make_unique<byte_filesystem_counter>();
    }
    virtual stream_counter_t new_stream_counter() const {
        return std::make_unique<byte_stream_counter>();
    }
};

This is quite clean, but it has drawbacks. After parsing the command-line arguments, I construct an std::vector<counter_policy> corresponding to what needs to be counted, and eventually an std::vector of counters. This is where the problems start: an std::vector<derived_class> cannot be used as an std::vector<base_class>, because even though all instances of derived_class are base_class also, this is not true for container types such as std::vector. So, if I create a vector of counter, and I pass it to a function which runs the stream-based state machine logic, then it needs to static_cast<> to stream_counter as it iterates through the vector, which is ugly and not type safe. Also, if I wanted to, I cannot mix different types of counters in the vector, since I don't know how to cast (was it a stream-based or file-based one?). Or, I can decide up-front that, due to the command-line arguments used, I will instantiate all stream-based or all file-based counters: in this case I don't need to do ugly casts, but now I have a different problem: the functions that display the results as a table, they don't care how the counts were constructed, they just care about the numbers, so they do want to just call counter::get_count(), so they want to get passed an std::vector<counter>, irrespective of whether the count was stream-based or file-based.

In the end, I decided that instead of passing counters to the display functions it's easier to extract the counts into typedef std::vector<unsigned long>s and pass that. I still have to have two different functions to extract, but that can be handled reasonably with templates:

template <typename T>
count_vector_t to_counts(const std::vector<T>& counters) {
    count_vector_t counts;
    std::for_each(counters.begin(), counters.end(),
            [&](auto& c) { counts.push_back(c->get_count()); });
    return counts;
}

So, for example, the function which handles counting things on stdin looks like this:

table_t process_stdin(const cl_args_t& cl_args) {
    table_t table;
    auto policies = policies_from_arguments(cl_args);
    stream_counter_vector_t counters;
    stream_counters_from_policies(policies, counters);
    process_stream(&std::cin, counters);
    auto counts = to_counts<stream_counter_t>(counters);
    table.push_back(tabulate(counts));
    return table;
}

The function countings things per file has two branches, one if everything can be counted on a file-based way, or it needs to get processed stream-wise, character-by-character:

count_vector_t process_file(const std::string fname, const cl_args_t& cl_args) {
    count_vector_t counts;
    auto policies = policies_from_arguments(cl_args);
    auto all_sfb = std::all_of(
        policies.begin(),
        policies.end(),
        [](auto& policy) { return policy->supports_file_based(); }
        );
    if (all_sfb) {
        filesystem_counter_vector_t counters;
        filesystem_counters_from_policies(policies, counters);
        std::for_each(counters.begin(), counters.end(),
            [&](auto& counter) {
                counter->process_file(fname);
            });
        counts = to_counts<filesystem_counter_t>(counters);
    } else {
        stream_counter_vector_t counters;
        stream_counters_from_policies(policies, counters);
        std::ifstream f(fname);
        if (f.is_open()) {
            process_stream(&f, counters);
        } else {
            std::cerr << "cannot read from file: " << fname << std::endl;
        }
        counts = to_counts<stream_counter_t>(counters);
    }
    return counts;
}

This second version of wc, with more abtracted classes, is here.

Complexity

Unfortunately, overall there is a hefty price to pay for these additional abstractions. The original version was 469 lines of code, this one is 567, so it's an extra 100 lines of code, or roughly 20% more. Unfortunately, I don't think it's cleaner, clearer, or easier to modify (in terms of lines of code to be touched). This is a case of too much complexity. A good sign of this is the long list forward declarations and typedefs I introduced to keep things somewhat readable:

class counter;
class counter_policy;
class stream_counter;
class filesystem_counter;

typedef std::unique_ptr<counter> counter_t;
typedef std::unique_ptr<counter_policy> counter_policy_t;
typedef std::unique_ptr<stream_counter> stream_counter_t;
typedef std::unique_ptr<filesystem_counter> filesystem_counter_t;
typedef std::vector<counter_policy_t> policy_vector_t;
typedef std::vector<counter_t> counter_vector_t;
typedef std::vector<stream_counter_t> stream_counter_vector_t;
typedef std::vector<filesystem_counter_t> filesystem_counter_vector_t;
typedef std::vector<unsigned long> count_vector_t;

// .. actual class definitions start here

The only "win" in all this was the realization that it's cleaner to just pass std::vector<unsigned long>s to the display functions, so they can be completely independent of however I decide to implement counters.

Performance

This version of wc, compared to my previous version, has a different code structure, but actual runtime performance is identical. At this point it started to bother me that I invested so much time, the performance is not too far off from my sytem wc, but it's still about 10% slower. So I did some profiling, and found that following: in the case that's relevant for benchmarking — the stream-based state machine branches — I lose time in the following for_each loop (this is true for both my previous and the current implementation):

unsigned process_block(
    const char* cp, unsigned n, stream_counter_vector_t& counters, bool eof) {
    long remaining = n;
    std::mbstate_t ps{};
    while (remaining > 0) {
        // ... code omitted
        std::for_each(counters.begin(), counters.end(),                    <-----
            [&](auto& c) { c->process_wchar(wc, num_bytes, error); });     <-----
        // ... code omitted
    }
    return remaining;
}

I tried multiple different ways of iterating, all had identical performance.

This was quite a blow, since the whole point of implementing wc in C++ is to abstract the state machines into their own classes, instantiate the state machines we need based on the command-like arguments, and perform only those counts. This — irrespective of the abstractions introduced in this post — does yield significantly cleaner code than the original C version in coreutils, which is one giant while loop with 10s of variables over 100s of lines of code, ie. all state machines are spilled into one another, with ugly, hard to follow, not-obviously-correct code.

Open-closed principle

After I wrote this version and in the end decided that it's over-engineered and all the astractions are not worth it, I realized that many otherwise good software engineering principles don't apply to wc. For example, the S and O in SOLID stand for (from Wikipedia):

The Single-responsibility principle: There should never be more than one reason for a class to change. In other words, every class should have only one responsibility.
The Open–closed principle: Software entities should be open for extension, but closed for modification.

Both cases are about what happens when code changes to incorporate features in the future. But wc is not like that: wc has been around with minimal changes since 1971, so more than 50 years! It's one of the few exceptions where you don't have to anticipate future changes to the code.

Conclusion

In the next post, I will show a 3rd variation of my C++ wc implementation, with a reasonable trade-off between C++ complexity, readability and run-time performance, based on the above considerations — this time with the explicit goal of beating my system wc in performance tests.

Wordcount I: Implementing the Unix command-line tool `wc` in modern C++

2023-09-10T00:00:00+02:00

Introduction

Like many programmers, I started programming on the Commodore 64, in BASIC, at age 6-7. A few years later, in elementary school, I graduated to Pascal, using the excellent Turbo Pascal compiler. Later in High School I learned C and C++ at about the same time. I remember being 16 and going on a trip with my family to Yosemite Park, where I chose my then-favorite Teach Yourself ANSI C++ in 21 Days book over going hiking with my parents.

I finished University in 2004 (Computer Science, Budapest University of Technology) and I got a job as a junior C++ programmer at the company Graphisoft, working on the architectural CAD product ArchiCAD. Later I had a startup called Scalien, where we wrote a distributed key-value database called ScalienDB (code on Github) in C++.

In all this reminiscing it's worth noting that I was not a good programmer initially. I was definitely not (and am still not) a natural talent like John Carmack, identifying problems worth solving and writing elegant and highly efficient solutions that millions of people still marvel at 20 years later (Quake source code on Github). In retrospect I would say I became a good programmer towards the end of my time at Graphisoft, and I wrote mostly good code at Scalien. However, this was in 2009 onwards, so, before the onset of modern C++. Back then my position was that the best way to use the C++ language is as a C-with-objects. Specifically, I did not enjoy using the Standard Template Library (STL), so in ScalienDB we rolled our own container classes, which is considered a big no-no per the C++ Core Guidelines (see below).

After Scalien I pivoted to working in the data field, where I was able to leverage my second University degree as a Physicist, which I earned while working at Graphisoft. However, this also meant leaving C++ behind, as the lingua franca of data in the last 10 years became Python, and SQL for querying relational databases.

Lately I realized I miss writing pedal-to-the-metal shotgun-to-the-foot C++ code. Part of this is the lively progress the language has made, with C++11, C++14, C++17 and the last C++20 standard introducing exciting new features into my favorite programming language. So I ordered a bunch of books on modern C++ development a few months ago, among them the latest version of Bjarne Strousrup's book (of which I owned an earlier edition before I gave it to my University library in 2015), and started to devour them.

I started by reading the excellent Davidson & Gregory book Beautiful C++ the last couple of weeks — to my surprise it's one of the best technical books I've read in a long time, in parallel discussing hard earned programming wisdom with new language features. The book is built on top of the C++ Core Guidelines, a set of rules for writing good modern C++ code, edited by Bjarne Stroustrup and Herb Sutter.

After reading the book cover-to-cover I had a strong urge to go and use all the new features and write some modern C++ code. I didn't want to write purely toy code, I wanted to approximate solving a real-world problem — but I also have limited time so anything big is out of the question. I settled on re-implementing, to a satisying (to me) degree the standard Unix command-line tool wc, most commonly used to count lines in a file using wc -l <file>.

The out of the box wc --help output on my Debian devbox is:

$ wc --help
Usage: wc [OPTION]... [FILE]...
  or:  wc [OPTION]... --files0-from=F
Print newline, word, and byte counts for each FILE, and a total line if
more than one FILE is specified.  A word is a non-zero-length sequence of
characters delimited by white space.

With no FILE, or when FILE is -, read standard input.

The options below may be used to select which counts are printed, always in
the following order: newline, word, character, byte, maximum line length.
  -c, --bytes            print the byte counts
  -m, --chars            print the character counts
  -l, --lines            print the newline counts
      --files0-from=F    read input from the files specified by
                           NUL-terminated names in file F;
                           If F is - then read names from standard input
  -L, --max-line-length  print the maximum display width
  -w, --words            print the word counts
      --help     display this help and exit
      --version  output version information and exit

GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Full documentation at: <http://www.gnu.org/software/coreutils/wc>
or available locally via: info '(coreutils) wc invocation'

I set of off to re-implement all the 5 basic modes of wc, in C++, while trying to maintain compatibility with the standard C implementation, to a reasonable degree.

The full source code of my C++ implementation is up on Github. You can compile like:

$ g++ -o wcpp -O3 wcpp_v1.cpp

The C implementation I most often used to peak to understand classic behaviour is in the coretuils Github repo.

Command line arguments

wc only has a few command line arguments, but each has a - and a -- way to specify it. Additionally, wc supports passing in -xyz, which is equivalent to passing in -x -y -z, and passing in the special key=value type argument --files0-from=F. To my surprise, there is no C++ standard library for parsing arguments, and even boost::program_options does not support this -xyz behaviour. So I wrote a few helper functions utilizing the STL to implement this:

struct cl_args_t {
    std::set<std::string> flags;
    std::map<std::string, std::string> key_values;
    std::vector<std::string> filename_args;
};

cl_args_t get_cl_args(int argc, char** argv) {
    cl_args_t cl_args;
    for(auto i = 1; i < argc; i++) {
        auto arg = std::string(argv[i]);
        if (arg[0] == '-') {
            if (cl_args.filename_args.size() > 0) {
                std::cerr << "error: switches must precede filename arguments"
                << std::endl;
            }
            auto eqi = arg.find('=');
            if (eqi == std::string::npos) {
                // no '=' in arg
                cl_args.flags.insert(arg);
            } else {
                auto k = arg.substr(0, eqi);
                auto v = arg.substr(eqi+1, arg.length());
                cl_args.key_values[k] = v;
            }
        } else {
            cl_args.filename_args.push_back(arg); 
        }
    }
    return cl_args;
}

void normalize_flags(cl_args_t& cl_args) {
    // convert eg. -lm to -l -m
    std::set<std::string> new_flags;
    for (auto f : cl_args.flags) {
        if (f[0] == '-' && f[1] != '-' && f.length() > 2) {
            for (auto i = 1; i < f.length(); i++) {
                std::string new_f = std::string("-") + f[i];
                new_flags.insert(new_f);
            }
        } else {
            new_flags.insert(f);
        }
    }
    cl_args.flags = new_flags;
}

void check_bad_args(const cl_args_t& cl_args) {
    std::set<std::string> accepted_flags = {
        "-l", "--lines",
        "-w", "--words",
        "-m", "--chars",
        "-c", "--bytes",
        "-L", "--max-line-length",
              "--help",
              "--version"
    };
    for (auto f : cl_args.flags) {
        if (!accepted_flags.count(f)) {
            std::cerr << "invalid option: " << f << std::endl;
            exit(1);
        }
    }
    for (auto k : cl_args.key_values) {
        std::string must_be = "--files0-from";
        if (k.first.substr(0, must_be.length()) != must_be) {
            std::cerr << "invalid option: " << k.first << std::endl;
            exit(1);
        }
    }
}

Counting architecture

wc supports counting 5 quantities:

bytes
characters
words
lines
the longest line

To support this, I defined an abstract base class called counter, and 5 child classes defining each counting logic:

class counter {
    public:
    virtual bool has_optimization() const { return false; }
    virtual void optimized_count(const std::string fname) { }
    virtual void process_wchar(const wchar_t, unsigned, bool) = 0;
    unsigned long get_count() const { return count; }
    void add(unsigned long i) { count += i; }
    void clear() { count = 0; }
    protected:
    unsigned long count = 0;
};

Essentially, each counter is assumed to be a state machine, and each invocation of the process_wchar(const wchar_t wc, unsigned size, bool error) triggers the state machine to process a new wide character wc of length size bytes that was read; if the current bytes to not form a valid character per the current locale then error is set and 1 byte is consumed. The has_optimization() and optimized_count() is a special path for supporting wc -c byte counting, in this case we don't have to actually count the bytes, we can just use fstat() to ask the operating system what the size of the file is — assuming we are processing files and not stdin.

As an example, here is the implementation for byte counting, the most trivial of the 5:

class byte_counter : public counter {
    public:
    bool has_optimization() const { return true; }
    virtual void optimized_count(const std::string fname) {
        struct stat st;
        stat(fname.c_str(), &st);
        count = st.st_size;
    }
    void process_wchar(const wchar_t, unsigned num_bytes, bool) {
        count += num_bytes;
    }
};

Handling wide characters

Although I've been using wc for 25 years, I never used it to count characters or words, so I was surprised to learn that a signiciant fraction of complexity comes from differentiating between bytes and (non-ASCII) characters. Essentially, in Unix a locale can be set, and wc honors this setting. On my Debian box, running locale shows:

$ locale
LANG=en_US.utf8
LANGUAGE=
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

This means that bytes are interpreted as UTF-8 characters by wc. So valid UTF-8 byte sequences are counted as 1 character by wc, and this is also honored by the word counting logic. However, if we set the locale to C, we switch to chars=bytes mode, and in this case wc -c and wc -m always outputs the same number; you can trigger this like LC_ALL="C" wc -c -m -l -w -L <file>.

However, not all byte sequences are valid UTF-8 (or whatever else the locale is set to), so invalid sequences are skipped one byte at a time, and a new attempt is made to read a valid character. The workhorse here is the C function mbrtowc(), which attempts to read a character per the current locale, and return it, or set an error. This logic is in my process_block() function, which after reading a block of bytes, repeatedly tries to extract wide characters. This function is quite C-like, and the core logic is similar to the standard wc implementations:

unsigned process_block(
    const char* cp, unsigned n, counter_vector& counters, bool eof) {
    long remaining = n;
    std::mbstate_t ps{};
    while (remaining > 0) {
        wchar_t wc;
        long num_bytes = 0;
        if (is_basic(*cp)) {
            wc = *cp;
            num_bytes = 1;
        } else {
            num_bytes = mbrtowc(&wc, cp, remaining, &ps);
        }
        bool error = false;
        if (num_bytes == 0) {
            // encountered null character
            num_bytes = 1;
        } else if (num_bytes == static_cast<std::size_t>(-1)) {
            // encountered bad wide character
            num_bytes = 1;
            error = true;
        } else if (num_bytes == static_cast<std::size_t>(-2)) {
            // encountered incomplete wide character, get more bytes if possible
            if (!eof)
                return remaining;
            num_bytes = remaining;
            error = true;
        } else {
            // success, read a wchar_t
        }
        std::for_each(counters.begin(), counters.end(),
            [&](auto& c) { c->process_wchar(wc, num_bytes, error); });
        cp += num_bytes;
        remaining -= num_bytes;
    }
    return remaining;
}

Bugs

Getting the locale-specific conversion of bytes to characters took at least half of all development time. I used large zipped files as a test, since these contain random characters, some of which are valid UTF-8, many are not, so this allowed me to test many branches of my code. In the end, I got everything to match except one case: U+2029, the Unicode paragraph separator character. My implementation calls std::iswspace(wc) which considers this as white-space, so it separates words and triggers an extra word count. However, my Debian system's wc does not consider this character to be a word separator. I reduced the difference to a test file containg: aU+2029a (5 bytes long, as U+2029 is represented in 3 bytes with UTF-8) , on which wc -w returns 1 word from the system wc, but my imlementation counts 2 words (test file is here). I believe mine is correct, so I left this difference.

Performance

I tested the performance by running all 5 counters on fairly large zip files repeatedly and using time. To my surprise, without any additional optimizations, just by compiling with -O3, my C++ implementation is only about 10% slower than my system's highly optimized C version from 20-30 years ago:

$ time wc -c -m -l -w -L playground/MNIST_data/train-images-idx3-ubyte.gz
  36468  207566 5259183 9912422     448 playground/MNIST_data/train-images-idx3-ubyte.gz

real    0m0.560s
user    0m0.548s
sys     0m0.008s
$ time ./wcpp -c -m -l -w -L playground/MNIST_data/train-images-idx3-ubyte.gz
   36468  207567 5259183 9912422     448 playground/MNIST_data/train-images-idx3-ubyte.gz

real    0m0.619s
user    0m0.620s
sys     0m0.000s

Note the difference in word counts due to the issue mentioned above.

Conclusion

I had a lot of fun writing wcpp.cpp and learned a lot about Unix, locale, wide character handling, and new features of C++ such as auto and for_each. I'm sure there are still a lot of optimization possibilities in terms of C++ language usage in my code, I'm happy to take feedback. I'm planning to continue on this path, and as time allows, implement other members of coreutils such as grep or find.

Better be first 99% of the time than second 100% of the time

2023-08-25T00:00:00+02:00

Introduction

This post is a review of the Donald MacKenzie's book Trading at the Speed of Light, which gives an excellent history and inside-peek of the world of High Frequency Trading, or HFT. Donald MacKenzie is a Professor of Sociology at the University of Edinburgh, Scotland, his main focus is the social studies of finance.

I've been interested in HFT for 20 years, and found this book interesting and a pleasure to read — highly recommended!

I took notes on the book and then used ChatGPT to generate most of the text in this article.

The Evolution of Trading Systems

The book delves deep into the history of trading systems, tracing the journey from pit trading to electronic platforms. Pit trading involved open outcry, where traders shouted orders and clerks recorded them in physical order books. Electronic trading emerged alongside, but the transition was far from smooth. It required overcoming cultural, technical, and regulatory hurdles, such as exchanges resisting algorithmic trading and early HFT firms resorting to robotic arms to punch keys as modern electronic API were not available.

Though HFT generates estimated revenues of \$3-5 billion in the U.S., it's a high-cost, slim-margin game. Even a large HFT firm making about \$1 billion in revenue faces extensive costs, making profitability precarious. The difference between success and failure often comes down to fractions of a cent per trade.

The books shows that today's HFT landscape is a culmination of history, physics, computation, politics, market structures and laws. Despite its efficiency, HFT has not significantly lowered the cost of financial transactions for retail investors. Regulatory bodies like the SEC and CFTC play vital roles but are often navigating uncharted waters.

Early Pioneers

One of the first electronic exchanges was Island ECN, introduced around 1998. Unlike its predecessors like Instinet, Island had an algorithmic matching engine. Early HFT firms like Automated Trading Desk (ATD) were among the first to realize the importance of milliseconds and started to move their servers closer to the Island's data centers to gain a speed advantage. ATD was bought by Citigroup for $680M in 2007. For a list of HFT firms today, see here.

Financial Evolution and Market Politics

The evolution of trading systems was politically fraught. Each step of automation led to job losses, inciting resistance from incumbents. For example, share prices initially quantized to 1/8 dollars to ensure market makers could maintain a wider bid-ask spread, thus securing profits: market makers offered to buy a stock at \$100, and offered to sell at \$100+1/8 (or more), since that was the minimum quantum of the market. When this changed to finer-grained price units of \$0.01, legacy exchanges and market makers lost business to more modern competitors.

Market makers create liquidity for other market sellers by always having buy and sell orders in the market book. Thus, if a seller $S$ wants to sell at time $T$, and a buyer $B$ wants to buy some later time $T'$, they don't have to wait for each other, since the market maker will buy from $S$ at time $T$, and later sell to $B$ at time $T'$, for a "small profit" related to the bid-ask spread, which itself is a function of the market quantum.

The Order Book and Matching Engine

A critical component of trading systems is the matching engine, responsible for pairing buy and sell orders. The evolution has led from human-operated matching engines to fully algorithmic systems. With algorithmic matching, an open order book became a possibility, allowing market participants to view buy and sell orders at various prices. An open order book could either be anonymous or not, affecting strategies and politics among trading entities.

The non-anonymity of some open order books presents a unique challenge for HFT firms. They need to be cautious not to make too much money off big firms like big banks. Banks, having the leverage to influence exchange policies, can block trades if they suspect an HFT firm is disproportionately benefiting from them. Therefore, HFT firms often limit their gains from a single identifiable firm.

High-Frequency Signals and Strategies

HFT relies heavily on signals like futures lead, where price changes in futures markets (the Chicago Mercantile Exchange, located in Chicago) often precede similar movements in the underlying asset markets (NYSE and NASDAQ, located in New York). Other signals include order book dynamics, market fragmentation, and correlated share price movement. HFT algorithms trade primarily with each other, as well as with execution algorithms that break down large institutional orders into smaller child orders.

HFT activities are concentrated in about 25 global data centers, with four major ones in the U.S., an Equities Triangle in New Jersey consisting of Nasdaq, NYSE, and NY4/5, and the Chicago Mercantile Exchange. Connections between these data centers, such as CME-NYC, become lifelines for HFT firms, who pay a premium for the fastest possible links.

Hugging the Geodesic

Connectivity has always been at the core of HFT. The first fiber optic Gold Line from the CME datacenter to New Jersey was a significant milestone, but companies like Spread Networks invested hundreds of millions to lay cables that more closely "hugged the geodesic" to reduce latency. Spread Networks pushed the limit to 6.65ms, a significant improvement from the previous 8ms but still above the Einstein limit of 3.94ms. AB Services revolutionized the game by introducing microwave links that could perform at 3.98ms, nearly meeting the theoretical limit.

Laying these microwave links wasn't easy. For example, one of the main challenges was crossing 80km across Lake Michigan. The solution was to shoot signals from skyscraper tops across the lake. Many firms switched to higher frequencies, up to 23Ghz, which are more vulnerable to environmental conditions like fog or rain. Yet, in HFT, it's "better to be first 99% of the time than second 100% of the time."

As HFT operations expanded, the race extended within cities like New Jersey. In these densely populated areas, millimeter wave technology replaced microwaves due to bandwidth requirements. AOptics introduced a dual-mode millimeter plus laser technology, each with its susceptibilities balanced by the other, to combat issues like rain and fog.

The Physicality of Computers in HFT

CME's datacenter in Chicago serves as a linchpin in the complex web of HFT. The facility moved from its original location in Cermak in 2012, which was originally a printing plant, to a more advanced, separate facility. Cermak could handle 100MW of electricity, all eventually transformed into heat — a concern when power density is a crucial metric in datacenter operations. Security is stringent, with most doors protected by biometrics.

In the world of HFT, even the physical attributes of a computer become crucial. Programmers must consider the physics of computation, focusing on elements like wires, photons and electrons, instructions, and data packets, to minimize latency. This brings in concepts from physics, emphasizing the literal speed of light as a limiting factor in trade execution times. Initially, firms relied on C++ programming for a speed advantage, but the bar has been raised significantly. To remain competitive today, HFT operations must be executed on Field-Programmable Gate Arrays (FPGAs). This has brought down response time to as low as 42 nanoseconds.

Two interesting strategies that try to beat others to be first are scout orders and speculative triggering. Small scout orders (a loss worth it) are used to detect price changes microseconds before the competiton, and then trade on it. Speculative triggering is a technique where HFT firms employ custom packet processing in FPGAs. This approach preemptively guesses the correct trading action based on incoming bits and initiates the sending of response bits even before the full incoming packet has been processed. If the guess turns out to be incorrect, the final bits of the response checksum are deliberately altered to ensure that the exchange's network stack discards the packet.

Market Structure, Regulation, and Effects

The datacenter houses key components such as trading firms' systems, the exchange's order gateway, and matching engine. When this engine finds a matching pair of orders from two firms, it sends a fill message. If there's no match, a confirm message is returned. Trading firms also have the flexibility to send cancel or modify messages to their existing orders.

Trading in HFT is a "cat and mouse game," where firms try to decode the logic behind other trading algorithms to anticipate their moves and trap them for profit. This involves offline analysis of massive amounts of anonymized market data to statistically identify algorithmic behaviors. One possible strategy is market impact trading, where the algorithm itself places orders that influence market prices. Firms anticipate the reactions of other algorithms to these price moves and exploit them before canceling their original orders. Sweeping involves selling against existing bids to lower prices, causing smaller firms to liquidate their holdings, which are then purchased by the sweeper at the lower price. Finally, spoofing entails sending large sell offers above the market price, along with a smaller buy order at a lower price. This temporary imbalance drives the price down, enabling the buy order to succeed; the original spoof orders are then canceled, and the market readjusts. These strategies often blur ethical lines and are subject to regulatory scrutiny, as they can manipulate market dynamics to the advantage of HFT firms.

Exchanges have attempted to level the playing field by introducing protective mechanisms like a minimum quote lifespan of 250ms to counteract spoofing. Large market participants often have the ability to lobby exchanges to block entities that engage in predatory trading activities against them.

Conclusion

The world of high-frequency trading is a high-stakes game that involves heavy investment in technology, innovation, and data analysis. From the architecture of data centers to the microsecond battles fought in the trading algorithms, HFT is a realm where the pursuit of speed and efficiency is relentless, and the financial implications are enormous.

The layers of complexity and the speed at which changes occur in this field underscore the need for ongoing scrutiny, ethical considerations, and potentially, further regulatory oversight. It's a world where microseconds can mean the difference between millions gained or lost, and where the technological arms race shows no signs of slowing down.

Introduction to Marketing Mix Modeling

2023-07-23T00:00:00+02:00

Introduction

Imagine a Data Scientist asked to evaluate the effectiveness of an online marketing campaign. The golden standard in this situation is to scope the campaign as a Randomized Controlled Trial (RCT), also known as an A/B test:

Create a control and a test group which are statistically equal by selecting a large enough sample size, stratification, CUPED, etc; see my earlier post Five ways to reduce variance in A/B testing
Send the marketing campaign to the test group, and don't send it to control. Note that control may still receive other marketing campaign from the company, but not the one being tested.
Compare the Overall Evaluation Metric (OEM), usually sales between control and test, and report a % lift (and potentially extra sales).

This approach works great if we want to evaluate a single campaign, and if the Data Scientist is involved before the campaign runs, so she can re-scope it as an A/B test. But what if the question is about the overall effectiveness of marketing, not just online but also TV, radio, print, billboards, etc), and over a longer period of time. In this case, there are still two options for our Data Scientist:

For the online marketing component, the Data Scientist can recommend to scope it as a large, year-long test. This is likely to get strong pushback from the business, since not sending marketing to even 10% of customers (the control group) could negatively impact sales and cause the company to miss targets. Another difficulty may be technically implementing a "block" for the control group in a large, complex organization (with many business units and divisions spread across multiple countries).
The above approach of a large, year-long test doesn't work for offline marketing such as TV, radio, print, billboards, etc. because here randomization is not possible. We cannot tell people "you're in our control group, close your eyes and ears if you see or hear one of our ads". Also, in an RCT, the experimental units are not supposed to know which variant or group they are in, since that in itself may affect their behaviour and bias the results. The only way to test this would be if the Data Scientist would have access to a parallel universe where the company doesn't do any marketing for a year (but just this year), and then compare the OEM...

So, without access to parallel universes, what can the Data Scientist do? The industry answer is Marketing Mix Modeling (MMM):

Marketing mix modeling (MMM) is statistical analysis such as multivariate regressions on sales and marketing time series data to estimate the impact of various marketing tactics (marketing mix) on sales and then forecast the impact of future sets of tactics. It is often used to optimize advertising mix and promotional tactics with respect to sales revenue or profit.

The answer is to take ad spend data, promotions data, competitor ad spend data, competitor promotions, revenues, macro trends, etc. (everything we can get) and look for correlations, ie. build a model that explains historic sales resulting from a sum of:

a seasonal baseline (the sales the model believes we would have without any marketing)
marketing activities (ad campaigns, pricing, promotions)
competitor effects (competitor's campaigns, pricings, promotions)
macro influences (economy, exchange rates, Covid-19 lockdowns)

Some points worth noting:

the most difficult task is likely to collect reliable and accurate input data, both about the company's own marketing and sales, as well as competitor's
once some input data is available, we can use existing techniques and libraries (see below) to quickly get a model and answers to basic questions
MMM is typically run using weekly level observations
extra time and money should be spent on getting more and better quality data, not building more complicated models!
if the model is built on low quality data then the results will be directional at best
if our inputs are garbage, so will our conclusions — Garbage In, Garbage Out (GIGO)

The Marketer

From the perspective of the user — the Marketer — the objective is to understand how different marketing activities contribute to the outcome and then use that knowledge to optimize the allocation of marketing resources. MMM is particularly useful in multi-channel marketing strategies where various marketing tactics work in unison. These can range from traditional methods like TV, radio, print ads, to modern digital campaigns such as social media, email, and content marketing. By understanding the contribution of each of these elements, a marketer can make informed decisions on where to invest, what to optimize, and where to cut back.

The concept of MMM can be likened to a chef perfecting a recipe. Each marketing input represents an ingredient in the recipe. Just as a chef adjusts the ingredients to perfect the taste, a marketing analyst adjusts the marketing inputs to optimize sales. By evaluating historical data, MMM helps identify the effectiveness of each marketing input, providing insights on how to allocate budgets effectively and forecast future results.

Core MMM inputs

At the heart of a Marketing Mix Model are a variety of inputs that capture the full spectrum of marketing activities and external factors influencing sales or market share:

Marketing variables: These are the different marketing efforts put forth by a company across various channels. This can include spending on television ads, online marketing, radio, print, social media, direct mail, email campaigns, SEO, SEM, PR, promotions, sponsorships, and any other marketing channel the company is utilizing.
Sales data: This is the dependent variable that the model attempts to explain. It could be total sales, market share, or any other key performance indicator (KPI) that the organization uses to measure success.
Economic indicators: This includes factors like inflation rate, unemployment rate, GDP growth, and consumer sentiment, which can impact consumer buying behavior and in turn affect sales.
Competitive information: Details about competitor activities, such as their marketing spends, pricing changes, product launches, promotions, can influence your sales and should be included in the model.
Seasonality and trend variables: These capture predictable fluctuations in sales that are due to the time of year (like holiday seasons) or broader market trends.
Pricing and distribution information: Details on the pricing strategy and the distribution reach (number of stores, online presence, etc.) also play a critical role in influencing sales.
Product variables: Changes in product characteristics or the introduction of new products can impact sales. This can include changes in product features, packaging, branding, or the introduction of new SKUs.

These inputs help quantify the impact of each marketing activity on sales, allowing for more efficient allocation of marketing resources. By adjusting these inputs, companies can predict the potential impact of different marketing strategies on their overall sales performance.

Core MMM outputs

The Marketing Mix Model generates several important outputs that aid in understanding the performance and impact of marketing efforts:

Contribution by marketing input: This output quantifies the impact of each marketing channel on the target variable (e.g., sales or market share). This provides a breakdown of how much each marketing activity contributed to the total sales.
Return on Investment (ROI): ROI measures the profitability of each marketing activity by comparing the amount of profit generated to the cost of the activity. This allows marketers to understand which activities are most profitable.
Elasticities: This is a measure of how sensitive the target variable (e.g., sales) is to a change in a marketing input. For example, if the elasticity of sales with respect to TV advertising is 0.8, it means that a 1% increase in TV advertising spend would lead to a 0.8% increase in sales, all else being equal.
Baseline sales and incremental sales: Baseline sales are the sales that would have been achieved without any marketing activity, while incremental sales are the additional sales gained due to marketing efforts.
Synergies: Marketing activities often interact with each other, and the model can measure these synergies. For instance, a TV ad campaign may make an online campaign more effective, and the model would quantify this effect.
What-if scenarios/forecasts: Using the model, marketers can predict the impact of future marketing plans on sales. These scenarios can provide insights on how changes in the marketing mix can affect outcomes.
Optimal spending levels: Based on the ROI, the model can suggest the optimal level of spending for each marketing channel to maximize profit or sales.

These outputs enable marketers to evaluate their past marketing efforts and guide future investment decisions. They can also help identify areas of improvement, and optimize the marketing mix for better performance.

Google's MMM library

To make MMM more concrete, let's look at Google's opensource Lightweight MMM (LMMM) library. LMMM fits a bayesian additive model to the data in attempt to decompose the target variable (eg. sales) into a baseline (including a trend and seasonality) and the effect of marketing channels:

$ kpi = \alpha + trend + seasonality + media\ channels + other\ factors $

Where $kpi$ is typically the volume or value of sales per time period, $\alpha$ is the model intercept, $trend$ is a flexible non-linear function that captures trends in the data, $seasonality$ is a sinusoidal function with configurable parameters that flexibly captures seasonal trends, $media\ channels$ is a matrix of different media channel activity (typically impressions or costs per time period) which receives transformations depending on the model used (see Media Saturation and Lagging section) and $other\ factors$ is a matrix of other factors that could influence sales. The full model is explained in the model documentation.

The below code runs LMMM with synthetic sample data (the code is from the LMMM Github repo):

import jax.numpy as jnp
import numpyro
from lightweight_mmm import lightweight_mmm
from lightweight_mmm import optimize_media
from lightweight_mmm import plot
from lightweight_mmm import preprocessing
from lightweight_mmm import utils

SEED = 105
data_size = 104 + 13
n_media_channels = 3
n_extra_features = 1
number_warmup=1000
number_samples=1000

media_data, extra_features, target, costs = utils.simulate_dummy_data(
    data_size=data_size,
    n_media_channels=n_media_channels,
    n_extra_features=n_extra_features)


split_point = data_size - 13
media_data_train = media_data[:split_point, ...]
media_data_test = media_data[split_point:, ...]
extra_features_train = extra_features[:split_point, ...]
extra_features_test = extra_features[split_point:, ...]
target_train = target[:split_point]

Here, we have 3 media channels, and 1 extra feature. An extra feature is just an extra timeseries that we think could be helpful for modeling, it could be something like inflation or a Covid-19 factor. Each of these is a timeseries of floats, with weekly or daily granularity. We can then fit the model:

media_scaler = preprocessing.CustomScaler(divide_operation=jnp.mean)
extra_features_scaler = preprocessing.CustomScaler(divide_operation=jnp.mean)
target_scaler = preprocessing.CustomScaler(divide_operation=jnp.mean)
cost_scaler = preprocessing.CustomScaler(divide_operation=jnp.mean, multiply_by=0.15)

media_data_train = media_scaler.fit_transform(media_data_train)
extra_features_train = extra_features_scaler.fit_transform(extra_features_train)
target_train = target_scaler.fit_transform(target_train)
costs = cost_scaler.fit_transform(costs)

mmm = lightweight_mmm.LightweightMMM(model_name="carryover")
mmm.fit(
    media=media_data_train,
    media_prior=costs,
    target=target_train,
    extra_features=extra_features_train,
    number_warmup=number_warmup,
    number_samples=number_samples,
    seed=SEED)

Now we can see how LMMM explains our target variable (eg. sales) using a generic baseline and the contributions from our marketing channel (3 channels in this example), ie. how sales is decomposed into a baseline, and 3 additional factors due to the 3 marketing channels:

media_contribution, roi_hat = mmm.get_posterior_metrics(
    target_scaler=target_scaler,
    cost_scaler=cost_scaler)
plot.plot_media_baseline_contribution_area_plot(media_mix_model=mmm,
    target_scaler=target_scaler,
    fig_size=(30,10))

Next, what % of sales is coming from each marketing channel (unfortunately colors don't match with above chart):

plot.plot_bars_media_metrics(metric=media_contribution, metric_name="Media Contribution Percentage")

Each marketing channel's return on investment (ROI), ie. effectiveness:

plot.plot_bars_media_metrics(metric=roi_hat, metric_name="ROI hat")

Last, the response curves for the marketing channels, ie. how does their effectiveness vary with spend:

plot.plot_response_curves(media_mix_model=mmm, target_scaler=target_scaler, seed=SEED)

In this synthetic example, it's linear per channel, so every additional \$ spend marketing gets us the same additional \$ sales, although with a different ROI (slope) per channel.

Conclusion

Google's LMMM library does a lot more, both in terms of handling inputs, outputs and modeling, but for this introduction I will stop here. In the next post I will look at articles evaluating the accuracy of modeling advertising effectiveness.

Other good sources on MMM:

Leadership models V: The Hero's Journey

2023-06-23T00:00:00+02:00

Introduction

This is the fifth article in my series on useful mental models in leadership and self-management:

The Hero's Journey, or Monomyth, is a narrative pattern identified by scholar Joseph Campbell that appears across a wide range of cultures and eras. At its heart, the Hero's Journey is a story of transformation, depicting the protagonist's path from the familiar to the unknown, through trials and revelations, and ultimately back to their world, forever changed. It's a blueprint for storytelling that not only speaks to our shared human experiences of growth, learning, and self-discovery, but also offers a profound metaphor for personal and professional development.

Call to Adventure

The Hero's Journey is a narrative structure common to countless stories across time and culture. This universal blueprint traces the path of a hero as they depart from the ordinary world, venture into a realm of fantastic wonders and daunting challenges, and return transformed, bearing gifts for their society.

From ancient mythological epics to modern cinematic blockbusters, the Hero's Journey pervades our collective storytelling. Its stages can be seen in the narrative arc of Gilgamesh, in the adventures of Odysseus, and in the celebrated saga of a young farm boy on Tatooine named Luke Skywalker. The enduring appeal of this narrative pattern has been ascribed to its deep resonance with our shared human experience.

Despite the fantastical settings and extraordinary events that often characterize these stories, the Hero's Journey reflects our own journeys through life. It’s not just a plot device, but a metaphor for personal transformation, growth, and self-discovery. As we accompany heroes like Luke Skywalker on their quests, we vicariously confront our own fears, overcome our own obstacles, and discover our own latent potential. The Hero's Journey, then, is more than just a story. It's a journey of the self, towards the self, a narrative pattern that echoes our universal human quest for meaning, purpose, and identity.

Mental model

The Hero's Journey, or the Monomyth, describes a universal pattern found in many narratives around the world. In relation to leadership, the Hero's Journey can serve as a useful model for leaders and managers in several ways:

Self-understanding: Leaders can compare their own personal and professional development to the stages of the Hero's Journey. Each stage can correspond to various aspects of a leader's journey, such as facing new challenges (Call to Adventure), hesitation in taking up new responsibilities (Refusal of the Call), acquiring new skills and knowledge, and so on. This can help leaders gain a better understanding of their own growth and development.
Development of empathy and understanding: By viewing their team members' journeys through the lens of the Hero's Journey, leaders can gain insights into the personal and professional challenges that their team members may be facing. This can lead to more empathetic and effective leadership.
Vision and Strategy Development: The Hero's Journey provides a narrative structure that leaders can use to craft their organizational vision and strategy. For instance, the Call to Adventure could represent the need for organizational change, while the Road of Trials could symbolize the challenges that the organization needs to overcome to achieve its goals.
Change Management: The Hero's Journey can serve as a model for managing change. Each stage of the journey involves changes and challenges that need to be overcome, providing a useful analogy for the process of implementing organizational change.
Motivation and Inspiration: The Hero's Journey, with its narrative of overcoming challenges and achieving a worthy goal, can serve as a powerful source of motivation and inspiration for both leaders and their team members.
Coaching and Mentoring: The journey model can serve as a guide for coaching and mentoring, helping leaders to identify the stages of their team members' personal and professional development and provide appropriate support and guidance at each stage.

In sum, Campbell's Hero's Journey offers a universal narrative structure that can provide a rich source of insights and inspiration for leaders and managers.

Hero's Journey

Joseph Campbell's Hero's Journey is typically divided into three main sections: Departure, Adventure and Resolution. Each of these sections contains various stages:

Sure, let's examine the Hero's Journey in detail, organized into three broad episodes:

Episode I: Departure

Ordinary World: This is the hero's normal world, where we learn crucial details about our hero, their true nature, capabilities, and outlook on life. This anchors the hero as a relatable, sympathetic character.
Call to Adventure: The adventure begins with the disruption of the comfort of the hero's ordinary world. This is a challenge or quest that the hero is compelled to undertake.
Refusal of the Call: Although the hero may be eager to accept the quest, at this stage, they will have fears that need overcoming. This refusal is often because of fear, duty, or insecurity.
Meeting with the Mentor: The hero comes across a seasoned traveler of the worlds who gives him or her training, equipment, or advice that will help on the journey.
Crossing the Threshold: This is the turning point where the hero commits wholeheartedly to the adventure and finally steps into the unknown, marking the end of the Departure phase.

Episode II: Adventure

Tests, Allies, Enemies: Now in a world of problems and challenges, the hero is tested, makes allies or enemies, and learns the rules of this new world.
Approach to the Innermost Cave: The hero arrives at the edge of a dangerous place, often deep underground, where the object of the quest is hidden.
Ordeal: The hero faces the greatest challenge yet, with the potential for death. This is a critical moment in the hero's journey, and everything is at stake.
Reward: After facing death, the hero achieves the quest's goal or part of it, taking possession of the treasure or the reward that they've been seeking.

Episode III: Resolution

Road Back: The hero is driven to complete the adventure, yet must deal with the consequences of confronting the dark forces in the ordeal.
Resurrection: This is the climax, where the hero has their final and most dangerous encounter with death. The hero must use everything they've learned to overcome their most difficult challenge.
Return with Special Knowledge: The hero returns to their original world, but the hero's journey has changed them. This change often manifests as a newfound knowledge or insight that the hero can now use to improve their world. This marks the resolution of the journey, where the hero emerges transformed, symbolizing their mastery of life.

By following this template, you can easily construct a captivating narrative that engages audiences due to its familiarity, emotional resonance, and satisfying resolution.

Luke Skywalker

In the annals of contemporary cinema, there are few stories as universally recognizable as the saga of Luke Skywalker in the Star Wars franchise. Not only has the character become a cultural icon, but his journey also serves as a textbook example of Joseph Campbell's "Hero's Journey." Skywalker's journey from a young farm boy on Tatooine to the savior of the galaxy follows Campbell's model almost to a tee, providing an intriguing case study of this universal narrative structure.

Ordinary World: Luke begins his journey on the desert planet of Tatooine. He lives a mundane life as a farm boy, longing for adventure.

Call to Adventure: When Luke stumbles upon two droids carrying secret plans that can help rebel forces defeat the oppressive Galactic Empire, he realizes there is a world beyond his desert home that requires his help.

Refusal of the Call: Initially, Luke hesitates to join the Rebellion. He feels he can't leave his family and their farm, despite his desire for adventure.

Meeting with the Mentor: Luke meets Obi-Wan Kenobi, a Jedi Knight who begins to teach him about the Force and his Jedi lineage, providing guidance and the lightsaber that belonged to his father.

Crossing the Threshold: Luke's threshold moment comes when his aunt and uncle are killed by the Empire. With nothing left for him on Tatooine, he decides to join Obi-Wan on his mission to deliver the droid R2-D2 and the Death Star plans to the rebel forces.

Tests, Allies, Enemies: Luke undergoes several tests, forms alliances, and faces enemies. He learns to pilot a spacecraft, meets Han Solo and Princess Leia, and faces conflicts with the forces of the Empire, including the menacing Darth Vader.

Approach to the Innermost Cave: The innermost cave for Luke is the Death Star, where he embarks on a rescue mission to save Princess Leia.

Ordeal: His ordeal is the climactic Battle of Yavin, where Luke, relying on his connection to the Force, has to destroy the Death Star.

Reward: With the Death Star destroyed, Luke and his friends are celebrated as heroes. Luke has proven himself a worthy Jedi and a formidable force against the Empire.

Road Back: The road back is the ongoing battle against the Empire. Although the Death Star is destroyed, the Empire remains a powerful threat.

Resurrection: Luke faces a significant ordeal in "The Empire Strikes Back" when he battles Darth Vader and learns the truth about his lineage. It's a form of death and rebirth for his character as he struggles with this truth and continues his Jedi training.

Return with Special Knowledge: By "Return of the Jedi", Luke has evolved significantly. He is now a skilled Jedi Knight and has come to terms with the revelation that Darth Vader is his father. He possesses the special knowledge of the Force and uses it to confront the Emperor, leading to the ultimate defeat of the Empire. He brings hope back to the galaxy, symbolizing his complete transformation from farm boy to galactic hero.

Leadership

In the realm of Leadership, the Hero's Journey can serve as an invaluable roadmap for personal development and transformation. The leader's "Ordinary World" could be their initial state of managing without much strategic thought or direction. The "Call to Adventure" represents the realization that more effective leadership is required, and "Refusal of the Call" shows the initial resistance to adopting new leadership styles or methods. "Meeting with the Mentor" could be a period of learning and development, perhaps through mentorship or leadership training. As the leader "Crosses the Threshold", they begin to apply what they've learned, leading their team with renewed vision and skills. The trials they face and allies they gather represent the everyday challenges of leadership and the team members who support them.

When it comes to Team Management, the Hero's Journey provides an apt metaphor for team development and goal achievement. The "Ordinary World" is the team's initial state, while the "Call to Adventure" represents the team's project or mission. As the team moves through the stages of the journey, they face various tests, in the form of project hurdles and deadlines, and make allies, in the form of supportive team members or other teams within the organization. The "Approach to the Innermost Cave", "Ordeal", and "Reward" stages reflect the process of tackling the project's most significant challenges and ultimately completing the project successfully. The "Road Back", "Resurrection", and "Return with Special Knowledge" stages can represent the team's review of the project, their learning from any mistakes made, and the application of these lessons to future projects.

In driving Change Management, the Hero's Journey can illustrate the process of implementing and managing change. The "Ordinary World" represents the initial state of the organization, while the "Call to Adventure" is the realization that change is necessary. The "Refusal of the Call" symbolizes initial resistance to change, while "Meeting with the Mentor" might represent consultation with change management experts or gathering information about how to manage the change. As the organization "Crosses the Threshold", they officially initiate the change. The ensuing stages, such as "Tests/Allies/Enemies", "Approach the Innermost Cave", and "Ordeal", symbolize the challenges encountered during the change process, while "Reward" represents the successful implementation of the change. Finally, "The Road Back", "Resurrection", and "Return with Special Knowledge" signify the process of solidifying the change, dealing with any lingering issues, and applying the lessons learned to future change initiatives.

Conclusion

The Hero's Journey, as expounded by Joseph Campbell, remains a compelling framework for understanding not only the structure of countless stories from around the world, but also our personal and professional journeys. Whether you're a leader seeking to inspire your team, an educator shaping young minds, or a creative professional striving to craft a captivating narrative, the stages of the Hero's Journey can provide profound insights. It’s a timeless blueprint for transformation, highlighting the challenges we face, the mentors and helpers we encounter, and the ultimate boon we seek in our quest for growth and self-realization. It speaks to the enduring human capacity for change, underscoring that the path to becoming a hero lies within each one of us.

Real-world experiments I: 5 Lessons from Google, Bing, Netflix and Alibaba

2023-06-18T00:00:00+02:00

Introduction

I have written more than 20 articles on A/B testing on Bytepawn, but most of them are about the technical, statistical aspects. In this post, I will discuss five lessons from large-scale experiments conducted by Google, Bing, Netflix and Alibaba. Each story briefly describes a specific experiment conducted by these companies and the consequent lessons learned. From Ronny Kohavi's enlightening "1 out of 3 rule" at Microsoft that reshapes the understanding of failure in the realm of innovation, to Google's seemingly trivial yet significantly impactful experiment on the '41 shades of blue', these narratives underline how experimentation often leads to surprising revelations and success. Similarly, Alibaba's large-scale randomized field experiment and Netflix's personalized movie image testing provide evidence for the importance of personalization. These stories prove the power of A/B testing and can be an inspiration for practicioners.

Kohavi's 1 out of 3 rule

Ronny Kohavi, formerly Vice President, Analysis & Experimentation on Bing, author of the excellent book Trustworthy Online Controlled Experiments famously writes in Online Experimentation at Microsoft: When we first shared some of the above statistics at Microsoft, many people dismissed them. Now that we have run many experiments, we can report that Microsoft is no different. Evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric!

In the later presentation titled Lessons from Running A/B/n Tests for 12 years states that 1/3 of ideas were positive ideas and statistically significant, 1/3 of ideas were flat: no statistically significant difference, 1/3 of ideas were negative and statistically significant. At Bing, the success rate is lower; the low success rate has been documented many times across multiple companies.

Lesson: most of your ideas will not work. This is learning, not failure.

Google's 41 shades of blue

The 2014 Guardian article titled Why Google has 200m reasons to put engineers over designers describes the following experiment at Google.

Roughly half a decade ago, Google initiated its venture into Gmail advertising. As was the case with their search engine, ads were presented as small blue links leading to various websites. However, it was observed that the shade of blue used in these two distinct products varied when linked to advertisements. Traditionally, the choice of color in such a scenario would have been determined by a chief designer or a marketing director, in what might be referred to as 'the hippo approach'. But in the data-driven world, Google decided to approach this differently.

They conducted a series of '1%' experiments, exposing one percent of users to a specific shade of blue, with each test incorporating a different hue. To ensure thoroughness, forty different variations of blue were tested in this manner. The objective was to discern the most popular shade among users, measured by the frequency of clicks. The results led to the discovery that a marginally purpler blue was more click-friendly than a slightly greener shade. This seemingly minor adjustment, given the magnitude of Google's business operations, had a significant financial impact: an additional $200 million per year in ad revenue.

Lesson: if you have the sample size, don't test 2 or 3 variants, test a lot of them to find the optimum.

Bing's unexpected big win

In Chapter 1 of Trustworthy Online Controlled Experiments, Kohavi opens with the following experiment, also descibed in the HBR article titled The Surprising Power of Online Experiments:

In the year 2012, an innovative idea emerged from within the Bing team at Microsoft. An employee proposed a change in the way ad headlines were displayed on the search engine. It was a straightforward implementation that wouldn't take more than a few days of engineering effort. However, being one amongst a sea of proposed ideas, it was deemed low priority by program managers. As a result, the idea remained untouched for over six months. It wasn't until an engineer, noting the minor cost of writing the necessary code, decided to launch an A/B test to evaluate the potential impact of this seemingly inconsequential change.

Within a matter of hours, the newly introduced headline variation started generating revenue at an unusually high rate, tripping the "too good to be true" alarm. While such alerts often indicate a glitch, in this case, it pointed towards a genuine increase in revenue. Analysis revealed a staggering 12% increase in revenue, translating to an annual boost of over $100 million in the United States alone, without any negative impact on key user-experience metrics. Despite being one of the most lucrative ideas in Bing's history, its value remained unrecognized until this experiment. This serves as a humbling reminder of the challenge in assessing the potential of new ideas and underscores the benefits of possessing the ability to run numerous inexpensive tests concurrently, a practice more businesses are beginning to appreciate.

Lesson: you don't know which experiment will yield big results.

Alibaba's personalization experiment

In the paper The Value of Personal Data in Internet Commerce researchers from Alibaba share surprising results.

In collaboration with Alibaba's E-commerce platform, a large-scale randomized field experiment was conducted involving 555,800 customers. Upon opening the Alibaba app, these users were exposed to the homepage recommendation. Half of the user base was randomly selected and assigned to a treatment group. For these users, personal data like demographics, past clicks, and purchase behaviors were not employed in the product recommendations on the homepage. The other half, constituting the control group, experienced the standard product recommendation process, which utilized their personal data. The stakes of this experiment were significant, given the reliance of the platform on homepage recommendations to match its vast user base with countless merchants and considering that customers spend a substantial amount of time interacting with recommended products.

The experiment illuminated the impact of personal data regulation on user engagement and transaction outcomes. The findings revealed a notable increase in product recommendation concentration and a significant decrease in the alignment of product recommendations with customer preferences in the treatment group compared to the control group. Moreover, recommendations made without personal data led to a drastic drop in matching outcomes, as measured by both customer engagement and market transactions. There was an immediate 75% decrease in customers' click-through rate (CTR) on the recommended products, and a 33% reduction in homepage browsing as measured by product views (PV). The combined effect resulted in a substantial 81% decline in customer purchases (gross merchandise volume, GMV) facilitated by homepage recommendations.

Lesson: personalized recommendations are worth high double-digit % lifts.

Netflix' movie image personalization

In their blog post titled Selecting the best artwork for videos through A/B testing Netflix describes how the artwork or images associated with movies or television shows significantly influence a user's decision on what to watch. To provide the best personalized user experience, Netflix decided to conduct an A/B test on its artwork in different regions to understand the kind of images that resonated with its diverse user base.

One of the more notable findings was that images featuring expressive facial emotions of the main character had more appeal. For example, in one A/B test, Netflix used different thumbnails for the movie "Good Will Hunting." The artwork with Robin Williams showing a strong emotional expression drove more views than artwork with other characters or less expressive faces. However, Netflix didn't release the specific numeric outcomes of these tests, but they were confident in stating the effectiveness of this approach in increasing engagement.

Following these A/B tests, Netflix has implemented a dynamic system that personalizes the artwork shown to individual users based on their viewing history and preferences. For instance, a user who watches a lot of romantic movies might see artwork emphasizing a love story, while a user who frequently watches action films might see artwork highlighting an action scene. This A/B testing has allowed Netflix to optimize user engagement, personalize content more effectively, and drive longer viewing sessions.

Lesson: personalization and experimentation opportunities lurk in unexpected areas.

Conditional Probabilities and Simpson's Paradox

2023-06-11T00:00:00+02:00

Introduction

Recently I was reading the excellent book Patterns, Predictions and Actions, which mentions the Simpson's paradox on page 176, in the chapter on Causality. I've encountered Simpson's paradox before, but like any good paradox, I have to take a step back, think things through and convince myself I understand what's going on each time. This time I decided to write it down to make it more sticky!

University admissions

Simpson's paradox is usually explained using the toy example of University admissions and gender bias. Suppose there is a University, and there are $N=1000$ applicants, 650 are admitted, leading to an admission rate of 65%. So far so good.

Now, suppose that of the $N=1000$ applicants, half are male and half are female. Out of 500 males, 400 were admitted, so the admission rate was 80%. Out of 500 females, 250 were admitted, so the admission rate was 50%. Note that the numbers add, up, $ 400+250=650 $ total admissions.

The differing rates of admission may lead us to think that there is gender bias, and the University is discriminating against females. A χ² hypothesis test tells us that the difference in acceptance rates in the two branches (male and female) are statistically significant at $ p < 0.001 $.

Can we stop here? Let's dig deeper: when students apply for University, they actually apply to department, and it's the department that accepts or rejects them. For simplicity, let's assume that each student only applies to one department, and that there are 2 departments, A and B. Simpson's paradox can now manifest itself.

No discrimination

First, let's look at an example where, once we look at the per-deparment admissions, we can convince ourselves that there is no discrimination:

Here, department A has a 90% acceptance rate across both males and females, and department B has a 40% acceptance rate across both males and females. Clearly, there is no discrimination at acceptance. The differing acceptance rate at the male-female level is because 400 out of 500 females apply to the more competitive B department, but only 100 out of 500 males do so.

Reversed discrimination

By playing around with the numbers, it's easy to create a situation where the perceived discrimination is reversed (changed numbers in red):

In this toy example, at the male-female level males have a higher acceptance rate (70% vs 50%), but at the department level, females actually have higher acceptance rates (90% vs 80% for department A and 40% vs 30% for department B). The root cause is the same: even though females are actually more successful at the department level, more of them apply to the more competitive department, leading to lower overall acceptance rates. Note that the same way if would have been erroneous to conclude discrimination without understanding the admission process and seeing the per-department data, it may equally be erroneous to conclude that there is reverse discrimination at the department level; perhaps there are other factors, more depth at play that lead to this outcome.

Same rates

For completeness, we can of course also have a situation where the per-department acceptance rate happen to match the University level (changed numbers in red):

Conditional probability

The "root cause" of these scenarion is simple: in general, there is no relationship between $ P(X=x) $ and $ P(X=x|Y=y) $. All three of the relations equals, less-than or greater-than are possible, and the same goes for $P(X=x|Y=y)$ and $P(X=x|Y=y,Z=z)$.

In our example, if we define $A$ as admission, $G$ as Gender and $D$ as deparment, and treat admission rate as a probability, than $P(A=a), P(A=a|G=g)$ and $P(A=a|G=g,D=d)$, as the above examples show, can be in any relation with each other.

Observation, not experiment

A novice Data Scientists may get confused and wonder: if an run an A/B test and detect a statistically significant difference, how do I know there is no Simpson's paradox lurking in the background? If you're a Data Scientist, I recommend you pause and try to answer this for yourself!

First of all, the example we described above was observational: we didn't interact with the system (applicants or departments), we simply observed the outcomes. In an A/B test (Randomized Controlled Experiment), we conduct an experiment, which means we interact with the system:

We randomly split our population into Treatment (A) and Control (B)
We treat units in Treatment, but leave units in Control alone.

The whole point of an A/B test is that with enough sample size $N$ (or good stratification), the two populations in A and B will be:

the same
the same as the overall population

This means that Simpson's paradox cannot occur (other than by variation, which can be limited by the afore-mentioned methods), because sub-population sizes will be the same! Ie. M and F were to branches of an A/B test (which they are not), than the number of people applying to the two departments would be roughly the same; that's the whole point of randomization!

Conclusion

One important conclusion is that a statistical significance test didn't help here; all it told us that the two are in fact different (the normal distributions of the means are far away), but in general a statistical test does not tell us about the reasons two means are different. It is an error of (statistical and logical) reasoning to say "I think X and Y are different because of Z", then use a statistical significance test to establish that "X and Y are indeed different", and then conclude, "therefore Z".

Another suspision is that Simpson's Paradox is more likely to occur if the rates we see reinforce a social construct or stereotype. For example, it's common today to look for gender bias and discrimiation, so it's easier to draw erroneous conclusions around this topic. In other words, we can construct an identical toy example with different words, and the result is less surprising: Ants of colony A have lower lifespans than ants of colony B, even though warrior and harvester ants of both colonies have equal survival probabilities, because colony A has more warrior ants, which have lower lifespans than harvester ants. This, to me is less surprising and less counterintuitive than the original University toy example with genders.

Testing GPT-4 spatial reasoning and comprehension

2023-05-21T00:00:00+02:00

Introduction

In an earlier post, I tested GPT-3 spatial reasoning abilities:

Testing GPT-3 spatial reasoning and comprehension

While on some tasks it did fairly well, I suspect that was due to the question being similar to training data. It did very poorly on the "people in a room" task, it couldn't answer a single question correctly, so my overall conclusion was that GPT-3 has not developed a sense for spatial reasoning. Let's see how GPT-4 (ChatGPT-4) does.

People in a room

Q: There is square room.
Alice is standing in the north-west corner.
Bob is standing in the south-west corner.
Charlie is standing in the south-east corrner.
David is standing in the north-east corner.
Ed is standing in the center of the room, looking at Alice.
How many people are there in the room?

A: There are five people in the room: Alice, Bob, Charlie, David, and Ed.

Q: Who is standing to the left of Ed?

A: If Ed is facing Alice who is in the north-west corner, then his left would be towards the west direction. Therefore, Bob, who is standing in the south-west corner, is to the left of Ed.

Q: Who is standing to the right of Ed?

A: If Ed is facing Alice who is in the north-west corner, then his right would be towards the north direction. Therefore, David, who is standing in the north-east corner, is to the right of Ed.

Q: Who is standing behind Ed?

A: If Ed is facing Alice who is in the north-west corner, then behind him would be the south direction. Therefore, Charlie, who is standing in the south-east corner, is behind Ed.

GPT-4 got all of them right. Very impressive! Let's spice it up:

Q: Now everybody goes around clockwise 90 degrees. Who is now in the north-west corner?

A: If everyone in the room moves 90 degrees clockwise, they would move as follows:
• Alice, originally in the north-west corner, would move to the north-east corner.
• Bob, originally in the south-west corner, would move to the north-west corner.
• Charlie, originally in the south-east corner, would move to the south-west corner.
• David, originally in the north-east corner, would move to the south-east corner.
• Ed, originally in the center, would remain in the center (assuming he also turns 90 degrees clockwise).
Therefore, Bob is now in the north-west corner.

Correct!

Clock faces

Can GPT-4 reason about the hands of a clock?

Q: It is half past eleven. What number is the minute hand of the clock pointing at?

A: The minute hand of a clock points to "6" when it is half past the hour. This is because a traditional analog clock is divided into 12 sections, and each section corresponds to 5 minutes. Therefore, when 30 minutes have passed, the minute hand points to the 6th section.

Note: correct.

Q: I have a watch. The hour hand is pointing to 11. The minute hand is at a right angle to it, on the right side. What number is the minute hand pointing at?

A: If the hour hand is pointing to 11, and the minute hand is at a right angle to the hour hand on the right side, the minute hand is pointing to 2. This corresponds to 10 minutes past the hour. Therefore, the time would be approximately 11:10.

Note: correct.

Q: I have a watch. The hour hand is pointing to 11:30. The minute hand is at a right angle to it, on the right side. What number is the minute hand pointing at?

A: If the hour hand is pointing to 11:30, it means it is halfway between 11 and 12. If the minute hand is at a right angle to the hour hand on the right side, the minute hand is pointing to approximately 3. This corresponds to 15 minutes past the hour. Therefore, the time would be approximately 11:15.

Note: not completely correct, but close; the minute hand is pointing at 2:30, the time is 11:12:30

Here GPT-3 also did fairly well on the first questions, but not on the later ones.

Triangles

Q: We have a triangle. One of the angles is 70 degrees, another is 80 degrees. What is the third angle?

A: To find the third angle of a triangle, we can use the fact that the sum of the angles in any triangle is always 180 degrees. Let's denote the third angle as x. We know that one angle is 70 degrees, another is 80 degrees, and the sum of all three angles is 180 degrees. We can set up the equation:
70 + 80 + x = 180
Simplifying the equation, we have:
150 + x = 180
To isolate x, we can subtract 150 from both sides:
x = 180 - 150
x = 30
Therefore, the third angle is 30 degrees.

Note: correct. It's impressive how GPT-4 derives the answer; unlike with GPT-3, where it just stated the answer, and I wasn't sure whether it's seen this exact question in the training data.

Q: We have a triangle. One of the angles is 70.51 degrees, another is 80.49 degrees. What is the third angle?

A: To find the third angle of a triangle when two angles are known, we can use the fact that the sum of the angles in any triangle is always 180 degrees. Let's denote the third angle as x. We know that one angle is 70.51 degrees, another is 80.49 degrees, and the sum of all three angles is 180 degrees. We can set up the equation:
70.51 + 80.49 + x = 180
Simplifying the equation, we have:
151 + x = 180
To isolate x, we can subtract 151 from both sides:
x = 180 - 151
x = 29
Therefore, the third angle is 29 degrees.

Note: correct.

Q: We have two identical right-angled triangles. We snap them together along their longest edges. What is the resulting shape?

A: When two identical right-angled triangles are snapped together along their longest edges, the resulting shape is a rectangle.

Note: correct.

Q: We have a very small triangle on a very large sphere. What is the sum of its angles, in degrees?

A: The sum of the angles of a triangle is always 180 degrees, regardless of the size or shape of the triangle. This property holds true for triangles on a sphere as well. Therefore, the sum of the angles of a very small triangle on a very large sphere would still be 180 degrees.

Note: partially correct; if the triangle is very small, the surface of the sphere is locally flat, so we can approximate with Euclidean geometry.

Q: We have a triangle on a sphere. What is the sum of its angles, in degrees?

A: On a sphere, the sum of the angles of a triangle is greater than 180 degrees. This property is known as the spherical excess or Gauss-Bonnet theorem. The formula for calculating the spherical excess (E) of a triangle on a sphere with angles A, B, and C (measured in radians) is:
E = A + B + C - π
To convert the angles from degrees to radians, we need to multiply them by π/180.
Let's say the angles of the triangle are A, B, and C (in degrees). The sum of the angles on the sphere is:
A + B + C - 180
Note that if the angles are given in radians, the sum of the angles would be:
A + B + C - π
So, in short, the sum of the angles of a triangle on a sphere is greater than 180 degrees (or π radians).

Note: correct; GPT-3 got this wrong.

Q: We have a triangle on a sphere. The sum of it's angles is 200 degrees. We scale up the sphere by 10x. What is the sum of the triangle's angles now?

A: When you scale up the sphere by 10x, the shape and proportions of the triangle on the sphere will also scale accordingly. However, the sum of the angles of the triangle will remain the same, regardless of the size of the sphere. Therefore, even after scaling up the sphere by 10x, the sum of the triangle's angles will still be 200 degrees.

Note: correct; GPT-3 got this wrong.

Unlike GPT-3, GPT-4 is able to correctly reason about triangles on a sphere. Very impressive!

Areas and volumes

Q: We have a white cube, each side is 1 unit long. We paint it red using 1 liter of paint.
We now have a white cube, each side is 2 units long. How many liters of paint do we need to paint it red?

A: The amount of paint needed is proportional to the surface area of the cube. The surface area of a cube is given by 6 times the square of the edge length (since a cube has six identical square faces).
1. For the cube with an edge length of 1 unit, the surface area is 6 * (1)^2 = 6 square units.
2. For the cube with an edge length of 2 units, the surface area is 6 * (2)^2 = 24 square units.
If it takes 1 liter of paint to cover 6 square units, then it should take 24/6 = 4 liters of paint to cover 24 square units. Therefore, you'd need 4 liters of paint to paint the larger cube.

Note: correct.

Q: We have a square. We cut up the square into pieces and rearrange the pieces to form a really long rectangle. Which has bigger area, the original square, or the really long rectangle?

A: The area of a shape remains the same regardless of how its pieces are rearranged. When you cut up a square and rearrange its pieces to form a rectangle, the total area remains the same. That's because area is a measure of the amount of "space" inside the shape, and cutting and rearranging the pieces doesn't create or remove any of this space. So the original square and the really long rectangle have exactly the same area.

Note: correct; GPT-3 got this wrong.

Q: There is a cube, and a sphere inside the cube. Which has bigger volume?

A: The volume of the cube is larger than the volume of the sphere inside it, provided the sphere is inscribed in the cube (meaning the diameter of the sphere is equal to the side of the cube).

Note: correct.

Q: The area of a unit square in 2D is 1, the area of the contained circle is Pi/4.
The volume of a unit cube in 3D is 1, the volume of the contained sphere is Pi/6.
What is the volume of the sphere contained in a unit cube in 4D?

A: In 4D, a "unit cube" would be a hypercube (also known as a tesseract) with edges of length 1, and a "sphere" would be a hypersphere. The general formula for the volume of a hypersphere in n dimensions (also known as an n-sphere), with a radius r, is:
V = (pi^(n/2) / Gamma(n/2 + 1)) * r^n
where Gamma is the gamma function. The gamma function of n is equal to (n-1)! for positive integers.
For the unit hypercube (a cube in 4D), each side length is 1, so the "radius" of the inscribed hypersphere is 0.5 (since the diameter is 1).
Let's substitute n=4 and r=0.5 into the formula:
V = (pi^(4/2) / Gamma(4/2 + 1)) * (0.5)^4
This simplifies to:
V = (pi^2 / 3) * (1/16)
Further simplifying gives:
V = pi^2 / 48 ≈ 0.06544984695
So, the volume of the hypersphere inscribed in a unit hypercube in 4D is approximately 0.06544984695.

Note: incorrect, the correct answer would be $\frac{\pi^2}{32}$.

Q: In higher dimensions what is the ratio of the volumes of a cube and the sphere contained in the cube? Do not give an exact mathematical answer, give a short qualitative answer.

A: As the number of dimensions increases, the volume of the sphere inscribed in a cube occupies a smaller and smaller proportion of the volume of the cube. This happens because the corners of the cube, which are outside the sphere, contribute more volume in higher dimensions. The more dimensions you have, the more "corner" there is relative to the total volume, and so the sphere takes up less of the space. In practical terms, the higher the dimension, the less "efficient" a sphere becomes at filling the space within a cube.

Note: correct.

ASCII art

Can GPT-4 read or write ASCII art? This is the question that originally triggered my exploration into spatial comprehension. Let's see:

Q: What is the letter in the following ASCII art?

██╗░░██╗
██║░░██║
███████║
██╔══██║
██║░░██║
╚═╝░░╚═╝

A: The ASCII art you've posted represents the capital letter "A".

Note: incorrect.

Q: What is the text in the following ASCII art?

██╗░░██╗███████╗██╗░░░░░██╗░░░░░░█████╗░
██║░░██║██╔════╝██║░░░░░██║░░░░░██╔══██╗
███████║█████╗░░██║░░░░░██║░░░░░██║░░██║
██╔══██║██╔══╝░░██║░░░░░██║░░░░░██║░░██║
██║░░██║███████╗███████╗███████╗╚█████╔╝
╚═╝░░╚═╝╚══════╝╚══════╝╚══════╝░╚════╝░

A: The ASCII art you've posted represents the word "GREAT".

Note: incorrect.

Q: What is the text in the following ASCII art?

███╗░░░███╗░█████╗░██████╗░████████╗░█████╗░███╗░░██╗
████╗░████║██╔══██╗██╔══██╗╚══██╔══╝██╔══██╗████╗░██║
██╔████╔██║███████║██████╔╝░░░██║░░░██║░░██║██╔██╗██║
██║╚██╔╝██║██╔══██║██╔══██╗░░░██║░░░██║░░██║██║╚████║
██║░╚═╝░██║██║░░██║██║░░██║░░░██║░░░╚█████╔╝██║░╚███║
╚═╝░░░░░╚═╝╚═╝░░╚═╝╚═╝░░╚═╝░░░╚═╝░░░░╚════╝░╚═╝░░╚══╝

A: The ASCII art you've posted represents the word "FUTURE".

Note: incorrect.

Q: Generate ASCII art for the letter H.

A: Sure, here is a simple ASCII art representation of the letter "H":

H   H
H   H
HHHHH
H   H
H   H

Note: correct.

Q: Generate ASCII art for the word HELLO.

A: Sure, here's a simple ASCII representation of the word "HELLO":

H   H  EEEEE  L     L      OOO  
H   H  E      L     L     O   O 
HHHHH  EEEE   L     L     O   O 
H   H  E      L     L     O   O 
H   H  EEEEE  LLLLL LLLLL  OOO

This is a simple form of ASCII art where each letter is 5 units tall and about 5 units wide (not counting spaces between letters). More complex or stylized versions could be created with different shapes or patterns.

Note: correct. Let's see if it can follow my ASCII style.

Q: Generate ASCII art for the word HELLO, in the style I used above.

A:Here's the ASCII representation of the word "HELLO" similar to the style you provided:

██╗░░██╗███████╗██╗░░░██╗███████╗
██║░░██║██╔════╝██║░░░██║██╔════╝
███████║█████╗░░╚██╗░██╔╝█████╗░░
██╔══██║██╔══╝░░░╚████╔╝░██╔══╝░░
██║░░██║███████╗░░╚██╔╝░░███████╗
╚═╝░░╚═╝╚══════╝░░░╚═╝░░░╚══════╝

Each letter is represented in a style similar to the style you provided. Please note that ASCII art can vary widely in complexity and style!

Note: incorrect, but it got the HE right!

Conclusion

Based on these experiments, GPT-4 is a lot better at spatial reasoning than GPT-3. If these models keep advancing at this pace, GPT-5 or 6 will be flawless at these tasks.

Leadership models IV: the Pareto Principle, the Peter Principle, the Rumsfeld Matrix, Servant Leadership and Pygmalion Effect, and Goleman's Emotional Intelligence Model

2023-05-19T00:00:00+02:00

Introduction

This is the fourth article in my series on useful mental models in leadership and self-management:

In this post, I will describe the Pareto Principle, the Peter Principle, the Rumsfeld Matrix, Servant Leadership and Pygmalion Effect, and finally Goleman's Emotional Intelligence Model.

Pareto Principle

The Pareto Principle, also known as the 80/20 Rule, was introduced by Italian economist Vilfredo Pareto in the late 19th century. Pareto observed that roughly 80% of the land in Italy was owned by 20% of the population. He developed the principle after observing that 20% of the pea pods in his garden contained 80% of the peas.

The principle has since been applied to a variety of fields, including business and productivity, where it is often interpreted as follows:

80% of outcomes (outputs) come from 20% of causes (inputs).
80% of results come from 20% of effort.
80% of a company's profits come from 20% of its customers or products.

It's important to note that the 80/20 ratio is not fixed; the distribution could just as well be 90/10 or 70/30, depending on the situation. The key idea is that a minority of causes, inputs, or effort often leads to a majority of the results, outputs, or rewards. Some practical applications of the Pareto Principle:

Business: In terms of customers, a common rule of thumb is that 80% of a company's revenue comes from 20% of its customers. Therefore, it can be beneficial to focus on the needs and preferences of that 20%.
Productivity: When it comes to personal productivity, the Pareto Principle suggests that 20% of our work activities will account for 80% of our results. Therefore, identifying and focusing on these high-impact tasks can significantly improve productivity.
Quality control: The Pareto Principle is also used in quality control, where it's known as the Pareto Chart. This tool suggests that by resolving the 20% of the issues causing 80% of the problems, overall quality can be significantly improved.

The Pareto Principle encourages individuals and organizations to identify and focus on the inputs that lead to the majority of their results, thereby enhancing efficiency and effectiveness.

Peter Principle

The Peter Principle is a concept in management theory formulated by Dr. Laurence J. Peter in his 1969 book, "The Peter Principle." This principle suggests that in a hierarchical organization, employees tend to rise to their level of incompetence. Here are its core tenets:

Promotion Based on Current Performance: The principle asserts that employees in a hierarchy are typically promoted based on their performance in their current role, rather than their potential to perform in the promoted role. This is based on the assumption that performance in one role directly translates to potential performance in a higher role.
Rise to the Level of Incompetence: As employees are promoted based on their performance, they will continue to rise through the ranks until they reach a position where they no longer perform competently. At this level, they do not have the skills or abilities to perform effectively, and thus their incompetence becomes evident.
Stagnation in Incompetent Positions: Once an employee has risen to a level of incompetence, they typically stay in that position. This is because their poor performance in the new role prevents further promotions, but it's not poor enough to warrant demotion or dismissal.
Organizational Inefficiency: The accumulation of employees in roles in which they are incompetent can lead to inefficiencies and ineffectiveness within the organization.

The Peter Principle is a reminder for organizations to consider potential capability and suitability for the future role when making promotion decisions, not just performance in a current role.

Rumsfeld Matrix

The Rumsfeld Matrix was popularized by former United States Secretary of Defense, Donald Rumsfeld, during a news briefing in 2002, although the concept has roots in the field of epistemology. The Rumsfeld Matrix consists of four categories of knowledge:

Known Knowns: These are things we know that we know. They represent our existing knowledge, facts, or information we are aware of and understand.
Known Unknowns: These are things we know that we don't know. They represent the gaps in our knowledge, areas where we are aware that we lack information or understanding.
Unknown Knowns: These are things we don't know that we know. They represent tacit knowledge or information we possess but may not be consciously aware of, such as unconscious biases, ingrained habits, or learned behaviors.
Unknown Unknowns: These are things we don't know that we don't know. They represent the areas of complete ignorance, where we are not even aware of our lack of knowledge or understanding.

The purpose of the Rumsfeld Matrix is to encourage individuals and organizations to consider and explore the different aspects of knowledge, helping them to identify blind spots, ask better questions, and make more informed decisions.

Servant Leadership

Servant Leadership is a leadership philosophy in which the primary goal of the leader is to serve others. This is starkly different from traditional leadership where the leader's main focus is on the thriving of their company or employees. A Servant Leader shares power, puts the needs of the employees first and helps people develop and perform as highly as possible.

This concept was first coined by Robert K. Greenleaf in his 1970 essay, "The Servant as Leader." Greenleaf suggested that the best leaders were servants first, and the key tools for a servant leader are listening, empathy, healing, awareness, persuasion, conceptualization, foresight, stewardship, and commitment to the growth of people and building community.

Here are the core tenets of Servant Leadership:

Listening: Leaders have traditionally been valued for their communication and decision-making skills. Although these are also important skills for the servant-leader, they need to reinforce these with a deep commitment to listening intently to others.
Empathy: The servant-leader strives to understand and empathize with others. Workers may be considered not only as employees, but also as people who have complex personal lives.
Healing: Servant-leaders recognize that they also have an opportunity to help make whole those with whom they come in contact.
Awareness: General awareness, and especially self-awareness, strengthens the servant-leader.
Persuasion: Servant-leaders rely on persuasion, rather than positional authority in making decisions.
Conceptualization: Servant-leaders nurture their abilities to dream great dreams.
Foresight: Foresight is a characteristic that enables servant-leaders to understand lessons from the past, the realities of the present, and the likely consequence of a decision in the future.
Stewardship: Servant leadership, like stewardship, assumes first and foremost a commitment to serving the needs of others.
Commitment to the Growth of People: Servant-leaders believe that people have an intrinsic value beyond their tangible contributions as workers.
Building Community: The servant-leader senses that much has been lost in recent human history as a result of the shift from local communities to large institutions as the primary shaper of human lives.

The overall goal of Servant Leadership is to build a more just, caring and sustainable world.

Pygmalion Effect

The Pygmalion Effect is a psychological principle that describes how our expectations about the abilities of others can influence their actual performance. It was first identified by psychologist Robert Rosenthal and school principal Lenore Jacobson in the late 1960s.

Here are the main aspects of the Pygmalion Effect:

Expectation Influence Performance: If you have high expectations of someone, they are likely to perform better. Conversely, if you have low expectations of someone, they are likely to perform poorly. This happens because our expectations subtly communicate what we think the other person is capable of, affecting their self-confidence and effort.
Self-fulfilling Prophecy: The Pygmalion Effect is a type of self-fulfilling prophecy. When we believe someone will behave in a certain way, we may treat them in ways that induce them to behave as expected, thereby confirming our initial expectations.
Application in Leadership and Education: The Pygmalion Effect is especially relevant in leadership and educational settings. Leaders or teachers who believe in their team members' or students' abilities to succeed can inspire those individuals to meet those expectations. However, if a leader or teacher has low expectations, it may negatively affect performance.
Importance of Positive Expectations: The Pygmalion Effect underscores the power of positive expectations. By showing others that you believe in their potential, you can motivate them to reach their goals, improving individual and collective performance.

In summary, the Pygmalion Effect demonstrates the power of expectation and the role it plays in performance. It's an important concept to understand for anyone in a leadership, teaching, or mentoring position.

Goleman's Emotional Intelligence Model

Goleman's Emotional Intelligence Model was proposed by psychologist and science journalist Daniel Goleman in his 1995 book "Emotional Intelligence: Why It Can Matter More Than IQ." The model outlines five key components of emotional intelligence that are crucial to leadership, personal development, and interpersonal effectiveness.

Here are the core components of Goleman's model:

Self-Awareness: This refers to the ability to recognize and understand your own emotions, strengths, weaknesses, values, and drives, as well as their impact on others. Self-awareness also includes recognizing your own emotional reactions to people or situations around you.
Self-Regulation (or Self-Management): This is about managing your internal states, impulses, and resources. This includes being able to control or redirect disruptive emotions and adapting to changing circumstances. Essentially, it’s about managing your emotions and keeping them in check.
Motivation: This refers to being driven to achieve for the sake of achievement. Motivation includes your personal drive to improve and achieve, commitment to your goals, initiative, or readiness to act on opportunities, and optimism and persistence in pursuing goals, even if things don’t go as planned.
Empathy: This is about considering other people's feelings, especially when making decisions. It involves understanding the emotional makeup of other people and treating them according to their emotional reactions.
Social Skills (or Relationship Management): This refers to managing relationships to move people in the desired direction. Social skills involve the ability to influence, to communicate clearly, to inspire and influence others, to manage conflict, to foster cooperation, and to build and maintain good personal relationships.

Goleman’s model underlines that emotional intelligence is as important, if not more important, than cognitive abilities for personal and professional success, especially in leadership roles.

Conclusion

This concludes my series on mental models useful in Leadership.

Leadership models III: First Principles Thinking

2023-05-16T00:00:00+02:00

Introduction

This is the third article in my series on useful mental models in leadership and self-management:

I dedicate a full post to First Principles Thinking, popularized by Elon Musk, CEO of SpaceX and Tesla, who has credited First Principles Thinking as a key factor in his ability to create innovative solutions and businesses. He contrasts it with reasoning by analogy, or taking what's already given and making incremental improvements.

First Principles Thinking

First Principles Thinking is a problem-solving method that involves breaking down complex problems into their most basic, fundamental elements and then reasoning up from there. The idea is to shed assumptions and received wisdom in order to understand the foundational principles at the heart of the problem:

Identify and Define Your Assumptions: Start by identifying what you think you know about a given problem or situation. These are your assumptions or beliefs that may or may not be true.
Break Down the Problem into Its Fundamental Principles: Once you have identified your assumptions, start breaking down the problem into its basic, fundamental parts. These are the truths or principles that are so fundamental that they cannot be broken down further.
Create New Solutions from Scratch: Using these fundamental principles, start to build up a solution from scratch. This is where the innovative solutions can come from, as you're no longer constrained by existing methods or conventional wisdom.

For example, when Elon Musk was faced with the high cost of space travel, he didn't just accept the established "fact" that rockets are expensive. Instead, he broke the problem down to its first principles, asking why a rocket is expensive and what materials make up a rocket. He discovered that the materials cost of a rocket was about 2% of the typical price. This insight led him to the idea of building cost-effective rockets, a key factor in SpaceX's success.

First Principles Thinking can be challenging as it forces you to abandon preconceived ideas and assumptions. However, it can also lead to innovative solutions that would have been missed with traditional problem-solving methods.

Below are illustrations of First Principles thinking: - SpaceX and Elon Musk - Berkshire and Warren Buffet - Google's organization design - Python - Random Forests - Convolutional Neural Networks

SpaceX

One of the most well-known examples of First Principles Thinking applied by Elon Musk is his approach to building reusable rockets at SpaceX.

Traditionally, rockets have been single-use. After delivering their payload, they either burn up in the Earth's atmosphere or fall back to Earth and land in the ocean, rendering them unusable for future launches. This method was considered a "given" in the industry. It contributed significantly to the high cost of space travel, as a new rocket had to be built for every launch.

Musk, however, didn't accept this as a given. Instead, he applied First Principles Thinking. He started by questioning the fundamental premise: Why are rockets single-use? Is it a fundamental law of physics that they have to be? It turned out that it wasn't.

Next, he broke down the cost of a rocket into its fundamental principles. He found that the raw materials that make up a rocket only account for about 2% of the final cost of a rocket. This meant that the significant cost was coming from the manufacturing and assembly of these materials into a rocket. If a rocket could be reused, then the cost per launch could be dramatically reduced.

From these first principles, Musk and his team at SpaceX began developing rockets that can return to Earth and land vertically, allowing them to be reused for future launches. Despite significant skepticism and several early failures, SpaceX has successfully landed multiple rockets back on Earth and reused them, significantly reducing the cost of each individual launch and revolutionizing the space industry in the process. This example illustrates how First Principles Thinking can challenge established norms and lead to innovative solutions, even in industries that have been operating the same way for decades.

Berkshire

Warren Buffet and Charlie Munger, the leaders of Berkshire Hathaway, are well known for their application of First Principles Thinking in their investment strategies. They have developed a distinctive approach to investing, which involves careful analysis of a company's fundamentals rather than following market trends or short-term fluctuations.

A fundamental principle of their investing philosophy is the concept of "intrinsic value". Instead of focusing on external factors like market sentiment or speculative trends, they look at the underlying value of a business. They assess factors such as the company's earnings, the quality of its management, its competitive position, and its potential for growth. If the market price of a company is significantly below its intrinsic value, they see it as a good investment opportunity.

This approach requires deep understanding and analysis of a business – its operations, its industry, its competitors, and its risks. This is in contrast to some other investment approaches that focus more on technical analysis of stock price movements or macroeconomic trends.

Another example of their First Principles Thinking is their focus on "circle of competence". They believe in investing only in businesses that they understand thoroughly. This means they often avoid hot trends or sectors they are not familiar with, even if these are popular with other investors.

Their First Principles approach to investing has helped Buffet and Munger achieve consistent returns over the long term and build Berkshire Hathaway into one of the largest and most successful investment companies in the world. Their success shows how First Principles Thinking can lead to superior results, even in fields like investing where there is a lot of uncertainty and many different approaches can be successful.

Google

Google's unique approach to organizational design and management can be seen as an application of First Principles Thinking. Instead of accepting traditional management structures and practices as given, they questioned the fundamental principles of how to organize and manage a company for innovation and productivity.

20% time policy: traditionally, companies expect employees to dedicate 100% of their work time to their assigned tasks and projects. Google, however, applied First Principles Thinking to this issue. They started with the premise that creative and innovative ideas can come from anyone in the organization, and that employees may have interests and abilities that aren't being used in their regular work. From this, they developed the "20% Time" policy, which allows engineers to spend 20% of their time working on any project they choose. The idea is that by giving employees time and freedom to explore their own ideas, they can foster innovation and creativity. This policy has led to the development of some of Google's most successful products, including Gmail, Google News, and AdSense.
Hiring and team formation: Instead of simply looking at a candidate's credentials and experience, Google looks at attributes like problem-solving ability, conscientiousness, and comfort with ambiguity. They recognize that the challenges they face are often unique and require innovative thinking, so they value these qualities over specific experience or expertise. They also pay attention to team dynamics, understanding that the success of a team depends not just on the skills of its members, but on their ability to work together effectively.
Objective and Key Results (OKRs): Google has adopted and popularized the use of OKRs, a goal-setting system that requires every team and individual in the company to set ambitious goals and measurable key results each quarter. The OKRs are publicly available to all employees in the organization. This practice promotes transparency, alignment, and focus throughout the organization, ensuring everyone is working towards the same strategic objectives. The public nature of OKRs also fosters a culture of accountability and open communication. Rather than accepting traditional goal-setting or performance management systems, Google's use of OKRs reflects a First Principles approach to driving organizational performance.
Flattened Hierarchy and Openness: Google has questioned the traditional hierarchical structure found in many organizations. Instead of a rigid hierarchy, Google promotes a more flattened organizational structure, where the distance between executives and employees is minimized. This is evident in practices such as 'TGIF' meetings (now called 'TGiT' since they moved to Thursdays), where founders Larry Page and Sergey Brin, and later Sundar Pichai, held weekly all-hands meetings where they shared updates and fielded questions - both in person and submitted online - from Googlers around the world. This level of transparency and open communication is unusual in large corporations but reflects Google's First Principles approach to leadership and management.
Focus on Data: Google is known for its data-driven decision-making culture. They apply First Principles Thinking in this area by challenging the idea that business decisions are best made based on experience or intuition. Instead, they argue that decisions should be based on data and rigorous analysis. This can be seen in their "People Analytics" function, which applies data analysis methods to HR and management issues. For example, they used data analysis to develop their "Project Oxygen", where they identified the key behaviors of effective managers at Google, challenging preconceived notions of what makes a good manager.

These examples illustrate how Google applies First Principles Thinking to various aspects of its organizational design and management practices, questioning traditional assumptions and developing innovative, effective practices based on fundamental principles.

Python

Python was created in the late 1980s by Guido van Rossum. At the time, many programming languages were designed with a focus on machine efficiency, resulting in code that could be difficult to write and even more difficult to read. Syntax was often complex and verbose, and different tasks could require vastly different conventions or paradigms. This made many languages difficult to learn and code difficult to maintain.

Rather than accepting these characteristics as a given, van Rossum took a First Principles approach when designing Python. He started with the premise that code is read more often than it is written, and therefore readability counts. This led to a focus on simplicity and clarity in Python's design. Python's syntax is straightforward and consistent, making it easier to read, write, and learn than many other languages.

Another first principle was that there should be one, and preferably only one, obvious way to do it. This contrasts with languages like Perl, which proudly proclaims that there is more than one way to do it. This principle guided the design of Python's features and syntax, resulting in a language where different programmers will often write very similar code to perform the same task, which greatly aids readability and maintainability.

These principles have guided Python's development for over three decades, and have helped it become one of the most popular programming languages in the world. This demonstrates how First Principles Thinking can be applied to software design, leading to innovative solutions that challenge the status quo.

Random Forests

Before the development of ensemble methods like Random Forest, decision trees were a popular method for both classification and regression tasks. They were easy to understand and interpret, but they often suffered from high variance, meaning that they were prone to overfitting the training data and performed poorly on unseen data.

Instead of trying to tweak or adjust decision trees to make them more robust, Leo Breiman, the inventor of the Random Forest algorithm, went back to the first principles. He realized that the high variance was a fundamental issue with decision trees, and that the solution was not to fix individual trees but to change the way they were used.

This led him to the concept of bagging (bootstrap aggregating), where multiple decision trees are trained on different subsets of the data, and their predictions are aggregated to produce a final result. This approach effectively reduces the variance without increasing the bias, making the model more robust and reliable.

Random Forests added another layer to this concept: feature randomness. In addition to training each tree on a different subset of the data, Random Forests also use a subset of features at each split in the decision tree. This further increases the diversity of the individual trees in the forest, making the model even more robust.

The development of Random Forests has had a significant impact on the field of machine learning. They are one of the most powerful and widely used machine learning algorithms today, and their development has inspired a range of other ensemble methods. This shows how First Principles Thinking can lead to major breakthroughs and innovations.

Convolutional Neural Networks

First Principles Thinking in data science, machine learning, and AI involves starting from foundational truths about data and algorithms and building up from there. This can lead to more effective models and methods of analysis. Consider the development of convolutional neural networks (CNNs) for image recognition as an example.

In the early days of AI and machine learning, image recognition was a significant challenge. Traditional algorithms struggled with changes in orientation, size, and lighting conditions in images. The common approach was to engineer features manually, which was very time-consuming and not particularly effective.

Instead of trying to incrementally improve these methods, researchers applied First Principles Thinking. They started by considering the fundamental nature of the problem: recognizing patterns in pixel data that are invariant to scale, orientation, and other transformations.

Researchers, notably Yann LeCun and his team, drew inspiration from the human visual system, where different neurons specialize in recognizing different patterns and these patterns are combined to understand the whole image. This led them to design a completely new kind of neural network architecture, the Convolutional Neural Network, which incorporates convolutional layers that can automatically learn a variety of spatial hierarchies of features, effectively learning from the fundamental features of the image data.

The convolutional layer uses a set of learnable filters. Each filter is spatially small but extends through the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the entries of the filter and the input and producing a 2-dimensional activation map of that filter. As a result, the network learns filters that activate when they see some type of visual feature such as an edge of some orientation or a blotch of some color on the first layer.

CNNs revolutionized the field of image recognition and now form the basis for most image recognition technology, including facial recognition and self-driving cars. This is a great example of how First Principles Thinking can lead to significant breakthroughs in technology and AI.

Conclusion

First Principles Thinking is a problem-solving approach that involves breaking down complex problems into their fundamental, self-evident truths and then reasoning up from these base truths to create new understanding or solutions. This method encourages innovative thinking by challenging existing assumptions and conventions, and building new solutions from the ground up.

Leadership models II: Growth Mindset, Eisenhower Matrix, Tuckman Model, Cynefin Framework, SCARF Model

2023-05-14T00:00:00+02:00

Introduction

This year I'm attending a multi-day leadership course that spans 6 months. One of the things the coaches talk about is various mental models useful in leadership and self-management. These are some of the mental models mentioned in the course.

Growth Mindset

The Growth Mindset is a concept introduced by psychologist Carol Dweck in her book "Mindset: The New Psychology of Success." It is centered around the belief that intelligence, abilities, and talents can be developed through effort, learning, and persistence. The Growth Mindset contrasts with the Fixed Mindset, which assumes that these traits are innate and unchangeable. The core tenets of the Growth Mindset are:

Embrace challenges: People with a growth mindset see challenges as opportunities for growth and development. They are more likely to take risks, try new things, and persist in the face of difficulties.
Value effort: In a growth mindset, effort is seen as a critical factor in achieving success. Hard work, dedication, and practice are valued as essential components of improvement and growth.
Learn from criticism and feedback: Individuals with a growth mindset are open to constructive criticism and feedback, viewing it as an opportunity to learn and improve their skills and understanding.
Be inspired by others' success: Instead of feeling threatened by the success of others, people with a growth mindset are inspired by it. They see the achievements of others as evidence that they too can improve and succeed.
Focus on continuous learning: A growth mindset prioritizes learning and self-improvement over simply proving oneself or maintaining a specific image. People with a growth mindset are lifelong learners, always looking for ways to expand their knowledge and abilities.
Develop resilience and persistence: Individuals with a growth mindset are more likely to persevere in the face of setbacks and obstacles. They understand that failure is a natural part of the learning process and an opportunity to learn, adapt, and grow.

Cultivating a growth mindset can lead to increased motivation, higher achievement, and greater overall satisfaction in both personal and professional aspects of life. By embracing the principles of the growth mindset, individuals can unlock their full potential and foster a love of learning and self-improvement.

Eisenhower Matrix

The Eisenhower Matrix is a time management tool that helps individuals prioritize tasks by urgency and importance. The tool is named after Dwight D. Eisenhower, the 34th President of the United States, who is attributed with saying, "I have two kinds of problems, the urgent and the important. The urgent are not important, and the important are never urgent."

The matrix consists of a 2x2 grid, creating four quadrants:

Quadrant I - Important and Urgent: These are tasks that need to be done immediately. They often involve critical issues that carry significant consequences if not addressed promptly. Examples could include crises, deadlines, or problems that need immediate attention.
Quadrant II - Important but Not Urgent: These are tasks that contribute to long-term goals and values but do not need to be done immediately. This quadrant often involves strategic planning, relationship building, personal growth, and preventive activities. Spending more time on these tasks can reduce the frequency of urgent issues arising.
Quadrant III - Not Important but Urgent: These tasks require immediate attention but do not contribute significantly to achieving long-term goals or values. They often involve interruptions or distractions, such as unimportant emails, phone calls, or meetings. These tasks can often be delegated to others.
Quadrant IV - Not Important and Not Urgent: These tasks do not contribute to long-term goals and do not require immediate attention. They often involve time-wasting activities or tasks that offer little to no value. These tasks should be minimized or eliminated.

The core tenets of the Eisenhower Matrix include:

Distinction between Urgency and Importance: Recognizing that urgent tasks are not always important, and important tasks are not always urgent, is key to effective use of the matrix.
Prioritization: The matrix helps individuals prioritize tasks based on their importance and urgency, leading to more effective time management and productivity.
Delegation: Tasks that are urgent but not important can often be delegated to others, freeing up time for more important activities.
Elimination: Unimportant and non-urgent tasks should be eliminated as they consume time and resources without contributing to long-term goals or values.
Focus on Quadrant II: The goal is to maximize time spent on Quadrant II activities, which contributes most to long-term success and can prevent issues from becoming urgent crises.

By using the Eisenhower Matrix, individuals can manage their time more effectively, focusing on what truly matters and reducing stress and burnout.

Tuckman Model

The Tuckman Model, also known as Tuckman's Stages of Group Development, is a framework proposed by psychologist Bruce Tuckman in 1965 to describe the path that most teams follow on their way to high performance. The model outlines four distinct stages:

Forming: The initial stage of team development where team members are polite and positive, but also anxious as they haven't fully understood what work the team will do. There is typically a focus on defining the team's goals, structure and leadership, with team members starting to take on roles and responsibilities.
Storming: As team members start to work together, they begin to push against the boundaries established in the forming stage. This can result in conflict within the team as individuals jockey for position, resist control by group leaders, and start voicing differing opinions about the team's direction. Despite its name, the "storming" phase isn't always a negative space -- it can also be a time of healthy, constructive debate.
Norming: Gradually, the team moves into the norming stage, where people start to resolve the conflicts that surfaced during the storming stage. The team becomes more aware of their collective goals, and they begin to appreciate each other's skills and experience. Norms and roles are established, and the team becomes more cohesive and functional.
Performing: The team reaches the performing stage when hard work leads, without friction, to the achievement of the team's goal. The structures and processes that you have set up support this well. As a leader, you can delegate much of your work, and you can concentrate on developing team members. It's often characterized by high levels of trust, autonomy, and interdependence.
Later, Tuckman added a fifth stage, Adjourning (or "Mourning"), where the team completes the work and moves on from the project. This can be a challenging stage as team members may have developed strong bonds and may feel a sense of loss upon the project's completion.

The core tenets of the Tuckman Model are:

Sequential Development: Teams tend to develop in stages, each of which presents unique challenges and opportunities for growth and development.
Conflict as a Natural Part of Development: Conflict isn't necessarily a bad thing, and can be a natural part of the team development process.
The Role of Leadership: The leader's role may change at each stage, from giving direction in the forming stage to fostering independence and competence in the performing stage.
The Importance of Clear Goals and Roles: Clear goals and well-defined roles can facilitate the team's progress through each stage.
Transition and Closure: Teams should acknowledge and plan for the end of the project or task, helping team members transition and bring closure to the experience.

Cynefin Framework

The Cynefin Framework, developed by Dave Snowden, is a decision-making model that helps leaders and organizations navigate the complexity of different problems and determine the most appropriate approach for each situation. The framework categorizes problems or contexts into five domains, each with its unique characteristics and recommended strategies:

Simple: In this domain, problems have clear cause-and-effect relationships and can be solved using established best practices. The recommended approach is to sense the situation, categorize the problem, and then respond using known solutions.
Complicated: Problems in this domain have multiple possible solutions, and cause-and-effect relationships can be determined with expert analysis. The recommended approach is to sense the situation, analyze the problem using expert knowledge, and then respond with the most suitable solution.
Complex: In this domain, cause-and-effect relationships can only be understood in retrospect due to the unpredictable nature of the problem. The recommended approach is to probe the situation by conducting experiments, sense the emerging patterns, and then respond by adapting to the evolving conditions.
Chaotic: Problems in this domain are characterized by high levels of uncertainty, with no apparent cause-and-effect relationships. The recommended approach is to act immediately to stabilize the situation, sense the new conditions, and then respond by adapting or creating novel solutions.
Disorder: This domain represents situations where it is unclear which of the other four domains applies. The objective in this context is to gather more information, break the problem down into smaller components, and then assign each component to the appropriate domain for further action.

By understanding the nature of a problem and determining which domain it falls into, leaders can use the Cynefin Framework to choose the most appropriate decision-making approach and improve their ability to address complex and unpredictable challenges.

SCARF Model

The SCARF Model is a social cognitive neuroscience model devised by Dr. David Rock. SCARF stands for Status, Certainty, Autonomy, Relatedness, and Fairness. These are the five key domains that influence our behavior in social situations:

Status: This refers to relative importance to others. Status is about where you rank in relation to those around you. Threats to status, such as feeling inferior or being disrespected, can be incredibly powerful in driving behavior.
Certainty: This pertains to our ability to predict the future. The brain craves certainty to make accurate predictions about the world. Uncertainty provokes a threat response and can therefore be stressful and draining.
Autonomy: This concerns the sense of control over events. Having a sense of autonomy, the feeling that you have some control over your environment, is crucial for our sense of wellbeing. A lack of autonomy can lead to stress, which impairs performance and morale.
Relatedness: This involves the sense of safety with others, of being 'in' or 'out' of a social group. It reflects our need to feel connected to others. When we perceive someone as being in our 'in-group', we work more collaboratively and cooperatively with them.
Fairness: This reflects the perception of fair exchanges between people. The perception of unfairness can provoke a strong threat response and can significantly impact workplace performance and engagement.

These concepts are all drawn from neuroscience research which suggests that the brain treats many social threats and rewards with the same intensity as physical threats and rewards. For instance, a perceived threat to one's status activates the same brain networks as a threat to one's life. By understanding these triggers, leaders and managers can learn to minimize threats and maximize rewards to help their teams be more effective, engaged, and collaborative.

Leadership models I: Iceberg Model, Six Thinking Hats, Trust Equation, Circle of influence, OODA Loop

2023-05-12T00:00:00+02:00

Introduction

Iceberg Model

The organizational Iceberg Model is used to understand organizational culture and change management. It emphasizes the visible and tangible aspects of an organization (above the water) and the more subtle, intangible, and often unconscious elements (below the water) that contribute to an organization's overall culture and performance.

Above the water:

Organizational Structure: The formal hierarchy, roles, and reporting relationships within an organization.
Processes: The systems, procedures, and protocols that govern how work is carried out, including decision-making, communication, and problem-solving.
Skills: The abilities, competencies, and expertise of individuals within the organization.
Behaviors: The observable actions and interactions of individuals and teams, reflecting how people conduct themselves in the workplace.
Vision: The long-term direction, goals, and aspirations of the organization.
Strategy: The plan or approach to achieving the organization's vision, including the allocation of resources, priorities, and milestones.
Stated Values: The explicit principles, beliefs, or ethical guidelines that the organization claims to prioritize and uphold.

Below the water:

Feelings: The emotions, moods, and attitudes of individuals within the organization, which can influence behavior, decision-making, and overall performance.
Norms: The informal, unwritten rules or expectations that govern behavior and interactions within the organization.
Values: The deeply-held beliefs and priorities that influence decision-making, actions, and attitudes, often operating at an unconscious level.
Unwritten Rules: The implicit guidelines, habits, or customs that influence behavior and decision-making within the organization, even though they may not be formally documented or acknowledged.
Shared Assumptions: The commonly-held beliefs or premises that individuals within the organization take for granted, which can shape their perceptions, actions, and interpretations of events.
Symbols: The visual, verbal, or physical representations that convey meaning, values, or identity within the organization, such as logos, jargon, or office layouts.
Stories: The narratives, anecdotes, or legends that circulate within the organization, reflecting its history, culture, and values, and helping to shape its collective identity.

The organizational Iceberg Model highlights the importance of considering both the overt, visible aspects of an organization and the underlying, often hidden elements that influence its culture and effectiveness. By addressing both the "above water" and "below water" components, leaders can develop a more comprehensive understanding of their organization and implement more targeted and sustainable change initiatives.

Six Thinking Hats

The Six Thinking Hats is a parallel thinking process and decision-making tool developed by Dr. Edward de Bono. The model encourages individuals or teams to approach problems or discussions from six different perspectives by "wearing" different metaphorical hats. Each hat represents a specific mode of thinking, and the purpose is to facilitate more efficient, creative, and comprehensive problem-solving, decision-making, and communication. The six hats are:

Blue Hat: Focused on managing the thinking process. When wearing the blue hat, participants oversee the overall discussion, ensuring that the group stays on track, follows the thinking process, and reaches a conclusion or decision.
White Hat: Focused on data and information. When wearing the white hat, participants gather, analyze, and share facts, figures, and objective information related to the topic.
Red Hat: Focused on emotions and intuition. When wearing the red hat, participants express their feelings, emotions, and gut reactions to the issue without the need for justification or rationalization.
Green Hat: Focused on creativity and innovation. When wearing the green hat, participants generate new ideas, alternative solutions, and innovative approaches, encouraging lateral thinking and brainstorming.
Yellow Hat: Focused on optimism and benefits. When wearing the yellow hat, participants explore the positive aspects, advantages, and opportunities associated with an idea or proposal, considering the best-case scenarios and potential value.
Black Hat: Focused on critical thinking and potential problems. When wearing the black hat, participants assess potential risks, challenges, and obstacles, identifying why a proposed solution may not work or what could go wrong.

By using the Six Thinking Hats approach, individuals and teams can explore different aspects of a problem, reduce conflict and biases, and make more balanced and well-informed decisions.

Trust Equation

The Trust Equation, developed by Charles H. Green is a model that defines the key components of trustworthiness in professional relationships. It highlights four elements that contribute to building and maintaining trust between individuals. The Trust Equation is represented as follows:

Trustworthiness = (Credibility + Reliability + Intimacy) / Self-Orientation

Credibility: This component is about the perception of your expertise, knowledge, and competence in a given area. Credibility is established through demonstrating your skills, providing accurate information, and offering well-informed opinions.
Reliability: This component refers to the consistency of your actions and the extent to which you can be counted on to deliver on your promises. Being dependable, punctual, and accountable for your commitments contributes to building reliability.
Intimacy: This component relates to the emotional safety and closeness that others feel when they interact with you. Establishing intimacy requires active listening, empathy, and the ability to create a comfortable environment where others feel safe to share their thoughts and feelings.
Self-Orientation: This component represents the extent to which you prioritize your own interests over those of others. High self-orientation can undermine trust, as it suggests that you may act in a self-serving manner. Lowering self-orientation involves demonstrating genuine concern for others' needs and focusing on building mutually beneficial relationships.

The Trust Equation provides a framework for understanding and developing trust in professional relationships. By enhancing credibility, reliability, and intimacy while reducing self-orientation, individuals can improve their trustworthiness and foster stronger, more productive relationships with colleagues, clients, and stakeholders.

Circle of Influence

The Circle of Influence and Circle of Concern concept is a mental model popularized by Stephen Covey. It is a tool designed to help individuals focus their time and energy on the areas where they can make the most impact, leading to increased effectiveness and personal growth. The model consists of two components:

Circle of Concern: This circle encompasses all the things that an individual may be worried or concerned about, but over which they have little or no control. Examples include global events, economic conditions, or the actions and decisions of other people. Focusing too much on the Circle of Concern can lead to feelings of helplessness and frustration, as it involves situations where the individual cannot directly influence the outcome.
Circle of Influence: This circle represents the things that an individual can directly affect or control, such as their own behavior, decisions, and reactions. Examples include personal habits, work performance, or relationships with others. Focusing on the Circle of Influence empowers individuals to take responsibility for their actions and proactively address the challenges they face.

The key to using the Circle of Influence and Circle of Concern concept effectively is to concentrate on the areas within one's Circle of Influence, rather than dwelling on the concerns that are beyond their control. By doing so, individuals can make a tangible impact on their lives and may even expand their Circle of Influence over time. This approach helps to cultivate a proactive mindset, leading to increased productivity, effectiveness, and personal satisfaction.

OODA Loop

The OODA Loop (Observe, Orient, Decide, Act) is a decision-making model developed by U.S. Air Force Colonel John Boyd. It provides a framework for understanding and responding to rapidly changing situations, particularly in competitive or adversarial environments. The OODA Loop emphasizes the importance of agility, adaptability, and quick decision-making. The model consists of four components:

Observe: The first step involves gathering data and information about the current situation, including the actions of competitors or adversaries, changes in the environment, and any other relevant factors. This step requires maintaining situational awareness and staying up-to-date with the latest developments.
Orient: In this step, you analyze the collected data to understand the context, identify patterns or trends, and evaluate the implications for your goals or objectives. This may involve updating your mental models, considering various perspectives, and recognizing your own biases and assumptions.
Decide: Based on your understanding of the situation, you develop a plan or make a decision about the best course of action. This step requires weighing different options, evaluating their potential risks and rewards, and choosing the most appropriate response given the available information.
Act: Once a decision has been made, you implement the chosen course of action as quickly and effectively as possible. This may involve coordinating with team members, allocating resources, or executing specific tasks or maneuvers.

After completing the Act step, the OODA Loop continues as you return to the Observe phase, monitoring the effects of your actions and gathering new information about the evolving situation. This iterative process allows for continuous learning, adaptation, and improvement, enabling individuals or organizations to respond effectively to dynamic and uncertain environments.

GPT News Poet: silly AI poems based on today's news

2023-05-07T00:00:00+02:00

Introduction

I think one of the coolest (and most useless) uses of Large Language Models is generating poetry. This resonates strongly with me, because I could never write poetry of any quality, but I do appreciate silly 4-liners. It's live here!

The code is up on Github.

News

There are multiple sites to get news from. For this toy project I did not want to scrape, I wanted to use a site that has a nice API and already returns structured JSON. GNews (not Google News) is exactly like that, and it's free up to 100 requests/day, and this only takes 1 request/day.

The endpoint is:

https://gnews.io/api/v4/top-headlines?category={category}&lang=en&country=us&max=10&apikey={apikey}

It returns a list of JSON dictionaries that looks like this:

[
    {
        "title": "Loaf-size mission to improve hurricane forecasting is ready to launch",
        "description": "A new NASA mission called TROPICS, designed to improve hurricane forecasting, is ready to launch ahead of the June 1 arrival of the 2023 Atlantic hurricane season.",
        "content": "Sign up for CNN’s Wonder Theory science newsletter. Explore the universe with news on fascinating discoveries, scientific advancements and more.\nCNN —\nA new mission designed to improve hurricane forecasting is ready to launch, just ahead of the June ... [3592 chars]",
        "url": "https://www.cnn.com/2023/05/07/world/nasa-tropics-mission-launch-scn/index.html",
        "image": "https://media.cnn.com/api/v1/images/stellar/prod/230428155358-01-nasa-tropics-mission-042023.jpg?c=16x9&q=w_800,c_fill",
        "publishedAt": "2023-05-07T10:53:00Z",
        "source": {
        "name": "CNN",
        "url": "https://www.cnn.com"
        },
    },
    ...
,

GPT-3

ChatGPT-4 generates significantly better quality poems that GPT-3, but there is no official API yet for it. So I used GPT-3 for this toy project, specifically text-davinci-003. The prompt I use is very simple and mostly works:

Write a witty 4 line poem about the following news: {articles_text}

Since I'm a lazy programmer in the age of AI, I used ChatGPT-4 to generate the Python code to access GPT-3 over the OpenAI API. I chose to save the generated poems into the JSON returned above, and then dump the whole thing to disk. Generating the HTML for the website will be a second, completely removed step.

The complete code for the first step of the pipeline, which downloads the news and generates the poems is less than 50 LOC:

import json
import openai
import urllib.request
import urllib.parse
from datetime import datetime
import secrets

def gnews_top_news(category='general', q=None):
    apikey = secrets.gnews_apikey
    url = f"https://gnews.io/api/v4/top-headlines?category={category}&lang=en&country=us&max=10&apikey={apikey}"
    if q is not None:
        url += "&" + urllib.parse.urlencode({"q": q})
    with urllib.request.urlopen(url) as response:
        data = json.loads(response.read().decode("utf-8"))
        return data["articles"]

def query_gpt_35(prompt, model="text-davinci-003", max_tokens=4000):
    openai.api_key = secrets.openai_apikey
    response = openai.Completion.create(
        engine=model,
        prompt=prompt, 
        max_tokens=max_tokens-len(prompt),
        n=1,
        stop=None,
        temperature=0.8,
    )
    generated_text = response.choices[0].text.strip()
    return generated_text

articles = gnews_top_news()
for i, a in enumerate(articles):
    articles_text = f"{a['title']}. {a['description']}"
    poet_prompt = f"Write a witty 4 line poem about the following news: {articles_text}"
    for _ in range(5):
        response = query_gpt_35(poet_prompt)
        if len(response.split("\n")) == 4:
            break
    a['poem'] = response.strip()

directory = "articles/"
filename = f"articles-{datetime.now().strftime('%Y-%m-%d')}.json"
with open(directory + filename, "w") as f:
    json.dump(articles, f)

Frontend

At this point I just need a web front-end. Initially I was planning to write a Flask app which reads the above JSON dynamically and convert it to HTML, but then I realized it's easier to just generate static HTML with Jinja templates and serve that up. I again used ChatGPT-4 to generate skeleton HTML+JS+CSS code, which needed heavy editing to actually make it work. The Python code to generate the HTML is quite short:

import os
import re
import json
from datetime import date, datetime
from jinja2 import Environment, FileSystemLoader

def get_latest_articles(directory="/home/mtrencseni/gpt-news-poet/articles/"):
    pattern = re.compile(r'articles-\d{4}-\d{2}-\d{2}\.json')    
    files = [f for f in os.listdir(directory) if pattern.match(f)]
    files.sort(reverse=True)
    latest_file_path = os.path.join(directory, files[0])
    with open(latest_file_path, 'r') as f:
        return json.load(f)

env = Environment(loader=FileSystemLoader('templates'))
html = env.get_template('index.jinja.html').render(
    articles=get_latest_articles(),
    dt=date.today().strftime("%A, %B %d, %Y")
    )
directory = "www/archive/"
filename = datetime.now().strftime('%Y-%m-%d') + ".html"
with open(directory + filename, "w") as f:
    f.write(html)
directory = "www/"
filename = "index.html"
with open(directory + filename, "w") as f:
    f.write(html)

Other files:

Crontab

First, I wrote a quick shell script to run the above pipeline:

#!/usr/bin/env bash

venv/bin/python3 generate_articles.py
venv/bin/python3 generate_html.py

I then put a line in my crontab so the cron daemon runs the pipeline every day at 9AM NYC time (13 UTC):

0 13 * * * cd /home/mtrencseni/gpt-news-poet && /home/mtrencseni/generate.sh

Conclusion

I think it's very cool that you can generate reasonable quality poetry from a cronjob in 2023. Once ChatGPT-4 becomes available over the OpenAI API, the quality of the poems will be even better!

Can ChatGPT write the C++20 async message queue codes from the previous posts?

2023-04-15T00:00:00+02:00

Introduction

In the previous articles I wrote a simple async message queue server in Python:

I went through the same steps with Python. There, I concluded with trying to get ChatGPT-3.5 to write the code:

In the Python case, ChatGPT-3.5 was able to produce working code, but it wasn't logically correct: for example, it mixed up delivery semantics during caching.

In this post, I will try the same with C++20. This time, I will use ChatGPT-4.

First version

Prompt:

Write a simple, asynchronous, event-driven, in-memory message queue TCP/IP server in C++20 using the asio library. The server should accept the listening port as a command line argument. Cache 100 messages per topic. Here is the message format for subscribing to a topic:

{'command':'subscribe', 'topic':'foo'}
{'command':'subscribe', 'topic':'foo', 'last_seen': '5'}

If last_seen is specified, only replay cached messages with higher id. Sent messages look like this:

{'command':'send', 'topic':'foo', 'msg':'hello', 'delivery':'all'}
{'command':'send', 'topic':'foo', 'msg':'world', 'delivery':'one'}

When the message is forwarded to subscribers, also insert an id field into the message, so subsciber clients can maintain their own last_seen.

ChatGPT's code:

#include <asio.hpp>
#include <iostream>
#include <unordered_map>
#include <vector>
#include <deque>
#include <string>
#include <nlohmann/json.hpp>

using asio::ip::tcp;
using json = nlohmann::json;

struct Subscriber {
    tcp::socket socket;
    std::string last_seen;

    Subscriber(asio::io_context& io_context)
        : socket(io_context) {}
};

class MessageQueueServer {
public:
    MessageQueueServer(asio::io_context& io_context, int port)
        : acceptor_(io_context, tcp::endpoint(tcp::v4(), port)) {
        do_accept();
    }

private:
    void do_accept() {
        acceptor_.async_accept(
            [this](std::error_code ec, tcp::socket socket) {
                if (!ec) {
                    auto subscriber = std::make_shared<Subscriber>(std::move(socket));
                    read_command(subscriber);
                }
                do_accept();
            });
    }

    void read_command(std::shared_ptr<Subscriber> subscriber) {
        asio::async_read_until(subscriber->socket, asio::dynamic_buffer(buffer_),
                               '\n',
                               [this, subscriber](std::error_code ec, std::size_t) {
                                   if (!ec) {
                                       std::istream is(&buffer_);
                                       json command;
                                       is >> command;
                                       buffer_.consume(buffer_.size());

                                       handle_command(subscriber, command);
                                       read_command(subscriber);
                                   }
                               });
    }

    void handle_command(std::shared_ptr<Subscriber> subscriber, const json& command) {
        std::string cmd = command["command"];
        std::string topic = command["topic"];

        if (cmd == "subscribe") {
            subscribers_[topic].push_back(subscriber);
            if (command.contains("last_seen")) {
                subscriber->last_seen = command["last_seen"];
            }
            replay_messages(subscriber, topic);
        } else if (cmd == "send") {
            std::string msg = command["msg"];
            std::string delivery = command["delivery"];
            cache_message(topic, command);

            if (delivery == "all") {
                for (const auto& sub : subscribers_[topic]) {
                    send_message(sub, command);
                }
            } else if (delivery == "one") {
                if (!subscribers_[topic].empty()) {
                    send_message(subscribers_[topic].front(), command);
                }
            }
        }
    }

    void cache_message(const std::string& topic, const json& command) {
        auto& messages = message_cache_[topic];
        json cached_msg = command;
        cached_msg["id"] = next_message_id_++;
        messages.push_back(cached_msg);

        if (messages.size() > max_messages_per_topic_) {
            messages.pop_front();
        }
    }

    void replay_messages(std::shared_ptr<Subscriber> subscriber,
                         const std::string& topic) {
        for (const auto& msg : message_cache_[topic]) {
            if (subscriber->last_seen.empty() || msg["id"] > subscriber->last_seen) {
                send_message(subscriber, msg);
            }
        }
    }

    void send_message(std::shared_ptr<Subscriber> subscriber, const json& message) {
        std::string msg = message.dump() + "\n";
        asio::async_write(subscriber->socket, asio::buffer(msg),
                          [](std::error_code, std::size_t) {});
    }

private:
    tcp::acceptor acceptor_;
    std::unordered_map<std::string, std::vector<std::shared_ptr<Subscriber>>> subscribers_;
    std::unordered_map<std::string, std::deque<json>> message_cache_;
    std::size_t next_message_id_ = 1;
    const std::size_t max_messages_per_topic_ = 100;
    asio::streambuf buffer_;
};

int main(int argc, char* argv[]) {
    if (argc != 2) {
        std::cerr << "Usage: message_queue_server <port>\n";
        return 1;
    }

    int port = std::stoi(argv[1]);

    try {
        asio::io_context io_context;
        MessageQueueServer server(io_context, port);
        io_context.run();
    } catch (std::exception& e) {
        std::cerr << "Exception: " << e.what() << "\n";
    }

    return 0;
}

Compiling ChatGPT's code

The code does not compile, so I asked ChatGPT for help:

I'm trying to compile. In read_command(), I get this compile-time error:

no instance of overloaded function "asio::dynamic_buffer" matches the argument list

ChatGPT gives me a new version:

I apologize for the confusion. To resolve this error, you should change the asio::dynamic_buffer function call to asio::dynamic_buffer(buffer_.data()). Here's the updated read_command() function:

void read_command(std::shared_ptr<Subscriber> subscriber) {
    asio::async_read_until(subscriber->socket, asio::dynamic_buffer(buffer_.data()),
                           '\n',
                           [this, subscriber](std::error_code ec, std::size_t) {
                               if (!ec) {
                                   std::istream is(&buffer_);
                                   json command;
                                   is >> command;
                                   buffer_.consume(buffer_.size());

                                   handle_command(subscriber, command);
                                   read_command(subscriber);
                               }
                           });
}

This still does not compile, I ask again:

I apologize for the confusion earlier. I made a mistake in the suggestion. We should use asio::streambuf instead of asio::dynamic_buffer. Here's the updated read_command() function:

void read_command(std::shared_ptr<Subscriber> subscriber) {
    asio::async_read_until(subscriber->socket, buffer_, '\n',
                           [this, subscriber](std::error_code ec, std::size_t) {
                               if (!ec) {
                                   std::istream is(&buffer_);
                                   json command;
                                   is >> command;
                                   buffer_.consume(buffer_.size());

                                   handle_command(subscriber, command);
                                   read_command(subscriber);
                               }
                           });
}

Now I get this:

'Subscriber::Subscriber': no overloaded function could convert all the argument types

ChatGPT again gives me new code:

struct Subscriber {
    tcp::socket socket;
    std::string last_seen;

    explicit Subscriber(tcp::socket&& socket)
        : socket(std::move(socket)) {}
};

// ...

class MessageQueueServer {
public:
    MessageQueueServer(asio::io_context& io_context, int port)
        : acceptor_(io_context, tcp::endpoint(tcp::v4(), port)) {
        do_accept();
    }

// ...

Putting everything together, the code now compiles! I put json.hpp next to the .cpp file from the json C++ library and change that one #include line to:

#include "json.hpp"

The fact that ChatGPT was able to guide me to through these compile time errors to get a program that compiles is quite impressive. Looking at the code, there are some obvious problems, but let's try to run it anyway. It used JSON, which means double-quotes " for the messages rather than single quotes ' like I specified in the prompts, but let's ignore that.

Running ChatGPT's code

After compiling the code, I ran it, and connected 2 clients. The first subscribes:

{"command":"subscribe", "topic":"foo"}

A second client sends a message:

{"command":"send", "topic":"foo", "msg":"0", "delivery":"one"}

The message is delivered, and the server crashes immediately. There is none of our own code on the stack trace, the program is doing an async send() operation:

I tell ChatGPT what happened, it responds:

The issue might be related to the lifetime of the buffer_. Since buffer_ is a member variable of the MessageQueueServer class, it's shared between all connected clients. The ideal approach would be to have a separate buffer for each client. We can achieve this by adding a buffer to the Subscriber struct. Here's the updated Subscriber struct and the corresponding changes in the MessageQueueServer class:

struct Subscriber {
    tcp::socket socket;
    asio::streambuf buffer;
    std::string last_seen;

    explicit Subscriber(tcp::socket&& socket)
        : socket(std::move(socket)) {}
};

// ...

class MessageQueueServer {
public:
    MessageQueueServer(asio::io_context& io_context, int port)
        : acceptor_(io_context, tcp::endpoint(tcp::v4(), port)) {
        do_accept();
    }

// ...

void read_command(std::shared_ptr<Subscriber> subscriber) {
    asio::async_read_until(subscriber->socket, subscriber->buffer, '\n',
                           [this, subscriber](std::error_code ec, std::size_t) {
                               if (!ec) {
                                   std::istream is(&subscriber->buffer);
                                   json command;
                                   is >> command;
                                   subscriber->buffer.consume(subscriber->buffer.size());

                                   handle_command(subscriber, command);
                                   read_command(subscriber);
                               }
                           });
}

The good news is, the program still compiles with these changes. Unfortunately, it still crashes. The final ChatGPT .cpp file is here.

Logical bugs

Putting aside the buffer management bug which is causing the server to crash, there are other problems as well:

in the example, I used single quotes ', not double-quotes ", which is not valid JSON; in my own implementation in the previous posts, I avoided including a large library by implementing (with the help of ChatGPT) a ~10 line parser function for flat string dictionaries
caching and delivery is buggy: in handle_command(), it calls cache_message() irrespective of delivery semantics. So if a message has delivery=one, it can get delivered to a single recipient now, but also later from the cache
in replay_messages() it does not implement delivery semantics at all
only cached messages receive an id field, messages sent as they are received do not contain this field; this also means clients that receive non-cached messages cannot maintain their last_seen counters
the last_seen field is stored as an std::string, and is then compared to the id lexicographically (instead of as ints)
there are probably numerous other bugs, including memory management issues that would cause further crashes...

Conclusion

Overall I think it's quite impressive that ChatGPT-4 can write this approximate solution. In C++ it's much harder to get a correct program (due to memory management, types, compiling), and this shows. But other than language issues, as described above, ChatGPT also produced numerous logical bugs. Overall, the code is like what a student of C++ would produce with lots of help from Stackoverflow and Google: along the right lines, but with lots of problems. Since this was a just a ~100 line toy problem, and it couldn't handle it, C++ programmers do not (yet) need to fear for their jobs!

Writing a simple C++20 async message queue server - Part II

2023-04-08T00:00:00+02:00

Introduction

In 2023, I want to get back into playful systems programming. My plan is to write a simple message queue (MQ), and for my own education write the same toy implementation in modern Python, modern C++ and Rust. To make things interesting, I will use async / event-driven programming and avoid multi-threading.

In the first 3 posts, I have written Python implementations, and played around with ChatGPT-3.5 to see if it can write code of this (not too great) complexity:

In the previous post, I wrote the simple version of the message queue in C++:

Writing a simple C++20 async message queue server - Part I

In this article, similar to Part II in the Python series, I will add features:

cache messages
specificy delivery semantics (all or one)
specify last_seen in subscribe message to pick up where the client left off

Additionally, even short C++ code like this is better factored into multiple files. Unlike Python code, where types can be omitted, in C++ types must be declared. This makes code mixing different levels of abstractions noisy and hard to read. In this case, networking code such as tcp::socket, co-routine and asio:: usage is best separated. To achieve this, the code is separated into 3 files: Utils.hpp for utility functiouns, MessageQueue.hpp has the MQ logic, and the main program in Main.cpp handles networking.

A good (but old) book which also covers the topic of physical design (breaking programs into files) is Large-Scale C++ Software Design by John Lakos, recommended reading.

The code shown in this article is up on Github:

Utils

I will omit the function bodies to save space, as it's very similar/identical to the functions in the previous post, with a few additions:

#ifndef __UTILS_HPP__
#define __UTILS_HPP__

#include <map>
...
using namespace std;

void trim(string& str)
{
    ...
}

bool startswith(string_view s, string_view prefix)
{
    ...
}

vector<string> split(const string& s, char delim)
{
    ...
}

typedef map<string, string> Dict;
Dict parse_dict(const string& dict_str)
{
    ...
}

string serialize_dict(const Dict& d)
{
    ...
}

#endif

The most interesting about these functions is that they are so simple, ChatGPT-4 can write flawless implementations using modern C++20!

MessageQueue

The MessageQueue implementation starts by declaring a Client class to contain the specifics of our TCP/IP networking. The main MessageQueue class will just use this Client, ignoring the specifics of I/O:

class Client
{
    tcp::socket& socket;

public:
    Client(tcp::socket& s) : socket(s) {}

    void Write(const Dict& msg)
    {
        auto s = serialize_dict(msg) + "\r\n";
        asio::write(socket, asio::buffer(s));
    }
};

One thing to note here is that unlike in the previous post, here we're using synchronous asio::write(). This is because almost always, even this asio::write() will never block since the operation just hands over the buffer to the OS, which will perform the write at a later time. The bonus is, this way we keep all co_awaits out of the rest of the MessageQueue class, so it doesn't know that this is an asynchronous program with co-routines! Now the main MessageQueue class:

class MessageQueue
{
    const unsigned short cache_length = 100;
    unordered_map<string, unordered_set<Client*>> subscribers;
    unordered_map<Client*, unordered_set<string>> subscriptions;
    unordered_map<string, deque<Dict>> cache;
    unordered_map<string, long> max_index;

public:
    void OnConnect(Client& client)
    {
    }

    void OnDisconnect(Client& client)
    {
        for (const string& topic : subscriptions[&client])
            subscribers[topic].erase(&client);
    }

    void OnMessage(Client& client, string& line)
    {
        Dict msg = parse_dict(line);

        if (msg["command"] == "subscribe")
            OnSubscribe(client, msg);
        else
            OnSend(client, msg);
    }

    void OnSubscribe(Client& client, Dict& msg)
    {
        long last_seen;
        subscribers[msg["topic"]].insert(&client);
        subscriptions[&client].insert(msg["topic"]);
        if (msg["last_seen"] == "")
            last_seen = -1;
        else
            last_seen = stoi(msg["last_seen"]);
        CachePlayback(client, msg, last_seen);
    }

    void OnSend(Client& client, Dict& msg)
    {
        AddIndex(msg);
        if (msg["delivery"] == "")
            msg["delivery"] = "all";
        for (auto& client : subscribers[msg["topic"]])
        {
            client->Write(msg);
            if (msg["delivery"] == "one")
                break;
        }
        if ((subscribers[msg["topic"]].size() == 0) || (msg["delivery"] == "all"))
            CachePush(msg);
    }

private:
    void AddIndex(Dict& msg)
    {
        msg["index"] = to_string(max_index[msg["topic"]]);
        max_index[msg["topic"]] += 1;
    }

    void CachePush(Dict& msg)
    {
        cache[msg["topic"]].push_back(msg);
        if (cache[msg["topic"]].size() > cache_length)
            cache[msg["topic"]].pop_front();
    }

    void CachePlayback(Client& client, Dict& msg, long last_seen)
    {
        bool reCache = false;
        for (auto& cached_msg : cache[msg["topic"]])
        {
            if (stoi(cached_msg["index"]) > last_seen)
            {
                client.Write(cached_msg);
                if (cached_msg["delivery"] == "one")
                    reCache = true;
            }
        }
        if (reCache)
        {
            deque<Dict> newCache;
            for (auto& cached_msg : cache[msg["topic"]])
                if ((stoi(cached_msg["index"]) <= last_seen) || (cached_msg["delivery"] == "all"))
                    newCache.push_back(cached_msg);
            cache[msg["topic"]] = newCache;
        }
    }
};

The code is very similar to the previous Python code, except it's nicely factored into functions.

Main

The code in the main file is very similar to the previous version, so I will just show the function which utilizes the new MessageQueue class. In this setup, the only thing the main code knows is that each line is a message (but no further formatting is assumed here), and that it needs to call MessageQueue::OnConnect() (which is actually unused), MessageQueue::OnMessage() and MessageQueue::OnDisconnect(). As previously, the code is not handling all errors to keep things clean and readable.

static MessageQueue mq;

awaitable<void> session(tcp::socket socket)
{
    try
    {
        Dict msg; 
        string data;
        Client client(socket);

        cout << "Client connected: " << socket.remote_endpoint() << '\n';
        mq.OnConnect(client);
        while (data != "quit")
        {
            data.clear();
            co_await asio::async_read_until(socket, asio::dynamic_buffer(data), '\n', use_awaitable); // line-by-line reading
            trim(data);
            auto lines = split(data, '\n');
            for (auto& line : lines)
            {
                trim(line);
                if (line != "") {
                    cout << "Received: " << line << endl;
                    mq.OnMessage(client, line);
                }
            }
        }
        mq.OnDisconnect(client);
        cout << "Client disconnected: " << socket.remote_endpoint() << '\n';
    }
    catch (const std::exception& e)
    {
        cout << "Exception in session: " << e.what() << '\n';
    }
}

awaitable<void> listener(io_context& ctx, unsigned short port)
{
    ...
}

int main(int argc, char* argv[])
{
    ...
}

To compile this code, you will need to get a ASIO library. You can test it by running it in a terminal, like:

$ ./AsyncEchoServer 7777

Then, in a second terminal, connect to it:

$ telnet
> open localhost 7777
{'command':'send', 'topic':'foo', 'msg':'0', 'delivery':'one'}
{'command':'send', 'topic':'foo', 'msg':'1', 'delivery':'all'}
{'command':'send', 'topic':'foo', 'msg':'2', 'delivery':'all'}
{'command':'send', 'topic':'foo', 'msg':'3', 'delivery':'one'}
{'command':'send', 'topic':'foo', 'msg':'4', 'delivery':'one'}
{'command':'send', 'topic':'foo', 'msg':'5', 'delivery':'all'}
{'command':'send', 'topic':'foo', 'msg':'6', 'delivery':'all'}
{'command':'send', 'topic':'foo', 'msg':'7', 'delivery':'one'}
{'command':'send', 'topic':'foo', 'msg':'8', 'delivery':'all'}

Then, in a third terminal:

$ telnet
> open localhost 7777
{'command':'subscribe', 'topic':'foo'}
quit
> open localhost 7777
{'command':'subscribe', 'topic':'foo', 'last_seen': '5'}
quit

The message(s) will appear in the second terminal!

Conclusion

With C++20, the code ended up being significantly more complicated than Python. In terms of LOC, it's about 5x, due to having to include utility funcitons, #includes, namespaces, class declarations, and because the code is factored into files and classes, a must in C++. The actual logic is identical and equally readable. In the third part of this series, I will write the message queue in Rust.

Writing a simple C++20 async message queue server - Part I

2023-04-02T00:00:00+02:00

Introduction

In the previous 3 posts, I have written Python implementations, and played around with ChatGPT-3.5 to see if it can write code of this (not too great) complexity:

In this article, I will write code similar to the first article: first I will write a simple TCP echo server, and then move on to a MQ server — but this time in modern C++. Similarly to the Python code, I will use co-routines, and event-driven async programming. With modern C++20, apart from helper functions, the main code looks very similar to the Python version; the implementations match almost line by line.

The code shown in this article is up on Github:

Simple async echo server

Let's start with writing the simplest possible skeleton server, which listens on a port, accepts TCP connections, reads incoming bytes, and echoes back whatever was sent:

#include <iostream>
#include <string>
#include <asio.hpp>
#include <asio/io_context.hpp>

using asio::awaitable;
using asio::co_spawn;
using asio::detached;
using asio::ip::tcp;
using asio::use_awaitable;
using asio::io_context;
using namespace std;

awaitable<void> session(tcp::socket socket)
{
    string data;
    cout << "Client connected: " << socket.remote_endpoint() << '\n';
    while (data != "quit")
    {
        data.clear();
        co_await asio::async_read_until(socket, asio::dynamic_buffer(data), '\n', use_awaitable); // line-by-line reading
        if (data != "")
            co_await asio::async_write(socket, asio::buffer(data), use_awaitable);
    }
    cout << "Client disconnected: " << socket.remote_endpoint() << '\n';
}

awaitable<void> listener(io_context& ctx, unsigned short port)
{
    tcp::acceptor acceptor(ctx, { tcp::v4(), port });
    cout << "Server listening on port " << port << "..." << endl;
    while (true)
    {
        tcp::socket socket = co_await acceptor.async_accept(use_awaitable);
        co_spawn(ctx, session(move(socket)), detached);
    }
}

int main(int argc, char* argv[])
{
    if (argc < 2)
    {
        cerr << "Usage: " << argv[0] << " <port>" << endl;
        return 1;
    }

    io_context ctx;
    string arg = argv[1];
    size_t pos;
    unsigned short port = stoi(arg, &pos);

    asio::signal_set signals(ctx, SIGINT, SIGTERM);
    signals.async_wait([&](auto, auto) { ctx.stop(); });
    auto listen = listener(ctx, port);
    co_spawn(ctx, move(listen), asio::detached);
    ctx.run();
}

The code is straightforward, assuming one has read the previous Python code. It uses the ASIO library, which is built on top of the C++20 language feature of co-routines, and adds asynchronous I/O functionality. It starts the co-routine loop by co_spawning the listener() co-routine, which in turn listens on the given port, and spawns a session() co-routine for each new connected client. For each client, we use async_read_until() to read a new line of text, and then write it back with async_write(). The client can disconnect by typing quit. Note that I skipped proper error handling here (which could be achieved by using try/catch blocks) to make the code more readable.

Note that co-routines are a language feature in C++20; co_await is a language keyword. co_spawn() is not a language feature, it's a function within ASIO that internally constructs awaitable objects and then uses co_await to trigger the co-routines.

To compile this code, you will need to get a ASIO library. You can test it by running it in a terminal, like:

$ ./TCPEchoServer 7777

Then, in another terminal, connect to it:

$ telnet
> open localhost 7777
foo
foo
bar
bar

Simple async message queue in C++20

The above skeleton can be extended to a simple bi-directional message queue. Bi-directional means that clients can both subscribe to topics and receive messages sent to that topic, as well as send messages to arbitrary topics. The server accepts two kinds of structured messages, subscribe and send.

{'command':'subscribe', 'topic':'foo'              }
{'command':'send',      'topic':'foo', 'msg':'blah'}

The semantics are simple: after subscribe, that client receives messages sent to that topic by other clients, sent after the client connected. In this simple implementation, all subscribers get the message.

The implementation is barely longer than the echo server, since the problem is very similar: we just have to maintain a list of tcp:sockets for each topic, and when we receive a message for a topic, we go through those sockets and send the message. Unlike the Python implementation, which came in just a bit longer than the echo server, the C++20 version is a fair amount longer, because we need to write some helper functions to keep the main code clean. For example, in Python parsing a dictionary like the ones we use was "free" with ast.literal_eval() which is a built-in, in C++ we need to write this function ourselves.

Let's see the boilerplate and the helper functions first:

#include <map>
#include <regex>
#include <iostream>
#include <sstream>
#include <vector>
#include <string>
#include <unordered_map>
#include <unordered_set>
#include <algorithm>
#include <cctype>
#include <asio.hpp>
#include <asio/io_context.hpp>
#include <signal.h>

using asio::awaitable;
using asio::co_spawn;
using asio::detached;
using asio::ip::tcp;
using asio::use_awaitable;
using asio::io_context;
using namespace std;

void trim(string& str)
{
    auto is_space = [](char c) { return std::isspace(static_cast<unsigned char>(c)); };
    str.erase(str.begin(), std::find_if(str.begin(), str.end(), std::not_fn(is_space)));
    str.erase(std::find_if(str.rbegin(), str.rend(), std::not_fn(is_space)).base(), str.end());
}

bool startswith(string_view s, string_view prefix)
{
    return s.find(prefix, 0) == 0;
}

vector<string> split(const string& s, char delim) {
    vector<string> result;
    stringstream ss(s);
    string item;

    while (getline(ss, item, delim))
        result.push_back(item);

    return result;
}

typedef map<string, string> Dict;
Dict parse_dict(const string& dict_str) {
    Dict result;
    regex dict_pattern(R"(\s*'([^']+)'\s*:\s*'([^']+)'\s*)");
    smatch matches;

    auto search_start = dict_str.begin();
    while (regex_search(search_start, dict_str.end(), matches, dict_pattern)) {
        result[matches[1].str()] = matches[2].str();
        search_start = matches[0].second;
    }

    return result;
}

We implement four helper functions, which will help us keep the main code clean, and similar to the Python code. void trim(string& str) removes whitespace from both ends of a string. bool startswith(string_view s, string_view prefix) returns true if s starts with prefix. vector<string> split(const string s, char delim) splits s into a vector of strings on delim. Dict parse_dict(const string& dict_str) returns a map of strings to string based on what's in dict_str, assuming it's flat and well-formed, Dict is typedef'd to map<string, string>.

Now the main code:

awaitable<void> session(tcp::socket socket)
{
    try
    {
        string line;
        static unordered_map<string, unordered_set< tcp::socket* >> subscribers;
        unordered_set<string> subscriptions;
        Dict mesg;
        cout << "Client connected: " << socket.remote_endpoint() << '\n';
        while (line != "quit")
        {
            line.clear();
            co_await asio::async_read_until(socket, asio::dynamic_buffer(line), '\n', use_awaitable); // line-by-line reading
            mesg = parse_dict(line);
            if (mesg["command"] == "subscribe")
            {
                subscribers[mesg["topic"]].insert(&socket);
                subscriptions.insert(mesg["topic"]);
            }
            else if (mesg["command"] == "send")
            {
                for (auto& subscriber : subscribers[mesg["topic"]])
                    co_await asio::async_write(*subscriber, asio::buffer(line), use_awaitable);
            }
        }
        for (const string& topic : subscriptions)
            subscribers[topic].erase(&socket);
        cout << "Client disconnected: " << socket.remote_endpoint() << '\n';
    }
    catch (const std::exception& e)
    {
        cout << "Exception in session: " << e.what() << '\n';
    }
}

awaitable<void> listener(io_context& ctx, unsigned short port)
{
    tcp::acceptor acceptor(ctx, { tcp::v4(), port });
    cout << "Server listening on port " << port << "..." << endl;
    while (true)
    {
        tcp::socket socket = co_await acceptor.async_accept(use_awaitable);
        co_spawn(ctx, session(move(socket)), detached);
    }
}

int main(int argc, char* argv[])
{
    if (argc < 2)
    {
        cerr << "Usage: " << argv[0] << " <port>" << endl;
        return 1;
    }

    io_context ctx;
    string arg = argv[1];
    size_t pos;
    unsigned short port = stoi(arg, &pos);

    asio::signal_set signals(ctx, SIGINT, SIGTERM);
    signals.async_wait([&](auto, auto) { ctx.stop(); });
    auto listen = listener(ctx, port);
    co_spawn(ctx, move(listen), asio::detached);
    ctx.run();
}

The only thing that changed compared to the echo server is the session() function, and the logic is identical to the Python version. First, the dictionary is parsed. If the client is subscribing, a pointer to the client's tcp::socket object is inserted into subscribers. If the client is sending a message, the appropraite tcp::socket pointers in subscribers are iterated and we write back the original message with async_write(). As before, I skipped proper error handling to keep the code more readable.

You can test it by running it in a terminal, like:

$ ./SimpleAsyncMessageQueue 7777

Then, in a second terminal, connect to it:

$ telnet
> open localhost 7777
{'command':'subscribe', 'topic':'foo'}

Then, in a third terminal:

$ telnet
> open localhost 7777
{'command':'send', 'topic':'foo', 'msg':'blah'}

The message will appear in the second terminal!

ChatGPT

Many years ago I was a full-time / professional C++ programmer. In the meantime, a lot has happened in the language, and I've become rusty, so I used a combination of Stackoverflow and ChatGPT-4 get back up to speed. I had ChatGPT generate an initial version of the TCP server using co-routines and ASIO. This was 95% right, but it didn't compile, so I fixed it by hand. From then on, I wrote the MQ implementation without AI help, but I wrote a version that accepted plain text commands (instead of the dictionary format, since I didn't want to write the parsing code). Then I had ChatGPT write the parse_dict() function, it produced this correct version. Then I asked it to review my complete code, and it came back with 7 suggestions, out of which 5-6 were good suggestions which I implemented. Mostly it suggested more idiomatic versions of the helper functions like trim() and startswith() that use STL features to make it shorter and faster. Overall ChatGPT-4 was a big help and I recommend experts (who can validate the responses) to use it.

Conclusion

In the next article, I will add some features to this simple skeleton.

Can ChatGPT write the Python async message queue codes from the previous posts?

2023-03-05T00:00:00+01:00

Introduction

In the previous articles I wrote a simple async message queue servers in Python:

In the first one, I wrote a very simple message queue using the asyncio library. In the second, I added a few features to it, such as delivery=one|all semantics, caching, message sequencing and last_seen specified by the subscriber. The whole thing remained very simple at less than 100 lines of code.

With the advent of ChatGPT, and it being able to write code, it's interesting to check whether the AI could write these simple codes. Personally I think anything ChatGPT can write is a linear combination of what's already out there on the public Internet, so it's not worth a blog post. Having said that, I do not claim the opposite, just because ChatGPT cannot reproduce something does not mean it's worthy or writing down.

I also tried using OpenAI's Codex, but it produced poorer results than ChatGPT, so I'm not showing that.

Note: the cover image is from DALL-E, the prompt was an ai writing python code.

Simple echo server

Prompt:

Write a simple TCP echo server in Python using the asyncio library. It should bind to localhost, but take the listening port as a command-line argument.

ChatGPT's code:

import asyncio
import argparse

async def handle_echo(reader, writer):
    while True:
        data = await reader.read(1024)
        if not data:
            break
        writer.write(data)
        await writer.drain()
    writer.close()

async def main():
    parser = argparse.ArgumentParser(description='TCP echo server')
    parser.add_argument('port', type=int, help='listening port')
    args = parser.parse_args()

    server = await asyncio.start_server(handle_echo, '127.0.0.1', args.port)
    async with server:
        await server.serve_forever()

asyncio.run(main())

This code runs and works! It's great start. It's showing what everybody knows by now, ChatGPT is super-useful for getting started with a topic.

Simple message queue

Prompt:

Write a simple TCP messsage queue server in Python using the asyncio library that accepts the listening port as a command-like argument. The messages are Python dicts followed by newlines. It should accept subscribe messages like {'command':'subscribe', 'topic':'foo'} and send messages like {'command':'send', 'topic':'foo', 'msg':'blah'}. There is no need to cache, just send messages to currentlly connected subscribers.

ChatGPT's code:

import asyncio
import json

class MessageQueueServer:
    def __init__(self):
        self.subscriptions = {}
        self.loop = asyncio.get_event_loop()

    async def handle_client(self, reader, writer):
        while True:
            data = await reader.readline()
            if not data:
                break
            message = json.loads(data.decode().strip())
            if message['command'] == 'subscribe':
                topic = message['topic']
                if topic not in self.subscriptions:
                    self.subscriptions[topic] = []
                self.subscriptions[topic].append(writer)
            elif message['command'] == 'send':
                topic = message['topic']
                msg = message['msg']
                if topic in self.subscriptions:
                    for subscriber in self.subscriptions[topic]:
                        subscriber.write(json.dumps({'msg': msg}).encode() + b'\n')
                        await subscriber.drain()

    async def start_server(self, port):
        server = await asyncio.start_server(self.handle_client, 'localhost', port)
        async with server:
            await server.serve_forever()

if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('port', type=int)
    args = parser.parse_args()
    server = MessageQueueServer()
    server.loop.run_until_complete(server.start_server(args.port))

It kindof works, but it commits some mistakes the way a novice programmer would: it assumes the structures are JSON, and uses json.loads(). This will not parse the messages as I specified it, as json requires double-quotes and not single-quotes. Otherwise, this is a working message queue with asyncio!

Delivery

Prompt:

Now, let's add a 'delivery' field to the send command, it can either be 'one' or 'all'. In the case of 'one', only one connected subscriber of that topic gets the message, in the case of 'all', all connected subscribers of that topic get the message.

ChatGPT's code:

import asyncio
import json

class MessageQueueServer:
    def __init__(self):
        self.subscriptions = {}
        self.loop = asyncio.get_event_loop()

    async def handle_client(self, reader, writer):
        while True:
            data = await reader.readline()
            if not data:
                break
            message = json.loads(data.decode().strip())
            if message['command'] == 'subscribe':
                topic = message['topic']
                if topic not in self.subscriptions:
                    self.subscriptions[topic] = []
                self.subscriptions[topic].append(writer)
            elif message['command'] == 'send':
                topic = message['topic']
                msg = message['msg']
                delivery = message.get('delivery', 'all')
                if topic in self.subscriptions:
                    subscribers = self.subscriptions[topic]
                    if delivery == 'one':
                        if subscribers:
                            subscriber = subscribers.pop(0)
                            subscriber.write(json.dumps({'msg': msg}).encode() + b'\n')
                            await subscriber.drain()
                            subscribers.append(subscriber)
                    else:
                        for subscriber in subscribers:
                            subscriber.write(json.dumps({'msg': msg}).encode() + b'\n')
                            await subscriber.drain()

    async def start_server(self, port):
        server = await asyncio.start_server(self.handle_client, 'localhost', port)
        async with server:
            await server.serve_forever()

if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('port', type=int)
    args = parser.parse_args()
    server = MessageQueueServer()
    server.loop.run_until_complete(server.start_server(args.port))

It works, but it's very ugly. On the one branch, it removes a subscriber from the array and then appends it back for some reason.

Caching

Prompt:

Now, let's add simple in-memory caching to it. Let's cache 100 messages per topic. When a new subscriber connects, we can send them whatever we have cached for that topic. Be careful to maintain the delivery semantics, ie. messages with delivery=one should only be delivered to 1 subscriber.

ChatGPT's code:

import asyncio
import json
from collections import deque

class MessageQueueServer:
    def __init__(self):
        self.subscriptions = {}
        self.caches = {}
        self.loop = asyncio.get_event_loop()

    async def handle_client(self, reader, writer):
        while True:
            data = await reader.readline()
            if not data:
                break
            message = json.loads(data.decode().strip())
            if message['command'] == 'subscribe':
                topic = message['topic']
                if topic not in self.subscriptions:
                    self.subscriptions[topic] = []
                if topic in self.caches:
                    for cached_msg in self.caches[topic]:
                        writer.write(json.dumps(cached_msg).encode() + b'\n')
                        await writer.drain()
                self.subscriptions[topic].append(writer)
            elif message['command'] == 'send':
                topic = message['topic']
                msg = message['msg']
                delivery = message.get('delivery', 'all')
                if topic not in self.caches:
                    self.caches[topic] = deque(maxlen=100)
                self.caches[topic].append({'msg': msg})
                if topic in self.subscriptions:
                    subscribers = self.subscriptions[topic]
                    if delivery == 'one':
                        if subscribers:
                            subscriber = subscribers.pop(0)
                            subscriber.write(json.dumps({'msg': msg}).encode() + b'\n')
                            await subscriber.drain()
                            subscribers.append(subscriber)
                    else:
                        for subscriber in subscribers:
                            subscriber.write(json.dumps({'msg': msg}).encode() + b'\n')
                            await subscriber.drain()

    async def start_server(self, port):
        server = await asyncio.start_server(self.handle_client, 'localhost', port)
        async with server:
            await server.serve_forever()

if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('port', type=int)
    args = parser.parse_args()
    server = MessageQueueServer()
    server.loop.run_until_complete(server.start_server(args.port))

ChatGPT is able to add the caching logic, but is not able to maintain the delivery=one semantics. Once a message is cached, it's sent to all subscribers who connect thereafter.

Prompt:

The way you wrote it, messages with delivery of one will be sent to multiple subscribers who connect after the message was cached. Fix it.

ChatGPT's code:

import asyncio
import json
from collections import deque

class MessageQueueServer:
    def __init__(self):
        self.subscriptions = {}
        self.caches = {}
        self.loop = asyncio.get_event_loop()

    async def handle_client(self, reader, writer):
        while True:
            data = await reader.readline()
            if not data:
                break
            message = json.loads(data.decode().strip())
            if message['command'] == 'subscribe':
                topic = message['topic']
                if topic not in self.subscriptions:
                    self.subscriptions[topic] = []
                if topic in self.caches:
                    for cached_msg in self.caches[topic]:
                        writer.write(json.dumps(cached_msg).encode() + b'\n')
                        await writer.drain()
                self.subscriptions[topic].append(writer)
            elif message['command'] == 'send':
                topic = message['topic']
                msg = message['msg']
                delivery = message.get('delivery', 'all')
                if topic not in self.caches:
                    self.caches[topic] = deque(maxlen=100)
                self.caches[topic].append({'msg': msg, 'delivery': delivery})
                if topic in self.subscriptions:
                    subscribers = self.subscriptions[topic]
                    if delivery == 'one':
                        for i, subscriber in enumerate(subscribers):
                            if i == 0:
                                subscriber.write(json.dumps({'msg': msg}).encode() + b'\n')
                                await subscriber.drain()
                            else:
                                self.caches[topic][-1]['delivery'] = 'all'
                    else:
                        for subscriber in subscribers:
                            subscriber.write(json.dumps({'msg': msg}).encode() + b'\n')
                            await subscriber.drain()

    async def start_server(self, port):
        server = await asyncio.start_server(self.handle_client, 'localhost', port)
        async with server:
            await server.serve_forever()

if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('port', type=int)
    args = parser.parse_args()
    server = MessageQueueServer()
    server.loop.run_until_complete(server.start_server(args.port))

It's not able to produce correct code, this problem statement is too complex for ChatGPT!

Conclusion

In this scenario, ChatGPT is like a very smart search engine. Before ChatGPT, to get a skeleton version of an asyncio server, you had to look at the documentation and/or Stackoverflow. Now, you still have to do that, but you can get a working skeleton from ChatGPT. As these examples show, ChatGPT won't get you much farther, but this is already impressive. And who knows how much better newer models will do that have 10x or 100x as many parameters and are trained on more code. Having said that, for the time being I suspect we programmers still don't have to fear for our jobs, and we still have to read the documentation.

Writing a simple Python async message queue server - Part II

2023-03-02T00:00:00+01:00

Introduction

In the previous article, I wrote a simple async message queue server in Python. Let's add features to it!

Delivery

Let's add a feature so the sender can specify whether messages get delivered to all subscribers or just one. So we extend the message with a delivery field, which can be all or one:

{'command':'send', 'topic':'foo', 'msg':'blah', delivery:'all'}
{'command':'send', 'topic':'foo', 'msg':'blah', delivery:'one'}

The implementation for this is simple. We just have to select a random writer in topics to get the message in case of one, and send it to all writers in the case of all, while watching for the empty list case:

import sys, asyncio, ast, random
from collections import defaultdict

topics = defaultdict(lambda: [])

async def handle_client(reader, writer):
    print('New client connected...')
    line = str()
    while line.strip() != 'quit':
        line = (await reader.readline()).decode('utf8')
        if line.strip() == '': continue
        print(f'Received: {line.strip()}')
        cmd = ast.literal_eval(line)
        if cmd['command'] == 'subscribe':
            topics[cmd['topic']].append(writer)
        elif cmd['command'] == 'send':
            if cmd['delivery'] == 'all':
                writers = topics[cmd['topic']]
            else: # if delivery == 'one':
                if len(topics[cmd['topic']]) == 0:
                    writers = []
                else:
                    which = random.randint(0, len(topics[cmd['topic']])-1)
                    writers = [topics[cmd['topic']][which]]
            for w in writers:
                w.write(line.encode('utf8'))
    writer.close()
    print('Client disconnected...')

async def run_server(host, port):
    server = await asyncio.start_server(handle_client, host, port)
    print(f'Listening on {host}:{port}...')
    async with server:
        await server.serve_forever()

asyncio.run(run_server(host='localhost', port=int(sys.argv[1])))

Caching

Now, let's add a feature where the server will cache 100 messages per topic, while honoring the delivery semantics, ie. messages with delivery of one will never be sent to more than one client.

The implementation is still simple: if an incoming message has all delivery, just push it into the topic's cache. If if has one delivery, only push it if there are no current subscribers. On the other hand, if a new subscriber connects, send them all cached messages for that topic, and then drop any messages that had one delivery:

import sys, asyncio, ast, random
from collections import defaultdict, deque

CACHE_LENGTH = 100

topics = defaultdict(lambda: [])
caches = defaultdict(lambda: deque(maxlen=CACHE_LENGTH))

def send_cached(writer, topic):
    for line in caches[topic]:
        writer.write(line.encode('utf8'))
    # construct new cache, removing elements which had delivery of one
    # since we just delivered those
    new_cache = deque(maxlen=CACHE_LENGTH)
    for line in caches[topic]:
        cmd = ast.literal_eval(line)
        if cmd['delivery'] == 'all':
            new_cache.append(line)
    caches[topic] = new_cache

async def handle_client(reader, writer):
    print('New client connected...')
    line = str()
    while line.strip() != 'quit':
        line = (await reader.readline()).decode('utf8')
        if line.strip() == '': continue
        print(f'Received: {line.strip()}')
        cmd = ast.literal_eval(line)
        if cmd['command'] == 'subscribe':
            topics[cmd['topic']].append(writer)
            send_cached(writer, cmd['topic']) # if there are any cached messages to send, send it
        elif cmd['command'] == 'send':
            if cmd['delivery'] == 'all':
                writers = topics[cmd['topic']]
                caches[cmd['topic']].append(line) # cache it
            else: # if delivery == 'one':
                if len(topics[cmd['topic']]) == 0:
                    writers = []
                    caches[cmd['topic']].append(line) # cache it
                else:
                    which = random.randint(0, len(topics[cmd['topic']])-1)
                    writers = [topics[cmd['topic']][which]]
                    # no need to cache it
            for w in writers:
                w.write(line.encode('utf8'))

    writer.close()
    print('Client disconnected...')

async def run_server(host, port):
    server = await asyncio.start_server(handle_client, host, port)
    print(f'Listening on {host}:{port}...')
    async with server:
        await server.serve_forever()

asyncio.run(run_server(host='localhost', port=int(sys.argv[1])))

Last seen semantics

Let's add support for clients (re)connecting and starting to read from a topic where they left off, by specifying a last_seen index in the subscribe message, like:

{'command':'subscribe', 'topic':'foo', 'last_seen': '123'}

To support this, we have to:

add an index to each message (starting at 0), which increases sequentially
the index is stored in the command itself, so it now makes more sense to cache the parsed cmd insteaf of the flat line
this index is added to the cmd structure and sent out to subscribers
when a new client connects and specifies last_seen (default -1), only send out messages which have a bigger index

import sys, asyncio, ast, random
from collections import defaultdict, deque

CACHE_LENGTH = 100

topics = defaultdict(lambda: [])
caches = defaultdict(lambda: deque(maxlen=CACHE_LENGTH))
indexs = defaultdict(lambda: 0)

def handle_subscribe(cmd, writer):
    topics[cmd['topic']].append(writer)
    last_seen = int(cmd['last_seen']) if 'last_seen' in cmd else -1
    send_cached(writer, cmd['topic'], last_seen) # if there are any cached messages to send, send it

def send_cached(writer, topic, last_seen):
    for cmd in caches[topic]:
        if cmd['index'] > last_seen:
            writer.write((str(cmd)+'\n').encode('utf8'))
    new_cache = deque(maxlen=CACHE_LENGTH)
    for cmd in caches[topic]:
        if cmd['index'] <= last_seen:
            new_cache.append(cmd)
        else:
            if cmd['delivery'] == 'all':
                new_cache.append(cmd)
    caches[topic] = new_cache

def handle_send(cmd):
    cmd['index'] = indexs[cmd['topic']]
    indexs[cmd['topic']] += 1
    if cmd['delivery'] == 'all':
        writers = topics[cmd['topic']]
        caches[cmd['topic']].append(cmd) # cache it
    else: # if delivery == 'one':
        if len(topics[cmd['topic']]) == 0:
            writers = []
            caches[cmd['topic']].append(cmd) # cache it
        else:
            which = random.randint(0, len(topics[cmd['topic']])-1)
            writers = [topics[cmd['topic']][which]]
            # no need to cache it
    for w in writers:
        w.write((str(cmd)+'\n').encode('utf8'))

async def handle_client(reader, writer):
    print('New client connected...')
    line = str()
    while line.strip() != 'quit':
        line = (await reader.readline()).decode('utf8')
        if line.strip() == '': continue
        print(f'Received: {line.strip()}')
        cmd = ast.literal_eval(line)
        if cmd['command'] == 'subscribe':
            handle_subscribe(cmd, writer)
        elif cmd['command'] == 'send':
            handle_send(cmd)
    writer.close()
    print('Client disconnected...')

async def run_server(host, port):
    server = await asyncio.start_server(handle_client, host, port)
    print(f'Listening on {host}:{port}...')
    async with server:
        await server.serve_forever()

asyncio.run(run_server(host='localhost', port=int(sys.argv[1])))

Handling client disconnects

Up until this point, the code did not handle client disconnects. So writers were added to lists, but never removed, which means if a subscriber disconnects, the next time that topic receives a message, there will be a stale writer in the topic list, and the Python runtime will throw an exception. Handling this easy:

when a client disconnects, an exception is thrown
we maintain a reverse lookup dictionary so we know which writer was added to which topic list

I will show the basic logic on the simplest message queue implementation:

import sys, asyncio, ast, random
from collections import defaultdict

topics = defaultdict(lambda: [])
topics_reverse = defaultdict(lambda: [])

async def handle_client(reader, writer):
    print('New client connected...')
    try:
        line = str()
        while line.strip() != 'quit':
            line = (await reader.readline()).decode('utf8')
            print(f'Received: {line.strip()}')
            cmd = ast.literal_eval(line)
            if cmd['command'] == 'subscribe':
                topics[cmd['topic']].append(writer)
                topics_reverse[writer].append(cmd['topic'])
            elif cmd['command'] == 'send':
                if cmd['delivery'] == 'all':
                    writers = topics[cmd['topic']]
                else: # if delivery == 'one':
                    if len(topics[cmd['topic']]) == 0:
                        writers = []
                    else:
                        which = random.randint(0, len(topics[cmd['topic']])-1)
                        writers = [topics[cmd['topic']][which]]
                for w in writers:
                    w.write(line.encode('utf8'))
    except Exception as e:
        print(e)
    if writer in topics_reverse:
        for topic in topics_reverse[writer]:
            topics[topic].remove(writer)
            print(f'Removing writer from topic {topic}')
        del topics_reverse[writer]
    writer.close()
    print('Client disconnected...')

async def run_server(host, port):
    server = await asyncio.start_server(handle_client, host, port)
    print(f'Listening on {host}:{port}...')
    async with server:
        await server.serve_forever()

asyncio.run(run_server(host='localhost', port=int(sys.argv[1])))

Conclusion

Of course this is not ready for production use, but it's quite surprising how much functionality fits into a 67 line, async message queue implementation which is bi-directional, supports caching and indexing.

Writing a simple Python async message queue server - Part I

2023-02-27T00:00:00+01:00

Introduction

In this first article, I will explore how to write a simple Python MQ server using the Python asyncio library. asyncio allows programmers to write code as if it would be multi-threaded, but the run-time is actually using co-routines and switching between them at special await points. Also, the library has special data structures that allow synchronization between co-routines witout having to use synchronization primitives such as locks.

Simple async echo server

Let's start with writing the simplest possible skeleton server, which listens on a port, accepts TCP connections, reads incoming bytes, and echoes back whatever was sent:

import sys, asyncio

async def handle_client(reader, writer):
    print('New client connected...')
    line = str()
    while line.strip() != 'quit':
        line = (await reader.readline()).decode('utf8')
        if line.strip() == '': continue
        print(f'Received: {line.strip()}')
        writer.write(line.encode('utf8'))
    writer.close()
    print('Client disconnected...')

async def run_server(host, port):
    server = await asyncio.start_server(handle_client, host, port)
    print(f'Listening on {host}:{port}...')
    async with server:
        await server.serve_forever()

asyncio.run(run_server(host='localhost', port=int(sys.argv[1])))

You can test it by running it in a terminal, like:

$ python async_echo.py 7777

Then, in another terminal, connect to it:

$ telnet
> open localhost 7777
foo
foo
bar
bar

asyncio.run() is the top-level entry-point in an async Python program, it's like main(). We pass in our own run_server() function, which kicks of the main event loop. This then uses asyncio.start_server(), which sets up a TCP server listening on the specified port, and will fire off a new instance of handle_client() on every new incoming connection. We can pretend each handle_client() is a new thread, but actually the run-time is single-threaded, and uses co-routines. server.serve_forever() continues the main event loop, this is where the handle_client() calls will come from.

Interacting with the client connection is simple. handle_client() receives a reader and a writer, and in these simple examples we use reader.readline() and writer.write() to receive and send data.

What about the funny async and await keywords? Essentially what happens in this example is that multiple copies of handle_client() will be running, but not at the same time. The Python runtime will potentially switch between these co-routines whenever one of the co-routines gets to an await statement. As an example, assume a client connects, and the handle_client() starts running. It will get to the line = (await reader.readline()).decode('utf8') line, which has await. At this point, this co-routine is waiting for input, but it may take any time for the data to arrive. Meanwhile, another client may connect, in this case this waiting co-routine is paused, and the main co-routine run_server() runs again to handle the new connection, and to spawn another handle_client(). Now we have 1 server listening/awaiting and 2 clients reading/awaiting, whichever event happens first, that co-routine will resume next. So, async tells Python that the function is a co-routine (anything with await inside is a co-routine), and await tells Python that the function being called is a (blocking) co-routine.

Simple async message queue in Python

The above skeleton can be extended to a simple uni-directional message queue. It accepts two kinds of structured messages, subscribe and send.

{'command':'subscribe', 'topic':'foo'              }
{'command':'send',      'topic':'foo', 'msg':'blah'}

The implementation is barely longer than the echo server, since the problem is very similar: we just have to maintain a list of writers for each topic, and when we receive a message for a topic, we go through those writers and send the message.

import sys, asyncio, ast
from collections import defaultdict

topics = defaultdict(lambda: [])

async def handle_client(reader, writer):
    print('New client connected...')
    line = str()
    while line.strip() != 'quit':
        line = (await reader.readline()).decode('utf8')
        if line.strip() == '': continue
        print(f'Received: {line.strip()}')
        cmd = ast.literal_eval(line)
        if cmd['command'] == 'subscribe':
            topics[cmd['topic']].append(writer)
        elif cmd['command'] == 'send':
            writers = topics[cmd['topic']]
            for w in writers:
                w.write(line.encode('utf8'))
    writer.close()
    print('Client disconnected...')

async def run_server(host, port):
    server = await asyncio.start_server(handle_client, host, port)
    print(f'Listening on {host}:{port}...')
    async with server:
        await server.serve_forever()

asyncio.run(run_server(host='localhost', port=int(sys.argv[1])))

You can test it by running it in a terminal, like:

$ python async_unidir.py 7777

Then, in a second terminal, connect to it:

$ telnet
> open localhost 7777
{'command':'subscribe', 'topic':'foo'}

Then, in a third terminal:

$ telnet
> open localhost 7777
{'command':'send', 'topic':'foo', 'msg':'blah'}

The message will appear in the second terminal!

Note that in traditional multi-threaded programming, the queue objects would have to be protected by locks to avoid corrupting the data structure. With Python async, this is not an issue, there is no way to corrupt the internal state of the queues (or the topics hashmap), since the program is actually single-threaded, and will only switch co-routines at await points.

Conclusion

In the next article, I will add some features to this simple skeleton.

Lifelong purchases

2023-02-05T00:00:00+01:00

Introduction

Recently there was an Ask HN on Hacker News entitled Ask HN: What are some of your buy-it-for-life purchases? I really enjoyed the question and gave my own answer, which I'm reproducing here.

My lifelong purchases

I don't really know whether these will be lifelong items, but their quality suggests they could be.

Furniture. Solid wood furniture like Crate & Barrel. Our current apartment has a lot of C&B wood furniture since my company owns the store and I get 40% off.

Knives. High quality knives like Winkler, Böker and Victorinox. Also, local knifemakers like Toth Laszlo in Hungary make beautiful knives that will last forever.

Pens. High-quality metal pens and pencils, like the ones Rotring makes.

Chess sets. A high-quality ebony staunton chess set, like the ones sold on Chessbazaar.

Notebooks. High quality leather stuff like notebook sleeves, for example from Ryoko.

Watches. I'm not an expert, but I don't see a reason for a watch (expensive or affordable) to break, if you wear it reasonably. I have cheap Casio and Seiko watches from 20-30 years ago that are just fine. Good not-so-expensive (compared to Rolex) brands are Casio, Seiko, Tissot, Victorinox, Porsche Design. I also enjoy buying fashion watches like Skagen, About Vintage and Sternglas; watch experts look down on these, but I think they're beautiful.

Bikes. Almost any modern bike will last you a very long time. Carbon ones possibly not for a lifetime, steel ones will last for a lifetime. Get a mechanical full Ultegra group set, they are awesome. (The electric is also awesome, but not sure about for-life.) If you want an all-arounder, get a cyclocross bike, you can use it like a road bike and also go off-road a bit. These days I prefer Bianchi, Pinarello and Cervelo in terms of shape and color, but other brands like Trek, BMC and Cube are also great. I currently own a Bianchi Oltre XR4 road bike and a Cervelo P5 timetrial bike.

Books. Most print books will last my lifetime in my estimation. Specifically, I predict that physical books in my posession will last me longer than DRM'd books. I expect that the platforms (eg. Kindle) and my personal storage discipline (of PDFs) will not last as long as physical books on my shelf.

Die-cast toys. Die-cast toys like the ones made by Bburago are virtually indestructible.

Speakers. High-quality speakers like Klipsch RF-3s. Mine are back in Hungary, so I'm using a stock photo.

Crypto V: On NFTs

2023-01-27T00:00:00+01:00

Introduction

This is the fifth post in a series on crypto. Previous articles:

Here I will talk about NFTs. First, some definitions from the Wikipedia entry for Non-fungible tokens:

A non-fungible token (NFT) is a unique digital identifier that cannot be copied, substituted, or subdivided, that is recorded in a blockchain, and that is used to certify authenticity and ownership. The ownership of an NFT is recorded in the blockchain and can be transferred by the owner, allowing NFTs to be sold and traded. NFTs can be created by anybody, and require few or no coding skills to create. NFTs typically contain references to digital files such as photos, videos, and audio. Because NFTs are uniquely identifiable assets, they differ from cryptocurrencies, which are fungible.

Proponents of NFTs claim that NFTs provide a public certificate of authenticity or proof of ownership, but the legal rights conveyed by an NFT can be uncertain. The ownership of an NFT as defined by the blockchain has no inherent legal meaning and does not necessarily grant copyright, intellectual property rights, or other legal rights over its associated digital file. An NFT does not restrict the sharing or copying of its associated digital file and does not prevent the creation of NFTs that reference identical files.

Virtual goods on blockchains

Fundamentally there is no problem with paying money for virtual goods. For example, software is also a virtual good, although it's a useful virtual good, because it can be run, and presumably some non-virtual value can be extracted (for example, a lawyer uses Microsoft Word to write court documents, get paid, and buys food). Movies and games have value, since they provide entertainment to gamers, beyond their ownership.

Virtual goods were common inside games and gamers paid money for them long before NFTs. A typical example is a skin for your character in a game. An in-game skin can still be argued to be provide non-virtual value if it provides entertainment for the gamer purchasing it.

However, for a digital good such as an in-game skin to be useful, it has to be tied to an online game that is controlled by a publisher; the publisher has to write and support the software that displays the skins. Games themselves are centralized online serviced, so it doesn't make sense for in-game content to be on a blockchain — and in fact, most of it is not on a blockchain.

The types of virtual goods that are traded on a blockchain, such as bored ape images, are useless. There is no value apart from the image itself, which can be viewed without owning the associated NFT anyway.

Legal problems

A lot of the content in this section is from LegalEagle's excellent video about NFTs. I highly recommend watching it if you're interested in NFTs.

One of the problems is around the concept of privity: a legal term for when two parties have some sort of linking legal relationship. The problem with NFTs, it's not clear what the contrast between the initial minter and the Nth owner (or even between the Nth owner and the N+1th owner). For example, if you buy a Bored Ape Yacht Club (BAYC) NFT on the Ethereum blockchain, what is the legal contract between you and Yuga Labs, the minter of the NFT? Do any terms and conditions travel with the NFT, or which do? What if the image pointed to in the NFT disappears from the Internet? What if the Ethereum blockchain (not run by BAYC) disappears? What is the blockchain forks and somebody else owns the token on the other chain?

A good counter-example is NBA Topshots, a walled-garden NFT platform run by NBA. Here you can buy "collectible" moments from NBA games. Here, to buy anything you have to have an account, and you constantly have a legal contract with the NBA. However, in this walled-garden scenario, there is no need for a blockchain or the concept of an NFT! The whole thing could work just as well as a plain old web-app — in reality, that is what this is, with a useless blockchain in the backend.

It's good to remember that NFTs (and any language associated with them, whether in terms & conditions or elsewhere) do not supplant copyright law. If you buy an NFT, you may or may not get the copyright, for example NBA Topshots does not give a copyright to the video, only a (very limited) license — you cannot do anything you want with the video, in fact there is very little you can do with it (NBA changed language from "owning" to "scoring" or "collecting" an NFT). In the case of the apes, BAYC holds the IP centrally and only gives license, which is actually held off-chain. In general, with digital art, if you don't have the copyright, you effectively have nothing [that is protectable under the law]. What is scarce about digital art is the copyright, but this is usually not part of the NFT — which defeats the whole point of NFTs!

In general, NFTs do not change or supplant established law, and if NFTs run into established law, the law takes precedence. A good example is when Qentin Tarantino created NFTs relating to his movies, and Miramax, who owns most of the rights, sued him in court. It doesn't matter what's on the blockchain, a court (or settlement) decision takes precedence. Similar to this, sometimes the idea of tracking and trading real estate ownership on the blockchain comes up, but there is already an established title system, which takes precedence. If the centralized records show that X is the owner, then X is the owner.

It's also not clear how adownstream purchaser can enforce terms against the original issuer or previous owners. Imagine a band creating an NFT that grants meet & greet privileges with the band. Do all owners get this privilege? If the NFT is traded once a day, the band has to meet a new NFT owner every day? Every hour? Who's responsible if the blockchain breaks or forks? If we need intermediares like OpenSea to clarify terms, give terms & conditions, etc, then it's a walled-garden anyway (it is), and why do we need a blockchain? We can already do these things off-chain.

In conclusion, web3 only makes things worse and more complicated.

Conclusion

To conclude, I will quote Devin from LegalEagle:

Programmers are bad at planning for contingencies, and don't understand contract law. Lawyers exist for a reason.

Crypto IV: On stock investing vs. crypto investing

2023-01-21T00:00:00+01:00

Introduction

In the last 3 posts I argued against the real-world usefulness of cryptocurrencies and pointed to its ponzi-like nature. One counter-argument of a sort could be that regular stock-investing is also ponzi-like; after all, if I buy a piece of Microsoft stock (MSFT) at \$1, and then sell it at \$2, somebody now has to enter the market at \$2 to buy it from me. So for that person to make money and sell at \$3, somebody else has to enter the market... and so on. So it sounds like from an investor perspective, there's no difference in the basic dynamic — or is there?

All articles in this series:

Profitability

I would say that for most investors, from their own perspective as an investor, there is no major difference first-order difference. You buy a stock or a crypto-something for \$1 and you would like to sell it for \$2 in the future; what happens to the underlying asset is secondary, as long as the price went up.

But I would argue that there is a big difference. When buying MSFT stock, you are buying fractional ownership in an economic machine that generates value — profits. With stock, you can always run the following thought experiment: if you buy up 100% of the stock, you now own and control the company. From now on, you can keep 100% of the annual profits. You can hire and fire people, change the company's product strategy, enter new markets or leave markets, and potentially make even more profits. The point is, with stock, you buy fractional control of a company that is (hopefully) generating profits. A good example of this, at the time of writing, is Elon Musk buying 100% of Twitter, taking over the company and taking it private.

With cryptocurrencies, this is not true. There is no underlying company or other asset. If you buy 100% of a cryptocurrency, there is nothing that you can control, there are no profits to keep.

Okay, but what about companies whose stock has a positive USD value, but they are not profitable? For example, Uber is its 14th year, is not profitable, but you can buy (and sell) a piece of Uber stock, for about \$30 at the time of writing. Some investors are buying this stock because they believe eventually Uber will be profitable; for these investors, the above thought experiment still works, it's "just" that after buying 100% of the company, they have to wait some time to start collecting profits. This sort of belief of course also plays an important role for already profitable companies; different investors have different ideas (fantasies) about future profitability of their portfolio companies. Other investors are of course speculative, irrespective of the profitability of the underlying asset.

Fractional ownership vs market cap

The previous thought experiment gets even more interesting when applied to cryptocurrencies. First, let's go back to the stock example. The price of a stock multiplied by the number of shares outstanding is the market cap(italization) of a company (note that not all stock may be available for purchase on open stock markets). In our thought experiment, if we buy 100% of the stock, does it change the market cap of a company? The market cap is related to the (future) profitability of the company, so fundamentally it is not affected by ownership changes.

However, I would argue this is not the case with a cryptocurrency like Bitcoin. Bitcoin also has a market cap calculated in a similar fashion as above. However, if you buy 100% of Bitcoin, I would argue the market cap would actually be \$0, because:

there is underlying asset, so there is no value coming from there
if a single person controls all of a currency, it's useless as a currency, so it can't be worth anything

As a result, a cryptocurrency like Bitcoin has a weird property: the more you buy of it, the less valuable it becomes!

Crypto III: On perceivd crypto advantages such as anonymity and irreversibility

2023-01-20T00:00:00+01:00

Introduction

This is the third in a series on cryptocurrencies. All articles in this series:

Here I will argue that anonymity, irreversibility and decentralization, while interesting technical features that implemented in a fascinating way using cryptographic primites, are not practical for real-world use.

A simple google search for properties of crypocurrencies shows many results where key properties are summarized. This Medium blog lists the following:

Decentralized (no central authority)
Irreversible and immutable (cannot be undone)
Anonymous
Limited supply and scarcity

Here's a longer list from another page:

Anonymity
No intermediary or supervisory bodies
Security
No centralization
Sending cryptocurrencies
Irreversable transactions
Fast development

Here I will talk about Anonymity, Irreversibility and Decentralization.

Anonymity

Anonymity refers to the fact that crypto transactions happen between wallets, wallets are like accounts, but don't care any additional meta information, such as name, phone number or address. A blockchain is a public database of transactions between wallets, so there are many websites that show it, for the Bitcoin blockchain, Ethereum, etc. Here is an example transactions from Bitcoin:

The point here is that the sender and receiver are identified by long unreadable strings. In the above, the sender is bc1qc09zgy8hd9ddy5mp2q4jq3gffgjvcqmn6a26lv. Clicking on the link reveals all wallet transactions (since the blockchain itself is public, since it's decentralized).

The problems can be categorized into 2 areas, technical and social.

Technically, one of the problems is that once a wallet's owner is revealed, then all past transactions involving that wallet can also be tied to the owner. So anonymity on the Blockchain is an all-or-nothing deal. If your identity gets revealed one in connection with 1 transaction, then suddenly your entire transaction history, including amounts and partner wallets, are also revealed. Suppose for example you tell a friend your wallet address bc1q9yn6zdkjjlh0z5y6sqpdvwq7pwkeh5r0ka28ad, that person sends 1 BTC; then 5 years later the FBI ask the person to reveal all identities of Bitcoin recipients, and she reveals that it's you; now the police will trivially see your complete transaction history just by visiting the above link, even if you never set foot in the US.

The main social problem is that anonymity is just not a sustainable feature. Democratic governments have legitimate reasons to know the financial (and non-financial) transactions of their citizens, for example to collect taxes or track criminals. Also, there is no technological solution to this; a nation state can simply outlaw anonymous transactions, and citizens must comply, otherwise men with guns will take them away. A common use-case for Blockchains that I hear is tracking the ownership of physical objects such as cars or land. In this case again, anonymity is a non-starter: the government needs to know who each wallet belongs to, so it can issue fines for speeding or identify the person to contact if a highway needs to be built on top of a piece of land.

Irreversibility

Irreversibility in the context of blockchains means that once a transaction is committed to the Blockchain, from a technical standpoint, it cannot be undone. This is due to the decentralized nature of the Blockchain; simply put, no single actor or node can go in and do it. Having said that, transactions are reversible in the sense that the receiver can choose to send back whatever she received to the original sender — but there is no way (technically) to force her.

The main problem here is social — sometimes you want to reverse a transaction, for example because a mistake was made and somebody typed extra zeros! It's worth pointing out that legally, if A accidentally sends too much money to B (eg. 1,000 BTC instead of 1,000 USD), B cannot just keep it.

A good example is when a crypto exchange accidentally sent $10m to an Australian woman, and then sued her to get the money back. I guess in these cases having a centralized government that can force transactions to be reversed is not so bad. It'd be great if we had software that supports this! Note that in this case, the only reason suing was even possible was that the exchange sent this money to one of its own users, whose identity was known. If this was a regular on-chain transaction to a wallet that is not managed by the exchange, undoing the mistake would be much harder or impossible.

Other, similar cases:

Decentralization

The positive arguments for decentralization are: no one person or entity controls the Blockchain, so it can't be frozen (the way a bank can freeze your assets) or tampered with. One example is that central banks, which control the supply of fiat money, can "print" new money, hence creating inflation. The argument is that, since there is no central bank with cryptocurrencies, there can be no inflation, at least by a single entity's decision to create new money.

There are multiple practical problems with this reasoning. As argued above, sometimes it's actually quite useful is an "administator" can reverse transactions, for example because the amount field contains a clerical mistake, or the destination address is non-existent and the cryptocurrencies are now lost.

Another problem is, as argued in the second article, inflation or not, the exchange rate of cryptocurrencies like BTC varies wildly, including in the inflationary direction (getting less valueble compared to USD). To a user of the currency, it doesn't really matter what the cause is, if their currency is worth 3x less real-life goods in a year's time.

Finally, as shown in the first article, for many cryptocurrencies [implemented as smart contracts on Ethereum, such as FTX and many stablecoins], it's not true that they are not inflationary. In these cases, when the initial coin is created, a large amount of reserves are also created, which are held by the creator, usually a company. This, in effect, creates control and leverage to cause inflation: if the reserves of a stablecoin start to be circulated without backing USDs, that causes the "stablecoins" to actually be worth less than the intended exchange rate (usually 1 USD), similar to inflation.

Conclusion

I have argued that anonymity, irreversibility and decentralization, while interesting technical features that implemented in a fascinating way using cryptographic primites, are not practical for real-world use.

Crypto II: On fiat currencies vs. crypto currencies

2023-01-09T00:00:00+01:00

Introduction

In the previous article, I discussed some aspects of cryptocurrencies through the example of FTX Tokens. Here I will compare cryptocurrencies to traditional fiat currencies.

All articles in this series:

Bartering, money and the Big Mac index

One of the core functions of money is to replace a barter system. In a barter system, if A makes apples and B makes bananas, they can agree on an apple/banana exchange rate, and trade between each other — this is called bartering. A third actor in the economy, C is producing cucumbers. Suppose it so happens that A wants B's bananas, B wants C's cucumbers and C wants A's apples (a directed cyclic graph) — in this scenario pairwise trades are not possible, the economy is blocked. It would be nice if they could use some intermediate asset to trade and unblock the economy. In the real world, we use fiat money such as USD for this: A, B and C set their prices in USD, and assuming everybody has sufficient reserves of USD, they can trade, even if they don't want each others goods that day.

For a currency to be acceptable to the participants of the economy, it has to be somewhat stable. Suppose D is a Data Scientist, and gets her salary in the common currency, let's say it's 100 USD. For USD to be a useful currency, in the sense that D accepts it as payment, she has to be able to buy basic goods, such as apples, bananas, cucumbers — and Big Macs at a relatively stable price.

The Big Mac Index is maintained by The Economist since 1968, and tracks the price of a Big Mac hamburger across time in various countries; the price is recorded twice a year. Let's see what the price of a Big Mac looks like in USD and BTC, for the past 5 years (10 datapoints):

The image shows that getting a (relatively) fixed salary in USD and then buying basic goods is a safe bet. The hi/lo ratio on the USD chart is 1.12, this is the inflationary effect, and was 12% in these 5 years. However, the hi/lo ratio on the BTC chart is 17.5, which shows that getting a salary in BTC, or exchanging goods in BTC is not a safe bet. Your BTC salary, or the BTC you receive for your goods could be worth significantly less in a couple of months or years.

BTC/USD

But wait, the value of BTC actually went up! In the case of USD, the hi/lo is 1.12, and the hi is at the tail end, meaning 1 USD buys less Big Macs than previously. With BTC, it's the other way around, 1 BTC buys 17.5x more Big Macs than 5 years ago! So does this invalidate the arguments above?

No, it does not. If you look at the Big Mac price in BTC, yes, overall it's "favorable" to BTC holders in this snapshot, but there were also times when it the price of a Big Mac in BTCs went up 3-4x in a year. The problem is, with a speculative asset, wide swings in the positive directions are always eventually matched by wide swings in the negative direction — and if this affects your livelihood, such as being able to buy basic goods with your salary, that is unacceptable.

This shows what everybody knows anyway — cryptocurrencies like BTC are not a useful currency for trading goods or getting salaries, because they are speculative.

Stablecoins

Okay, but BTC is just one type of cryptocurrency, there are 1000s of others. Maybe another one is better? Most cryptocurrencies suffer from the same problem as BTC, they are speculative, so they are not a useful currency for trading goods or getting salaries.

One exception to this could be stablecoins. However, stablecoins are not really useful in the above scenario, since stablecoins assume the presense of a fiat currency like USD to which it is pegged — so the actors in the economy might as well use USD. You could say that that is right, but cryptocurrencies have other desirable technical advantages, like anonymity, irreversibility, etc. Regarding these perceived advantages, let's see how stablecoins work out in practice (repeated from the previous post). This portion is repeated from the previous article.

A stablecoin is a token whose value is pegged to a traditional fiat currency. Most stablecoins are pegged to \$1 USD. The most popular stablecoins at the time of writing are:

The way stablecoins are supposed to work is roughly like this: a company releases a new stablecoin, let's say it's pegged to \$1. The company mints a large amount of the stablecoins for itself (reserve), and then sells some of them to users for \$1 (in circulation). So users give the company \$1 and get 1 stablecoin in exchange. The company guarantees that it holds on to the \$1 in its bank accounts, ie. it doesn't squander it away. Users can then transact with their stablecoins, enjoying the benefits of cryptocurrencies. If they ever need USD, for example because they want to buy bread, or they need to pay their US taxes, they can sell their stablecoins back to the minting company and get real dollars back. Sounds great and is a very useful concept in the crypto world.

So, in this case, is there new value created? If everybody plays their part (the price of the token remains \$1 and the company holds on to the reserves), users can safely pretend that 1 stablecoin is worth \$1. So in this sense, user value is created, but the stablecoin itself has no value detached from the reserves. In reality, the company has \$1 and a liability of \$1, which nets to 0. And the user has a stablecoin, which is the other side of that liability. So the net value in the system is still \$1 (not \$2). For this reason, the minting company cannot pretend that its reserve stablecoins are worth real dollars.

I came up with the following analogy to stablecoins: imagine there is a cult of finance people called Purists, living in a village called Puretown. Purists really like the feel and smell of freshly printed USDs. So they go to the bank, get 1 million \$1 bills, and bring it home to marvel at it. But they don't want to use it in circulation, because then it won't be as nice and fresh. So they photocopy each one, and agree that in their village they will use the photocopied bills to pay each other, while the Major holds on to the real USD bills in a safe at city hall. And, if they ever need a real dollar bill to transact outside their village (ecosystem), they can go back to the Major of Puretown, who holds the real USD reserves, give him a photocopied bill, and get back a real USD bill. We can agree that no value was created (they didn't magically make \$2M out of \$1M), we just have a bunch of cultists using monopoly money in their own ecosystem!

Also, it's not true that the reserves of the stablecoin (the non-circulating tokens that are still with the company) are now worth real dollars. Those are unbacked tokens, worth nothing.

The problem with stablecoins always comes from the Psychology circle in the Venn-diagram above. There are two (related) failure modes:

The minting company's executives cannot resist and will (secretly) give themselves some of the stablecoins, without putting in the equivalent backing USDs — they want to make money out of thin air. There are now unbacked stablecoins in the system, so in reality each coin is now worth less, say \$0.99 instead of \$1.00. But they attempt to keep this a secret, so everybody believes and/or pretends the coin is worth \$1 (the exchange rate charts still show \$1), and withdraws per that exchange rate when getting back real dollars. Eventually the company runs out of money. This could be caused because somebody suspects or finds the unbacked trades, tweets about it, and a bank run happens.
The minting company is supposed to hold on to the USD reserves, but the company's executives cannot resist and start making risky investment bets with the money, with the hope of turning the reserves into more money (and keeping the difference). Eventually one of the trades doesn't work out, and the reserves are now under water compared to the stablecoins in circulation. As in the previous failure case, a stablecoin is now worth less, say \$0.99 instead of \$1.00. But they attempt to keep this a secret, so everybody believes and/or pretends the coin is worth \$1... same story.

Note: there are also algorithimic stablecoins, which do not have any backing USDs at all. These are supported pure fantasy disguised as complex algorithmic mumbo-jumbo. The most famous of these was Terra, which got "de-pegged" this year:

So, in practice, stablecoins have also not proven to be a useful currency for trading goods or getting salaries.

So far I have argued that:

In practice, BTC is not a useful currency for trading goods or getting salaries, fiat currencies like the USD are.
Since stablecoins assume the presence of the USD, the actors in the economy might as well use USD.
In practice, stablecoins have proven to be hot beds for fraud.

Laws and taxes

The fundamental difference between fiat money and crypocurrencies is that fiat money's stability is backed by men with guns. Fiat money is issued by nation states; nation states have laws and collect taxes; if you don't obey the laws, or don't pay your taxes, men with guns will take your money and/or freedom away; there is no opt-out. Note: in some countries, you can opt-out by leaving the country; in others, such as the US, you have to send taxes home even if you're abroad.

This stabilizes nation state's fiat currencies in 3 ways:

Nation states levy taxes on its citizens and corporations, which must be paid in its fiat currency, thus creating demand for it.
Nation states issue laws that its own fiat currency must be accepted as payment in its territories.
Nation states issue laws that salaries must be paid in its own fiat currency.

Because of these factors, there is no scenario where all participants lose hope in the fiat currency, stop using it, causing the currency to be useless/worthless — at least not without a major regime change or (civil) war, which also changes the loyalty of men with guns. But it happens all the time with crypto (multiple times a year in the US and western countries), much more frequently then wars. The USD has been around since 1792, the GBP since 1707.

Ironically, many cryptocurrency users are trying to avoid laws and taxes, which are the exact things which would make their currency stable.

Real uses for BTC

Having said the above, there have been cases when cryptocurrencies have provided real (non-speculative) value for people. A good example is a mismanaged country where citizen's money in the bank (whether denominated in the nation's currency or USD) is frozen and eventually lost. In these cases, one of the options to fled one's money is to convert it to BTC, and move it abroad.

However, I would argue that this is a temporary use-case. Nation states (and their money) fail much less frequently than cryptocurrencies, and even the most long-lived cryptocurrency BTC had a 17.5x hi/lo ratio in the last 5 years — so it can act as a good technical vehicle for moving money out of a troubled country for short period of time, but it is not a stable store of value or a useful currency.

Greater fool theory

At the end of the day, both fiat and cryptocurrencies have no intrinsic value — they depend on the participants of the economy agreeing to use it for exchange, at certain exchange rates for goods.

Fiat money has attributed value because a government declares it legal tender and enforces its use through laws and taxes, enforced by men with guns; cryptocurrencies are purely belief based. And history shows the former is stronger than the latter.

Because cryptocurrencies are not useful as actual everyday money, their users (try to) use it as a speculative store of value, which is essentially a ponzi scheme or greater fool theory. The only reason it makes sense for me to buy BTC at $100 if I believe there will be a greater fool who will buy it at $101 from me. For him, it only makes sense to buy from me at $101 if she believes there will be an even greater fool at $102, and so on. Any profit made on cryptocurrencies is someone else's loss, since no economic value is created. Note that this is not the case for stocks, where the underlying asset, the company is creating value through its revenues and profits.

Conclusion

In the next and final article on cryptocurrencies, I will talk about the perceived technical advantages of crypto.

Crypto I: On cryptocurrencies, explained using FTX Tokens

2022-12-24T00:00:00+01:00

Introduction

Disclaimer: I have never bought, sold or owned crypto. I have no skin in the game, I just have a semi-informed opinion. If you're technical and into crypto, you may not find anything interesting here. If you find any bugs in my thinking, send comments to mtrencseni-at-gmail-dot-com.

I've been interested in the crypto space since the mid 2010s. It's a fascinating intersection of technology, economics and human psychology.

My stance towards crypto, at a very high level is:

The use of cryptographic primitives (old and new) to create a blockchain of transactions is creative and fascinating technology.
I am not aware of any useful, currently existing real-world applications of blockchain [that cannot be better accomplished with plain-vanilla centralized database].
There may be useful applications of crypto technologies that we have not thought of yet.
I believe all crypto currencies (including Bitcoin) are worth \$0.

A good starting point to understand cryptographic primitives is one of the old(er) O'reilly books on the topic, Mastering Bitcoin (free to read) by Andreas Antonopoulos. It has simple 2 line Python examples which directly demonstrate how cryptographic primitives are used in Blockchains. I read this many years ago and it was a great way to understand the technological building blocks of Blockchains and Bitcoin — highly recommended. The same author also has a second O'reilly book on the topic, Mastering Ethereum (also free to read).

All articles in this series:

Nature of money

I find that thinking and reading about the nature of money fascinating. In the last 10 years, it's impossible to think this topic and not ponder about the value of crypto tokens and NFTs. At the time of writing, we have just witnessed the collapse of the crypto exchange FTX, so to make it interesting, I will use the FTX exchange token FTX Token — abbreviated FTT — as an example. However, my argumentation is independent of the collapse of FTX, it equally applies to still-running tokens such Binance's Binance Coin as well.

FTT, like many other coins, is implemented as an ERC20 smart contract on top of Ethereum. The Github repo for FTT is here. It's essentially boilerplate with the actual "implementation" of the token in the 11 line file ftt/contracts/ftt.sol:

pragma solidity ^0.5.0;

import 'openzeppelin-solidity/contracts/token/ERC20/ERC20.sol';
import 'openzeppelin-solidity/contracts/token/ERC20/ERC20Detailed.sol';
import "openzeppelin-solidity/contracts/token/ERC20/ERC20Burnable.sol";

contract FTT is ERC20, ERC20Detailed, ERC20Burnable {
    constructor() ERC20Detailed('FTT', 'FTX Token', 18) public {
        _mint(msg.sender, 350_000_000 * 10 ** 18);
    }
}

The smart contract is declaring a new token FTX Token, abbreviated FTT, with 18 decimals of accuracy and 350M of tokens minted on creation. It's good to remember that no USD valuation is set of implied here. The 18 digit accuraccy means that you can send 0.000000000000000001 FTT between wallets (17 zeros after the decimal), but you cannot send 0.0000000000000000001 (18 zeros after the decimal). The 350_000_000 * 10 ** 18 bit is a bit misleading, is it minting 350M tokens or 350M x $10^{18}$ tokens? Internally, these tokens use integer arithmetic, so technically there 350M x $10^{18}$ indivisible units. By convention 1 FTT token is $10^{18}$ units, and this determines what fractions of FTTs can be transferred between wallets.

I think it's good to look at this and remember: no value is being created here.

So who gets the 350M FTTs? Unsurprisingly, the wallet who executes this initial smart contract on Ethereum, msg.sender in the code. We can look at the initial transaction from 2019-04-21 13:21:27 UTC on the Ethereum blockchain to see who it is:

https://etherscan.io/tx/0x048906570650e3c282e42832de167a0270c7890c77192362a0bff0f5bf415d3b

This transaction shows up technically as coming the NULL address 0x0000000000000000000000000000000000000000 and going to the destination address 0x772589e99bc9c54dd40acb7d73f88ccbc9d9cf47 (FTX: Deployer). This is completely fine, this is just the technical act of printing 350M monopoly crypto money and calling it FTT. Note that there is a fee (called gas) of using the Ethereum blockchain and network, 0.000000036 Ether for this transaction. This initial transaction contains the smart contract that we saw on Github, with the base classes ERC20, ERC20Detailed and ERC20Burnable unwrapped:

https://etherscan.io/address/0x50d1c9771902476076ecfc8b2a83ad6b9355a4c9#contracts

Market cap

There are a lot of crypto sites that track the historic and actual USD exchange rate or ticker price (and various other metrics, such as trading volume and "market cap") of crypto tokens, including etherscan.io itself. Here's a chart for the historic FTT/USD exchange rate:

It starts off around \$1, briefly reaches a high of \$80, for a long time it moves between \$20 and \$40, and eventually crashes back to \$1 when FTX collapsed. Most of these tracker sites also display a "market cap" for coins, which for FTT would be the 350M tokens we saw in the initial transaction, times this USD rate. So initially it was around \$350M, and briefly it touched \$80 x 350M = \$28B, that's 28 billion USD.

Note: I don't know quite understand why the exchange rate is not \$0 right now. There is still trading volume, so even though FTX has crashed, some people and/or bots are still buying and selling FTTs. Since FTT is a smart contract running on Ethereum, this is certainly technically possible irrespective of the fate of FTX the company.

Value creation (or lack therof)

So here's the million dollar question: how do you go from writing an 11 line smart contract, which actually only has 2 significant lines to a \$28B valuation?

In my opinion, the simple answer is: you don't. FTT was always worth \$0, the value of these coins is just a collective delusion.

Let's use FTT as a specific example. FTT was the so-called Exchange Token for FTX. Crypto exchanges are a place to exchange fiat money like USD into crypto tokens like Bitcoin or FTT, and buy/sell between pairs of crypto tokens. Let's use the example that is most often cited as a real-world use-case (value creation) of crypto: fast hassle-free sending of money between Alice in country A and Bob in country B. Let's assume that with regular banking this is slow and cumbersome, burdened by old computer systems and KYC regulations. Let's also assume that FTX is present in both countries A and B and has bank accounts in both countries. Both Alice and Bob set up an FTX account. Bob is a Data Scientist and works for Alice, so at the end of the month Alice wants to send Bob his salary. Note that Bob wants to buy food with his \$1,000 salary, which is not possible with crypto, so he needs to get his salary in his local currency. With an exchange like FTX, the solution is:

Alice transfers \$1,000 to FTX
Irrespective of the current FTT/USD exchange rate (let's say it's \$2) Alice buys FTT; so she ends up with 500 FTT tokens (so now Alice owns 500 of the 350M FTT tokens)
Alice sends the 500 FTTs to Bob (Bob's wallet)
Bob sells the FTTs on FTX (to FTX or another user) to end up with \$999 USD
Bob transfers the \$999 to his local account, in the process converting it to his local currency
Bob buys a burger and eats it

Let's look at the following questions: 1. Why did Alice use FTT? Why not another token like Bitcoin? 2. Why did Bob end up with \$999 instead of \$1,000? 3. What is the actual value FTX/FTT has provided here? 4. Doesn't the above successful chain prove that FTT was worth \$2?

Why did Alice use FTT? Why not another token like Bitcoin? Per my understanding this is where Exchange Tokens are advantageous. FTX charges a fee for using its systems, the same way a traditional bank does, but less — this is completely fine. FTX tells customers that if they buy/sell FTT instead of other tokens (like Bitcoin), it will charge them less fees. This is how FTX creates demand for its tokens. It's also worth noting that this is pretty rational from Alice and Bob's perspective. In this use-case they are not speculating on the value of FTT, it is simply an in-flight tool to transfer money. Depending on the speed of the transaction, the \$1,000 will only spend a few minutes of its lifetime as FTT, then it gets converted back to USD by Bob, so the risk of FTT crashing is essentially zero.

Why did Bob end up with \$999 instead of \$1,000? This is a minor point, so I'll be brief. It's because at various points in the transaction chain FTX will charges a fee (to Alice and Bob), the same way as a traditional financial institution would — this is completely fine. In this use-case, the promise of crypto over traditional banking is that less fees will be charged and money moves faster. In my example fees totaled $1, I just made this up.

What is the actual value FTX/FTT was providing here? In this use-case, in my opinion, FTX is providing the exact same value as Wise (formely Transferwise): allowing Alice and Bob to move \$1,000 between banks in different countries, by first moving it to FTX's account in country A, and then moving it out of FTX's account in country B, for a fee — the same way Wise would charge a fee. In my opinion, that's it — the FTT token is no more important than the rows in a traditional relational database that Wise might use in its operations.

Note that what I write here does not imply that this is a useful real-world application of crypto, since the same real-world user value (transferring money between A and B) can be accomplished without a token, as demonstrated by Wise millions of times a day.

Doesn't the above successful chain of transactions prove that FTT was worth \$2? This is a key question. In my opinion, the answer is no. Let's run the following thought experiment. Let's assume the above transaction chain is the first one that is run of FTX, shortly after the initial 350M coins were minted. Hopefully we can agree that the minting process itself, those 2 lines of code, did not create value out of thin air, so the 350M FTT tokens are worth \$0. So what's the next step? Initially, FTX has \$0 in their bank account, plus they hold 350M FTTs, and Alice has \$1,000 in her bank account. The combined system has a value of \$1,000. After step 2, FTX has Alice's \$1,000 in their FTX bank account, Alice has \$0 and 1,000 FTT tokens. Hopefully we can agree that no new value was created out of thin air in the first 2 steps, so by conservation of value, the combined system still has a value of \$1,000 — which means the FTTs have to be worth \$0. In my opinion, repeating these types of transactions millions of times does not change that.

Fake money

So why do exchanges like FTX and Binance introduce their own Exchange Tokens? This is where the psychology circle of the Venn diagram comes into play, in my opinion. Many people ("crypto investors") would not agree with me on the above, they would in fact say that many-many transactions of the above type show that FTT has non-zero value — let's say it's \$1. But then, to these people, FTX can sell their remaining (of the 350M) FTT tokens for real money! From, the perspective of FTX, if there are people who believe this (and there are), then they have just created money out of thin air: they ran a few lines of smart contract on Ethereum, set up an exchange, and over and above the valuation of FTX itself (the company), they're now sitting on a bunch of monopoly money (FTT tokens) that can be sold for real money.

Note that in the above example, most of what I described is not illegal. This is not why (or, not only) FTX founders and executives are headed to jail right now. What I described above is completely standard in the crypto world. You can replace FTX with Binance and FTT with Binance Coin, it's the exact same thing. This sort of scheme also exists outside of exchanges, where something that possibly has value (a company like FTX, a game like CryptoZoo, an influncer like Logan Paul marketing reach) is "connected" to a coin by marketing, and "crypto investors" for some reason "assign" or "connect" the potentially real-world value of the original entity (the company's success/revenue/profits/valuation, the game's success/revenue/profits, the influencer's reach) to the token, and hence believe that the tokens are now worth real USDs — even though there is no contractual connection between the two!

To reflect on an aspect of the failure of FTX, and part of the reason the founders and executives are headed to jail: Even better, FTX can set up a sister company, let's call it Alameda, and Alameda can start buying and selling FTTs at increasingly higher prices! This way the USD exchange rates can climb all the way to \$80! "Crypto investors" who bought at \$1 or \$10 feel like they're making a killing because their FTX account is showing their holdings are worth a lot more USDs than they paid for it! They can even sell their FTTs at this higher price, and withdraw the USDs to their bank account, and end up with real money gains! This works only up to a point though; since no value is created in the system, if people keep selling FTTs and withdrawing USDs at a higher average price then they bought, eventually the FTX bank account will run out of money and fail. This is called a Ponzi scheme.

The value of FTX (the company) vs the value of FTT (the token)

Going back to my salary transfer example, I said that this is essentially what Wise (formely Transferwise) does. Wise is not a crypto company, however it's publicly traded, and currently has a market cap of \$5B. So it stands to reason that if FTX (the company) was doing something similar, wasn't FTX (the company) also valuable? I would say, yes!

I would agree that FTX the crypto exchange, for enabling certain useful use-cases for users, and being able to collects fees for that (just like Wise), should have a non-zero company valuation — let's say this number is \$1B. But, this has nothing to do with the FTT token's value (or lack thereof). The valuation of FTX are captured by the shares of FTX, which are held by the founders and VC investors — not by FTT token holders! There was no written or unwritten contractual connection between the valuation of FTX and FTT tokens. The proof is in the smart contract code of FTT, which was shown above! It's possible that "crypto investors" made a connection in their head, but there never was a contractual obligation backing that up.

Didn't FTX guarantee a (minimum) value of FTT?

No. If you look at the cached copy of the Terms of Service, with reference to FTT, it only says:

FTT is the exchange token of the FTX ecosystem. FTT is not being offered in the United States or to U.S. persons.

...

Note that, among other things, FTX does not guarantee the right to trade. While FTX will take all reasonable methods to ensure that accounts do not receive any unwanted fills outside of our standard liquidation procedures, risk engine, and the terms of service, FTX does not as a matter of principle have any obligation to honor people's desires for trades to have occurred that didn't.

The FTT FAQ (cached) also does not mention any guarantees.

So, you can try to sell your FTTs (or any other crypto token) on the FTX exchange, but if there are no buyers, FTX did not guarantee that they will be the buyer of last resort at some minimum exchange rate, say \$1. That's the whole point: they wanted to keep the real money for themselves! This is true for almost all token minting I believe: in the end, when all is done, the "founders" want to be left with some of the (real) USDs.

What about stablecoins?

A stablecoin is a token whose value is pegged to a traditional fiat currency. Most stablecoins are pegged to \$1 USD. FTT was not a stablecoin. The most popular stablecoins at the time of writing are:

The way stablecoins are supposed to work is roughly like this: a company releases a new stablecoin, let's say it's pegged to \$1. Technically it's the same spiel as above with FTT: the company mints a large amount of the stablecoins for itself (reserve), and then sells some of them to users for \$1 (in circulation). So users give the company \$1 and get 1 stablecoin in exchange. The company guarantees that it holds on to the \$1 in its bank accounts, ie. it doesn't squander it away. Users can then transact with their stablecoins, enjoying the benefits of cryptocurrencies. If they ever need USD, for example because they want to buy bread, or they need to pay their US taxes, they can sell their stablecoins back to the minting company and get real dollars back. Sounds great and is a very useful concept in the crypto world.

Also, it's not true that the reserves of the stablecoin (the non-circulating tokens that are still with the company) are now worth real dollars. Those are unbacked tokens, worth nothing.

The problem with stablecoins always comes from the Psychology circle in the Venn-diagram. There are two (related) failure modes:

The minting company's executives cannot resist and will (secretly) give themselves some of the stablecoins, without putting in the equivalent backing USDs — they want to make money out of thin air. There are now unbacked stablecoins in the system, so in reality each coin is now worth less, say \$0.99 instead of \$1.00. But they attempt to keep this a secret, so everybody believes and/or pretends the coin is worth \$1 (the exchange rate charts still show \$1), and withdraws per that exchange rate when getting back real dollars. Eventually the company runs out of money. This could be caused because somebody suspects or finds the unbacked trades, tweets about it, and a bank run happens.
The minting company is supposed to hold on to the USD reserves, but the company's executives cannot resist and start making risky investment bets with the money, with the hope of turning the reserves into more money (and keeping the difference). Eventually one of the trades doesn't work out, and the reserves are now under water compared to the stablecoins in circulation. As in the previous failure case, a stablecoin is now worth less, say \$0.99 instead of \$1.00. But they attempt to keep this a secret, so everybody believes and/or pretends the coin is worth \$1... same story.

Why is everything shown in USD?

One thing I always chuckle at is that all the crypto coins are always tracked per the USD exchange rate — that seems to be the topline metric. This is ironic, as crypto was supposed to be a better alternative to fiat money, and by now we have thousands of coins/tokens floating around — yet everything is always measured in USDs. I suspect that deep down the smart players know that the coins are worthless...

Would I buy BTC at \$1?

The last thought experiment for this article is the following. The BTC/USD exchange rate right now is \$16,821. Suppose somebody offers to sell me BTC at \$1. To make it even interesting, let's suppose this offer is coming from Satoshi, the original creator of Bitcoin, whose (untouched) wallet from 10+ years ago still holds about 1M Bitcoins. Would I take it, and then immediately re-sell it at \$16,821?, and make \$16,820 in profits? Or would I stick to what I argued in this article and not buy — because why would I pay \$1 for something that I believe is worth \$0?

Fundamentally I believe that buying Bitcoin is pure speculation. There is no value being being created, so any money being made when anybody sells their BTC for USD is somebody else's USD (without additional value being created in the process for either party). So my win is your loss, and vica versa. So any money I win, somebody loses. So in the above thought experiment, if I buy Satoshi's BTC for \$1, with previously untouched ("unbacked") BTC being introduced into the system, and then sell it immediately at \$16,821 to Alice, then overall, I'm taking \$16,820 off the collective gambling stakes — other people's money.

So in my opinion, this is like saying: there's a bunch of people playing poker, the pool money is already very high. Satoshi shows me a sure way to beat the other players and take some money off the table, risk-free — would I do it? Let's suppose there are just 2 other poker players, and I know them well, they are my friends. Would I do it? How about if there are a million faceless poker players, and the games are being conducted over the Internet?

The proof is in the pudding, so I don't know what I would do. But fundamentally I'm against taking other's people's money without a fair exchange of value.

Conclusion

Crypto is a beautiful, but unregulated and hence dangerous intersection of technology, economics and human psychology. I often use thought experiments to try to remove smoke and unnecessary complications when trying to understand the crypto world. Here I shared some of these, using FTX's FTT token as an example.

I plan to write follow-up posts, where I will:

Argue why I think most use-cases for smart contracts aren't practical in the real world — including tokens, NFTs and others.
Argue why I think that buying and selling traditional stock (such as Google or Microsoft stock) is fundamentally different than buying and selling tokens and NFTs.
Explain why I think some of the crypto community's criticism of fiat money, central banks and regulation are actually a good thing.

See next article on fiat money vs. cryptocurrencies.

Finding similar API functions between Pytorch and Tensorflow with Doc2Vec

2022-12-21T00:00:00+01:00

Introduction

In a series of previous posts I used Doc2Vec to add recommendations to this blog, which are now live (scroll to the bottom of any page, it's a blue box). These previous posts were:

I wanted to see how Dov2Vec performs out-of-the-box comparing pages from different domains, ie. pages that have a different structure. Since I don't have a labeled data set for this, I was thinking of some domain where there are obvious similarities, and I could manually check the quality of the results. It occured to me that Pytorch and Tensorflow are similar Deep Learning libraries, so I could use Docv2Vec to compute similarities between their API doc pages, and see if it finds obvious "pairs". By pairs I mean, both will have a library function for eg. Cross Entropy, and so on. Let's see how it goes!

The code is up on Github.

Crawling

First, let's download the API docs. Initially I tried to use scrapy for this, but after a few hours of usage, I grew disappointed and abandoned it, for the following reasons:

it does not (seem to) have a good default auto-crawl, I needed to specifically tell it what links to crawl
it does not (seem to) have good default document extraction, you're on your own with eg. BeautifulSoup
it does not (seen to) have good default error-handling, eg. handling a javascript: or mailto: links crashes it
it uses a multiprocessing library which does not allow multiple crawls/restarts when used from ipython on Windows; I had to restart the whole python kernel on every new crawl

After a few hours of not getting much bang for my buck, I realized I'm better off writing a simple crawler loop myself with requests, urllib and BeautifulSoup. At least for such a simple use-case, I was right. My solution is more robust when used from ipython, simples, and about the same amount of code as the scrapy driver class:

def extract_text_and_links(html):
    soup = BeautifulSoup(html, 'html.parser')
    text = ' '.join(soup.find('article').text.split())
    links = [link['href'] for link in soup.find_all('a', href=True)]
    return text, links

def link_prefix(link):
    if '#' in link:
        link = link.split('#')[0]
    if '?' in link:
        link = link.split('?')[0]
    return link

def resolve_links(base_url, links):
    return set([link_prefix(urljoin(base_url, link)) for link in links])

def filter_links(base_domain, links):
    return [link for link in links if link.startswith('https://') and base_domain in link and '.pdf' not in link]

def crawl_and_save(base_url, num_pages):
    print(f'Crawling {base_url}...')
    base_domain = base_url.split('/')[0]
    urls_queued, urls_crawled, saved_pages = ['https://' + base_url], set(), {}
    while len(urls_queued) > 0 and len(urls_crawled) < num_pages:
        url = urls_queued.pop(0)
        urls_crawled.add(url)
        print(f'Fetching {url}')
        try:
            html = requests.get(url).text
            text, links = extract_text_and_links(html)
            if base_url in url:
                saved_pages[url] = text
            links = resolve_links(url, links)
            links = filter_links(base_url, links)
            links = [link for link in links if (link not in set(urls_queued) and link not in urls_crawled)]
            urls_queued.extend(links)
        except:
            pass
    print(f'Done!')
    print(f'Crawled {len(urls_crawled)} total pages, saved {len(urls_crawled)} target pages')
    print(f'Total content extracted: {int(sum([len(v)/1000 for v in saved_pages.values()]))} kbytes')
    return urls_crawled, saved_pages

With this I can now crawl both API docs:

_, tf_saved_pages = crawl_and_save(
    base_url='tensorflow.org/api_docs/python/tf',
    num_pages=5000,
)
_, pt_saved_pages = crawl_and_save(
    base_url='pytorch.org/docs/stable',
    num_pages=5000,
)

The output looks something like:

Crawling tensorflow.org/api_docs/python/tf...
Fetching https://tensorflow.org/api_docs/python/tf
...
Fetching https://www.tensorflow.org/api_docs/python/tf/raw_ops/WriteScalarSummary
Fetching https://www.tensorflow.org/api_docs/python/tf/image/stateless_random_saturation
Done!
Crawled 5000 total pages, saved 5000 target pages
Total content extracted: 14570 kbytes

Crawling pytorch.org/docs/stable...
Fetching https://pytorch.org/docs/stable
...
Fetching https://pytorch.org/docs/stable/_modules/torch/ao/nn/qat/dynamic/modules/www.lfprojects.org/policies/
Fetching https://pytorch.org/docs/stable/_modules/torch/ao/nn/qat/dynamic/modules/www.linuxfoundation.org/policies/
Done!
Crawled 3912 total pages, saved 3912 target pages
Total content extracted: 7921 kbytes

Note: at num_pages=5000, there were still some Tensorflow pages left, since it crawled the full 5,000 pages. Pytorch stopped at 3,912, so it crawled the entire documentation.

The next step is to merge the saved pages, and similar to how we did it in the previous Doc2Vec post, build the model:

pages = {**tf_saved_pages, **pt_saved_pages}
tagged_posts = {url : TaggedDocument(word_tokenize(text), [idx]) for idx, (url, text) in enumerate(pages.items())}
idx_lookup = {idx : url for idx, url in enumerate(pages.keys())}
model = Doc2Vec(tagged_posts.values(), vector_size=100, alpha=0.025, min_count=1, workers=16, epochs=100)

Similarity extraction

First, let's re-use the similar_pages() function from the previous article and do a consistency check:

def similar_pages(which, n=3):
    if not isinstance(which, str):
        which = idx_lookup[which]
    # at this point which is the url
    if n == 'all':
        return model.dv.most_similar(positive=[model.infer_vector(tagged_posts[which][0])], topn=None)
    results = model.dv.most_similar(positive=[model.infer_vector(tagged_posts[which][0])], topn=len(pages))
    results = [(idx_lookup[idx], f'{score:.3f}') for idx, score in results
        if idx != tagged_posts[which][1][0] and 'www' not in idx_lookup[idx]]
    return results[:n]

Check:

similar_pages('https://tensorflow.org/api_docs/python/tf/linalg/adjoint', n=10)

Returns:

[('https://tensorflow.org/api_docs/python/tf/raw_ops/BatchMatrixSolve', '0.661'),
 ('https://tensorflow.org/api_docs/python/tf/linalg/det',               '0.651'),
 ('https://tensorflow.org/api_docs/python/tf/linalg/logdet',            '0.640')]

This looks reasonable. adjoint is a matrix operation, and it returns related matrix operations. Let's look at another one:

similar_pages('https://tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy')

Returns:

[('https://tensorflow.org/api_docs/python/tf/keras/losses/BinaryFocalCrossentropy',         '0.885'),
 ('https://tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy',   '0.826'),
 ('https://tensorflow.org/api_docs/python/tf/keras/losses/CosineSimilarity',                '0.820')]

This looks reasonable. So similar_pages() returns similar pages, from the same API docs, as expected.

The next step is to write a simple function which returns the top n pages from the other API docs, given a query URL:

def similar_pages_cross(which, n=3):
    if not isinstance(which, str):
        which = idx_lookup[which]
    # at this point which is the url
    results = model.dv.most_similar(positive=[model.infer_vector(tagged_posts[which][0])], topn=len(pages))
    exclude = 'tensorflow.org' if 'tensorflow.org' in which else 'pytorch.org'
    results = [(idx_lookup[idx], f'{score:.3f}') for idx, score in results
        if idx != tagged_posts[which][1][0] and exclude not in idx_lookup[idx] and 'www' not in idx_lookup[idx]]
    return results[:n]

Let's see:

similar_pages_cross('https://tensorflow.org/api_docs/python/tf/linalg/adjoint')

Returns:

[('https://pytorch.org/docs/stable/generated/torch.Tensor.tril_.html',          '0.581'),
 ('https://pytorch.org/docs/stable/_modules/torch/_C/_distributed_c10d.html',   '0.581'),
 ('https://pytorch.org/docs/stable/_modules/torch/testing/_creation.html',      '0.578')]

This does not look good, I would have liked to get https://pytorch.org/docs/stable/generated/torch.adjoint.html in there.

Let's try with BCE:

similar_pages_cross('https://tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy')

Returns:

[('https://pytorch.org/docs/stable/generated/torch.nn.functional.binary_cross_entropy_with_logits.html',    '0.559'),
 ('https://pytorch.org/docs/stable/generated/torch.Tensor.logit_.html',                                     '0.536'),
 ('https://pytorch.org/docs/stable/generated/torch.nn.HuberLoss.html',                                      '0.526')]

This looks reasonable.

Conclusion

After playing around with the results more, my conclusion is that the top recommendations from the other API docs are not always what I'd intuitively expect. Ie. similarly to how the pair of adjoint is not found, Doc2Vec does not reliably identify the matching/similar API call in the other API docs. Based on this very limited experiment, I suspect that this simple version would not be good enough for production use, ie. give recommendations for a programmer coming from one API, trying to use the other.

Testing GPT-3 spatial reasoning and comprehension

2022-12-17T00:00:00+01:00

Introduction

In recent months I've played around with OpenAI's GPT-3 and tested it in various domains I work in (have worked in):

My core conclusion in these experiments was that the AI does surprisingly well in these domains, with big caveats: it sounds equally confident in right and wrong answers. But, in retrospect, now that we're used to the idea of GPT-3, perhaps this makes sense, because GPT-3 is trained from Wikipedia articles and other knowledge bases, and a lot of these answer can be — as the system proves — answered just on a statistical pattern-matching / text-completion basis.

What I was wondering recently whether, or to what degree, GPT-3 has developed an understanding of something removed from textual knowledge: spatial reasoning and comprehension. Going into these experiments, I did not know what to expect. GPT-3 has done well in the other domains and surprised me, so my baseline expectation was that I will have a similar experience here: GPT-3 will do surprisingly well.

You can also talk to GPT-3, just create an account on OpenAI and go to the sandbox!

People in a room

The simplest test I could think of is to place people in 4 corners in a room, put a person in the middle, and ask who is to the left/right/behind the person in the middle. Let's start with an easy question:

A: Five people are in the room: Alice, Bob, Charlie, David, and Ed.

Q: There is square room.
...
Who is standing to the left of Ed?

A: Alice is standing to the left of Ed.

Note: incorrect, Bob is standing to the left of Ed.

Q: There is square room.
...
Who is standing to the right of Ed?

A: Bob is standing to the right of Ed.

Note: incorrect, David is standing to the right of Ed.

Q: There is square room.
...
Who is standing behind Ed?

A: Nobody is standing behind Ed.

Note: incorrect, Charlie is standing behind Ed.

Clearly GPT-3 cannot handle this task. Let's try a variation:

Q: There is square room.
...
Everybody goes around clockwise 90 degrees.
Who is now in the north-west corner?

A: Charlie is now in the north-west corner.

Note: incorrect, Bob is now in the north-west corner.

Perhaps the problem is with using north-south-west-east terminology. Let's simplify:

Q: There is square room.
Alice is standing in the upper left corner.
Bob is standing in the lower left corner.
Charlie is standing in the lower right corner.
David is standing in the upper right corner.
Ed is standing in the center of the room, facing Alice.
Who is standing to the left of Ed?

A: Bob is standing to the left of Ed.

Note: correct!

Q: There is square room.
...
Who is standing to the right of Ed?

A: Charlie is standing to the right of Ed.

Note: incorrect, David is standing to the right of Ed.

Q: There is square room.
...
Who is standing behind Ed?

A: Nobody is standing behind Ed.

Note: incorrect, Charlie is standing behind Ed.

The conlusion is clear: GPT-3 cannot handle this spatial comprehension task.

Clock faces

Can GPT-3 reason about the hands of a clock?

Q: It is half past eleven. What number is the minute hand of the clock pointing at?

A: The minute hand of the clock is pointing at 30.

Note: correct.

Q: I have a watch. The hour hand is pointing to 11. The minute hand is at a right angle to it, on the right side. What number is the minute hand pointing at?

A: The minute hand is pointing at 2.

Note: correct.

Q: I have a watch. The hour hand is pointing to 11:30. The minute hand is at a right angle to it, on the right side. What number is the minute hand pointing at?

A: The minute hand is pointing at 8.

Note: incorrect, the minute hand is pointing at 2:30.

Let's try the same question, phrased differently:

Q: I have a watch. The hour hand is pointing halfway between 11 and 12. The minute hand is at a right angle to it, on the right side. What number is the minute hand pointing at?

A: The minute hand is pointing to the number 8.

Note: incorrect again.

This is a mixed bag. The simpler questions GPT-3 can answer, probably because they occur in flat text it was trained on. But going beyond, it fails.

Triangles

Q: We have a triangle. One of the angles is 70 degrees, another is 80 degrees. What is the third angle?

A: The third angle is 30 degrees.

Note: correct. This is so simple, it may even have occured with these numbers in the training data. Also, it's known that GPT-3 can do simple arithmetic.

Q: We have a triangle. One of the angles is 70.51 degrees, another is 80.49 degrees. What is the third angle?

A: The third angle is 29 degrees.

Note: correct.

Q: We have two identical right-angled triangles. We snap them together along their longest edges. What is the resulting shape?

A: The resulting shape is a square.

Note: correct.

Q: We have a very small triangle on a very large sphere. What is the sum of its angles, in degrees?

A: 180 degrees

Note: correct, if the triangle is very small, the surface of the sphere is locally flat, so we can approximate with Euclidean geometry.

Q: We have a triangle on a sphere. What is the sum of its angles, in degrees?

A: The sum of the angles of a triangle on a sphere is always 180°.

Note: incorrect. On a sphere, the sum of angles of a triangle is between 180 and 3x90=270 degrees.

Q: We have a triangle on a sphere. The sum of it's angles is 200 degrees. We scale up the sphere by 10x. What is the sum of the triangle's angles now?

A: The sum of the triangle's angles now is 2000 degrees.

Note: incorrect. The angles don't change, the sum remains 200.

This is similar to the clock face thread: GPT-3 falls apart quickly beyond the simplest questions.

Areas and volumes

Q: We have a white cube, each side is 1 unit long. We paint it red using 1 liter of paint.
We now have a white cube, each side is 2 units long. How many liters of paint do we need to paint it red?

A: We would need 4 liters of paint to paint the cube red. This is because each side of the cube is now twice as long, so the total surface area of the cube has been multiplied by four.

Note: correct.

Q: We have a square. We cut up the square into pieces and rearrange the pieces to form a really long rectangle. Which has bigger area, the original square, or the really long rectangle?

A: The original square has a bigger area than the really long rectangle.

Note: incorrect. Area remains the same.

Q: There is a cube, and a sphere inside the cube. Which has bigger volume?

A: The cube has a larger volume.

Note: correct.

A: The volume of the sphere contained in a unit cube in 4D is 2π/15.

Note: incorrect, the correct answer would be $\frac{\pi^2}{32}$.

Q: In higher dimensions what is the ratio of the volumes of a cube and the sphere contained in the cube??

A: The ratio of the volumes of a cube and the sphere contained in the cube is 8 : π2.

Note: not a good answer. A good answer is that in higher dimensions the ratio if the volume of the contained sphere and the cube goes to zero. In other words, in higher dimensions — against our intuition — the sphere becomes arbitrarily small.

Random point processes

Can GPT-3 reason about random points in a square?

Q: We have a unit square. We randomly place points within the unit square, each point is independent from the rest, and it's position distribution is uniform in the square. What percent of the points falls within the square?

A: 100%. All points placed within the unit square will fall within the square.

Note: correct.

Q: We have a unit square. We randomly place points within the unit square, each point is independent from the rest, and it's position distribution is uniform in the square. We draw a diagonal from the top left corner to the lower right corner. What percent of points falls below this diagonal?

A: The answer is 50%. Since the points are placed randomly and independently, and the position distribution is uniform, an equal number of points will fall to the left and right of the diagonal line. Therefore, the percentage of points below the diagonal will be 50%.

Note: correct.

A: The percentage of points that fall within the circle contained in the square is approximately 78.54%.

Note: correct, $\frac{\pi}{4} = 0.7854$.

To my surprise, after the previous topics, GPT-3 got all these correct. My suspicion is that these questions are close enough to flat text knowledge covered in Wikipedia articles (and similar training documents).

ASCII art

Can GPT-3 read or write ASCII art? This is the question that originally triggered my exploration into spatial comprehension. Let's see:

Q: What is the letter in the following ASCII art?

██╗░░██╗
██║░░██║
███████║
██╔══██║
██║░░██║
╚═╝░░╚═╝

A: The letter is "E".

Note: incorrect.

Q: What is the text in the following ASCII art?

██╗░░██╗███████╗██╗░░░░░██╗░░░░░░█████╗░
██║░░██║██╔════╝██║░░░░░██║░░░░░██╔══██╗
███████║█████╗░░██║░░░░░██║░░░░░██║░░██║
██╔══██║██╔══╝░░██║░░░░░██║░░░░░██║░░██║
██║░░██║███████╗███████╗███████╗╚█████╔╝
╚═╝░░╚═╝╚══════╝╚══════╝╚══════╝░╚════╝░

A: Welcome!

Note: incorrect, but "Welcome" is eerily close to "Hello". I wonder what's going on here.

Q: What is the text in the following ASCII art?

███╗░░░███╗░█████╗░██████╗░████████╗░█████╗░███╗░░██╗
████╗░████║██╔══██╗██╔══██╗╚══██╔══╝██╔══██╗████╗░██║
██╔████╔██║███████║██████╔╝░░░██║░░░██║░░██║██╔██╗██║
██║╚██╔╝██║██╔══██║██╔══██╗░░░██║░░░██║░░██║██║╚████║
██║░╚═╝░██║██║░░██║██║░░██║░░░██║░░░╚█████╔╝██║░╚███║
╚═╝░░░░░╚═╝╚═╝░░╚═╝╚═╝░░╚═╝░░░╚═╝░░░░╚════╝░╚═╝░░╚══╝

A: No text

Note: incorrect.

Q: Generate ASCII art for the letter H.

  ____ 
 |    |
 |    |
 |    |
 |____|

Note: incorrect.

Q: Generate ASCII art for the word HELLO.

██╗   ██╗██╗ ██████╗ ███████╗    ██╗      ██████╗ ██████╗ 
██║   ██║██║██╔════╝ ██╔════╝    ██║     ██╔═══██╗██╔══██╗
██║   ██║██║██║  ███╗███████╗    ██║     ██║   ██║██║  ██║
╚██╗ ██╔╝██║██║   ██║╚════██║    ██║     ██║   ██║██║  ██║
 ╚████╔╝ ██║╚██████╔╝███████║    ███████╗╚██████╔╝██████╔╝
  ╚═══╝  ╚═╝ ╚════

Note: incorrect, but it's impressive that it's able to generate some ASCII art with letters.

Conclusion

Based on these experiments, my conlusion is that during its training GPT-3 has not developed the ability to reason spatially. However, based on the recent flow of models released that can paint (DALL-E), code (Codex), chat (ChatGPT), and so on, I assume that given enough training data it would be possible to train models that perform well on spatial reasoning tasks.

Fred Brooks' The Mythical Man-Month

2022-12-15T00:00:00+01:00

Introduction

In November of this year Frederick P. Brooks passed away. I know Brooks from 2 of his books, both of which I recommended highly:

In this post I reflect on The Mythical Man-Month (usually abbreviated MMM), and the impact it had on me. But first, who was Fred Brooks?

From his Wikipedia page:

Frederick Phillips Brooks Jr. (April 19, 1931 – November 17, 2022) was an American computer architect, software engineer, and computer scientist, best known for managing the development of IBM's System/360 family of computers and the OS/360 software support package, then later writing candidly about the process in his seminal book The Mythical Man-Month. In 1976, Brooks was elected as a member into the National Academy of Engineering for "contributions to computer system design and the development of academic programs in computer sciences". Brooks received many awards, including the National Medal of Technology in 1985 and the Turing Award in 1999.

The original edition of MMM was published in 1975 as a collection of essays. They are a record of Fred Brooks' insights gained from the OS/360 software engineering project. OS/360 was the operating system for the System/360, IBM's flagship mainframe computer in 1960s and 70s.

The first time I read the book was in the early 2000s, when I was working my first job as a (junior) C/C++ Developer at Graphisoft. I was writing code for Archicad versions 7-10, one of the first versions to have the new C++ codebasem versus the older versions' C codebase. Fun fact, Archicad is currently on version 26. What struck me first and foremost was that Brooks identified some of the core challenges and fallacies as far back as 1975, working on one of the first large-scale software engineering efforts! Second, throughout my career I noticed that I repeatedly hit on the same issues as described by Brooks, at different companies, on a different technological stacks. It seemed to me — and still seems — that our industry is continuously repeating these mistakes.

I will not try to repeat all the points in the book. The book is very well written, broken into bite-sized essays, it deserves to be read in its original form. As a crutch, the Wikipedia page of the book has a good one-paragraph summary of each essay. I will only reflect only on the core points that I often recall and actually apply in my own work. The points I make below are not necessarily the original points by Brooks, but my interpretation of them, and how I "filled it content" in the past 15 years.

The mythical man-month

Brooks' point here is that adding people to a project does not improve velocity after a break-even saturation point. The basic pattern is this: an estimate is made that a project takes 120 man-months. A naive manager comes along and says, "Great, let's get 20 software engineers, and we will be done in 120/20 = 6 months!" In reality, it is not true that <man-month-estimate> = <number-of-people> x <months-taken>. The reasons are many, I will enumerate a few:

With $n$ people on the project, there are potentially $n(n-1)/2$ communnication edges; in general communication scales like $O(n^2)$, unless special care is taken in the org to avoid this. At some break-even point, adding new people to the project does not make it faster, in fact it makes it slower.
Product Managers and Software Engineers need a fixed amount of learning time to explore the domain and to identify the right solution. Adding more people does not reduce this fixed initial time, in fact it makes it longer, since more time will be wasted with too many people trying to contribute to the design.
With too many people on the project, more of the work goes to the bottom 50% (or the bottom 80%); this work needs more revision (potentially rewrites); usually there is an optimal number of senior engineers that should work on a project, perhaps aided by a few less senior contributors.

No silver bullet

Often, some new technology — a silver bullet — is identified that project managers believe will significantly increase productivity. Although such cases exist, it is very rare. (A clear case of a silver bullet in my experience is AWS.) For example, on one product I worked on, it was mistakenly believed that converting the codebase from C to C++ would improve developer productivity and product stability significantly — but it did not, in fact both decreased, because senior C programmers were converted to novice C++ programers; and even putting that aside, I believe there is no step-function jump in productivity going from C to C++. I believe that functional programming languages like Haskell were also believed to be silver bullets, but did not end up changing the game. (Having said that, functional programming patterns added to traditional programming languages like C++, C# and Python are a clear win.)

The second-system effect

The second-system effect happens when once a software engineering team has successfully shipped and stabilized the first version of a system, and is thinking of the next step. The first version is mature and in users' hands, but in the process inevitably architectural trade-offs were made and hacks made it into the codebase. The team now wants to do a rewrite, a clean system-design incorporating all the lessons learned from the first system. Often the second system will becone over-architected, will take significantly longer than planned, and in the end will still include (a different set of) architectural trade-offs and hacks. As this becomes clear, the team loses some of their motivation, and development slows. This is the second-system effect. In fact, many second systems end up being delayed (or abandoned altogether), and development on the first version is resumed, at least for the time being.

Surgical team

The core insight in this essay is that not all software engineers are created equal, and the team structure should reflect this. Some engineers are more senior, have more domain experience and a knack for software architecture. The other main point is that software architecture needs to be coherent and consistent, so a minimum number of people should design it to avoid unnecessary complexity. The idea then is to model the software engineering team as a surgical team, with the surgeon making the cuts, and various other team members playing various support roles. So, have a lead programmer come up with the core architecture and write most of the core code, with others playing a support function (writing non-core sub-systems, tooling, documentation, etc). Although the terminology breaks, I always patch on the idea of a "co-pilot" to the main "pilot" (the surgeon); the co-pilot should always have full context and can take over if the pilot leaves. The way I run teams today is heavily influenced by this idea.

Tooling teams

Following on from the idea of a surgical team, Brooks realized that this pattern can also be applied at the organization level. It's amazing that he had this insight in the 1960s, but the idea of tooling teams and DevUX only took off at tech companies in the 2010s. And non-tech companies still don't have tooling teams!

10x developer

Brooks mentions that the difference in productivity between the best and the worst developers can be as much as 10x. This pops up periodically on Internet forums such as Hacker News. Personally I believe in this concept. I have worked with many people where I was a 10x developer compared to them, simply because they were working on the wrong problems, or made naive architectural choices, or they blocked for days on problems that I was able to solve much faster (eg. how to resolve a memory leak or a server process crashing). But, I have also worked with developers who were 10x compared to me, esp. in my early days! In a way, this is what hiring is all about: trying to keep the gap between your best and your worst engineers as tight as possible, to not let the downward 10x developers in!

Conclusion

I hope the article gave a reasonable account of the ways Brooks' book The Mythical Man-Month influenced me. If you have not, read the book!

How good an astrophysicist is GPT-3?

2022-12-12T00:00:00+01:00

Introduction

Recently I have been playing around with OpenAI's GPT-3 and I am very impressed by it. It reminds of the famous Arthur C. Clarke quote, “Any sufficiently advanced technology is indistinguishable from magic.” If you don't know what GPT-3 is, this is a good introduction with a lot of links.

You can also talk to GPT-3, just create an account on OpenAI and go to the sandbox!

Here I will show a "conversation" with GPT-3 to gauge how good an astrophysicist — or an illusion of an astrophysicist — it is. Mostly these are questions one might ask a student in an exam situation. The questions are independent of one another, so they work with a context-free system such as GPT-3. By context-free I mean that GPT-3 does not maintain conversation state, each question and answer are indepedent from the AI's perspective.

GPT-3 is trained on large amounts of text downloaded from the Internet. So I expect it to do well on questions where there are high-quality answers on the Internet, and it "just" has to return it, in the correct english structure. For questions that require more reasoning and combining facts (from the question and training data) I expect to get less impressive answers.

Also see these previous Bytepawn posts on GPT-3:

Basics

Q: In cosmology, what is the experimental evidence for the non-zero value of the cosmological constant lambda?

A: The experimental evidence for the non-zero value of the cosmological constant lambda comes from several observations, including: 1. The observed accelerated expansion of the universe, which was first detected in 1998 when observations of Type Ia supernovae showed that the universe was expanding at an accelerating rate.
2. The Cosmic Microwave Background (CMB), which shows that the universe is dominated by a component that behaves like a cosmological constant.
3. Large-scale structure observations, which indicate that the universe is dominated by a component that behaves like a cosmological constant.
4. Observations of distant supernovae, which indicate that the expansion of the universe is accelerating.
5. Observations of the anisotropies in the CMB, which indicate that the universe is flat, consistent with the presence of a cosmological constant.

Comment: this is a good answer, but not great. The first and the fourth point are the same. Also, there were prior observations to 1998, although with higher error margins, going back all the way to Hubble. Regarding the last point, it's not clear what the sentence is implying; in itself a flat Universe does not require non-zero lambda.

Q: In cosmology, what is the difference between a flat, open and closed cosmology?

A: A flat cosmology is a model of the universe in which the total energy density is equal to the critical density, and the spatial curvature of the universe is zero. This type of universe will expand forever, and its expansion will eventually slow down as its energy density decreases.

An open cosmology is a model of the universe in which the total energy density is less than the critical density, and the spatial curvature of the universe is negative. This type of universe will expand forever and its expansion will eventually accelerate as its energy density decreases.

A closed cosmology is a model of the universe in which the total energy density is greater than the critical density, and the spatial curvature of the universe is positive. This type of universe will eventually stop expanding and will eventually collapse in on itself, or recollapse.

Comment: this is a good answer.

Q: In cosmology, what is the difference between a flat, spherical and hyperbolic universe?

A: A flat universe is one where the average density of matter is just enough to make the universe flat, meaning that if you draw a triangle on a flat two-dimensional surface, the sum of the angles of the triangle will be 180 degrees. A spherical universe is one where the curvature of space-time is positive, meaning that if you draw a triangle on a spherical two-dimensional surface, the sum of the angles of the triangle will be greater than 180 degrees. A hyperbolic universe is one where the curvature of space-time is negative, meaning that if you draw a triangle on a hyperbolic two-dimensional surface, the sum of the angles of the triangle will be less than 180 degrees.

Comment: this is mostly a good answer, the geometric explanation is spot-on. I don't like the first sentence, although it is often said like this "... average density makes the Universe flat ...". The global topology of the Universe is what it is, it's one of the three, it cannot change (as the average density changes). Eg. if the Universe started out globally flat (or spherical), then it always remains globally flat (or spherical), it cannot go from one to the other. Local curvature of course can change in the presence of matter...

Q: In cosmology, is it possible to have globally spherical Universe that is expanding forever?

A: Yes, it is possible to have a globally spherical Universe that is expanding forever. This type of Universe is known as a "closed" or "bounded" Universe. In a closed Universe, gravity eventually stops the expansion, and the Universe begins to contract until it reaches a maximum size and then expands again. This cycle of expansion and contraction is known as the "Big Bounce" model.

Comment: this is a confused answer. In the first sentence it says it is possible, but then itr says that eventually expansion stops. Per my understanding, in a globally spherical (closed) Universe, you can get infinite expansion if you introduce a large enough cosmological constant Λ (just as in the flat case). Let's ask a follow-up question...

Q: In cosmology, is it possible to have globally flat Universe that is expanding forever?

A: Yes, it is possible to have a globally flat Universe that is expanding forever. This is known as a "flat Lambda-CDM universe" and is the most widely accepted model of the universe. It is based on the assumption that the universe is spatially flat (i.e. it has zero curvature), and that its expansion is driven by a cosmological constant (Lambda) and cold dark matter (CDM). This model predicts that the universe will expand forever, but the rate of expansion will gradually slow down.

Comment: great answer, since this is standard Wikipedia-level knowledge. Now, let's get back to probing it with curvature questions...

Topology, curvature

Q: In cosmology, what is difference between global and local curvature? Are the two always the same?

A: Global curvature refers to the overall curvature of the universe, based on its average density. Local curvature, on the other hand, refers to the curvature of space-time at any given point in the universe. The two are not always the same, as local curvature can be affected by the presence of matter or energy at that point, while global curvature is determined by the average density of the universe.

Comment: great answer, except the last sentence. Per my understanding, we use measurements of the average density to learn the global topology of the Universe, but it's not determined by the average density (which is changing, anyway). The global topology is what it is, and (probably) does not change. Let's see what it thinks on that topic...

Q: In cosmology, can a Universe jump between flat, spherical and hyperbolic global curvatures as matter density changes?

A: No, a Universe cannot jump between flat, spherical, and hyperbolic global curvatures as matter density changes. In the standard cosmological model, the Universe is thought to have a constant global curvature, and the amount of curvature is determined by the average density of matter in the Universe. If the matter density is greater than a certain value, the Universe will have a spherical curvature. If the matter density is less than that value, the Universe will have a hyperbolic curvature. If the matter density is exactly equal to that value, the Universe will have a flat curvature. The curvature of the Universe will not change as the matter density changes.

Comment: great answer!

Q: In cosmology, does the global curvature of the Universe ever change?

A: No, the global curvature of the Universe is assumed to be constant and does not change over time.

Let's ask some questions about the initial singularity in the Big Bang model:

Q: In cosmology, assuming the Universe is flat and infinite, in what sense was there a singularity at zero time. How does an infinite flat Universe become infinitely small?

A: The singularity at zero time in an infinite flat Universe is a mathematical concept rather than a physical one. It is a point in the mathematical description of the Universe where the density of matter, space and time become infinite and the laws of physics break down. A flat and infinite Universe cannot become infinitely small, since it is already infinitely large. The singularity at zero time is simply a mathematical artifact that arises from trying to describe an infinite Universe in finite terms.

Comment: great answer!

Isotropy

Q: In cosmology, can a periodically infinite flat Universe be isotropic?

A: Yes, a periodically infinite flat Universe can be isotropic. This means that the Universe has the same properties in all directions, and so the physical laws that govern it are the same no matter which direction an object is travelling.

Comment: it can be locally isotropic, but not globally. Let's explore this further...

Q: In cosmology, what is the difference between local isotropy and global isotropy?

A: Local isotropy is the idea that the universe looks the same in all directions when observed on a local scale. This means that when looking at a small region of the universe, it appears to be the same in all directions. Global isotropy, on the other hand, is the idea that the universe looks the same in all directions when observed on a global scale. This means that when looking at the entire universe, it appears to be the same in all directions.

Q: In cosmology, is a periodic flat Universe, ie. picture a large box with the sides connected, globally isotropic?

A: No, a periodic flat Universe is not globally isotropic. A periodic flat Universe is a flat universe with periodic boundary conditions, meaning that the universe is periodic in the spatial dimensions. This means that the universe appears to be finite in size, and because of this it is not globally isotropic.

Comment: this is correct. Imagining a box, with infinite copies of it placed in a grid closely touching, the x, y and z directions of the box are special, because an astronaut can get back to her homeworld in less time then traveling in any other direction.

Clustering

Q: In cosmology, how would you explain Baryon Acoustic Oscillations to a layperson?

A: Baryon Acoustic Oscillations are a type of pattern in the universe that are caused by sound waves traveling through the early universe. They can be seen in the distribution of galaxies in the sky today, which appear to be slightly lumpy, like ripples in a pond. The sound waves that caused these patterns were created shortly after the Big Bang, and allow us to study the universe's beginnings.

Comment: this is a good answer, as are the following in this section.

Q: In cosmology, what is the role of clustering and Baryon Acoustic Oscillations in determining cosmological parameters?

A: Clustering and Baryon Acoustic Oscillations (BAO) play an important role in determining cosmological parameters, such as the matter density, cosmological constant and Hubble constant. Clustering is used to measure the distribution of matter in the universe and to constrain the values of cosmological parameters, while BAO allow us to measure the angular diameter distance to galaxies and to determine the expansion rate of the universe and its age. Together, these two measurements can be used to constrain the values of cosmological parameters and test theories like the Big Bang model.

Q: In cosmology, what is acoustic about Baryon Acoustic Oscillations?

A: Baryon Acoustic Oscillations (BAO) are a phenomenon in cosmology where the distribution of matter in the universe is characterized by a series of peaks and valleys. This pattern is caused by sound waves traveling through the primordial plasma of the early universe. These sound waves created regions of higher and lower density, which eventually became galaxies, clusters of galaxies, and other large-scale structure in the universe. The BAO signature is a powerful tool for measuring the expansion of the universe and constraining cosmological parameters.

Q: In cosmology, how are Baryon Acoustic Oscillations measured?

A: Baryon Acoustic Oscillations (BAOs) are measured by looking at the correlation function of galaxies in the Universe. This correlation function is measured using data from surveys such as the Sloan Digital Sky Survey (SDSS). The correlation function describes how the galaxies are distributed in the Universe and allows us to measure the scale of the BAOs. This scale is then used to measure the distance to the galaxies and thus the expansion rate of the Universe. This measurement of BAOs has been used to measure the properties of dark energy, which has helped to confirm the current cosmological model.

Q: In cosmology, what is the difference between the two-point mass correlation function and the three-point mass correlation function?

A: The two-point mass correlation function measures the probability of finding two galaxies at a given distance from one another. It provides information about the clustering of galaxies on large scales. The three-point mass correlation function measures the probability of finding three galaxies at a given distance from one another. This can provide information about the clustering of galaxies on smaller scales, as well as the strength of interactions between them.

Q: In cosmology, what is the role of clustering and Baryon Acoustic Oscillations in determining cosmological parameters? What is the additional accuracy gained vs just looking at Planck CMB data?

A: Clustering and Baryon Acoustic Oscillations (BAO) are powerful methods used to measure cosmological parameters. Clustering refers to the spatial patterns of galaxies and other matter in the universe, and BAO refer to the ripples in the density of matter that were created by sound waves in the early universe. By measuring these features, it is possible to obtain precise information about the evolution of the universe.

Clustering and BAO provide an additional level of accuracy compared to Planck CMB data, because they measure different aspects of the universe. Planck CMB data measures the Cosmic Microwave Background radiation, which is a snapshot of the universe at a very early stage, whereas clustering and BAO measure the spatial patterns of galaxies and other matter in the universe at a much later stage. This additional information allows us to determine cosmological parameters with more accuracy.

Conclusion

Overall, most of the GPT-3 answers are okay, which is very impressive. On some of the trickier questions, the answer is a bit confused. This is the overall danger of GPT-3: if the reader is not an expert, s/he can't tell if they're getting a good answer (in this case, about half the time) or reasonable sounding non-sense.

Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

2022-11-30T00:00:00+01:00

Introduction

One of the most common Data Science tasks in a business setting is timeseries forecasting. Examples include:

given N years of historic daily sales, build a forecast for next year's daily sales
given N years of historic Daily Active Users (DAU), build a DAU forecast for the remainder of the year
given N years of historic hourly transaction counts per retail store, build a forecast for next month's hourly transaction count per store

I was curious what methods and libraries other Data Scientists use, so I posted an "Ask HN" on Hacker News: Data Scientists, what libraries do you use for timeseries forecasting? I expected zero to moderate engagement, I would have been happy with 10 answers. In the end, the post generated 89 comments, most of them high-quality. This is my summary of the discussion.

Hacker News comments

Below is my enumeration of main points made in the comments.

There were a number of recommendations for Darts, which is a Python super-library for forecasting, most notably by user hrzn, the library's creator. By "super-library", I mean that it implements its own models, but also wraps existing libraries (such as Prophet). I was not aware of Darts, I definitely plan to invest time to experiment with it.

User isoprophlex suggests to reframe the problem as a classical regression problem, and use XGBoost or LightGBM:

As an example, imagine you want to calculate only a single sample into the future. Say furthermore that you have six input timeseries sampled hourly, and you don't expect meaningful correlation beyond 48h old samples. You create 6x48 input features, take the single target value that you want to predict as output, and feed this into your run of the mill gradient boosted tree. The above gives you a less complex approach than reaching for bespoke time-series stuff; I've personally have had success doing something like this. If your regressor does not support multiple outputs, you can always wrap it in sklearns MultiOutputRegressor (or optionally RegressorChain; check it out). This is useful if, in the above example, you are not looking to predict only the next sample, but maybe the next 12 samples.

In a response, user em500 points out:

[Prophet] is mostly just regression though, with features for trend, yearly and weekly periodicity (smoothed a bit using trigonometric regressors), and holiday features. The only non-standard linear regression part is that it includes a flexible piecewise linear trend, with regularization to select where the trend is allowed to change.

To my surprise, there a quite a few comments critical of Prophet. In my experience at work (where we do forecasting of business timeseries, see examples in the introduction) Prophet does a good job. Prophet supports holidays, external regressors, growth trends, changepoints, and just the right seasonalities out of the box. In the majority of cases I don't feel a need for using another library. Back to the criticism, user dxbydt writes:

Peter Cotton has atleast a dozen very credible studies/results on Prophet vs other timeseries libraries. Before committing to prophet, please check out a few of these (all over linkedin). His tone is acerbic because he believes prophet is suboptimal & makes poor forecasts compared to the other contenders. That said, you can ignore the tone, just download the packages & test out the scenarios for yourself. I personally will not use Prophet. Like most stat tools in the python ecosystem, it is super easy to deploy & code up, but often inaccurate if you actually care about the results. ofcourse, if its some sales prediction forecast where everything’s pretty much made up & data is sparse/unverifiable, then Prophet ftw.

Note: I don't understand why he/she says sales forecasting is "made up".

User qsort writes:

[Prophet is] not as good as they'd have you believe, but it's fast and analysts can play with it to some extent.

Another interesting comment from user tfehring:

For cases that Prophet doesn't cover I recommend bsts, which is much more flexible and powerful. Anything too complicated for bsts, I'll typically implement in Stan.

What's interesting here is that Prophet uses Stan internally to make its forecast.

Multiple users mention ensemble methods, ie. building multiple forecasts with different models and combining them. The most interesting comment came from user d4rti:

I did some anomaly detection work, in business transactions, and found the best way was to create a sort of ensemble model, where we applied all the models, and kept any anomalies, then used simple rules to only alert on 'interesting' anomalies, like: 2-3 anomalies in a row, high deviation from expected, multiple models detected anomaly, to improve signal vs noise.

User jll29 is of many who mentions neural network models:

Former Reuters Research Director here. When modeling time series, you will want a model that is sensitive both to short term and longer term movements. In other words, a Long Term Short Term Memory (LSTM).

And finally, a mention for classical ARIMA from user crimsoneer:

For time series, classical methods (ARIMA etc) still continue to perform very well for most problems.

Methods and libraries mentioned

A list of methods/libraries mentioned in this thread (excluding R libraries):

ARIMA (method)
Exponential smoothing (method)
Darts
Prophet
Stan: a statistical library used internally by Prophet
XGBoost: how to use XGBoost's regressor to build a timeseries forecast
LightGBM: how to use LightGBM's regressor to build a timeseries forecast
PyTorch-forecasting
Pmdarima
State Space Model and Kalman Filters
Pybsts: stands for Python Bayesian Structural Time Series
Sktime
Tsfresh: timeseries feature extraction
statsmodels for timeseries forecasting

Conclusion

The comments are very interesting and informative. I plan to look into Darts, Pytorch-forecasting and some of the other methods mentioned here.

Estimating mathematical constants with Monte Carlo simulations

2022-10-09T00:00:00+02:00

Introduction

Over the past years I've written many blog posts where I used Monte Carlo simulations to compute a quantity or illustrate a concept. A Monte Carlo simulation is just a simulation involving random numbers. Usually it's involved when directly computing a quantity is not feasible, but it can efficiently be estimated by some random method. Typically, estimation with Monte Carlo simulations becomes more accurate the longer we run it.

Past articles using Monte Carlo simulations:

Here I will use simple (<10 LOC) Monte Carlo simulations to estimate famous mathematical constants: $\sqrt{2}, \phi, e$ and $\pi$. The code is on Github.

Estimate $\sqrt{2}$

$\sqrt{2}$ is the number $x$ such that $ x^2 = 2 $. We can get an estimate by randomly selecting numbers, and checking which one's square is closest to 2:

from random import random
from math import pi

N = 1000*1000
xs = sorted([(1.4 + random()/10) for _ in range(N)])
for x in xs:
    if x**2 > 2: break

print('actual:   %.5f' % sqrt(2))
print('estimate: %.5f' % x)

Output:

actual:   1.41421
estimate: 1.41421

Note: this is suprisingly simple, but quite inefficient. There is no advantage to randomization here, it would be just as well to start at 1.4, and increase by some small $\epsilon$ on each step, and stop if we are over 2, without using a list. The step-size could even be made dynamic.

# estimate sqrt(2) without randomization

eps = 0.00000001
x = 1.4
while True:
    x += eps
    if x**2 > 2:
        break
print('actual:   %.7f' % sqrt(2))
print('estimate: %.7f' % x)

Output:

actual:   1.4142136
estimate: 1.4142136

Estimating the golden ratio $\phi$

The golden ratio is the number $ \phi = \frac{a}{b} $ such that $ \frac{a+b}{a} = \frac{a}{b} $ and $ a < b $. Estimating $ \phi $ is straightforward: generate pairs of numbers $a$ and $b$, compute $ \frac{ \frac{a+b}{a} }{ \frac{a}{b} } $, and for the one closest to 1, return $ \phi = \frac{a}{b} $:

from random import random
from scipy.constants import golden

N = 10*1000*1000
gs = []
for _ in range(N):
    a, b = random(), random()
    if a < b: continue
    l, r = (a + b) / a, a / b
    gs.append((l / r, r))
gs.sort()
for g in gs:
    if g[0] > 1: break
print('actual:   %.5f' % golden)
print('estimate: %.5f' % g[1])

Output:

actual:   1.61803
estimate: 1.61803

Estimating $e$

One of the possible definitions of $e$ is: $e$ is the number such that the integral of the function $ \frac{1}{x} $ from 1 to $e$ is 1.

We can randomly generate numbers between 1 and $M$ (I will use $M=3$ here), order them, and estimate the integral for each random point. When the integral crosses from less than 1 to more than 1, stop:

from random import random
from math import e

N, Ni = 10*1000*1000, 0
ps = sorted([(1 + 3 * random(), random()) for _ in range(N)])
for i, p in enumerate(ps):
    if p[1] < 1 / p[0]: Ni += 1
    if (p[0]-1) * Ni / (i+1) > 1: break

print('actual:   %.5f' % e)
print('estimate: %.5f' % p[0])

Output:

actual:   2.71828
estimate: 2.71896

Here we also don't benefit from randomization: we can just start at 1, and increase by some small $\epsilon$ on each step, and stop if the integral is above 1, without using a list:

eps = 0.00000001
x = 1
integral = 0
while True:
    x += eps
    integral += eps * 1/x
    if integral > 1:
        break
print('actual:   %.5f' % e)
print('estimate: %.7f' % x)

Output:

actual:   2.7182818
estimate: 2.7182818

Estimating π

Estimating $\pi$ is easiest by using the definition of the area of a circle, $r^2 \pi$. We can randomly generate points $(x, y)$ in the square between $(-1, -1)$ and $(1, 1)$. The ratio of the number points such that $d^2 = x^2 + y^2 < r^2 = 1 $, ie. the points that fall within the circle with $r=1$, to the total number of points is equal to the ratio of the area of the circle $r^2 \pi$ to the area of the aquare $4r^2$:

from random import random
from math import pi

N = 10*1000*1000
Nc = sum([1 for _ in range(N) if random()**2 + random()**2 < 1])

print('actual:   %.5f' % pi)
print('estimate: %.5f' % (4 * Nc / N))

Output:

actual:   3.1412
estimate: 3.1416

As before, we can make it arbitrarily precise by using a grid:

eps = 0.0001
x = y = 0
Nc = N = 0
for _ in range(floor(1/eps)):
    for _ in range(floor(1/eps)):
        if x**2 + y**2 < 1: Nc += 1
        N += 1
        y += eps
    x += eps
    y = 0
print('actual:   %.5f' % pi)
print('estimate: %.5f' % (4 * Nc / N))

Output:

actual:   3.14159
estimate: 3.14199

Conclusion

Originally I wrote this article to demonstrate how very simple Monte Carlo simulations can be used to arrive at a reasonable estimate to these mathematical constants. This is true, but after I wrote the MC code, I realized that in most cases the approach can be re-formulated as a more efficient non-random grid search/integration. The grid methods are faster, use less (constant) storage, and the desired accuracy can be selected up front.

Common patterns in technical interviewing

2022-10-01T00:00:00+02:00

Introduction

Interviewing, both as an interviewee and interviewer, is part of everyday life in the tech industry. Median tenures are 2-4 years, it multiple interview loops to switch, so I estimate that at any given time at least half of all tech workers are doing one or the other. On the hiring side, typical hiring funnel conversion rates are 5-20%. (Of course, on average, the two sides' conversion rate must be exactly equal.) Even when not looking for a job, I believe in ABI - Always Be Interviewing; to be aware of opportunities, market rates and to stay sharp.

Interviewing is also a great learning opportunity: companies tend to compress a lot of what they think is important into interview loops. "They asked X, which I don't know much about; is this a blindspot for me? Why are we not doing X at my current job?"

As a hiring manager, I conduct about 5-10 interviews per month, and I have been doing so for the past 10+ years, so I think about these topics a lot. Although, per the above, this does not make special.

Here, I will attempt to enumerate all the categories of questions commonly asked in technical interview loops, and my experience with them.

Experience

A typical experience question is: "Walk me through a project you worked, what you did specifically on the project, what the end-result and impact of your work was." This is a great way for the candidate to showcase their past work and explain how they went over and beyond their roles to accomplish something awesome. However, there are many problems with experience-based questions:

Unfortunately, there is no way to verify what is being said. Even honest people can have selective memory and will bias the story in their own favor in an interview situation. It's also worth realizing that for many people, a job like Software Engineer at Google (in Silicon Valley or London) is a life-changing opportunity; life-changing not just for them, but for their family.
Candidates coming from lower tier companies (or coming from countries with a lower presence of tech companies) may have the right skills for the job, but may not have sexy projects under their belt to talk about. But experience questions discriminate against them, and increase pressure to inflate, which can then backfire.

Because of the above reasons, I don't ask any experience-based questions on my interview loops.

Baseline knowledge

The idea here is to test the candidate baseline knowledge on a topic. For example, if the candidate has Python on their CV and the role involves Python programming, implementing 2-3 line Python functions using if/elif/else, return, yield, for loops and basic Python functions like len() or range() demonstrates baseline Python knowledge. A non-coding example is asking the candidate to explain what the difference between pass-by-value and pass-by-reference is, and when does each happen in Python.

Although baseline knowledge questions seem too simple to be useful, they are not. The trick is to ask a lot of them, and cover breadth. I find that the right number of baseline knowledge questions give the strongest signal in technical interview loops.

Deep knowledge

Asking deep knowledge questions makes sense for more senior roles. An example coding task would be to implement a disk-persistent key-value store that support get(), set() and iteration, but is faster than linear. A non-coding example is asking about less common edge cases, tricky decorators, type hints, how Python runtimes differ.

It's worth noting that a senior candidate can be a Senior Generalist (has worked in many environment, can get up to speed quickly, can solve a wide variety of problems) vs Senior Specialist (has been coding in the same environment for 10 years, knows it inside-out). In my experience, big tech companies prefer to hire and grow Senior Specialists, since at 10k+ headcount and deep budgets they can afford to hire the best domain experts, there's less need for Senior Generalists.

As an example, I am a Senior Generalist: I have worked as a C++ programmer, as a Data Engineer, as a Data Scientist, I have managed all 3; but big tech doesn't care. If I interview for a Data Engineering role, I don't get any extra points for the other stuff.

Memory recall

These kind of questions ask the candidate to recall something that can be looked up. For example, a problem is given which can obviously be reduced to finding the shortest path in a directed weighted graph. Now, the question is whether the candidate can recall eg. Floyd's algorithm (and implement it).

I find these questions completely useless. Also, these questions heavily bias towards (i) candidates who have recently gone to school, and/or (ii) candidates who invest time to study for the interview loop. In my opinion, studying for the interview loop is not a problem, as it shows dedication. But the question is not meant to test (i) and (ii), it is meant to evaluate technical competence, which it does not.

Tricky question

Big tech companies have at various times used tricky questions in interview loops. A famous example is this Amazon question: There are two poles 50 meters apart, 30 meter tall. From the tops of the poles hangs a 40 meter rope between the poles. The lowest point of the rope is 10 meters from ground. How far are the poles? The trick is to realize that if the rope is 10 meters from the ground, and the poles are 30 meter high, then it takes just 2x20m=40m to get to the lowest point. Since the rope is 40 meters long, there is no play in the rope at all, it must go straight down, so the 2 poles must be right next to each other, 0 meters apart.

At Facebook, there are a lot of tricky coding questions that are variations of the Knapsack problem. If you know that Facebook interviews often have a question that is a knapsack, and do a few practice examples, it's easier to spot and solve; even though it's irrelevant to actual coding at Facebook.

I find these questions completely useless. Most candidates are nervous and stressed during the interview, potentially with high blood pressure, which impairs their cognitive abilities. Any questions which assumes a clear head will get low performance from people who tend to worry, even though this is unrelated to on-the-job performance (where people in general are not nervous and stressed).

Time challenge

When I was working at Facebook, one summer I and another senior engineer was tasked with building a simple interview loop for Engineering interns. We made the loop a coding loop, where they have to live code relatively simple Python function (2-5 lines each). There were hundreds of applications, which the recruiters narrowed down to 20. So we came up with a list of 10 coding questions for a 60 minute interview. We felt 10 questions is too many, we didn't expect candidates to get to more than 4-5 in the time allotted, but we wanted to make sure we don't run out of questions.

In the end there were multiple candidates who were easily able to complete all 10 questions within the 60 minute limit. It turns out that many of the intern candidates were competitive coders, so they were just able to write out the solutions, without having to think much about it. We the interviewers, although we were senior engineers, have never done competitive coding, so this was a surprise. The fastest candidate was able to solve all 10 in just 25 minutes. I remember thinking there is no way I could solve all 10 in 25 minutes...

In the end, although this was not the plan from the beginning, we ended up ranking them based on completion time, and hired the 2 candidates (fortunately it was a male and a female in the top 2 spots) who got all 10 right and were the fastest. (In reality there were 2 interviews, one for Python and one for SQL, and we combined the rankings.)

I would never do this on purpose (and have never repeated it), since in real life there is no time pressure when doing technical work.

Explicit pressure

In this scenario, explicit pressure is put on the candidate throughout the interview. Note that in the previous scenario, we evaluated candidates based on time, but we didn't put pressure on them during the interview.

Pressure can be created by nudging candidates to move fast(er), or by challenging their responses. When I was interviewing DE and DS candidates at Facebook, I had to follow a script, and as an interviewer I was expected to get throught the entire script in 60 minutes (even if it meant not getting a response for a part). The script, counting sub-items, consisted of 15-20 points. So as part of interview calibration, I was taught that if candidates are slow to get to a milestone in a question by a certain time, they have to be nudged, and eventually helped or somehow pushed along, since the entire script must be finished in the time alloted. The rationale was that different parts of the interview gave us signals on different attributes, and the hiring committee needs all these to make the call. If an interviewer didn't get through the script and didn't get all the signals, sometimes an additional (8th or 9th) interview would be scheduled with the candidate.

Pressure naturally occurs with other type of questions: deep knowledge questions (cannot remember decorator syntax), memory recall questions (cannot remember algorithm) and trick questions (cannot figure it out). For reasons given before, I don't use (or like) this pattern, in fact I do the opposite: I try to minimize pressure on the candidate as much as possible to get a realistic reading.

Bug/feature on existing code

Sometimes companies show the candidate a piece of code that is very close to what they would work on on the job, eg. a piece of Django code for a Python backend position at a company that uses Django. In the real world, most often you are not writing brand new code, but instead modifying existing code. So in this scenario you have to take an existing piece of code, which may have multiple bugs/issues to fix, and/or add new features to it. Although I don't use this type of question, I think this is a good way to gauge candidates for a variety of factors.

An interesting interview detail is whether the candidate is allowed to use Google (this appies to straighforward coding questions too). In my experience, most interview loops ask candidates not to use Google, and instead ask them to ask the interviewer if they have a question. In recent times I have experimented with asking candidates to share their screen when coding, and telling them it's okay to google — after all, we all constantly do it on the job. Interestingly, I found that this gives me valuable signal, but most often it's not in the candidate's favour: seeing how they google something (and chrome/google revealing past queries) communicates a lot. For example, does somebody google [how do I keep the first n characters of a string in python] or [python string prefix]. (To my surprise, many technical people don't know how to google effectively.)

Situational

Situational questions are no-code versions of the previous type: "It's Thursday 7p, you get an alert that the number of logins per minute on your SaaS has dropped significantly. What do you do?"

Situational questions are excellent for eg. senior backend, SRE or production engineer roles. Surprisingly, these questions are also great on Product Manager loops. The tricky thing is to get the amount and direction, guidance and nudging right. The candidate may not know what the interviewer is looking, and needs help to talk about the right topic at the right depth. For example, does the interviewer care in depth how the candidate sets up the crisis response team on Slack? or, is the interviewer want the candidate to talk more about which logs to check on the servers? A candidate may have the right depth in the desired portion, but without good prompting, she may misjudge the interviewer and contentrate on another portion.

Homework

This interview pattern is to ask the candidate to solve a 2-4 hour problem at home. I have been involved (on both sides) in multiple interview loops of this kind. On the hiring side, in all cases we eventually stopped giving homework assignments, because we had too many senior candidates who just didn't want to invest the time. Usually they didn't say so, they just never sent a solution, they silently churned.

The problem is, a good candidate, the kind you want to hire, will never spend 2-4 hours on a 2-4 hour problem. They will go overboard and will (want to) spend 20-40 hours on it. However, senior candidates will have a demanding full-time job and potentially family, so finding 20-40 hours over a week's period is tough.

Work with us for real

This is the best way to gauge a candidate: just have them come to the (remote) office, and work with us for a week as part of the interview loop. This sounds crazy, but at one startup I worked at, we did this for many years. That's also how I got the job, by working at the company for a full week, and presenting what I did Friday afternoon. I was coming off my previous failed startup and was unemployed, so I had the time. Technically, the candidate got payed for the week a competitive salary as a contractor. It goes without saying that this gives the maximum amount of accurate signal on the person, since the future team gets to see the full package, from the time the person arrives at the office, coding, having lunch, presenting, everything. The co-founders of the startup believed strongly in this system, but as the company matured it was first reduced to 3 days, then 1 day, then totally abandoned because of legal risks (and because many senior candidates were not willing to take a week off).

Conclusion

The goal in this first article on interviewing was to lay out (most of) the interview types for technical interviewing. I gave my own opinion, but I'm not looking to convince anyone. My opinion is based on what I've seen work, and what I feel is fair when I'm getting interviewed. Different roles at different companies require different interview loops.

How good a particle physicist is GPT-3?

2022-09-24T00:00:00+02:00

Recently, a Google engineer has made the claim that one of Google's proprietary internal AI systems called LaMDA is sentient. A more useful perspective on such advanced statistical AI systems is to say that they are an extremely useful, wonderful illusion.

You can also talk to GPT-3, just create an account on OpenAI and go to the sandbox!

Here I will show a "conversation" with GPT-3 to gauge how good a particle physicist — or an illusion of a particle physicist — it is. Mostly these are questions one might ask a student in an exam situation. The questions are independent of one another, so they work with a context-free system such as GPT-3. By context-free I mean that GPT-3 does not maintain conversation state, each question and answer are indepedent from the AI's perspective.

Also see these previous Bytepawn posts on GPT-3:

Q&A with GPT-3

Q: The Standard Model is a gauge theory with a U(1) x SU(2) x SU(3) symmetry. Is this SU(2) symmetry the same as the SU(2) symmetry of a fermion's spin?

A: No. The SU(2) symmetry of a fermion's spin is an internal symmetry, while the SU(2) symmetry of the Standard Model is a gauge symmetry.

This is a reasonably good answer.

Q: In particle physics, why is the weak force SU(2) and not U(2)? Why is the strong force SU(3) and not U(3)?

A: The U(1) subgroup of SU(2) is not a symmetry of the weak force, and the U(1) subgroup of SU(3) is not a symmetry of the strong force.

Here, GTP-3 is not answering the question, since it's talking about U(1). A better answer would be:

The matrix representation of U(n) has dimensionality $n^2$ (ie. n2 linearly independent matrices), whereas SU(n) has dimensionality $n^2-1$. Each linearly independent matrix corresponds to a gauge boson. So a gauge theory that is U(n) has $n^2$ gauge bosons, a gauge theory that is SU(n) has $n^2-1$. Note that U(1) is the same as SU(1). It is an experimental fact that:

electromagnetism has 1 gauge boson (the photon), so it's U(1)
the weak force has 3 gauge bosons (W+, W- and Z), so it's SU(2)
the strong force has 8 gauge bosons (8 gluons), so it's SU(3)

Q: In particle physics, why are we always using special unitary groups, like U(1), SU(2) and SU(3)?

A: There are many reasons for this. One reason is that special unitary groups are Lie groups, which means they are continuous groups with a well-defined algebra. This algebra can be used to construct representations of the group, which are useful for doing calculations in physics.

Another reason is that the special unitary groups are subgroups of the unitary group U(N), which is the group of all unitary matrices with determinant 1. The unitary group is important in physics because it is the group of all symmetry transformations that preserve the inner product of a vector space. The special unitary groups are important because they are the only subgroups of the unitary group that preserve the inner product up to a phase.

The special unitary groups are also important because they arise naturally in many physical situations. For example, the group SU(2) arises in the study of spin 1/2 particles, and the group SU(3) arises in the study of particles with color charge.

This is a reasonably good answer. On unitary operators, a better explanation would be:

Quantum mechanics (and QFT) is a probabilistic theory. When peforming measurements, there no certain outcomes, only probabilities. Furthermore, the "weights" associated with each outcome must be a complex number (complex amplitude), not a real number, because this is how we get our theory to reproduce the results of interference experiments (such as the two-slit experiment). So, the state of a quantum system is represented by a wave function, which consists of complex amplitudes, one for each possible measurement outcome. The sum of absolute squares of these amplitudes must sum to 1 — thus, we can imagine it like a unit vector in complex space. When we apply an operator to this state to (eg. the time evolution operator in the Schrodinger equation, or a gate in quantum computing, or a vertex in QFT), we want to preserve the property that the absolute squares sum to 1 — in other words, we want the operators to rotate around the unit complex sphere. Such operators are called unitary operators

Q: In particle physics, why does the strong force have 8 gauge bosons? There are 3 color charges, so why isn't there 3 gluons, one for each color charge?

A: The strong force has 8 gauge bosons because of the 3 color charges. The 3 gluons are the 3 color charges, so the strong force has 8 gauge bosons.

The answer is self-contradictory, in the first sentence it talks about 8 gauge bosons (the gauge bosons are the gluons for the strong force), but then in the second sentense it talks about 3 gluons. A good answer would be:

It is in general not true that there is one gluon for each charge. For the electromagnetic theory it's true, but it's an accident. Also for the weak force, where there are 3 gauge bosons, but there are not 3 weak charges. Also not that not all gauge bosons carry charge, examples: (i) QED's photon does not carry electric charge (or any charge) (ii) the weak force's Z boson does not carry any charge (iii) the W boson, although it's not part for QED (electron-proton-photons processes), carries an electric charge.

Q: What is the Hilbert space in Quantum mechanics?

A: The Hilbert space in quantum mechanics is the space of all possible states that a quantum system can be in.

The answer is not terrible, but could be more useful, like:

The complex vector space (with an inner product) where states describing a quantum mechanical system live (see above) is called a Hilbert space.

Q: In particle physics, what is the difference between helicity and chirality?

A: Helicity is a measure of the projection of the spin of a particle onto its momentum. Chirality is a measure of the handedness of a particle.

Good answer!

Q: In Quantum Field Theory, what is a current? What kind of currents are there?

A: In quantum field theory, a current is a conserved quantity associated with a symmetry of the Lagrangian. There are three types of currents in quantum field theory: vector currents, axial-vector currents, and tensor currents.

Good answer!

Q: In particle physics, what is the difference between the Cabbibo angle and the Kobayashi-Maskawa angle?

A: The Cabbibo angle is a parameter in the Standard Model of particle physics that governs the strength of the weak interaction between quarks. The Kobayashi-Maskawa angle is a parameter in the Standard Model that governs the strength of the weak interaction between quarks and leptons.

This is a bit of a trick question. There is such a thing as a Cabbibo angle, and Kobayashi-Maskawa are closely related, but there is no such thing as a Kobayashi-Maskawa angle; instead there is a Cabbibo-Kobayashi-Maskawa or CKM matrix. The CKM matrix describes the probability of a transition from one flavour quark to another flavour of quark, but not between quarks and leptons.

Q: What is the difference between the weak mixing angle and the Weinberg angle?

A: The weak mixing angle is a parameter in the Standard Model of particle physics that determines the strength of the weak force. The Weinberg angle is a parameter that determines the strength of the weak force in the electroweak theory.

This is a trick question — they are the same.

Q: In particle physics, for what kind of theories are Feynman-diagrams not useful?

A: In general, Feynman diagrams are not useful for theories that do not have a perturbative expansion. For example, in a theory with strong interactions, such as quantum chromodynamics (QCD), the Feynman diagrams become very complicated and are not very helpful in understanding the behavior of the theory.

Great answer!

Conclusion

Overall, 8 of the 10 of the GPT-3 answers are okay, which is very impressive. On some of the trickier questions, the answer is good english but incorrect or meaningless. This is the overall danger of GPT-3: if the reader is not an expert, s/he can't tell if they're getting a good answer (in this case, about half the time) or reasonable sounding non-sense.

Data Science Culture Doc

2022-09-23T00:00:00+02:00

Introduction

I wrote a Culture Doc for the Data Science team I lead. It consists of 24 values (or slogans) that is a good representation of what we believe, what we want to achieve, and how we want to get there.

It's no secret that a lot of the values are taken from other companies' culture docs (mainly Facebook and Amazon), with some re-wording; some of them are original.

The best-way to read is on the Data Science Culture mini-site, and I have also reproduced the document below.

Related Bytepawn posts worth checking out:

Data Science Culture Doc

The goal of this document is to solidify our culture by writing it down. The goal is not to create it — you cannot create culture by writing it down. Culture is people's existing everyday behaviour and communication, and which of those is rewarded by the organization.

Some of the phrasing we shamelessly stole from other companies' culture docs. As long as we're also living these values, it's fine. And Great artists steal is one of our values.

Mission

Focus on impact.
Jeff Bezos has said "customers are always beautifully, wonderfully dissatisfied, even when they report being happy and business is great". We believe that in the long-term Data Science can make customers more satisfied, make customers like our products more and in the process improve our revenues significantly.

Data is the new oil.
We have billions of rows of data about millions of customers. As long as our customers are receiving static emails about irrelevant offerings, can't find what they're looking for in our stores or struggle with widgets they never use in our apps — there is oil in the ground.

Our goal is not to get people to like us.
Our goal is to get customers to like our company's products. We are not our users. We build products that benefit the customer in the end.

We ❤ ️experiments
To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment.

Customers are more than data.
Build products around and for people, not data. We must never forget that data is a means to an end, and the "end" is to create better experiences for people.

Data wins arguments. Data enables us to win.
Data beats talking. Data beats slides. Data beats ties. Data beats being popular. Many of the important decisions we make can be made with data. A team making rational, high-quality decisions based on data and experiments will light a bonfire.

If we don't create the next version of our services, somebody else will.
We face great competition in the marketplace. Our team also faces great competition internally.

Do things that feel right. Measure beyond numbers.
It is dangerous to think everything can be measured, because it leads us to believe everything that cannot be measured isn't real. Emotions, feelings are real. People are not deterministic black boxes, nor independent, identically distributed random variables. People think, talk and feel. Building a sustainable brand means winning the trust, hearts and minds of people, not just their compulsive clicks (and buys).

Tactics

Great artists copy.
Steve Jobs invented the computer mouse by copying the idea from Xerox. We live with open eyes and an open mind and adopt good ideas we see elsewhere.

Do more with less.
Constraints breed resourcefulness, self-sufficiency and invention. We use simple but powerful tools to get our work done. There are no extra points for growing headcount or budget.

There is no point in having a 5 year plan.
Our industry is changing at a breakneck speed. We should have a 6 month plan, and know where we want to be in 5 years. And every 6 month, we take another look at where we are.

Ruthless prioritization.
Picking the right problem is more important than being able to solve problems. Focus on solving big problems. Focus on helping the most customers.

Stay focused and keep shipping.
Your product doesn't exist until it ships. And if it doesn't exist, it cannot have an impact. It doesn't matter how great your idea is if nobody is using it.

The quick shall inherit the earth.
Fast is better than slow. Fast means it's out in the real world. That means fast can learn from experience while slow can only theorize.

We treat reversible decisions as experiments.
Some decisions are consequential and irreversible or nearly irreversible – one-way doors – and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation (for example, picking the location of a new Carrefour hypermarket). But most decisions aren’t like that – they are changeable, reversible – they’re two-way doors (for example, the way we do code reviews). We don’t have to live with the consequences for that long. We can reopen the door and go back through.

Done is better than perfect.
Building a good enough product and shipping in 3 months is almost ways better than waiting to release the first version in 12 months. The learning and feedback from users in those 9 months will enable us to build a much better product in the end.

Team

Hire and develop the best.
We maintain a high bar and hire the best people to solve hard problems. You're good at what you do. The person in your team is also good at what they do. Insist on high standards. Talk to others. Don't pay too much attention to seniority and rank.

Clear, concise and consistent writing.
Good writing gets ideas noticed. It’s not true that only ideas matter. If your writing is sloppy, people will think you are the same. They won’t care about your message, they won’t work with you.

We are distributed.
We are a distributed team working from 5+ countries. We must be able to communicate efficiently online using asynchronous channels, while also creating opportunities to come together regularly and socialize. We prefer informal written communication (chat) to meetings and emails.

Always be learning and teaching.
We are some of the luckiest humans to ever be alive. We work in a high-demand field at the intersection of some of the most intellectually stimulating disciplines, and this field is continuously improving and re-inventing itself. We must always be reading and learning to keep up, and help others to learn and keep up. But, this is not a problem for us, since it's great fun.

We ❤ ️failures.
When something goes wrong, we don't blame each other, we remain patient, but quickly fix the problem. We do a post-mortem, identify root causes and look for systemic improvements. We do this for both social and technical issues.

We are not good at everything.
We don't have to be good at everything. For example, we are not good at Objective-C development or writing marketing copy, and we can be honest about it.

We trust each other.
We listen attentively, speak candidly, and treat others respectfully. We are vocally self-critical, even when doing so is awkward or embarrassing. We prefer transparency over office politics and information silos.

A team of humans.
We come together everyday to solve interesting problems, not to go skiing. We are a beautifully diverse team from many cultures, countries, religions, languages and family backgrounds. We respect each other, we help each other and we have fun while we solve our interesting problems. We serve each other and other teams.

How good a Data Scientist is GPT-3? - Part II

2022-09-03T00:00:00+02:00

In the previous blog post, I asked GPT-3 interview-level Data Science questions to gauge "how good a Data Scientist" it is. Here I continue this line of questioning, with questions that are about real life Data Science work I've done at previous jobs. These are still posed in the style of interview questions, but are a bit closer to how a Data Scientist might want to get help from GPT-3 on real Data Science work.

You can also talk to GPT-3, just create an account on OpenAI and go to the sandbox!

Product metrics

Q: Imagine you're a Product Analyst or Data Scientist working with a Product Manager at a social networking company like Facebook or Twitter. What kind of product metrics would you recommend the Product Manager look at?

A: There are many potential product metrics that a Product Manager could look at when working with a social networking company. Some examples of potential metrics could include:
- Number of active users
- Number of daily/weekly/monthly active users
- User engagement levels (e.g. time spent on site, number of page views, etc.)
- Number of new users
- Number of referrals
- Churn rate
- Demographics of users (e.g. age, gender, location, etc.)

Note: this is not a bad starting list. Having said that, demographics aren't metrics, but it's no big deal.

Q: Imagine you're a Product Analyst or Data Scientist working with a Product Manager at a social networking company like Facebook or Twitter. Give some examples of values of product metrics that might indicate a healthy or unhealthy product.

A: Some examples of values of product metrics that might indicate a healthy product are:
- The number of people using the product on a daily basis
- The number of people using the product on a monthly basis
- The number of people using the product on a yearly basis

Some examples of values of product metrics that might indicate an unhealthy product are:
- The number of people who have stopped using the product
- The number of people who have uninstalled the product
- The number of people who have given the product a low rating

Note: the first list is essentially worthless, the second one is good. Let's try the same question, but with more leading:

A: A healthy product metric might be something like DAU to MAU ratio being 1 in 10. This would indicate that the product is being used frequently enough. An unhealthy product metric might be something like a DAU to MAU ratio of 1 in 30. This would indicate that the product is not sticky enough and people are not using it as frequently as they should be.

Note: the answer is worthless mumbo-jumbo.

Q: Imagine you're a Product Analyst or Data Scientist working with a Product Manager. Your team owns a mobile app, imagine something like Twitter or Facebook. You are designing the Monthly Active User of MAU metric, which you will compute on a rolling lookback window basis. In other words, every day the metric counts the number of unique users in the last X days. You have a choice of picking X, you're options are 28 days, 30 days, 31 days. Which one would you pick and why?

A: 31 days. This is the longest month, so it will capture the most data.

Note: in a Product Analytics setting, usually the right number to pick is 28. Let's try the same question, but with more leading:

A: There are a few reasons why we might want to pick 28 days over the other options:
1. 28 days is a multiple of 7, which means that the metric will always count the same number of weekends. This can be helpful if we want to compare MAU week over week and don't want the metric to be skewed by differences in the number of weekends.
2. 28 days is a common month length, so it will be easy for people to understand and compare the MAU metric to other metrics that are also computed on a monthly basis.
3. 31 days is the longest possible month length, so choosing 28 days instead will help to reduce noise in the data.

Note: 1. is the expected answer here, and the reason internally we usually use a 28 day MAU definition. 2 and 3 are are false/mumbo-jumbo.

Checkout counters

Q: Imagine you are a Data Scientist. You want to build a model to recommend how many checkout counters should be open at a hypermarket, for upcoming days, in one hour slots. How would you go about it?

A: I would first look at the data to see if there are any patterns that can be observed. For example, are there more people shopping on certain days of the week or at certain times of the day? I would also look at the data to see how many people are typically checking out at the same time and see if there are any bottlenecks. Once I have a good understanding of the data, I would build a model that takes into account the number of people shopping and the number of people checking out at the same time.

A: The model pipeline would consist of the following parts:
1. Data cleaning and preparation: This would involve cleaning the data set to remove any invalid or missing data points.
2. Data exploration: This would involve exploring the data set to identify any patterns or trends.
3. Model training: This would involve training a model to predict the number of checkout counters that should be open for upcoming days, in one hour slots.
4. Model evaluation: This would involve evaluating the trained model to see how accurate it is in predicting the number of checkout counters that should be open for upcoming days, in one hour slots.

Note: the answer is not incorrect, but very generic. Also, what it describes are phases of a Data Scientist's work, but not the stages of a model pipeline.

Q: Imagine you are a Data Scientist. You want to build a model to recommend how many checkout counters should be open at a hypermarket, for upcoming days, in one hour slots. As a first step, you need to forecast the number of future checkouts, given historic data going back to 2018. How would you do this, given that we also had Covid-19 happening starting in 2020 May?

A: There are a few ways to approach this problem. One way would be to build a time series model that takes into account the effects of Covid-19 on the number of checkouts. Another way would be to build a model that predicts the number of checkouts based on a variety of factors, including the time of day, the day of the week, and whether or not there is a Covid-19 pandemic.

Note: while not incorrect, the answer contains no useful information.

A: There are many ways to approach this problem, but one possible approach would be to use a time series forecasting model such as SARIMAX. This model can take into account seasonality and trends in the data, as well as exogenous variables such as Covid-19.

Q: Imagine you are a Data Scientist. You want to build a model to recommend how many checkout counters should be open at a hypermarket, for upcoming days, in one hour slots. In this problem, how would you use an M/M/1 model from queueing theory?

A: If you were using an M/M/1 model from queueing theory, you would need to know the arrival rate of customers and the service rate of the checkout counters. With this information, you would be able to calculate the average number of customers in the system, the average time a customer spends in the system, and the average number of customers in the queue. From this, you would be able to recommend how many checkout counters should be open in order to minimize the average time a customer spends in the system.

Receipt scanning

Q: Imagine you're a data scientist. We want to build a model to parse out store, date, time and total amount from a picture of a receipt. How would you go about it?

A: There are a few ways to tackle this problem:
1. Use Optical Character Recognition (OCR) to extract the text from the image. This can be a difficult problem on its own, but there are many commercial and open-source OCR libraries available.
2. Use a trained machine learning model to classify the text in the image. This would require a lot of training data, but could be more accurate than OCR.
3. Use a combination of OCR and heuristics to parse out the relevant information. For example, you could use OCR to extract all the text from the image and then use heuristics to look for patterns that match store names, dates, times, and total amounts.

Q: Imagine you're a data scientist. We want to build a model to parse out store, date, time and total amount from a picture of a receipt. Suppose you have an OCR program that can extract the text in the image. How would you use it to build this model?

A: There are a few ways to build this model. One way would be to use a regular expression to parse out the store, date, time and total amount from the text. Another way would be to use a machine learning algorithm to learn the patterns in the data and then predict the store, date, time and total amount from new data.

Note: the fact that it recommends regular expressions is very cool. When I was working on this problem, the baseline version I was improving on, built by a previous team, was using regular expressions!

A: There are many ways to approach this problem, but some common libraries for machine learning include TensorFlow, scikit-learn, and Keras.

Note: since it recommends scikit-learn, let's ask about it.

A: There are a few ways to approach this problem. One way would be to use a regex model to extract the relevant information from the text. Another way would be to use a series of scikit-learn models to extract the store, date, time, and total amount separately.

Note: the answer is not specific to scikit-learn.

Q: Imagine you're a data scientist. We want to build a model to parse out store, date, time and total amount from a picture of a receipt. Suppose you have an OCR program that can extract bounding boxes and the text in each bounding box from the image. How would you use this to solve the problem?

A: There are a few different ways to solve this problem. One way would be to use the OCR program to extract the text in each bounding box, and then use a regular expression to parse out the store, date, time and total amount from the text. Another way would be to use the OCR program to extract the bounding boxes, and then use a computer vision algorithm to parse out the store, date, time and total amount from the image.

Note: the answer is not very useful. A more useful answer would be about using the positions and alignments of the bounding boxes to predict which contains useful information, or adding up individual amounts to increase confidence about total amount.

Conclusion

In this round, GPT-3 did significantly worse than on the previous, more straightforward interview questions. Based on these sessions, I don't think it's valuable as an assistant for Data Scientists in real world settings.

How good a Data Scientist is GPT-3?

2022-07-31T00:00:00+02:00

You can also talk to GPT-3, just create an account on OpenAI and go to the sandbox!

Here I will show a "conversation" with GPT-3 to gauge how good a Data Scientist — or an illusion of a Data Scientist — it is. Mostly these are questions I might ask a candidate in a Data Science interview. The questions are independent of one another, so they work with a context-free system such as GPT-3. By context-free I mean that GPT-3 does not maintain conversation state, each question and answer are indepedent from the AI's perspective.

Culture

Q: What does a good Data Scientist look like?

A: There is no one-size-fits-all answer to this question, as the ideal data scientist depends on the specific needs of the organization. However, some key qualities that a good data scientist should possess include strong analytical and problem-solving skills, the ability to think creatively, and experience with data-driven decision making. Additionally, a good data scientist should be able to effectively communicate their findings to both technical and non-technical audiences.

Note: This was an easy question because there are lots of sentences like this on the Internet, and the same applies to the following questions. The answer is still impressive and creates a strong illusion, because the quality of the answer is so strong, and fits the structure of the question.

Q: What's the difference between a Data Scientist and a Data Engineer?

A: A data engineer is responsible for designing, building, and maintaining the data infrastructure that a company uses to collect, store, and process data. A data scientist is responsible for analyzing and extracting insights from data.

Q: What's the difference between a Data Scientist and a Product Analyst?

A: A data scientist is someone who is responsible for extracting meaning from data. A product analyst is someone who is responsible for analyzing a product and determining its feasibility.

Q: What's the difference between a Data Scientist and an ML Engineer?

A: A Data Scientist is someone who is responsible for extracting insights from data. An ML Engineer is someone who is responsible for building and deploying machine learning models.

Q: In a tech company working on a SaaS product, what's a good ratio of Data Scientists and Product Managers?

A: There is no definitive answer to this question since it can vary depending on the specific needs of the company and product. However, a good rule of thumb is to have at least one Data Scientist for every two Product Managers.

Q: In a tech company working on a SaaS product, what's a good ratio of Data Scientists, Data Engineers, Software Engineers and Product Managers?

A: There is no definitive answer to this question, as it will vary depending on the specific needs of the company and product. However, a good general ratio to aim for is 1 Data Scientist, 1 Data Engineer, 2 Software Engineers, and 1 Product Manager. This will ensure that there is enough expertise and manpower dedicated to each area, while still allowing for some overlap and collaboration between teams.

Note: the AI is contradicting itself; in the first question, it recommended a 1:2 DS:PM ratio, here it's recommending a 1:1 ratio. I'm not sure where the 1:1 is coming from, in my experience at tech companies a 1:1 ratio would be too many DEs.

Coding

Q: Given a list li, write a function top_sum(li, n) that returns the sum of the top n elements: top_sum(li, n). Eg. top_sum(li=[5, 1, 5, 2, 0], n=3) returns 12.

def top_sum(li, n):
  sorted_list = sorted(li, reverse=True)
  return sum(sorted_list[:n])

Q: Given two lists keys and values of identical length containing key-value pairs, write a generator function yield_sorted_kvs(keys, values) which yields key-value pairs from the two lists, in ascending key order. Note that the input list keys may not be ordered. Eg. yield_sorted_kvs(keys=[3, 1, 2], values=[‘a’, None, 5]) yields (1, None) .. (2, 5) .. (3, ‘a’).

def yield_sorted_kvs(keys, values):
    d = {k: v for k, v in zip(keys, values)}
    for k in sorted(d.keys()):
        yield k, d[k]

Note: GPT-3 nailed both, even though these exact top_sum() and yield_sorted_kvs() functions are not on the Internet. Also note that GPT-3 is not specifically trained to write code, being able to implement small functions like this is just an "accidental sife-effect". OpenAI is working on a model called OpenAI Codex, specifically to write code.

Q: Can you show me some Python code that you consider beautiful?

def main():
    print("Hello, world!")

if __name__ == "__main__":
    main()

Note: This is a very weak answer, it breaks the illusion. Let's try again.

Q: Can you show me some Python code that you consider beautiful, that implements a famous algorithm of your choosing?

def fft(x):
    N = len(x)
    if N <= 1: return x
    even = fft(x[0::2])
    odd =  fft(x[1::2])
    T= [exp(-2j*pi*k/N)*odd[k] for k in range(N//2)]
    return [even[k] + T[k] for k in range(N//2)] + \
           [even[k] - T[k] for k in range(N//2)]

Note: This is a great answer, but it's copy/paste from this Stackoverflow answer. It would be interesting to know why the AI picked FFT, and this particular implementation!

Q: Can you show me some SQL code that you consider beautiful?

A: This is a difficult question since beauty is subjective. However, here is an example of SQL code that is well written and easy to read:

SELECT
    first_name,
    last_name,
    email
FROM
    users
WHERE
    active = 1
ORDER BY
    last_name,
    first_name

Note: this is actually a great answer. I agree with GPT-3 that the only way to interpret "beautiful" in the context of SQL is formatting, and this is nicely formatted SQL code. I follow the same formatting. But the illusion would work better if the sample query would be a bit longer:

Q: Can you show me some SQL code that you consider beautiful, that involves use of the WITH clause and also a JOIN?

WITH user_sessions AS (
  SELECT user_id, session_id, start_time, end_time
  FROM sessions
  WHERE user_id = 1
)
SELECT u.username, s.session_id, s.start_time, s.end_time
FROM users u
JOIN user_sessions s ON u.user_id = s.user_id

Note: here the illusion breaks, because now it's using a different style of formatting!

A/B testing

Q: Let's say we're running an A/B test with two variants. How should I split traffic to maximize the probability of getting a statistically significant result?

A: The traffic should be split evenly between the variants.

Note: a lot of people don't know this in interviews.

Q: Let's say we're running an A/B test with two variants. If we split traffic evenly between A and B, we maximize the probability of getting a statistically significant result. However, in real life usually traffic is not split evenly. Why?

A: There are many reasons why traffic might not be split evenly in a real-life A/B test. For example, if the variants are assigned randomly, there is a chance that one variant will be assigned to more users than the other. In addition, some users may be more likely to use one variant over the other, or one variant may be more popular than the other.

Note: this one is surprising. The answer I was expecting was around managing the risk of trying a new version. Let's see if we can lead it.

A: The reason is that if the new variant is worse than the old variant, we want to minimize the number of users who experience a worse experience. If the new variant is better than the old variant, we want to quickly find out and roll it out to all users. So in most A/B tests, the new variant gets a small amount of traffic at first. If the results look promising, the new variant gets more traffic. If the results are not promising, the new variant gets less traffic.

Note: by priming GPT-3 in the question, we were able to direct it to the expected answer.

Q: Let's say we're running an A/B test with two variants of an email. The evaluation metric is CTR. One branch has a CTR of 4%, the other branch has a CTR of 91%. What do you think?

A: There is a significant difference in CTR between the two variants of the email. The branch with a CTR of 91% is likely to be the more successful email.

Note: while technically correct, this is not a strong answer. A strong(er) answer would be to say that this is an unlikely result, almost certainly due to a bug or measurement error. See Twyman's law.

Q: Can you explain to me what a p-value is in A/B testing?

A: A p-value is the probability that the results of a hypothesis test are due to chance. A low p-value indicates that the results are not due to chance, and are therefore significant.

Q: Can you explain to me what the difference between frequentist and bayesian A/B testing is?

A: Frequentist A/B testing is based on the idea of hypothesis testing, where we test a null hypothesis (usually that there is no difference between the two groups) against an alternative hypothesis (usually that there is a difference between the two groups). We then calculate a p-value, which is the probability of seeing our data if the null hypothesis were true. If the p-value is less than a certain threshold (usually 0.05), we reject the null hypothesis and conclude that there is a difference between the two groups.

Bayesian A/B testing is based on the idea of posterior inference, where we use the data to calculate the posterior probability of the null hypothesis being true. We then compare this posterior probability to a prior probability, which represents our beliefs about the null hypothesis before we saw the data. If the posterior probability is greater than the prior probability, we conclude that the null hypothesis is more likely to be true.

Note: the last 2 questions were simple, but the illusion is still impressive!

Q: Can you recommend a good webpage to me that explains the difference between frequentist and bayesian A/B testing?

A: There is a lot of information out there on this topic. However, this webpage provides a clear explanation of the differences between frequentist and bayesian A/B testing: https://www.evanmiller.org/bayesian-ab-testing.html

Note: here the illusion is strong again, I would also recommend Evan's blog on this topic!

Q: In the context of A/B testing, can you explain to me how CUPED uses historical data?

A: CUPED uses historical data to help identify which variation of a given test is performing better. By looking at past performance, CUPED can more accurately predict which variation is more likely to lead to a successful outcome for the current test. This helps to ensure that the results of the test are more reliable and accurate.

Note: this is a very generic answer. It's interesting to note that originally I had a typo in my question, I wrote "used" instead of "uses". This totally tripped up GPT-3 and yielded a very generic answer:

Q: In the context of A/B testing, can you explain to me how CUPED used historical data?

A: CUPED used historical data to analyze the conversion rates for each variation of their A/B test. They then used this data to determine which variation was more effective in converting users.

Conclusion

Overall I find GPT-3 to be very impressive. It performs better as a question answering bot than many human Data Scientists I interview. Although this AI is not yet good enough to take away any technical contributor's job, it's not hard to imagine systems that are trained more specifically for a domain (such as Data Science and programming) and are potentially 10-100x bigger.

I will continue this post with a Part II, where I will ask it modeling questions. Overall, I am very impressed by GPT-3.

The Elements of Typographic Style

2022-07-23T00:00:00+02:00

The Elements of Typographic Style by Robert Bringhurst is the most beautiful book I've ever held in my hand. This stunning piece of readable art shows Bringhurst's love for the craft of design, typography and writing, and his mastery of these subject, a result of his life-long devotion to them. I am not a professional typographer, but I enjoyed glancing at, reading and appreciating every page of this book. I recommend you order your copy right now. I recommend the softcover version.

Sample pages

To increase your appetite, here are some sample pages from the book. These are from a scan of the previous edition, the actual book is much nicer due to the higher resolution of print (about 5-10x higher than your screen).

Typography

The text face is Minion Pro, designed by Robert Slimbach.
The captions are set in Scala Sans, designed by Martin Majoor.
The paper is Glatfelter Laid, archival quality and acid-free.

Quotes

Some of my favorite quotes from the book, which apply to not just to typography.

By all means break the rules, and break them beautifully, deliberately and well.

Using what there is to best advantage almost always means using less than what is available.

Consistency is one of the forms of beauty. Contrast is another.

Choose [your library of faces] slowly and well. Stay with your first choices long enough to learn their virtues and limitations before you move on.

With type as with philosophy, music and food, it is better to have a little of the best than to be swamped with the derivative, the careless, the routine.

Every alphabet is a culture.

On mathematics in typography: The mathematics are not here to impose drudgery upon anyone. On the contrary, they are here entirely for pleasure. They are here for the pleasure of those who like to examine what they are doing, or what they might do or have already done, perhaps in the hope of doing it still better.

Don't restate the obvious [regarding the use of running headers in books].

Architects build perfectly proportioned kitchens, living rooms and bedrooms in which their clients will make, among other things, a mess. Typographers likewise build perfectly proportioned pages, then distort them on demand. The text takes precedence over the purity of the design.

The state of the art has more by far to do with the knowledge and skill of its practitioners than with the subtleties of their tools, but tools can constrain that skill or set if free. The limitations of the tools are therefore also of some interest.

Like a forest or a garden or a field, an honest page of letters can absorb – and will repay – all the attention it is given. Much type now, however, is delivered to computer screens. The best computer monitors now sold have barely adequate resolution (220 dpi: roughly a third the current norm for laser printers and less than a tenth of the norm for professional offset printing). When texts disintegrate into pixels, the eye goes looking for distraction, which the screen is all too able to provide... The underlying problem is that the scvreen mimics the sky instead of the earth. Note: a flagship phone in 2022 has approximately 500 pixels per inch. The 4K monitor I'm typing on has abour 200 pixels per inch.

The Culture Map

2022-07-22T00:00:00+02:00

The Culture Map by Erin Meyer is a great business management book on how to communicate and manage people from different cultures, or just people with different ways of working. These are my notes of the book, which I recommend you should buy right now.

Introduction

The Culture Map by Erin Meyer is a system of 8 scales which can be used to determine how cultures vary along a spectrum. The scales can be used to analyse one culture relative to another and decode how culture influences your international collaborations. If you take the example of an Israeli executive who has been appointed to manage a newly purchased factory in Russia, and compare where both countries are on the scale, you can see where difficulties could arise.

The author's website has a non-free Culture Mapping Tool that draws the above plot for the countries selected, ie. the country composition of your team(s).

Let's now look at the 8 dimensions considered in the book.

Communicating

Low context: say it like it is, precise, simple clear, intended to be taken at face value.
High context: messages are nuanced, impled, need to be "decoded" or read between the lines.

Americans are the most explicit or low-context culture there is (low-context meaning their conversation assumes relatively little intuitive understanding). This is not surprising for a young country composed of immigrants that prides itself on straight-talking. Japan and other East Asian countries represent the other extreme.

Meyer offers strategies for negotiating these differences, but the most basic solution, as with all scales discussed in the book, is simply to be aware. Thus Americans in Japan should pay attention to what's not being said; while Japanese in America should brace themselves for direct language.

Evaluating

How directly (negative) feedback is given. Cultures that are direct wrt communication are not always direct wrt feedback, eg. US.

Americans may be very explicit communicators, but they are in the middle of the spectrum when it comes to giving negative feedback — as anyone who as been to an American school knows. Israelis, Russians, and Dutch are among the most direct when it comes to negative feedback. Japanese are among the most indirect.

Persuading

Concept first: like in a prussian school, first learn the theory, then apply it. In a business setting, this means first give theoretical arguments, then get to the actual business application.
Application first: focus on solving the problem at hand, and worry about the theory later. In a business setting, this means starting with the proposed action item, and then explaning why it's a good idea.

Some cultures, notably the French and Italians, tend toward deductive arguments, focusing on theories and complex concepts before presenting a fact, statement, or opinion. Others, notably Anglo-Saxon cultures, tend toward inductive arguments, starting with focusing first on practical application before moving to theory.

This trait shows up in everything from how people give presentations or lead meetings to how they write emails.

Leading

Egalitarian: the manager is a member of the team, a servant to the team. Decisions are based with more involvement from the team. Flat structure, titles are not that important internally.
Hierarchical: the manager is a boss, and tells team members what to do—and team members expect to be told what to do. Status and titles are important.

"In Denmark, it is understood that the managing director is one of the guys, just two small steps up from the janitor," a Danish executive told Meyer. This represents one extreme in attitudes toward leadership. On the other side of the spectrum in countries like Japan and Korea, however, the ideal boss should stand far above the workers at the top of a hierarchy. America's outlook on leadership falls somewhere in the middle.

Deciding

Consensual: team makes decisions together, as a quorum.
Top-down: the boss makes decisions and lets everybody know.

How organizations make decisions relates closely to how they view leadership, but with some important differences.

Notably, while Japan has a very hierarchical leadership system, it has a very consensual decision-making system. This is the famous ringi system, which involves building consensus at a lower level before bringing a proposal to a higher level, thus enabling broad corporate consensus.

Note: it's possible to have (a) an egalitarian team, where the manager doesn't tell people exactly what to do. It's possible to have a hierarchical structure with (b) consensual decision making, or (c) top-down decision making. In case (a) decision making on most issues is delegated to team members because of trust. In (b), decisions are made together, but then the managers tells everybody what to do. In (c), the manager decides on his own and then tells everybody what to do.

Trusting

Task-based: business relationships are just that, and do not require or assume personal friendship. Business partners may not trust each other (completely), but believe that contracts and the legal system creates boundary conditions that channels the other party to do their part and protect them if not.
Relationship-based: business relationships are only established with or after creating a personal connection and some personal trust, like getting to know each other, having lunch or dinner, going out for drinks, or in some cultures, getting connected to each other through trusted parties.

In some cultures, notably America, people don't worry so much about trusting each other because they trust their legal system to enforce contracts, and so business negotiations focus on what's practical.

In others, including many emerging market economies but also to a lesser extent Western Europe, personal relationships are much more important, in part because people don't trust their legal system to enforce contracts.

Disagreeing

Confrontational: open confrontation/disagreement is okay, and won't hurt the professional/personal relationship.
Non-confrontational: open confrontation/disagreement is seen as unproductive and/or impolite.

Some cultures embrace confrontation while others avoid it. This scale looks a lot like the scale showing the directness of negative feedback, though with some differences, such as Sweden being further to the left (direct) on negative feedback and further to the right (avoiding confrontation) on disagreeing.

Scheduling

Linear time: 10AM means 10AM, a few minutes late is okay. Time estimates are assumed to be somewhat reliable and taken seriously.
Flexible time: 10AM can mean anything from 10AM to 12PM. Appointment timings and time estimates should not be taken too seriously.

That different cultures treat time differently is one of the most common observations for anyone working or even traveling abroad. On one extreme you've got the exceedingly precise Germans and Swiss; Americans fall relatively close to this end of the spectrum; Western Europeans and Latin Americans tend to be more flexible; Africa, the Middle East, and India are extremely flexible.

Working Backwards

2022-07-10T00:00:00+02:00

This is a great business management book. These are my notes of the book, which I recommend you should buy right now.

Leadership principles and mechanisms

Amazon established a set of principles and mechanisms, enabling the company to grow from a single founder to several hundred thousand employees while remaining stubbornly true to its mission of obsessing over customers to create long-term shareholder value. Per Jeff Bezos:

You can write down your corporate culture, but when you do so, you’re discovering it, uncovering it—not creating it.

Amazon’s 14 Leadership Principles:

Customer Obsession: Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.
Ownership: Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say “that’s not my job.”
Invent and Simplify: Leaders expect and require innovation and invention from their teams and always find ways to simplify. They are externally aware, look for new ideas from everywhere, and are not limited by “not invented here”. Because we do new things, we accept that we may be misunderstood for long periods of time.
Leaders Are Right, A Lot: Leaders are right a lot. They have strong judgement and good instincts. They seek diverse perspectives and work to disconfirm their beliefs.
Learn and Be Curious: Leaders are never done learning and always seek to improve themselves. They are curious about new possibilities and act to explore them.
Hire and Develop the Best: Leaders raise the performance bar with every hire and promotion. They recognise people with exceptional talent and willingly move them throughout the organisation. Leaders develop leaders and are serious about their role in coaching others. We work on behalf of our people to invent mechanisms for development like Career Choice.
Insist on the Highest Standards: Leaders have relentlessly high standards – many people may think these standards are unreasonably high. Leaders are continually raising the bar and driving their teams to deliver high quality products, services and processes. Leaders ensure that defects do not get sent down the line and that problems are fixed so they stay fixed.
Think Big: Thinking small is a self-fulfilling prophecy. Leaders create and communicate a bold direction that inspires results. They think differently and look around corners for ways to serve customers.
Bias for Action: Speed matters in business. Many decisions and actions are reversible and do not need extensive study. We value calculated risk taking.
Frugality: Accomplish more with less. Constraints breed resourcefulness, self-sufficiency and invention. There are no extra points for growing headcount, budget size or fixed expense.
Earn Trust: Leaders listen attentively, speak candidly, and treat others respectfully. They are vocally self-critical, even when doing so is awkward or embarrassing. Leaders do not believe their or their team’s body odour smells of perfume. They benchmark themselves and their teams against the best.
Dive Deep: Leaders operate at all levels, stay connected to the details, audit frequently, and are sceptical when metrics and anecdote differ. No task is beneath them.
Have Backbone; Disagree and Commit: Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting. Leaders have conviction and are tenacious. They do not compromise for the sake of social cohesion. Once a decision is determined, they commit wholly.
Deliver Results: Leaders focus on the key inputs for their business and deliver them with the right quality and in a timely fashion. Despite setbacks, they rise to the occasion and never compromise.

Good intentions don't work; mechanisms do. No company can rely on good intentions. Amazon has put in place mechanisms to ensure that the Leadership Principles translate into action. Three foundational mechanisms:

Annual planning process: starts previous year summer and requires 1-2 months of managers' time. Executive team comes up with high level goals like "Grow revenue from $10M to $15M", which are then cascaded down the orgs. All goals are SMART goals. Each org builds its own goals and operating plan, which includes:

assessment of past performance, including goals achieves, goals missed, and lessons learned
key initiatives for the coming year
detailed income statement
requests and justifications for resources (hires, marketing spend, equipment, etc)

After the year completes, in January, the goals and plans are adjusted for what happened in Q4 in a phase 2. Once this narrative (goals and operating plan) is locked, changing it requires executive team approval.

Executive team's goals process: certain goals from orgs are selected by the executive team and elevated to company goal status, plus they come up with their own. Executive goals can number in the 100s at Amazon! These are mainly input-focused metrics that measure certain activities teams are performing during the year. Eg. "add 1,000 products to Store X" or "99.99% of API calls to service X are complete within 10ms". Executive team goals are tracked by the Finance dept and reviewed quarterly (red, yellow, green).

Compensation plan: re-inforces long-term thinking. The bulk of employee's compensation is always stock (vs base income or cash bonus) to avoid misalignment: (i) by rewarding short-term goals at the expense of the long-term and (ii) by rewarding localized department achievements irrespective of whether they're good for the company.

Hiring: Bar raiser process

When you consider the potential positive and negative impacts of an important hire, not to mention the precious time dedicated to it, it is shocking to consider how little rigor and analysis most companies put into their hiring process.

Bad hiring comes from:

personal bias: surround yourself with people similar to you
hiring urgency: filling headcount just to look good
relying on gut feeling, "I like him/her"

Good hiring: scalable, repeatable, formal process for consistently making appropriate and successful hiring decisions. Easy to teach to new people, has feedback loop to ensure continuous improvement.

Bar raisers: group of "senior" interviewers who must be involved in all hiring, and have veto power over the hiring manager. The bar raiser must come from outside of the team that is hiring.

Amazon hiring process:

Job description: You cannot hire the right person for the job if you don’t have a clearly defined job description. This is an essential point-of-reference for interviewers. A good description must be specific and focused.
Resume Review: Resumes should meet the requirements spelled out in the job description.
Phone Screen: 45-60 minute call with the hiring manager (once resumes are screened). Based on initial data, hiring manager decides if they would be inclined to hire the candidate based on preliminary data. If so, the candidate will be invited to the next step.
In-house interview: No participants will be more than one level below the position of the candidate’s position. Pool of interviewers should not include the prospective boss.
Written Feedback: Interviewers generate detailed notes as close to a verbatim record as possible. Written feedback must be thorough, detailed, and specific. The report should be written shortly after completion of the interview. Oral feedback is unacceptable. Written feedback includes hiring recommendation based on four possible options: strongly inclined to hire, inclined to hire, not inclined to hire, or strongly not inclined to hire. Interviewers do not discuss their feedback until submitting their own report (to avoid bias).
Debrief/Hiring Meeting: After written feedback is submitted, the interview team meets to debrief and make the hiring decision. Team reviews interview feedback and has a chance to change their vote based on the cumulative information.
Reference check
Offer

Every new hire should “raise the bar,” that is, be better in one important way (or more) than the other members of the team they join.

Behavioral interviewing: The process of evaluating a candidate’s past behavior and their compatibility with Amazon’s Leadership Principles. Each interviewer is assigned one or more of the 14 Principles to focus on in their interview (in order to learn how specifically the candidate aligns with the assigned principle). Questions are mapped to assigned principles.

Example: “Can you give me an example of a time when your team proposed to launch a new product or initiative and you pushed back on their plan because you didn’t think it was good enough?”

STAR questions (Situation, Task, Action, Result):

What was the situation?
What were you tasked with?
What actions did you take?
What was the result?

Organizing: Seperable, Single-threaded leadership

Velocity, a measure of speed and direction, is critical for a business. With all other things being equal, the organization that moves faster will innovate more, simply because it will be able to conduct a higher number of experiments per unit of time.

Dependencies can slow down innovation and the rate at which corporate teams operate. When a team cannot operate independently, its progress slows to whatever or wherever the gating dependency resides. As organizations become more interdependent and complex, an inordinate amount of time is spent on internal communication and coordination.

Too much of any kind of dependency not only slows down the pace of innovation but also creates a dispiriting second-order effect: disempowered teams. Amazon determined that improving coordination and communication doesn’t resolve the problem of dependencies. Amazon instead looked to eliminate dependencies. Per Jeff Bezos: Eliminate communication rather than encourage it in order to make Amazon a place where builders can build.

Single-Threaded Leadership: is Amazon’s mechanism whereby individuals are responsible for single, focused initiatives. These leaders have specialized responsibilities rather than a broad set of responsibilities. They run teams that are largely autonomous.

Two-Pizza Teams: An early Amazon approach to creating more autonomous teams. Teams were no bigger than the number of people who could be fed by two large pizzas (no more than 10 people). Amazon’s software architecture was highly influenced by this approach (e.g. modular APIs for services and data).

Autonomous teams are built for speed. Autonomous teams should have a clearly defined purpose, clearly defined boundaries, and useful metrics for tracking progress.It’s up to the team to figure out the specifics of how they will achieve their goal. The most successful teams made initial investments in removing dependencies and building infrastructure and instrumentation before adding new features (aka they laid the groundwork for future innovation).

Per Jeff Bezos: Most decisions should probably be made with somewhere around 70% of the information you wish you had. If you wait for 90%, in most cases, you’re probably being slow...if you’re good at course correcting, being wrong may be less costly than you think, whereas being slow is going to be expensive for sure. Be stubborn on the vision but flexible on the details.

Single-Threaded Leaders: Leaders whose focus is to get a specific job done. They don’t work on anything else. Separable, single-threaded teams have fewer organizational dependencies than conventional teams. The best way to fail at inventing something is by making it somebody’s part-time job.

Communication: Narratives and the 6-pager

Amazon relies far more on the written word to develop and communicate ideas than most companies, and this difference makes for a huge competitive advantage. Unlike most companies, Amazon does not use PowerPoint (or any kind of presentation software). Better informed people make higher-quality decisions, and can deliver better, more detailed feedback.

Written narratives are the primary communication tool used for proposals, plans, and process documentation. The company uses two main types of narrative:

The six-pager: A document used to describe, review, or propose an idea, process or business.
The PR/FAQ: A document used to develop and iterate on product ideas.

Per Jeff Bezos: The reason writing a good 4 page memo is harder than writing a 20 page PowerPoint is because the narrative structure of a good memo forces better thought and better understanding of what’s more important than what, and how things are related. PowerPoint-style presentations somehow give permission to gloss over ideas, flatten out any sense of relative importance, and ignore the interconnectedness of ideas.

Benefits of an effective six-pager:

Six pages is optimal for a 60 minute meeting where the meeting participants spend the first 20 minutes collectively reading the document individually (in silence while taking notes).
Document contains all essential information. Participants can review presentation, uninterrupted, in its entirety. The document is standalone and doesn’t require a presenter to fill in the blanks.
The document is portable and scalable. It can be circulated easily and read by anyone at any time.
Anyone can edit or add notes to the document.
The document serves as its own record.
Written narratives contain 7-9 times the information density of a PowerPoint.
People read three times faster than the typical presenter can talk.

A complete narrative should also anticipate the likely objections, concerns, and alternate points of view that we expect our team to deliver. Writers will be forced to anticipate smart questions, reasonable objections, even common misunderstandings—and to address them proactively in their narrative document.

The six-pager can be used to explore any argument or idea you want to present to a group of people—an investment, a potential acquisition, a new product or feature, a monthly or quarterly business update, an operating plan, or even an idea on how to improve the food at the company cafeteria.

Meeting format:

Presenter does not verbally repeat the document.
20 minutes of reading the narrative.
40 minutes of discussion.
Someone (not the presenter) should be charged with taking notes on behalf of the audience.

Start with the desired costumer experience

Amazon's approach is to product and business development is to start by defining the customer experience. Once the desired experience is defined, Amazon works backwards from there to figure out how to realize said experience.

PR/FAQ is a document that imagines a product release has occurred. The employee writes a press release that details the product, its features, pricing. rationale, and more. The FAQ portion answers questions that both customers and outside observers will have as well as questions that internal peers and managers might have.

Most companies work forward: leaders define the product and try generate consumer interest. This is the classic “a solution in search of a problem” mentality. PR/FAQs are less expensive product experiments. Teams can determine whether a project is worthwhile or not without spending precious R&D money in advance.

Basic format (follows the 6-page rule for 1-hour meetings):

1 page press release.
5 pages or fewer for frequently asked questions.

Press release format:

Heading (one sentence): “Blue Corp. announces the launch of Melinda, the smart mailbox.”
Subheading (on sentence) describing the benefits: “Melinda is the physical mailbox designed to securely receive and keep safe all your e-commerce and grocery deliveries.”
Summary paragraph: Give a more detailed summary of the product and benefit.
Problem paragraph: Describe the problem the service solves from the point of view of the customer.
Solution paragraph: Describe how the product solves the customer problem.
Quotes and getting started paragraph: Describe how a customer can obtain the product and how much it will cost. Include one quote from a company spokesperson and one quote from a hypothetical customer.

Sample FAQ questions:

How many consumers have this need or problem?
How big is the need?
How many consumers are willing to spend money to solve this problem?How much money will they spend?
What are the unit economics of the device?
What is the rationale for the price point?
What is the initial up-front investment required to build the product?
If 3rd party adoption is needed, how will they be induced to use the product?
Are there 3rd party technologies required?
What are the challenging engineering problems to solve?
What are the customer UI issues?
How can we manage the risk of the up-front investment needed?

Most PR/FAQs never make it past the exploration stage. This is a feature not a bug. Rejecting projects via the PR/FAQ process is far less expensive than actually developing failed products.

Metrics: Manage your inputs, not your outputs

Focus on the controllable input metrics, the activities you directly control, which ultimately affect output metrics such as share price. Output metrics show results. Input metrics provide guidance.

DMAIC (Six Sigma process improvement method) is as follows:

Define the metrics you want to measure. Amazon tracks things like selection, price, and convenience. These input metrics drive output metrics like orders, revenue, and profit. Amazon uses the “flywheel” concept from Jim Collins’ Good to Great. “You inject energy into any one element, or all of them, the flywheel spins faster.” The goal is to find the things that matter the most (but the process of identifying the optimal inputs is iterative).

Measure: Data must be collected and presented in a usable format. Align the metrics with the customer experience. Make sure you are able to regularly audit the metrics.

Analyze: The objective in this stage is separating signals from noise, and then identifying and addressing root causes. When Amazon teams come across a surprise or a perplexing problem with the data they are relentless until they discover the root cause. Use the “Five Whys” method developed by Toyota. Keep drilling down with “why” questions which may sit layers down a chain of probing questions.

Improve: This step can only happen once sufficient investments in the first three stages have occurred. Example: If you reach a weekly 95% in-stock rate for products, figure out how to get to a 98% rate.

Control: Ensure that processes are operating as expected and maintain performance levels. This is an opportunity for identifying automation opportunities.

The Weekly Business Review (WBR) is a tactical operational meeting to analyze performance trends of the prior week. At Amazon, it was not the time to discuss new strategies, project updates, or upcoming product releases. Amazon’s internal reporting that allows for effective input management. Unlike the Six-pager or PR/FAQ, this data package is heavy with graphs, tables, charts, and visuals. The document may include explanatory notes for the metrics. The deck represents a data-driven, end-to-end view of the business. Emerging patterns are a key point of focus. Graph plot results against comparable prior periods, this is critical for trend identification. A consistent format is maintained for ease of use and efficient interpretation. Anecdotes and exception reporting are woven into the deck. Highlight unusual situations or developments.

HBR Guide to Better Business Writing

2022-07-08T00:00:00+02:00

This is one of my favorite practical advice books. I read it first in 2015, and multiple times since then. These are my notes of the book, which I recommend you should buy right now.

Introduction

If your writing is sloppy and artless people will think you are the same. They won’t care about your message, they won’t do business with you. It’s not true that only ideas matter. Good writing gets ideas noticed. Those who write poorly create barriers between themselves and their readers. Those who write well connect with their readers, open their minds, and achieve goals. Don’t waste your reader’s time. If you’re in business and you’re writing, you’re a professional writer.

Qualities of good writing:

an intense focus on your reasons for writing, and on your reader’s needs
preference for the simplest words possible
feel for natural idioms
appreciation for the right words in the right places
an ear for tone

Know why you’re writing

Say clearly what your issue is and what you are trying to accomplish. With every sentence, ask yourself whether you’re advancing that cause. Consider your audience and purpose before starting to write, and let these guide what you say and how you say it. Plainly state the issue you’re addressing and what you hope to achieve. Keep your goal in mind: don’t undermine your efforts with hostile or inappropriate tone.

Understand your readers

Respect your readers’ time contraints. They are very busy. They have little sense of duty to read what you put in front of them. If they don’t get your point quickly, they’ll leave. At the slightest need to struggle to understand you, they’ll stop trying—and think less of you.

Prove quickly that you have something valuable to say—valuable to your readers. Waste no time in saying it. Write with such clarity and efficiency that reading your material is easy, even enjoyable. Use tone that makes you likable.

Tailor you message to you audience. A good trick is to connect with particular readers to connect with large audiences. Warren Buffet recommends:

When writing Berkshire Hathaway’s annual report, I pretend that I’m talking to my sisters. I have no trouble picturing them: though highly intelligent, they are not experts in accounting or finance. They will understand plain English, but jargon may puzzle them. My goal is simply to give them the information I would wish them to supply me if our positions were reversed. To succeed, I don’t need to be Skakespeare; I must, though, have a sincere desire to inform.

Recap:

Understand that your readers have no time to waste: get to the point quickly and clearly to ensure that your message gets read.
Use a tone appropriate for your audience.
Emphasize the items most important to your readers. If they can easily see how your message is relevant to them, they will be more likely to read it and respond.
Choose an intelligent, nonspecialist member of your audience to write for and focus on writing for that person. Your message will be more accessible and persuasive to all your readers as a result.

Divide the writing procss into four separate tasks (MACJ)

The Madman gathers material and generates ideas.
The Architect organizes information by drawing up an outline, however simple.
The Carpenter puts your thoughts into words, laying out sentences and paragraphs by following the Architect’s plan.
The Judge is your quality-control character, polishing the expression throughout—everything from tightening the language to correcting grammar and punctuation.

Three main points (Architect)

When writing a short piece like an email, write down your three main points first in full sentences. People can’t hold more than 3 things in their heads. Spell out your logic as clearly as you can. Force yourself to think through your reasons.

Recap:

Find your focus by first generating a list of topics to cover.
Develop these raw ideas into full sentences and categorize your main points in sets of threes.
Arrange these sets in a logical order, keeping your reader’s needs in mind.

Write in full—rapidly (Carpenter)

Write the first draft as quickly as you can. Once you’ve written your three main points so that you know where you’re going, you’re in Carpenter mode. Write as quickly as possible. Your sentences will be shorter than they otherwise should be, your idioms will be more natural. If there’s a painful part of writing, it’s doing the first draft. When you shorten the duration, it’s not as painful.

Speed writing: to prevent premature fussing, write against the clock. Allow yourself 5 or 10 minutes to draft each section. Don’t edit. This is not yet the time to let the Judge in. If you find yourself stumped, move on to a different section you’re more comfortable with and come back to the problem once you’ve found your flow.

Improve what you’ve written (Judge)

Once you have written a complete draft, you’ll revise first and then edit. Revising is a reconsideration of what you’re saying as a whole, and where you’re saying it. It’s rethinking the floor plan. Editing is more a matter of fine-tuning sentences and paragraphs. You need to allow time for both. On the one hand, don’t let some neurotic obsession with perfectionism delay important projects. On the other hand, don’t rashly send things out without proper vetting and improvement.

Revising:

Have I been utterly thruthful?
Have I said all that I need to say?
Have I been appropriately diplomatic and fair?
Do I have three parts—an opener, a middle, and a closer?
In my opener, have I made my points quickly and clearly?
In the middle, have I proved my points with specifics?
Is my closer consistent with the rest, expressed freshly?

Editing:

Where can I save some words?
Is there a better way of phrasing this idea?
Is my meaning unmistakable?
Can I make it more interesting?
Is the expression relaxed but refined?
Does one sentence glide into the next, without discontinuities?

Be relentlessly clear

Clarity can be a double-edged sword. When you’re forthright enough to take a position or recommend a course of action, you’re sticking your neck out. People who don’t want to commit make their writing muddy. Perhaps they’re trying to leave room for their views to evolve as events unfold. Or perhaps they’re hoping they can later claim credit for good results and deny responsibility for bad ones. The fact is though, that many readers will perceive them not as savvy wait-and-see participants but as spineless herd-followers who are slow to see opportunities within their reach. So clean up the mud.

Adopt the reader’s perspective: always judge clarity from the reader’s standpoint—not your own. Your goal should be to write so unmistakably that your readers can’t possibly misunderstand or misinterpret.

Keep your language simple: simplicity breeds clarity.

Show, don’t tell: be specific enough that you lead your readers to draw their own conclusions (that match yours), as opposed to simply expressing your opinions without support and hoping people will buy them. Not: “he was a bad boss”. Instad: “he got a promotion based on his assistant’s detailed reports, but then—despite the company’s record profits—denied that assistant even routine cost-of-living raises”. A short, vague sentense like “he was a bad boss” may register in the readers’ mind—but only as a personal impression that’s potentially biased. It’s credible only if its source (you) is credible. Concrete business writing is persuasive because it’s evidence-based, clear and memorable. When you supply meaningful, objective details you’re sharing information, not just your opinion.

Recap:

Put yourself in the reader’s shoes to assess your clarity. Better yet, see whether a colleague can accurately summarize the main points of your draft from a quick read-through.
Phrase your ideas as plainly and briefly as possible, aiming for an average sentence length of 20 or fewer words.
Pave your readers’ way with concrete details. Don’t try to push them there with abstract assertions.
Cultivate your letter writing to improve your writing skills more generally.

Learn to summarize—accurately

A good summary is focused and specific—and it’s at the beginning of your document so readers don’t have to dig. It gets to the point. It lays the foundation for what’s to follow. There’s no holding back on the crucial information.

People often assume that shorter is better when it comes to summaries. But brevity without substance is worthless. Never say more than the occasion demands—but never say less, either. Adopt the reader’s perspective: fill in as much information as it takes to get people up to speed.

Recap:

Summarize the vital information at the beginning of the document.
Summarize each section with a sentence that addresses “the five Ws” (who, what, when, where, why) and how—and use these sentences to build your general summary.
Provide only the information the reader needs to understand the issue—no more and no less.

Use chronology when giving a factual account

Create a chronology of relevant events to organize the narrative. Don’t jump in the middle. Once you’ve laid out a chronology of events, drafting the email becomes much easier—just a matter of stringing the events together and asking to meet with Sarah before next Tuesday’s meeting. But avoid rote repetition of unimportant details like dates.

Be a stickler for continuity

Smooth writing consists of a sequence of well-joined sentences and paragraphs, not a mere collection of them. This smooth sequencing requires good planning and skill in handling transitions, or links that help readers follow your train of thought. Good subheads steer the reader from one idea to the next. Types of connections:

establish a time sequence: then, at that point, as soon as...
establish place: there, up front, way back...
add a point: and, or, further...
underscore a point: after all, indeed, more important...
concede a point: although, admittedly...
return to a point: even so, nevertheless, still...
give an example: for example...
provide a reason: because, thus, since, therefore...
set up contrast: but, yet, conversely...
set up a conclusion: so, finally, to sum up...

No matter how smooth your transitions are between sentences and paragraphs, time-pressed readers will zone out if you place a solid wall of text in front of them. Break up your document with some signposts to lead people from section to section and let them quickly locate parts. Make your subheads as consistent as you can.

Recap:

Use well-placed transitional phrases to guide the reader to your next idea and indicate its relationship to what came before.
Break up documents with concise, descriptive subheads to increase readability and help readers quickly locate the information most important to them.
Use a “summary” subhead to point your readers to the document’s highlights.
Use consistent style and parallel syntax in your subheads to reinforce the document’s logical and rhetorical cohesion.

Learn the basics of correct grammar

Your readers may see your language—especially your use of your native language—as a reflection of your competence. Make lots of mistakes and you’ll come across as uneducated and uninformed. People will hesitate to trust your recommendation to launch a resource-intensive project, for example, or to buy goods or services. They may think you don’t know what you’re talking about.

Don’t anesthetize your readers

The best conversationists and lecturers, no matter how obscure their topic, make it fascinating. They avoid trite expressions. They use strong, simple words.

Recap:

Don’t overuse “I”. Use “we”, “our”, “you”, and “your” instead to add a personal touch and appeal to your reader.
Avoid stuffiness by overcoming any fear you might have of contractions.
For clearer, more straightforward writing, prefer active voice—unless the passive in a particular context sounds more natural.
Vary the length and structure of your sentences.
Make the reader’s job easier by avoiding acronyms when you can.

Watch your tone

Striking the right tone takes work—but it’s critical to the success of your business documents. If you sound likable and professional, people will want to work with you and respond to you. So adopt a relaxed tone, as if speaking directly to the recipient of your document. Avoid hyperformality: what do you think of colleagues who say or write “how may I be of assistance?” instead of “how may I help you?” When they choose overblown words over everyday equivalents, don’t they strike you as pompous?

Too much formality will spoil your style. Keep your writing down to earth and achieve a personal touch by:

Writing your message more or less as you’d say it, but without all the casualisms (“like” and “you know”).
Including courtesies such and “thank you”, “we’re happy to”, and “we appreciate”.
Using the names of the people you’re writing about.
Using personal pronouns (“you”, “he”, “she”—not “the reader”, “the applicant”).

Be collegial: you’ll have better luck delivering most kinds of messages, even tough ones, if you approach people collegially. Imagine that everything you write will be paraded before a jury in a lawsuit.

Drop the sarcasm: sarcasm expresses contempt and superiority. It doesn’t shame people into compliance. Rather, it’s a surefire way of irritating and alienating them.

Recap:

Arrive at a relaxed but professional tone by writing your message as if you were speaking to the recipient in person.
Refer to people by name, use personal pronouns as you naturally would, and shun fancy substitutes for everyday words.
Always use your best judgement and a collegial tone in composing your messages, even if the content isn’t positive. You’ll get better responses and keep yourself out of trouble.
Adopt a tone appropriate to your relationship with the recipient.
Never use sarcasm is professional messages. It will result in a step away from—not toward—your desired outcome.

Writing emails

Be as direct as possible while maintaining a polite tone. Come to the point of your email within the first two or three sentences.

Keep emails brief. Restrict yourself to one screen’s worth of text and keep the message tight and focused so your readers get the point fast. Write a concise subject line that tells your readers why you’re writing and what it means to them. If they need to act on your message, make that clear in the subject line.

Diligently adhere to standard writing conventions—even when typing with your thumbs on a handheld device. Never click “Reply All”. Pick your recipients!

A checklist for the four stages of writing (MACJ)

Madman (prepare):

Consider why you’re writing: What’s moved you to write? What’s the assignment? What you do hope to achieve?
Think about who your readers are and what they need to know.
Figure out how much time you have, and work out a rough schedule for gathering ideas and material, outlining, preparing a draft, and revising.
Research with imagination and gusto. Take notes on relevant information.
Push yourself to be creative. Don’t be content with obvious ideas that just anyone would think of.

Architect (skeleton):

Write down your three main points in complete sentences—with as much specificity as you can.
Consider the best order of the three main points and reorganize them if necessary.
Decide how to open and conclude the document.
Think about what visual aids might be helpful in conveying your ideas.

Carpenter (draft):

If possible, turn away from all distractions. Silence your phone and your computer alerts, and find an hour of solitude. You’ll be writing.
Use your three-point outline as a guide.
Start writing paragraphs that support the point your find easiest to start with—then move to the other points.
Write swiftly without stopping to edit or polish.
Try to write a full section in one sitting. If you must get up in the middle of a section, start the next sentence with a few words and then leave.

Judge (improve and iterate):

Immediately after completing your draft, read it through with the idea of amplifying ideas here and there.
Then let it cool off—overnight, if you can, or for a few minutes if you’re working under an urgent deadline.
When you return to your draft, consider it from the audience’s perspective. Will it be clear to everyone who looks at it, or does it require inside knowledge? Is it concise, or does it waste words and time?
Identify the draft’s two biggest flaws and try to fix them.
Ask yourself:
- Is anything essential missing?
- Are important points stressed?
- Is the meaning of each sentence clear and accurate?
- Are my transitions smooth?
- What can I trim without sacrificing important content?
- Are there any vague passages I can sharpen with specific facts?
- Are there boring passages I can word more vividly?
- Can I improve the phrasing?
- Can I improve the punctuation?
- Are there any typos?

Culture Docs: Facebook, Netflix and Valve

2022-06-18T00:00:00+02:00

Introduction

Many companies have some sort of "Culture Doc", a booklet or similar, which explains to new joiners what the company is about. I received Facebook's "Little Red Booklet" when I joined in 2016 February, and I was amazed how good it was. Recently I was researching other companies' Culture Docs, and found a version of Netflix's and Valve's online. It's interesting to compare and contrast what these different companies choose to put in their Culture Doc.

Facebook's Culture Doc is very mission and execution oriented and serious, Netflix is analytical and HR-focused, and Valve's is a lighthearted explanation of how the company works.

Disclaimer: I worked at Facebook in 2016-17. I currently own META stock.

Facebook

Facebook is the best company I ever worked at, mostly because of its awesome mission and impact oriented culture. This is reflected in the Culture Doc, which is easily the best of the three — but, I'm definitely biased. There are many principles (such Move Fast and Break Things) and mental models (such as the Perception vs Reality plot, which I can't find online unfortunately) that I learned at Facebook and I still refer to on a daily basis.

Facebook's Culture Doc is produced by the Facebook Analog Research Laboratory, a little team within the company responsible for making posters and booklets and other analog materials for employees and the office. Fortunately, Ben Barry, who created the booklet, has put a few scans online. All the images below are from his site.

Thinking along the Vision—Mission—Strategy—Tactics—Execution spectrum, Facebook's Culture Doc focuses on Mission and Execution.

Facebook was originally not created to be a company.

Changing how people communicate will always change the world.

What happens when everyone can put their message in front of a lot of people?

Each Facebook engineer is responsible for approximately 1,000,000 people using Facebook. (My old edition says 433,000...)

There is no point in having a 5-year plan in this industry.

Hacking can be playful — as long as it works.

Greatness and comfort rarely coexist.

Neither snow, nor rain, nor heat, nor gloom of night keeps these hackers from the swift completion of their code.

The quick shall inherit the Earth.

When you don't realize what you can't do, you can do some pretty cool stuff.

Remember, people don't use Facebook because they like us. They use it because they like their friends. 👍

We don't build services to make money. We make money to build better services.

If we don't create the thing that kills Facebook, some else will. #metaverse

Unfortunately most pages of the booklet are not available scanned, but in some cases the same message is available in scanned poster form:

Move fast and break things.

Done is better than perfect.

What would you do if you weren't afraid?

Fail harder.

The back still says the name of a different technology company, one that came before us, left as a reminder that if we fail, someday someone might replace us.

Other good sources of information about Facebook culture:

Founder's Letter, 2012
Founder's Letter, 2021
Facebook: The Inside Story (book) - by Steven Levy, highly recommended
Hack, HHVM and avoiding the Second-system effect

Netflix

The Netflix Culture Doc started out as a deck of 100+ slides, but over time it has become a minisite on the official Netflix website: jobs.netflix.com/culture.

Unlike Facebook's, which focuses on mission and execution, this feels more like a set of HR principles. However, there are many gems. Here are some of my favorite (from the 2012 version):

The real company values, as opposed to the nice-sounding values, are shown by who gets rewarded, promoted, or let go.

Netflix wants to work with people who embody these 9 values: judgement, communication, impact, curiosity, innovation, courage, passion, honesty, selflessness.

Increase talent density as company grows and continue to run informally.

Context, not control: If you want to build a ship, don't drum up the people to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.

Highly aligned, loosely coupled.

Annual compensation review is market based.

Valve

Valve's Culture Doc is up on their website, with translations to 8 languages. It's from 2012, I'm not sure whether there is a more up-to-date version internally. It's called Handbook for New Employees, it's mostly a light-hearted explanation of how the company works, with some HR principles sprinkled in. Whereas Facebook's Culture Doc is mission oriented and serious and Netflix's is analytical (with curves and intersections) and HR-focused, Valve's is more of a lighthearted explanation of how the company works and its history.

Valve is flat. It's our shorthand way of saying that we don't have any management, and nobody "reports to" anybody else.

You should always be considering where you could move yourself to be more valuable.

Valve's customers are who you're serving. Do what's right for them.

Cabals are really just multidisciplinary project teams.

My comment: it seems that being in the office and physically working together was a really important part of Valve's culture pre-Covid. Also see this article with Gabe Newell, Valve's President, on how Covid affected Valve.

Nobody has ever been fired at Valve for making a mistake.

Talk to someone in a {meeting, elevator, kitchen, bathroom}.

Stack-ranking.

Hiring is the most important thing in the Universe.

T-shaped employee model.

What is Valve not good at?

Similar books

(+) denotes books I've read and can whole-heartedly recommend.

About successful companies:

Facebook: The Inside Story (+) - by Steven Levy
Hack and HHVM: Programming Productivity Without Breaking Things (+) - by Owen Yamauchi, how Facebook rewrote the PHP website without rewriting the PHP website
Working Backwards: Insights, Stories, and Secrets from Inside Amazon (+) - by Colin Bryar and Bill Carr
Invent and Wander: The Collected Writings of Jeff Bezos (+) - by Jeff Bezos
Masters of Doom: How Two Guys Created an Empire and Transformed Pop Culture (+) - David Kushner
How Google Works - by Eric Schmidt
No Rules Rules: Netflix and the Culture of Reinvention - by Reed Hastings

Also see Amazon's official leadership principles and my blog post about the best parts of the Invent and Wander book.

Anti-patterns:

Bad Blood: Secrets and Lies in a Silicon Valley Startup (+) - by John Carreyrou
Billion Dollar Loser: The Epic Rise and Fall of WeWork (+)- by Reeves Wiedeman

Also see my blog post about Theranos and WeWork.

Conclusion

Every company should have a Culture Doc and publish it!

PS: Coincidentally my friend Jack Mardack's company Oyster just opensourced their Culture Doc today!

More Data Scientists should learn SQL

2022-05-29T00:00:00+02:00

Introduction

Our Data Scientist interview is a multi-stage process. As the hiring manager, my interview is somewhere in the middle. The candidates who make it this far have had their CVs screened to make sure SQL is on it, and had to answer a few SQL screening questions, like whether they know how many rows are produced when using UNION ALL. Still, most candidates I talk to struggle to write a relatively simple SQL query.

It's worth pointing out that, although we always ask SQL questions, this is about 10-20% of the interview process. We also ask candidates to write 2-3 line Python functions, ask them about metrics, stats, modeling/ML, and maybe even some Linux.

A relatively simple SQL interview question

For this post, I wrote a slightly modified version of our interview question, but it's the same structure and complexity. Let's say there is table transactions, with columns:

transactions:
- id BIGINT
- customer_id BIGINT
- ts TIMESTAMP
- amount_usd DECIMAL

Question: how many customers spent at least $1000 in all months of the year 2021?

The solution we're looking for:

WITH
monthly_spends AS
(
    SELECT
        customer_id,
        MONTH(ts) AS month,
        SUM(amount_usd)
    FROM
        transactions
    WHERE
        YEAR(ts) = 2021
    GROUP BY
        1, 2
    HAVING
        SUM(amount_usd) >= 1000
),
qualifying_users AS
(
    SELECT
        customer_id,
        COUNT(*)
    FROM
        monthly_spends
    GROUP BY
        1
    HAVING
        COUNT(*) = 12
)
SELECT COUNT(*) FORM qualifying_users

Extracting metrics from a Vertica DWH is a daily task for Data Scientists on our current team (same for the previous 3 jobs), it's not uncommon for us to write SQL queries that are longer than 100 lines, some are longer than 1000 lines. So expecting a candidate to be able to use GROUP BY, HAVING, COUNT(), SUM() is definitely not unfair, it's what we do every day.

First, I make sure they understand the question. I spend extra time stressing the "all months of the year" part, here english-as-a-second-language sometimes is a factor. So I usually repeat this requirement phrased in 2-3 different ways (in english), to make sure the candidate got it.

What to look for

Interviewing is nerve-wrecking, so I expect candidates to perform worse than they would do under normal, real-life conditions. It's the same for me. For this reason, I tend to help them, and ignore minor errors/omissions. For example, people tend to not remember MONTH() and YEAR() and maybe try to use DATE_PART(), but then struggle with the syntax. Or they forget to actually write out the GROUP BY, even though clearly that's what they're doing. In cases like this I help them, and don't count this against them. Or, if they write the correct query, but forget the WHERE YEAR(ts) = 2021 part, I would remind them and also not count it against them. Also, most problems can be solved in several different ways, I'll take anything that is roughly right: eg. in the above query, you can write HAVING ... like I did, or move that to the subsequent SELECT as a WHERE condition.

The thing we are looking for is, can candidates think in SQL, formulate the solution as an SQL query? Can they break down the question into a series of SELECTs, which progressively get closer to the answer we are looking for. This is where most people fail — to my surprise. This tells me that these candidates haven't yet figured out that knowing how to think in SQL and write queries more complicated than a single SELECT <columns> FROM <table> WHERE <condition> is worth it.

In our interview process, depending on the position and the person, a candidate can still get hired if they can't solve the above SQL, but they have to be really good at something else. Also, if the candidate is not a junior, I would be skeptical how they made it this far without learning to write SQL. This is simply because having to write SQL has been a daily part of my last 4 jobs in this field. In my current job I'm a hands-on manager running a 20-person team, but I still have about 10,000 lines of SQL in my draft, not counting code that I've commited to Github.

Indentation

Another thing I notice is that very few candidates (i) indent their SQL code (they're writing in on their own laptop in the interview) and (ii) don't use WITH, they write subqueries instead. For my brain, indenting is a major factor to keep track of what's going on in the SELECT and be able to scan it to make sure it's good, while using WITH allows me to think sequentially, and also keep the code sequential. Using sub-queries requires more indentation and reverses the order of the SELECTs, to me it's a mental overhead that hinders clarity.

See this past article on How I write SQL code.

Why SQL gets overlooked

There is no one answer to this, because Data Scientists come from many different educational backgrounds and companies.

People coming from Software Engineerings background may miss out on SQL because we (I also have a Comp.Sci. degree) spend a lot of time learning and coding in imperative languages like C, C++, Java, Python. In the last 10 years there was a lot of buzz around Functional Programming (FP) and languages like Haskell and F#, and a lot of good functional patterns made it into mainstream programming languages and practice. But there's not a lot of buzz around Declarative Programming and languages like Prolog and SQL, and I don't think it's taught widely at Universities ( I was lucky enough to learn Prolog and SQL at University). In a declarative language like SQL we declare what results we want, and we let the runtime (query optimizer and execution engine) figure out the best way to get it.

People coming from math and science background probably invest their time learning Python and all the interesting libraries and frameworks that we have today. After all, it's hard to get yourself to invest time in SQL, if you can spend the same time playing around with Reinforcement Learning in Pytorch.

Lastly, there are companies, where Data Science teams haven't figured this out yet, so the candidates haven't learned this through osmosis. In these teams the Data Scientists block waiting for other people to write SQL for them to extract data from a DWH and receive it in a flat file.

Why SQL is worth knowing

1. "SQL just another language, like Python or Java!" ... and you can't expect everybody to know all programming languages. I strongly disagree with this framing. SQL has been around since the 1970s, it's the de facto standard for getting data out of a (relational) database system. Even database systems that start out without SQL support usually end up with some sort of SQL support, because it's just so damn useful. In an analytics/DS/ML setting, the Data Warehouse (DWH) will almost always be queried with SQL (or something resembling SQL, like HiveQL). This even tends to happen if the underlying storage is not strictly relational, like when running a Presto engine on flat files stored on S3. Yes, there are ML projects that don't deal with relational data, like image, video and audio processing, but in my experience (i) in real life, there's always projects that do, and (ii) even something like image recognition will usually have additional input (eg. when recognizing faces on an image, it's worth knowing who was tagged on the image, what the location and time was, etc, and these additional fields will come from a DWH).

SQL is not just like a language like Python or Java. It's been around since the 1970s, and there is no sign that it will go away anytime soon, including humans directly writing SQL.

2. "I usually just get the data out and do it in Python!" This breaks if there is too much data "to get out", which can happen even at medium sized companies. Also, it's very inefficient and slow, since all the data has to be copied out from the database disks, over a network, to a laptop or devserver. Usually, the combined program to get the data out and do the processing is harder to maintain and more error-prone then a flat SQL query like above. And the biggest reason: if you do it in Python, things like JOINs and GROUP BYs will be executed by the query optimizer taking into account indexes, storage patterns, statistics, etc --- it will be a lot faster, since (i) only the relevant data needs to be read from disk (ii) the execution engine is highly optimized C/C++/Java/Assembly code (iii) instead of all data getting transfered out of the database, only the results are.

3. "Data Analysts / Data Engineers write SQL queries for me!" At many companies, there are separate roles that do most of the SQL. But, in my experience, even at companies like this (like Facebook), Data Scientists need to and should write SQL:

it's unreasonable to block and wait for somebody to write a 20-50 line SQL which takes 5-10 minutes to write
the Data Engineering person or team may be busy or have long turnaround times
at this or a future job, there may be no other team/person to do this
the other team/person may make a mistake, which could invalidate the Data Scientist's results

If my current team of Data Scientists would block on the Data Engineering team to write SQL queries, our productivity/impact would at least be 10x less that it is today.

4. It's just SELECTs! Data Scientists mostly just have to write SELECTs (versus designing tables, choosing indexes, writing upserts, etc.), which is a relatively small part of SQL overall!

5. Writing SQL absurdly increases the value of Data Scientists! In my experience, if a Data Scientist or a team of Data Scientists writes their own SQL, they will become experts at the data and metrics. Add to this Data Scientist's objectivity, understanding of statistics, and ability to put models on top. Since they write their own SQL, they don't block on other teams, so this means they can answer incoming questions very quickly. Eg. "the Finance team came up with this forecast for Q4, but it may be biased by company OKRs, what do you think?" Get the relevant metrics out with SQL, play around with it, put a Prophet model on top with some external regressors, and make a better forecast than the Finance team with a 1-day turnaround. Altough this is not the core job of a Data Science team, this makes us absurdly valuable to the company, which means good salaries, good raises, no layoffs, plus headcount, etc.

Conclusion

If you don't know SQL, learn it! 😀

Useful Python decorators for Data Scientists

2022-05-22T00:00:00+02:00

Introduction

In this post, I will show some @decorators that may be useful for Data Scientists. It may also be useful to revisit previous Bytepawn posts on decorators:

The ipython notebook is up on Github.

`@parallel`

Let's assume I write a really inefficient way to find primes:

from sympy import isprime

def generate_primes(domain: int=1000*1000, num_attempts: int=1000) -> list[int]:
    primes: set[int] = set()
    seed(time())
    for _ in range(num_attempts):
        candidate: int = randint(4, domain)
        if isprime(candidate):
            primes.add(candidate)
    return sorted(primes)

print(len(generate_primes()))

Outputs something like:

Then I realize that I could get a "free" speedup if I run the original generate_primes() on all my CPU threads in parallel. This is pretty common, it makes sense to define a @parallel:

def parallel(func=None, args=(), merge_func=lambda x:x, parallelism = cpu_count()):
    def decorator(func: Callable):
        def inner(*args, **kwargs):
            results = Parallel(n_jobs=parallelism)(delayed(func)(*args, **kwargs) for i in range(parallelism))
            return merge_func(results)
        return inner
    if func is None:
        # decorator was used like @parallel(...)
        return decorator
    else:
        # decorator was used like @parallel, without parens
        return decorator(func)

With this, with one line we can parallelize our function:

@parallel(merge_func=lambda li: sorted(set(chain(*li))))
def generate_primes(...): # same signature, nothing changes
    ... # same code, nothing changes

print(len(generate_primes()))

Outputs something like:

In my case, my Macbook has 8 cores, 16 threads (cpu_count() is 16), so I generated 16x as many primes. Notes:

The only overhead is having to define a merge_func, which merges the results of the different runs of the function into one result, to hide the parallelism from outside callers of the decorated function (generate_primes() in this case). In this toy example, I just merge the lists and make sure the primes are uniques by using set().
There are many Python libraries and approches (eg. threads vs processes) to achieve parallelism. This example uses process parallelism with joblib.Parallel(), which works well on Darwin + python3 + ipython and avoids locking on the Python Global Interpreter Lock (GIL).

`@production`

Sometimes we write a big complicated pipeline, with extra steps which we only want to run in certain environments. Eg. do something on our local dev environment, but not in production or vica versa. It'd be nice to be able to decorate functions and get them to only run in certain environments, and do nothing elsewhere.

One way to achieve this is with a few simple decorators: @production for stuff we want to only run on prod, @development for stuff we only want to run in dev, we can even introduce an @inactive which just turns the function off altogether. The benefit of this approach is that this way the deployment history and current state is tracked in code/Github. Also, we can make these changes in one line, leading to cleaner commits; eg. @inactive is cleaner than a big commit where an entire block of code is commented out.

production_servers = [...]

def production(func: Callable):
    def inner(*args, **kwargs):
        if gethostname() in production_servers:
            return func(*args, **kwargs)
        else:
            print('This host is not a production server, skipping function decorated with @production...')
    return inner

def development(func: Callable):
    def inner(*args, **kwargs):
        if gethostname() not in production_servers:
            return func(*args, **kwargs)
        else:
            print('This host is a production server, skipping function decorated with @development...')
    return inner

def inactive(func: Callable):
    def inner(*args, **kwargs):
        print('Skipping function decorated with @inactive...')
    return inner

@production
def foo():
    print('Running in production, touching databases!')

foo()

@development
def foo():
    print('Running in production, touching databases!')

foo()

@inactive
def foo():
    print('Running in production, touching databases!')

foo()

Output:

Running in production, touching databases!
This host is a production server, skipping function decorated with @development...
Skipping function decorated with @inactive...

This idea can be adapted to other frameworks/environments.

`@deployable`

At my current work, we use Airflow for ETL/data pipelines. We have a rich library of helper functions which internally construct the appropriate DAG, so users (Data Scientists) don't have to worry about it.

The most commonly used one is dag_vertica_create_table_as(), which runs a SELECT on our Vertica DWH and dumps the result into a table every night:

dag = dag_vertica_create_table_as(
    table='my_aggregate_table',
    owner='Marton Trencseni (marton.trencseni@maf.ae)',
    schedule_interval='@daily',
    ...
    select="""
    SELECT
        ...
    FROM
        ...
    """
)

This then becomes a query on the DWH, roughly like:

CREATE TABLE my_aggregate_table AS
SELECT ...

In reality it's more complicated: we first run the query for today, and conditionally delete yesterday's if today's was successfully created. This conditional logic (and some other accidental complexity specific to our environment, such as having to issue GRANTs) results in the DAG having 9 steps, but this is not the point here, and is beyond the scope of the article.

Over the last 2 years we have created almost 500 DAGs, so we scaled up our Airflow EC2 instances and introduced seperate development and production environments. It'd be nice to have a way to tag DAGs whether they should be running on dev or prod, track this in the code/Github, and use the same mechanism to make sure the DAGs don't accidentally run in the wrong environment.

There are about 10 similar convenience functions, such as dag_vertica_create_or_replace_view_as() and dag_vertica_train_predict_model(), etc, and we'd like all calls of these dag_xxx() functions to be switchable between production and development (or skip everywhere).

However, the @production and @development decorators from the previous section won't work here, because we don't want to switch dag_vertica_create_table_as() to never run on one of the environments. We want to be able to set it per invocation, and have this feature in all of our dag_xxxx() functions, without having to copy/paste code. What we want is to add a deploy parameter to all of our in all of our dag_xxxx() functions (with a good default), so we can just add this parameter in our DAGs for added security. We can achieve this with the @deployable decorator:

def deployable(func):
    def inner(*args, **kwargs):
        if 'deploy' in kwargs:
            if kwargs['deploy'].lower() in ['production', 'prod'] and gethostname() not in production_servers:
                print('This host is not a production server, skipping...')
                return
            if kwargs['deploy'].lower() in ['development', 'dev'] and gethostname() not in development_servers:
                print('This host is not a development server, skipping...')
                return
            if kwargs['deploy'].lower() in ['skip', 'none']:
                print('Skipping...')
                return
            del kwargs['deploy'] # to avoid func() throwing an unexpected keyword exception
        return func(*args, **kwargs)
    return inner

Then we can add the decorator to our function definitions (1 line added for each):

@deployable
def dag_vertica_create_table_as(...): # same signature, nothing changes
    ... # code signature, nothing changes

@deployable
def dag_vertica_create_or_replace_view_as(...): # same signature, nothing changes
    ... # code signature, nothing changes

@deployable
def dag_vertica_train_predict_model(...): # same signature, nothing changes
    ... # code signature, nothing changes

If we stop here, nothing happens, we don't break anything. However, now we can go to the DAG files where we use these functions, and add 1 line:

dag = dag_vertica_create_table_as(
    deploy='development', # the function will return None on production
    ...
)

`@redirect` (stdout)

Sometimes we write a big function, which also calls other code, and all sorts of messages are print()ed. Or, we may have a bug, have a bunch of print()s, and want to add line numbers to the printouts so it's easier to refer to them. In these cases, @redirect may be useful. This decorator redirects print() standard output to our own line-by-line printer, and we can do whatever we'd like with it (including throwing it away):

def redirect(func=None, line_print: Callable = None):
    def decorator(func: Callable):
        def inner(*args, **kwargs):
            with StringIO() as buf, redirect_stdout(buf):
                func(*args, **kwargs)
                output = buf.getvalue()
            lines = output.splitlines()
            if line_print is not None:
                for line in lines:
                    line_print(line)
            else:
                width = floor(log(len(lines), 10)) + 1
                for i, line in enumerate(lines):
                    i += 1
                    print(f'{i:0{width}}: {line}')
        return inner
    if func is None:
        # decorator was used like @redirect(...)
        return decorator
    else:
        # decorator was used like @redirect, without parens
        return decorator(func)

If we use redirect() without specifying an explicit line_print() function, it will print the lines, but with line numbers added:

@redirect
def print_lines(num_lines):
    for i in range(num_lines):
        print(f'Line #{i+1}')

print_lines(10)

Output:

01: Line #1
02: Line #2
03: Line #3
04: Line #4
05: Line #5
06: Line #6
07: Line #7
08: Line #8
09: Line #9
10: Line #10

If we want to save all printed text to a variable, we can also achieve that:

lines = []
def save_lines(line):
    lines.append(line)

@redirect(line_print=save_lines)
def print_lines(num_lines):
    for i in range(num_lines):
        print(f'Line #{i+1}')

print_lines(3)
print(lines)

Output:

['Line #1', 'Line #2', 'Line #3']

The actual heavy lifting of redirecting stdout is done by contextlib.redirect_stdout, as shown in this StackOverflow thread.

`@stacktrace`

The next decorator pattern is @stacktrace, which emits useful messages when functions are called and values are returned from functions:

def stacktrace(func=None, exclude_files=['anaconda']):
    def tracer_func(frame, event, arg):
        co = frame.f_code
        func_name = co.co_name
        caller_filename = frame.f_back.f_code.co_filename
        if func_name == 'write':
            return # ignore write() calls from print statements
        for file in exclude_files:
            if file in caller_filename:
                return # ignore in ipython notebooks
        args = str(tuple([frame.f_locals[arg] for arg in frame.f_code.co_varnames]))
        if args.endswith(',)'):
            args = args[:-2] + ')'
        if event == 'call':
            print(f'--> Executing: {func_name}{args}')
            return tracer_func
        elif event == 'return':
            print(f'--> Returning: {func_name}{args} -> {repr(arg)}')
        return
    def decorator(func: Callable):
        def inner(*args, **kwargs):
            settrace(tracer_func)
            func(*args, **kwargs)
            settrace(None)
        return inner
    if func is None:
        # decorator was used like @stacktrace(...)
        return decorator
    else:
        # decorator was used like @stacktrace, without parens
        return decorator(func)

With this, we can decorate the topmost function where we want tracing to start, and we will get useful output about the branching:

def b():
    print('...')

@stacktrace
def a(arg):
    print(arg)
    b()
    return 'world'

a('foo')

Output:

--> Executing: a('foo')
foo
--> Executing: b()
...
--> Returning: b() -> None
--> Returning: a('foo') -> 'world'

The only trick here is hiding parts of the callstack which is not interesting. In my case, I'm running this code in ipython over Anaconda, so I hide parts of the callstack where the code is in a file which is has anaconda in its path (otherwise I would get about 50-100 useless callstack entries in the snippet above). This is accomplished by the exclude_files parameter of the decorator.

`@traceclass`

Similarly to the above, we can define a decorator @traceclass which we use with classes, to get traces of its members' execution. This was included in the previous decorator post, there it was just called @trace and had a bug (since fixed in the original post). The decorator:

def traceclass(cls: type):
    def make_traced(cls: type, method_name: str, method: Callable):
        def traced_method(*args, **kwargs):
            print(f'--> Executing: {cls.__name__}::{method_name}()')
            return method(*args, **kwargs)
        return traced_method
    for name in cls.__dict__.keys():
        if callable(getattr(cls, name)) and name != '__class__':
            setattr(cls, name, make_traced(cls, name, getattr(cls, name)))
    return cls

We can use it like:

@traceclass
class Foo:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

f1 = Foo()
f2 = Foo(4)
f1.increment()
print(f1)
print(f2)

Output:

--> Executing: Foo::__init__()
--> Executing: Foo::__init__()
--> Executing: Foo::increment()
--> Executing: Foo::__str__()
This is a Foo object with i = 1
--> Executing: Foo::__str__()
This is a Foo object with i = 4

Conclusion

In Python functions are first class citizens, and decorators are powerful syntactic sugar exploiting this functionality to give programmers a seemingly "magic" way to construct useful compositions of functions and classes. These are 5 decorators that may useful specifically for Data Scientists working in ipython notebooks.

Thanks Zsolt for bugfixes and improvement suggestions.

Building a toy Python @dataclass decorator

2022-05-12T00:00:00+02:00

Introduction

Following the previous article on a toy implementation of a Python Enum class and the article on Python decorator patterns, this time I will write a toy implememtation of the built-in @dataclass decorator. The official documentation for dataclasses is here. The ipython notebook is up on Github.

`@dataclass` features

@dataclass is a very useful feature of the Python standard library. It's a simple way to declare a class with typed variables, and get useful helper functions added to the class "for free":

from dataclasses import *

@dataclass
class Fraction:
    numerator: int = 0
    denominator: int = 1

With @dataclass, we get free constructors for the member attributes. There are 3 ways to get a new Fraction object:

f = Fraction() # defaults, same as Fraction(0, 1)
f = Fraction(1, 2)
f = Fraction(numerator=1, denominator=2)

Note that the second and third would not work without @dataclass, we would get TypeError: Fraction() takes no arguments. We also get free equality checks (it compares each of the members), like:

f = Fraction(1, 2)
g = Fraction(1, 2)
f == g # True

Note that without @dataclass, the == equality would be False. We also get a useful string representation:

print(Fraction(1, 2)) # prints: Fraction(numerator=1, denominator=2)

By specifying order=True in the @dataclass decorator, we also get free <, <=, >, >= comparisons:

@dataclass(order=True)
class Fraction:
    numerator: int = 0
    denominator: int = 1

f = Fraction(1, 2)
g = Fraction(2, 3)
f <= g # True

The free comparison functions aren't very useful for the Fraction example, since they just do a attribute-wise comparison. So eg. f < g is the same as f.numerator < g.numerator and f.denominator < g.denominator, which mathematically isn't the right expression.

By specifying frozen=True in the decorator arguments, we get read-only objects, which are useful in eg. multithreading:

@dataclass(frozen=True)
class Fraction:
    numerator: int = 0
    denominator: int = 1

f = Fraction(1, 2)
f.numerator = 2 # FrozenInstanceError: cannot assign to field 'numerator'

Finally, at the top-level of dataclasses are 2 useful functions, asdict() and astuple():

asdict(Fraction(1, 2))  # {'numerator': 1, 'denominator': 2}
astuple(Fraction(1, 2)) # (1, 2)

The Python standard library @dataclass has a lot more features, but these are some of the most frequently used ones. Let's practice our Python fu and create a toy implementation of @dataclass. As seen before, the general skeleton will be:

def dataclass():
    def decorator(cls):
        # add useful features to cls
        ...
        return cls
    return decorator

Finding the annotated member attributes

First we have to locate the annotated attributes declared by the user in the class object. The trick is telling them apart from all the built-in functions and attributes and any user defined functions. We can check in cls.__dict__ and dir(cls):

class Fraction:
    numerator: int = 0
    denominator: int = 1
    def mul(self, x: int):
        self.numerator *= x

print(Fraction.__dict__)
print()
print(dir(Fraction))

Output:

{'__module__': '__main__', '__annotations__': {'numerator': <class 'int'>, 'denominator': <class 'int'>}, 'numerator': 0, 'denominator': 1, '__dict__': <attribute '__dict__' of 'Fraction' objects>, '__weakref__': <attribute '__weakref__' of 'Fraction' objects>, '__doc__': None}

['__annotations__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'denominator', 'numerator']

Long story short, the right place to look is cls.__annotations__:

print(Fraction.__annotations__)

Output:

{'numerator': <class 'int'>, 'denominator': <class 'int'>}

Building the constuctor

Now that we found the attributes, let's build the constructor:

def dataclass(...):
    def decorator(cls):
        cls._attribs = cls.__annotations__.keys()
        # define initializer, unless defined by the user
        if '__init__' not in cls.__dict__:
            def __init__(self, *args, **kwargs):
                if len(args) > 0:
                    for attrib, arg in zip(self.__class__._attribs, args):
                        # avoid our own __setattr__ in case it's a frozen dataclass:
                        object.__setattr__(self, attrib, arg)                       
                elif len(kwargs) > 0:
                    for attrib in self.__class__._attribs:
                        # avoid our own __setattr__ in case it's a frozen dataclass:
                        object.__setattr__(self, attrib, kwargs[attrib])
            cls.__init__ = __init__
        ...
        return cls

First we save the attribute keys in cls._attribs, so we can conveniently access it in subsequent functions.

We check if the user explicitly defined their own constructor with if '__init__' not in cls.__dict__. If not, then we define the function and assign it with cls.__init__ = __init__ at the end. The constructor function itself takes *args and **kwargs. If the user calls it like Fraction(1, 2), then args will be (1, 2) and kwargs will be an empty dict {}. If the user calls it like Fraction(numerator=1, denominator=2), then args will be () and kwargs will be a dict like {'numerator': 1, 'denominator': 2}. So we just need to get these, and assign them to the appropriate member variable with object.__setattr__(), to avoid using cls.__setattr__() in case this is a frozen dataclass (see later).

String conversion

String conversion is relatively straightforward:

    def decorator(cls):
        cls._attribs = cls.__annotations__.keys()
        ...
        # define string conversion, unless defined by th user
        if '__str__' not in cls.__dict__:
            def __str__(self):
                kv_tuples = [(attrib, getattr(self, attrib)) for attrib in self.__class__._attribs]
                kv_str = ', '.join([f'{k}={v}' for (k, v) in kv_tuples])
                return f'{self.__class__.__name__}({kv_str})'
            cls.__str__ = __str__

Equality and comparators

Next, let's write equality and comparators. These can be switched on and off by passing the appropriate arguments to the decorator, so we have to check these:

def dataclass(..., **kwargs):
    def decorator(cls):
        ...
        if kwargs.get('eq', True):
            # define ==, unless defined by the user
            if '__eq__' not in cls.__dict__:
                def __eq__(self, other):
                    for attrib in self.__class__._attribs:
                        if getattr(self, attrib) != getattr(other, attrib):
                            return False
                    return True            
                cls.__eq__ = __eq__

Operators for <, <=, >, >= follow the same logic, showing just the first:

        if kwargs.get('order', False):
            # define <, <=, >, >=, unless defined by the user
            if '__lt__' not in cls.__dict__:
                def __lt__(self, other):
                    for attrib in self.__class__._attribs:
                        if getattr(self, attrib) >= getattr(other, attrib):
                            return False
                    return True            
                cls.__lt__ = __lt__
        ...

Frozen

If frozen=True is passed to the decorator, we have to disallow assignment to attributes, ie. f.numerator = 1 should raise an exception. When we write f.numerator = 1, internally this becomes Fraction.__setattr__(f, 'numerator', 1). So to disallow this, we just have to define our own __setattr__():

        if kwargs.get('frozen', False):
            # don't allow changing attributes
            def __setattr__(self, attrib, value):
                if attrib not in self.__class__._attribs:
                    setattr(self, attrib, value)
                else:
                    raise AttributeError(f'dataclass is frozen, cannot assign to field \'{attrib}\'')
            cls.__setattr__ = __setattr__

`@dataclass` vs `@dataclass(...)`

We need one final trick. When using this decorator, we want to allow users to write both @dataclass and @dataclass(...). In the first case, the class after the decorator gets passed directly to the dataclass function. In the second case, dataclass() is first called with the arguments (without the following class), and that must return a function which accepts the class. We want to support both behaviours in a single decorator function. The solution is quite simple:

def dataclass(cls=None, **kwargs):
    def decorator(cls):
        cls._attribs = cls.__annotations__.keys()
        ...
        # this is where the code shown above is
        ...
        return cls
    if cls is None:
        # decorator was used like @dataclass(...)
        return decorator
    else:
        # decorator was used like @dataclass, without parens
        return decorator(cls)

Complete toy implementation

That's it! The complete toy implementation, including asdict() and astuple() is:

def asdict(x):
    return {a:getattr(x, a) for a in x.__class__._attribs}

def astuple(x):
    return tuple(getattr(x, a) for a in x.__class__._attribs)

def dataclass(cls=None, **kwargs):
    def decorator(cls):
        cls._attribs = cls.__annotations__.keys()
        # define initializer, unless defined by the user
        if '__init__' not in cls.__dict__:
            def __init__(self, *args, **kwargs):
                if len(args) > 0:
                    for attrib, arg in zip(self.__class__._attribs, args):
                        # avoid our own __setattr__ in case it's a frozen dataclass:
                        object.__setattr__(self, attrib, arg)                       
                elif len(kwargs) > 0:
                    for attrib in self.__class__._attribs:
                        # avoid our own __setattr__ in case it's a frozen dataclass:
                        object.__setattr__(self, attrib, kwargs[attrib])
            cls.__init__ = __init__
        # define string conversion, unless defined by th user
        if '__str__' not in cls.__dict__:
            def __str__(self):
                kv_tuples = [(attrib, getattr(self, attrib)) for attrib in self.__class__._attribs]
                kv_str = ', '.join([f'{k}={v}' for (k, v) in kv_tuples])
                return f'{self.__class__.__name__}({kv_str})'
            cls.__str__ = __str__
        if kwargs.get('eq', True):
            # define ==, unless defined by the user
            if '__eq__' not in cls.__dict__:
                def __eq__(self, other):
                    for attrib in self.__class__._attribs:
                        if getattr(self, attrib) != getattr(other, attrib):
                            return False
                    return True            
                cls.__eq__ = __eq__
        if kwargs.get('order', False):
            # define <, <=, >, >=, unless defined by the user
            if '__lt__' not in cls.__dict__:
                def __lt__(self, other):
                    for attrib in self.__class__._attribs:
                        if getattr(self, attrib) >= getattr(other, attrib):
                            return False
                    return True            
                cls.__lt__ = __lt__
            if '__le__' not in cls.__dict__:
                def __le__(self, other):
                    for attrib in self.__class__._attribs:
                        if getattr(self, attrib) > getattr(other, attrib):
                            return False
                    return True            
                cls.__le__ = __le__
            if '__gt__' not in cls.__dict__:
                def __gt__(self, other):
                    for attrib in self.__class__._attribs:
                        if getattr(self, attrib) <= getattr(other, attrib):
                            return False
                    return True
                cls.__gt__ = __gt__
            if '__ge__' not in cls.__dict__:
                def __ge__(self, other):
                    for attrib in self.__class__._attribs:
                        if getattr(self, attrib) < getattr(other, attrib):
                            return False
                    return True
                cls.__ge__ = __ge__
        if kwargs.get('frozen', False):
            # don't allow changing attributes
            def __setattr__(self, attrib, value):
                if attrib not in self.__class__._attribs:
                    setattr(self, attrib, value)
                else:
                    raise AttributeError(f'dataclass is frozen, cannot assign to field \'{attrib}\'')
            cls.__setattr__ = __setattr__
        return cls
    if cls is None:
        # decorator was used like @dataclass(...)
        return decorator
    else:
        # decorator was used like @dataclass, without parens
        return decorator(cls)

With this toy implementation, the examples shown at the beginning work just like the Python standard library @dataclass, whose implementation is in dataclasses.py (it's 1488 lines of code).

Conclusion

Similar to the previous articles, this is a good way to improve one's Python fu.

Python decorator patterns

2022-05-08T00:00:00+02:00

Introduction

In Python, functions are first class citizens: functions can be passed to other functions, can be returned from functions, and can be created on the fly. Let's see an example:

# define a function on-the-fly
pow2 = lambda x: x**2
print(pow2(2))

# take a function as a parameter
def print_twice(func: Callable, arg: Any):
    print(func(arg))
    print(func(arg))
print_twice(pow2, 3)

# take a function as a parameter and return a new function
def hello():
    print('Hello world!')
def loop(func: Callable, n: int):
    for _ in range(n):
        func()
loop(hello, 3)

Output:

4
9
9
Hello world!
Hello world!
Hello world!

Decorators in Pythons are syntactic sugar for passing functions to functions and returning a new function. Let's see how this works and how we can put it to use in practice. The code for this article is on Github.

`@measure`: decorator functions without arguments

Let's take a useful example of measuring how long it takes to execute a function. The best would be if we could easily annotate an existing function and get "free" measurements. Let's look at the following two functions:

from timeit import default_timer as timer
from time import sleep

def measure(func: Callable):
    def inner(*args, **kwargs):
        print(f'---> Calling {func.__name__}()')
        start = timer()
        func(*args, **kwargs)
        elapsed_sec = timer() - start
        print(f'---> Done {func.__name__}(): {elapsed_sec:.3f} secs')
    return inner

def sleeper(seconds: int = 0):
    print('Going to sleep...')
    sleep(seconds)
    print('Done!')

measure() is a function which takes a function func() as an argument, and returns a function inner() declared on the inside. inner() takes whatever arguments are passed in and passed them along to func(), but wraps this call in a few lines of to measure and print the elapsed time in seconds. sleeper() is a test function which explicitly sleeps for a while so we can measure it.

Given these, we can construct a measured sleeper() function like:

measured_sleeper = measure(sleeper)
measured_sleeper(3)

Output:

---> Calling sleeper()
Going to sleep...
Done!
---> Done sleeper(): 3.000 secs

This works, but if we're already using sleeper() in a bunch of places, we'd have to replace all those calls with measured_sleeper(). Instead, we can:

sleeper = measure(sleeper)

Here we are replacing the sleeper reference in the current scope to point to the measured version of the original sleeper() function. This is exactly the same thing as putting the @ decorator in front of the function declaration:

@measure
def sleeper(seconds: int = 0):
    print('Going to sleep...')
    sleep(seconds)
    print('Done!')

So @decorators are just syntactic sugar to passing a newly defined function to an existing decorator function, which returns a new function, and having the original function name point to this new function!

`@repeat`: parameterized decorator function

In the above example we took an existing function sleeper() and decorated it with a function-taking-and-returning-a-function measure(), ie. a @decorator. What if we want to pass arguments to the decorator function itself? For example, imagine we have a function, and we want to repeat it n times. To accomplish this, we just have to add one more inner function:

def repeat(n: int = 1):
    def decorator(func: Callable):
        def inner(*args, **kwargs):
            for _ in range(n):
                func(*args, **kwargs)
        return inner
    return decorator

@repeat(n=3)
def hello(name: str):
    print(f'Hello {name}!')

hello('world')

Output:

Hello world!
Hello world!
Hello world!

`@trace`: Decorating a class with a function

We can also decorate classes, not just functions. As an example, assume we have an existing class Foo, and we would like to trace it, ie. get a print() each time a method is called, without having to manually change each method. So we'd like to be able to put @trace before the class definition and get this functionality for free, like:

@trace
class Foo:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

What does trace() look like? It must accepts a cls argument (the newly defined class, Foo in our case), and return a new/modified class (with added tracing):

def trace(cls: type):
    def make_traced(cls: type, method_name: str, method: Callable):
        def traced_method(*args, **kwargs):
            print(f'Executing {cls.__name__}::{method_name}...')
            return method(*args, **kwargs)
        return traced_method
    for name in cls.__dict__.keys():
        if callable(getattr(cls, name)) and name != '__class__':
            setattr(cls, name, make_traced(cls, name, getattr(cls, name)))
    return cls

The implementation is quite straightforward. We go through all methods in cls.__dict__.items(), and replace the method with a wrapped method, which we manufacture with the inner make_traced() function. It works:

f1 = Foo()
f2 = Foo(4)
f1.increment()
print(f1)
print(f2)

Output:

Executing Foo::__init__...
Executing Foo::__init__...
Executing Foo::increment...
Executing Foo::__str__...
This is a Foo object with i = 1
Executing Foo::__str__...
This is a Foo object with i = 4

`@singleton`: The singleton pattern

A second example of decorating a class with a function is implementing the common singleton pattern:

In software engineering, the singleton pattern is a software design pattern that restricts the instantiation of a class to one "single" instance. This is useful when exactly one object is needed to coordinate actions across the system.

Our implementation as a Python decorator @singleton:

def singleton(cls: type):
    def __new__singleton(cls: type, *args, **kwargs):
        if not hasattr(cls, '__singleton'):
            cls.__singleton = object.__new__(cls) # type: ignore
        return cls.__singleton                    # type: ignore
    cls.__new__ = __new__singleton                # type: ignore
    return cls

As mentioned in the Enum articles, the __new__() class method is called to construct new objects, before __init__() is called on the newly created instance to initialize it. So, to get singleton behaviour, we just need to override __new__() to always return a single instance. Let's test it:

@singleton
class Foo:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

@singleton
class Bar:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

f1 = Foo()
f2 = Foo(4)
f1.increment()
b1 = Bar(9)
print(f1)
print(f2)
print(b1)
print(f1 is f2)
print(f1 is b1)

Output:

This is a Foo object with i = 5
This is a Foo object with i = 5
This is a Bar object with i = 9
True
False

`@Count`: Decorating a class with a class

The reason the above code works is that in Python, class declarations are really just syntactic sugar for a function which constructs a new type object. For example, a class Foo declared above can also be defined programatically like:

def make_class(name):
    cls = type(name, (), {})
    setattr(cls, 'i', 0)
    def __init__(self, i): self.i = i
    setattr(cls, '__init__', __init__)
    def increment(self): self.i += 1
    setattr(cls, 'increment', increment)
    def __str__(self): return f'This is a {self.__class__.__name__} object with i = {self.i}'
    setattr(cls, '__str__', __str__)
    return cls

Foo = make_class('Foo')

But, if that's the case, we can not just decorate a function with a function, a class with a function, but also a class with a class. Let's see an example of this with the @Count pattern, where we want to count the number of instances created. We have an existing class, and we'd like to be able to just put @Count before class definition, and get a "free" count of instances created, that we can then access using the decorator Count class. The solution:

class Count:
    instances: DefaultDict[str, int] = defaultdict(int) # we will use this as a class instance
    def __call__(self, cls): # here cls is either Foo or Bar
        class Counted(cls): # here cls is either Foo or Bar
            def __new__(cls: type, *args, **kwargs): # here cls is Counted
                Count.instances[cls.__bases__[0].__name__] += 1
                return super().__new__(cls) # type: ignore
        Counted.__name__ = cls.__name__
        # without this ^ , self.__class__.__name__ would
        # be 'Counted' in the __str__() functions below
        return Counted

The trick is that when a class is decorated with Count, its __call__() method is invoked by the runtime, and the class is passed in as cls. Inside, we construct a new class Counted, which has cls as its parent, but overrides __new__(), and increments a counter in the Count class variable instances (but otherwise created a new instance and returns it). The newly constructed Counted class (whose name is overridden) is then returned, and replaces the original defined class. Let's see it in action:

@Count()
class Foo:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'
@Count()
class Bar:
    i: int = 0
    def __init__(self, i: int = 0):
        self.i = i
    def increment(self):
        self.i += 1
    def __str__(self):
        return f'This is a {self.__class__.__name__} object with i = {self.i}'

f1 = Foo()
f2 = Foo(6)
f2.increment()
b1 = Bar(9)
print(f1)
print(f2)
print(b1)
for class_name, num_instances in Count.instances.items():
    print(f'{class_name} -> {num_instances}')

Output:

This is a Foo object with i = 0
This is a Foo object with i = 7
This is a Bar object with i = 9
Foo -> 2
Bar -> 1

`@app.route`: Building a Flask-like application object by decorating functions

Finally, many of us have used Flask, and have written HTTP handler functions along the lines of:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello, World!'

This is yet another creative use of decorator patterns. Here we're building up an app object by adding our custom handler functions, but we don't have to worry about defining our own class derived from Flask, we just write flat functions which we decorate. This functionality is straightforward to duplicate as a toy Router class:

class Router:
    routes: dict[str, Callable] = {}

    def route(self, prefix: str):
        def decorator(func: Callable):
            self.routes[prefix] = func
        return decorator

    def default_handler(self, path):
        return f'404 (path was {path})'

    def handle_request(self, path):
        longest_match, handler_func = 0, None
        for prefix, func in self.routes.items():
            if path.startswith(prefix) and len(prefix) > longest_match:
                longest_match, handler_func = len(prefix), func
        if handler_func is None:
            handler_func = self.default_handler
        print(f'Response: {handler_func(path)}')

The only trick here is that the Router::route() can act like a decorator, and returns a function. Example usage:

app = Router()

@app.route('/')
def hello(_):
    return 'Hello to my server!'

@app.route('/version')
def version(_):
    return 'Version 0.1'

app.handle_request('/')
app.handle_request('/version')
app.handle_request('does-not-exist')

Output:

Response: Hello to my server!
Response: Version 0.1
Response: 404 (path was does-not-exist)

`@decorator` vs `@decorator()`

In the @measure example, we wrote:

@measure
def sleeper(seconds: int = 0):
    ...

Could we also write @measure() before the def? No! We would get an error:

measure() missing 1 required positional argument: 'func'

But, in the app.route() example, we do write the () parentheses. The situation is simple: roughly speaking, @decorator def func gets replaced by func = decorator(func). If we write @decorator() def func, it gets replaced by func = decorator()(func). So in the latter case, decorator() is run, and it needs to return a function which accepts a function as as an argument, and returns a function. This is how all the examples where the decorator takes an argument are structured.

Conclusion

In Python functions are first class citizens, and decorators are powerful syntactic sugar exploiting this functionality to give programmers a seemingly "magic" way to construct useful compositions of functions and classes. This is an important language feature that sets Python apart from traditional OOP languages like C++ and Java, where achieving such functionality requires more code, or more complex templated code. This dynamic nature of Python creates more runtime overhead compared to a language like C++, but it makes the code easier to wrote and comprehend. This is a win for programmers and projects; in most real-world software engineering efforts runtime performance is not a bottleneck.

Thanks Zsolt for bugfixes and improvement suggestions.

Building a toy Python Enum class - Part II

2022-05-05T00:00:00+02:00

Introduction

In the previous article I started to write a toy implementation of Python's Enum class. Here I will continue and add more features. The posts are an exercise in how to use Python's language features to build an easy-to-use interface, in this case a class which resembles old-school C++ enums.

The code for this post is up on Github. You should also check the official cpython implementation of Enum here on Github (it's 2018 lines of code).

Leaner auto()

A friend pointed out that the way I implemented auto() is quite inefficient and unneceassary! It can be simplified to:

class auto:
    pass

Also, we can change our code slightly so users can write both auto() or just auto when using this feature. In one case we will find an object of type auto, in the other case we will find the type auto itself:

class EnumMeta(type):
    def __new__(metacls, cls, bases, classdict, **kwds):
        enumerations = {x: y for x, y in classdict.items() if not x.startswith('__')}
        # handle auto()
        next_value = 1
        for k, v in enumerations.items():
            if type(v) is auto or v is auto: # <--------------
                enumerations[k] = next_value
                next_value += 1
            else:
                next_value = v + 1
        enum = super().__new__(metacls, cls, bases, classdict, **kwds)
        enum._enumerations = enumerations
        return enum
    ...

class Color(Enum):
    RED: int = auto
    GREEN: int = auto
    BLUE: int = auto

String representations

We want both Enum classes (such as Color) and Enum objects (such as Color(1)) to have nice string representations, like with the standard library Enum. This is easy:

class EnumMeta(type):
    ...
    def __str__(cls):
        return f'<enum \'{cls.__name__}\'>'
    ...

class Enum(metaclass=EnumMeta):
    ...
    def __str__(self):
        return "%s.%s" % (self.__class__.__name__, self.__key)
    ...

Note that with EnumMeta, the first argument is cls (not self) which is a type object. This gets called when the Color class itself gets printed out:

class Color(Enum):
    RED: int = auto()
    GREEN: int = auto()
    BLUE: int = auto()

print(Color)          # __str__() in EnumMeta
print(type(Color(1))) # __str__() in EnumMeta
print(Color(1))       # __str__() in Enum

Output:

<enum 'Color'>
<enum 'Color'>
Color.RED

Pretty!

Accessors

Next, let's make it so Color['RED'] and Color.RED returns a appropriate color object. The first one, Color['RED'] is quite simple: when we use [] on an object, the class' __getitem__() is called. For Color['RED'], the object is the class itself, it's type is EnumMeta. So all we have to do is:

class EnumMeta(type):
    def __getitem__(cls, key):
        return cls(cls._enumerations[key])

Making Color.RED work is a bit more tricky. The basic idea is simple: when we use the dot . on an object, the class' __getattr__() and/or __getattribute__() is called. To understand the difference between these two, check this post. __getattr__() is called if the attribute is not found, and the user wishes to return a computed value. In our case, Color.RED is defined, so we have to override __getattribute__(), which is always called by Python when accessing an attribute with dot operator. So, it's natural to think that a copy/paste version of the above __getitem__() will work here:

class EnumMeta(type):
    ...
    def __getattribute__(cls, key):
        return cls(cls._enumerations[key]) # recursively call itself, kills the kernel!

The problem is that we use the dot operator all over the place, including the EnumMeta constructor:

enum._enumerations = enumerations

What we have to do is to explicitly use Python's built-in __getattribute__() function, like this:

class EnumMeta(type):
    def __getattribute__(cls, key):
        if key.startswith('_'):
            return object.__getattribute__(cls, key)
        else:
            return cls(object.__getattribute__(cls, '_enumerations')[key])
    ...

What this does is, if we are trying to access an attribute which starts with an underscore, such as __class__ or our own _enumerations, it routes it through the usual channel of the built-in Python attribute look-up. But in other cases, such as Color.RED, it returns a new object constructed on the fly: cls(...) in the last line would be Color(...), object.__getattribute__() is used to avoid recursively entering this function, and _enumerations[key] is our own helper dictionary where we store the Enum's cases. In the case of Color.RED, key is RED, _enumerations[key] is 1, so the whole thing becomes return Color(1).

Membership and equality

Another useful feature of the standard library Enum is the ability to check Color.RED in Color, and of course Color.RED == Color.GREEN checks. These are easy:

class EnumMeta(type):
    ...
    def __contains__(cls, other):
        if type(other) == cls:
            return True
        else:
            return False  

print(Color['RED'] in Color)    # True
print(Color.RED in Color)       # True
print(Color(1) in Color)        # True
print(Color.RED in WeekendDay)  # False
print(Color.BLUE in WeekendDay) # False

Equality checks:

class Enum(metaclass=EnumMeta):
    ...
    def __eq__(self, other):
        if type(self) != type(other):
            return False
        else:
            return (self.__key == other.__key and self.__value == other.__value)

print(Color.RED == Color)               # False
print(Color.RED == Color.RED)           # True
print(Color.RED == Color['RED'])        # True
print(Color.RED == Color(1))            # True
print(Color.RED == Color(2))            # False
print(Color.RED == WeekendDay.SATURDAY) # False

Static objects

There is an issue with the implementation so far:

print(Color.RED is Color.RED)    # False
print(Color.RED is Color['Red']) # False
print(Color.RED is Color(1))     # False

At least the first one should be True, and probably all of them (in the standard library version, they're all True). Unlike equality, Python's is checks whether the variables are referring to the same object. But in our implementation so far, all of these calls return a new object (by doing cls(...)). This both breaks the above checks, and is also wasteful. Let's fix this.

Let's create one instance for each of the Enum cases (RED, GREEN, BLUE in the example) up front, and always return references to these objects, in a transparent way, so the user doesn't notice it. The first step is relatively easy, we create these static instances when the class is defined:

class EnumMeta(type):
    def __new__(metacls, cls, bases, classdict, **kwds):
        ...
        # make "static" instances of each enumeration object
        enum._instances = {k: object.__new__(enum) for k, v in enumerations.items()}
        # initialize static instances
        for k, instance in enum._instances.items():
            instance.__init__(enumerations[k])
        ...

Now we have to change our code to always return these saved instances. In our toy Enum class, there are several ways to get an Enum object such as Color: Color.RED, Color['RED'], Color(1). Making the first two return the saved instances is easy, because we already explicitly control what happens in those cases with our __getitem__() and __getattribute__() implementation. But what about when the user explicitly calls the constructor like Color(1), which we also do currently in our functions when returning Enum objects. If we get the constructor to return the static instances, we're done.

In Python, when the user calls the constuctor, two things happen. First, the class' __new__() is called to construct the object (so this has a return value), and then __init__() is called to initialize the already created object, which is conventionally called self (this has no return value, as self is already a given). Clearly here we have to write our custom __new__(), and also avoid unneceassary duplicate initialization in __init__(). The cleanest way to make this work that I have found is to override the newly created's Enum's' __new__() when the type is being created:

class EnumMeta(type):
    def __new__(metacls, cls, bases, classdict, **kwds):
        ...
        # overwrite the new Enum's __new__() so that is returns the static instances
        enum.__new__ = lambda cls, value: enum._instances[reverse_enumerations[value]]
        return enum

Short circuiting the __init__() is an easy further optimization:

class Enum(metaclass=EnumMeta):    
    def __init__(self, value):
        if hasattr(self, '_Enum__key') and hasattr(self, '_Enum__value'):
            return
        ...

The attributes are called like _Enum__key because of Python's mangling of "private class members".

By inserting some print() statements, we can verify that our code works as intended:

class Color(Enum):
    RED: int = auto
    GREEN: int = auto
    BLUE: int = auto

class WeekendDay(Enum):
    SATURDAY: int = auto
    SUNDAY: int = auto

print(Color.RED == Color)               # False
print(Color.RED == Color.RED)           # True
print(Color.RED is Color.RED)           # True
print(Color.RED == Color['RED'])        # True
print(Color.RED == Color(1))            # True
print(Color.RED == Color(2))            # False
print(Color.RED == WeekendDay.SATURDAY) # False
print(Color.RED is Color.RED)           # True
print(Color.RED is Color['RED'])        # True
print(Color.RED is Color(1))            # True
for _ in range(100):
    c = Color(3)
    d = WeekendDay(2)

Output:

Constructing new type Enum
Constructing new type Color
Initializing new <enum 'Color'>
Initializing new <enum 'Color'>
Initializing new <enum 'Color'>
Constructing new type WeekendDay
Initializing new <enum 'WeekendDay'>
Initializing new <enum 'WeekendDay'>
False
True
True
True
True
False
False
True
True
True

First the Enum type itself is constructed, then Color, the 3 Color instances are created, then the WeekendDay type is constructed, 2 WeekendDay instances are created. The actual code itself doesn't create any more objects!

Final version

The final version is 71 lines of code:

class auto():
    pass

class EnumMeta(type):
    def __new__(metacls, cls, bases, classdict, **kwds):
        enumerations = {x: y for x, y in classdict.items() if not x.startswith('__')}
        # handle auto() and auto
        next_value = 1
        for k, v in enumerations.items():
            if type(v) is auto or v is auto:
                enumerations[k] = next_value
                next_value += 1
            else:
                next_value = v + 1
        enum = super().__new__(metacls, cls, bases, classdict, **kwds)
        enum._enumerations = enumerations
        reverse_enumerations = {y: x for x, y in enumerations.items()}
        # make "static" instances of each enumeration object
        enum._instances = {k: object.__new__(enum) for k, v in enumerations.items()}
        # initialize static instances
        for k, instance in enum._instances.items():
            instance.__init__(enumerations[k])
        # overwrite the new Enum's __new__() so that is returns the static instances
        enum.__new__ = lambda cls, value: enum._instances[reverse_enumerations[value]]
        return enum

    def __str__(cls):
        return f'<enum \'{cls.__name__}\'>'

    def __len__(cls):
        return len(cls._enumerations)

    def __iter__(cls):
        return (cls(value) for value in cls._enumerations.values())

    def __getitem__(cls, key):
        return cls(cls._enumerations[key])

    def __getattribute__(cls, key):
        if key.startswith('_'):
            return object.__getattribute__(cls, key)
        else:
            return cls(object.__getattribute__(cls, '_enumerations')[key])

    def __contains__(cls, other):
        if type(other) == cls:
            return True
        else:
            return False

class Enum(metaclass=EnumMeta):    
    def __init__(self, value):
        if hasattr(self, '_Enum__key') and hasattr(self, '_Enum__value'):
            return
        # make sure the passed in value is a valid enumeration value
        if value not in self.__class__._enumerations.values():
            raise ValueError(f'{value} is not a valid {self.__class__.__name__}')
        # save the actual enumeration value
        for k, v in self.__class__._enumerations.items():
            if v == value:
                self.__key = k
                self.__value = v

    def __str__(self):
        return "%s.%s" % (self.__class__.__name__, self.__key)

    def __eq__(self, other):
        if type(self) != type(other):
            return False
        else:
            return (self.__key == other.__key and self.__value == other.__value)

You should also check the official cpython implementation of Enum here on Github (it's 2018 lines of code).

Conclusion

This was a great exercise to improve and exercise my Python Fu. Highly recommended! Thanks Zsolt for bugfixes and improvement suggestions.

Building a toy Python Enum class - Part I

2022-05-03T00:00:00+02:00

Introduction

Enumerations, or enums for short, are part of the core language in traditional statically typed languages like C, C++ and Java. For example, in C++ we can write:

enum Color { RED, GREEN, BLUE };
Color r = RED;

switch(r)
{
    case RED  : std::cout << "red\n";   break;
    case GREEN: std::cout << "green\n"; break;
    case BLUE : std::cout << "blue\n";  break;
}

When the C++ compiler sees an enum definition, it allocates a special memory structure wide enough to hold the possible values (in the above case, there are 3 possible values, so 2 bits would be enough, but the compiler would actually allocate at least 1 byte, usually 4 bytes). Since C++ is statically typed, if we accidentall write a case statement like case PURPLE : std::cout << "purple\n"; break; we would get an error from the compiler, since it knows that PURPLE is not a valid/possible value of Color, as defined above.

Python is a very different language from C++: enumerations are not part of the core Python language, unlike say tuples. In Python, Enums (upper case) are part of the standard library, and are implemented using Python code, using Python classes in a tricky way. For example, here is enum.py (link) from cpython, and on line 1077 you can read the (very tricky) implementation of Enums. It looks something like:

class Enum(metaclass=EnumType):
    ...

As a former C++ programmer, I find this weird, cool and intriguing. So I decided to practice my Python Fu and write my own toy implmenentation of Python's Enum class, specifically for int values. Emphasis on "toy"; the full source code for enum.py linked above is 2018 lines of code (!), which also includes related classes such as IntEnum, StrEnum, Flag, etc.

The ipython notebook is up on Github.

Enumerations in Python

In Python, to use standard library Enums we have to first import it from the enum module, like:

from enum import Enum

class Color(Enum):
    RED: int = 1
    GREEN: int = 2
    BLUE: int = 3

Alternatively, we can skip giving values by hand, and use the auto() magic function:

from enum import Enum, auto

class Color(Enum):
    RED: int = auto()
    GREEN: int = auto()
    BLUE: int = auto()

The magic auto() function starts numbering at 1, so the above two are equivalent.

There are various ways to create enums:

c1 = Color(1)
c2 = Color(Color.RED)
c3 = Color(Color['RED'])
print(type(Color), type(c1), type(Color.RED), type(Color['RED']))

Output:

<class 'enum.EnumMeta'> <enum 'Color'> <enum 'Color'> <enum 'Color'>

Truthiness tests:

Color(1) == 1             # false
Color.RED == Color(1)     # true
Color.RED == Color['RED'] # true

An Enum itself (not an instance) has a useful iteration interface (this sort of convenience does not exist in C++):

print(len(Color)) # 3
# can iterate:
for c in Color:
    print(c)

Output:

3
Color.RED
Color.GREEN
Color.BLUE

Examples of things that don't work with enums:

Color(4)      # ValueError: 4 is not a valid Color
Color('RED')  # ValueError: 'RED' is not a valid Color
c1 = Color(1) # okay
c1 += 1       # TypeError: unsupported operand type(s) for +=: 'Color' and 'int'

Metaclasses in Python

So, as an exercise in Python meta-programming, let's write a simple Enum class that accomplishes the above. In the code above, notice that type(Color) is <class 'enum.EnumMeta'>, this is a big clue. Python has a feature called metaclasses, which is a way to construct classes the way we construct objects. In Python, classes in fact are just objects of type type:

class Color():     # not deriving from Enum
    RED: int = 1
    GREEN: int = 2
    BLUE: int = 3

print(type(Color))

Output:

<class 'type'>

But then, when we printed the type for Color(Enum), we got <class 'enum.EnumMeta'>, so what's going on? Let's check:

from enum import EnumMeta, Enum, auto

class Color(Enum):
    RED: int = 1
    GREEN: int = 2
    BLUE: int = 3

print(type(Color), type(Enum), type(EnumMeta))

Output:

<class 'enum.EnumMeta'> <class 'enum.EnumMeta'> <class 'type'>

So both Enum and Color are of type EnumMeta, and EnumMeta itself is of type type. We will follow this pattern in our own implementation of Enum.

In Python, with metaclasses, we can create classes that are of our own metaclasses' type, and we can have "constructor" code run when the class is defined (not when instances are created). Let's see an example:

class Enum(type):
    def __new__(metacls, cls, bases, classdict, **kwds):
        print(f'Defining new Enum type {cls}:')
        print(f'- metacls = {metacls}')
        print(f'- bases = {bases}')
        print(f'- classdict = {classdict}')

class Color(metaclass=Enum): # specifying our own Enum metaclass
    RED: int = 1
    GREEN: int = 2
    BLUE: int = 3

Output:

Defining new Enum type Color:
- metacls = <class '__main__.Enum'>
- bases = ()
- classdict = {'__module__': '__main__', '__qualname__': 'Color',
  '__annotations__': {'RED': <class 'int'>, 'GREEN': <class 'int'>,
  'BLUE': <class 'int'>}, 'RED': 1, 'GREEN': 2, 'BLUE': 3}

Note that the code in __new__() ran when we declared Color! We never created an instance of Color in the snippet above! This is where we start our toy implementation, and this is also how the standard library Enum works: by constructing a special class when a class deriving from Enum is declared. Here we are actually not deriving, we are metaclassing, which we will fix later.

Basic `Enum` functionality

The example above was just an illustration, a Color defined like this is not useful. To accomplish the standard library functionality, we will use a chain of classes similar to enum.py:

class EnumMeta(type):
    ...

class Enum(metaclass=EnumMeta):
    ...

class Color(Enum):
    ...

This way, when we define an Enum like Color, it derives from Enum, which is of metaclass EnumMeta, so this way we can use both inheritance and metaclassing features of Python. As a first order of business, in EnumMeta's __new__(), let's go through the enumerations defined by the user and save them into the dictionary. As seen in the output above, these are available in the passed in classdict object:

class EnumMeta(type):
    def __new__(metacls, cls, bases, classdict, **kwds):
        enumerations = {x: y for x, y in classdict.items() if not x.startswith('__')}
        enum = super().__new__(metacls, cls, bases, classdict, **kwds)
        enum._enumerations = enumerations
        return enum

class Enum(metaclass=EnumMeta):
    pass

class Color(Enum): # specifying our own Enum metaclass
    RED: int = 1
    GREEN: int = 2
    BLUE: int = 3

Now let's check how this works:

print(Color._enumerations)                     # {'RED': 1, 'GREEN': 2, 'BLUE': 3}
print(type(Color), type(Enum), type(EnumMeta)) # ...
Color(1)                                       # TypeError: Color() takes no arguments

The last call to the Color() constructor will fail, because the default constructor in Python does not take arguments. Let's fix this in the Enum base class:

class Enum(metaclass=EnumMeta):
    def __init__(self, value):
        # make sure the passed in value is a valid enumeration value
        if value not in self.__class__._enumerations.values():
            raise ValueError(f'{value} is not a valid {self.__class__.__name__}')
        # save the actual enumeration value
        for k, v in self.__class__._enumerations.items():
            if v == value:
                self.__key = k
                self.__value = v

Now we can try again:

Color(1) # okay
Color(4) # ValueError: 4 is not a valid Color

We have provided our own automatic constructor, which takes just the values the user-defined Enum should. We can pick some low hanging fruit and get string and iteration related functionality working:

class EnumMeta(type):
    ...

    def __len__(cls):
        return len(cls._enumerations)

    def __iter__(cls):
        return (cls(value) for value in cls._enumerations.values())

class Enum(metaclass=EnumMeta):
    ...

    def __str__(self):
        return "%s.%s" % (self.__class__.__name__, self.__key)

print(Color(1))
print(len(Color))
for c in Color:
    print(c)

Output:

Color.RED
3
Color.RED
Color.GREEN
Color.BLUE

Adding `auto()`

An easy feature to add is the magic auto() function, which enables us to avoid writing out int values, and increments automatically in the class definition. auto() is just a function that runs when the class is defined, so it needs to return something. Then the results can be "cleaned up" in the metaclass's __new__() function. We could use None or -999 as the auto() a return value, but that would conflict with the user using that value in their own Enums, so let's create a class just for this:

class _Auto():
    pass

def auto():
    return _Auto()

Then:

class EnumMeta(type):
    def __new__(metacls, cls, bases, classdict, **kwds):
        enumerations = {x: y for x, y in classdict.items() if not x.startswith('__')}
        # handle auto()
        next_value = 1
        for k, v in enumerations.items():
            if type(v) != _Auto: # if auto() was used, v will be an _Auto
                next_value = v + 1
            else:
                enumerations[k] = next_value
                next_value += 1
        enum = super().__new__(metacls, cls, bases, classdict, **kwds)
        enum._enumerations = enumerations
        return enum
    ...

Now we can do:

class Color(Enum):
    RED: int = auto()
    GREEN: int = auto()
    BLUE: int = auto()

The values will be replaced with 1, 2, 3.

The final version so far:

class _Auto():
    pass

def auto():
    return _Auto()

class EnumMeta(type):
    def __new__(metacls, cls, bases, classdict, **kwds):
        enumerations = {x: y for x, y in classdict.items() if not x.startswith('__')}
        # handle auto()
        next_value = 1
        for k, v in enumerations.items():
            if type(v) != _Auto:
                next_value = v + 1
            else:
                enumerations[k] = next_value
                next_value += 1
        enum = super().__new__(metacls, cls, bases, classdict, **kwds)
        enum._enumerations = enumerations
        return enum

    def __len__(cls):
        return len(cls._enumerations)

    def __iter__(cls):
        return (cls(value) for value in cls._enumerations.values())

class Enum(metaclass=EnumMeta):
    def __init__(self, value):
        # make sure the passed in value is a valid enumeration value
        if value not in self.__class__._enumerations.values():
            raise ValueError(f'{value} is not a valid {self.__class__.__name__}')
        # save the actual enumeration value
        for k, v in self.__class__._enumerations.items():
            if v == value:
                self.__key = k
                self.__value = v

    def __str__(self):
        return "%s.%s" % (self.__class__.__name__, self.__key)

Conclusion

This is good progress, but some things are still missing: Color['RED'] doesn't work, Color.RED returns an int, equality doesn't work, etc. In the next part I will add more features to this toy class to cover the most commonly used functionality of the standard library Enum.

"Over 70% of all Porsche vehicles ever built are still on the road today"

2022-04-30T00:00:00+02:00

Introduction

In 2021 November I went to a Porsche event in Dubai called Icons of Porsche. It was a very cool (free) event, where 100s of iconic Porsche cars were on display, both from local owners, and also many transported from the official Porsche museum in Stuttgart for this event! Near the entrance of the event I snapped this picture:

This has been tingling my curiosity ever since. Is this a testament of the quality and longevity of Porsche cars, or simply a result of the brand switching from niche sportscar manufacturing to mass production around the year 1999?

For those who don't know much about Porsches, the brand exploded when the company started to manufacture less expensive and more practical cars around 2000:

Cayman/Boxster (2 seater), introduced in 1996
Cayenne (big SUV), introduced in 2003
Panamera (wagon), introduced in 2009
Macan (small SUV), introduced in 2014
Taycan (electric), introduced in 2019

The iconic Porsche 911 also had a significant switch in 1998. Models before this were air-cooled, models after were water-cooled. The air-cooled 911s are much more of a collector's item than the later generations. Also, the first water-cooled 911, the 996 had the famous broken-egg shaped lights, which many enthusiasts (including me) find ugly. The 996 and later models are not considered collector's items, while the previous generations, including the last air-cooled 993, are. If you remember the original Bad Boys movie (with Will Smith and Martin Lawrence), he famously drove a black 993 generation 911 Turbo. That exact car happened to be on display at the Mall of Emirates a few years ago, where I snapped this picture:

So the question is: is the seemingly high 70% statistic because a lot of Cayenne/Macans were built in the last 15 years, and these cars simply haven't had time yet to die? Or do Porsche cars in fact have a relatively long lifespan?

The ipython notebook is up on Github.

Porsche production timeseries

This page has annual production numbers from 1998 to 2021. Scrolling down there is also a break-down by model, this shows that the sportscar segment (911, Cayman) hasn't grown much, the growth is coming from the "practical" segment (Cayenne, Macan, Panamera).

This official Porsche page has some scattered annual and cumulative production numbers for the preceeding 50 years. These are:

1947: 0 produced
1948: 52 produced that year
1956: 10,000 total produced up to this year
1963: 11,000 produced
1969: 14,000 produced
1977: 300,000 total produced up to this year
1996: 1,000,000 total produced up to this year

I wrote a quick function to compute a piecewise linear fit given these constraints (it's not a simple fit, since the "integral" is also involved):

Number = Union[float, int]
TimeSeries = Sequence[tuple[int, Number]] # type alias

def piecewise_linear_fit(f: TimeSeries, c: TimeSeries) -> TimeSeries:
    # compute a piecewise linear fit to f, also taking
    # into account the cumulative sum constraints in c
    r = [f[0]]
    fi, ci = 1, 0
    while True:
        if ci >= len(c) and fi >= len(f):
            break
        elif ci >= len(c) or f[fi][0] <= c[ci][0]:
            # the next closest data point is in f, use that to get the slope
            slope = (f[fi][1] - r[-1][1]) / (f[fi][0] - r[-1][0])
            to = f[fi][0]
            fi += 1
        elif fi >= len(f) or f[fi][0] > c[ci][0]:
            # the next closest data point is in c, use that to get the slope
            ts = range(1, c[ci][0] - r[-1][0] + 1)
            slope = (c[ci][1] - sum([t[1] for t in r]) - r[-1][1]*len(ts)) / sum(ts)
            to = c[ci][0]
            ci += 1
        (x_0, y_0) = r[-1]
        for x in range(r[-1][0]+1, to+1):
            r.append((x, y_0 + slope * (x - x_0)))
    return r

Note that given that we're doing a simple survival analysis, the shape of the curve in the 20th centure won't matter much, and one of the last cumulative constraints is at 1996 anyway.

Running this, we get the following fit for the annual production curve from 1948 to 2021:

Same, but showing cumulatives:

Naive survival

First, let's do a naive survival model. Let's assume that cars are manufactured, are alive for lifespan years, and then they die:

def ratio_alive(f: list, lifespan: int) -> float:
    total = sum(f)
    alive = sum(f[-lifespan:])
    return alive/total

ds = []
for lifespan in range(10, 25):
    r = ratio_alive([x[1] for x in f], lifespan)
    ds.append((lifespan, r))
    print(f'{lifespan} -> {r:.3f}')

Output:

10 -> 0.519
11 -> 0.545
12 -> 0.567
13 -> 0.585
14 -> 0.603
15 -> 0.625
16 -> 0.647
17 -> 0.668
18 -> 0.688
19 -> 0.705
20 -> 0.720
21 -> 0.733
22 -> 0.745
23 -> 0.756
24 -> 0.766

We can read off the output that in this simple model, we have to assume a lifespan of 19 years to get a 70% survival rate for today.

Monte carlo survival

The above gives a good initial indicator, but it's too naive even for a naive model. The problem is, even if we say that it suggests an average lifespan of 19 years, it's biased. Old cars (eg. produced in 1975), even if their lifespan is off from the average, by 2021 most will have died. But for recent cars, we could get early deaths probabilistically, which would drive down the survival rate. Ie. if a car produced from 1975 dies at 5 years, it doesn't affect our survival rate in 2021, but if a car produced in 2015 dies "too early" in 2020, it does. So let's model the lifespan of the cars as a normal distribution centered around an average lifespan:

avg_lifespan, sigma = 20, 5 # years
total, num_alive = 0, 0
for (year, num_produced) in f:
    for _ in range(round(num_produced)):
        expiry_year = round(year + max(0, normal(avg_lifespan, sigma)))
        total += 1
        if expiry_year >= 2022:
            num_alive += 1
survival_ratio = num_alive/total
print(f'Assuming avg_lifespan = {avg_lifespan} years and sigma = {sigma}, survival % = {survival_ratio*100:.1f}')

Output:

Assuming avg_lifespan = 20 years and sigma = 5 years, survival % = 70.4

So introducing a normal distribution doesn't change the result much. It's worth noting that an increasing sigma with a fixed average lifespan yields a lower survival %. So if we assume a 10 year sigma, to get 70% we'd have to assume an average lifespan of 22 years.

We can also try to divide the cars into a classics vs mass-produced, by assuming cars before 1999 are classics, and after are mass-produced, and using two separate normal distributions for their lifespans. Here we can play around with the assumption that the classic cars get a lot of care and loving, don't get driven much, and hence have very long lifespans, while the mass produced cars have lower lifespans:

# avg_lifespan_classics, sigma_classics = 33, 10 # years
# avg_lifespan_massprod, sigma_massprod = 15, 5  # years
avg_lifespan_classics, sigma_classics = 25, 10 # years
avg_lifespan_massprod, sigma_massprod = 17, 5  # years
classics_year_end = 1999
total, num_alive = 0, 0
for (year, num_produced) in f:
    for _ in range(round(num_produced)):
        if year < classics_year_end:
            expiry_year = round(year + max(0, normal(avg_lifespan_classics, sigma_classics)))
        else:
            expiry_year = round(year + max(0, normal(avg_lifespan_massprod, sigma_massprod)))
        total += 1
        if expiry_year >= 2022:
            num_alive += 1
print(f'Assuming avg_lifespan_classics = {avg_lifespan_classics} years and sigma_classics = {sigma_classics} years')
print(f'Assuming avg_lifespan_massprod = {avg_lifespan_massprod} years and sigma_massprod = {sigma_massprod} years')
print(f'Survival % = {survival_ratio*100:.1f}')

Output:

Assuming avg_lifespan_classics = 25 years and sigma_classics = 10 years
Assuming avg_lifespan_massprod = 17 years and sigma_massprod = 5 years
Survival % = 70.4

Of course there are lot of parameter choices of the 4 four parameters (2 averages, 2 sigmas) that yield a 70% survival rate. But taking all 3 versions into account, it suggests that modern Porsches have a 15-20 year average lifespan, which does sound pretty good. Google suggests that the average car's lifespan is 8 years.

Conclusion

The analysis is quite naive, but because all 3 models were in roughly the same range, I think the conclusion of modern Porsches having a 15-20 year average lifespan is directionally sound. Having said that, some caveats and comments:

Lifespan in itself is not just a property of the car/brand, it's also a function of the owners. Many Porsche owners love their cars and take good care of it, which increases their lifespans (compared to a Suzuki). Also, for similar reasons many Porsche cars don't get driven as many miles as other brands, which reduces opportunities for the car to die.
Even if a car brand has good reliability and owners take care of it, if it's been mass producing cars for a long time, the survival rate would be significantly lower. For example, the Ford Motor Company has been mass producing cars since 1901.
Lifespan is not necessarily related to low fault rates. In my experience, Porsche cars have just as many issues (or more) as other similar brands (BMW, Land Rover/Jaguar). Official statistics are in disagreement, some suggest Porsches are reliable, other suggest it's below-average.
Here lifespan-in-years was the primary input metric, but a better analysis would also take into account lifespan-in-miles.
I don't know how realistic a normal distribution is for modeling the lifespan, I didn't research how cars die. The sigmas don't seem to matter much.
Doing a per model (eg. Macan, Cayenne, 911, etc) model doesn't seem to be worth it, since already the classic vs. mass-produced had too many parameters.

"The company is all hot air"

2022-04-26T00:00:00+02:00

Theranos

In 2013, user car posted a link titled Theranos to Hacker News, pointing to the company's website. A few minutes later a throaway user account medman77 was created, and posted the following comment:

"The company is all hot air. They have a board full of retired military figure heads that have no experience in medical devices or retail services. Additionally, they do not have any products to show. Look at their patents. They are all very general and broad. There has been NO FDA CLEARANCE for anything they are doing, which raises legal questions. Speaking of legal, search for lawsuits they are involved in. Their core technology is not even theirs. They stole it from someone else."

Two years later, John Carreyrou published his first article on Theranos in the Washington Post, Hot Startup Theranos Has Struggled With Its Blood-Test Technology. This was the first blow to the house of cards that was Theranos, a company that raised \$1B and was valued at \$9B at its peak. The company was dissolved September 4, 2018, 15 years after its founding. Since then, CEO Elizabeth Holmes and COO Sunny Balwani were charged with wire fraud and conspiracy, with Holmes being found guilty on four counts in January 2022. Her sentencing is set for late 2022. Balwani's trial began in March 2022.

I first learned about Theranos sometime in 2018 when John Carreyrou's excellent book Bad Blood popped up in my Amazon recommendations. I bought the book, read it, and I've been hooked on the story ever since. In case you haven't heard about Theranos, there are so many great accounts, I will not repeat the story here. It's a fascinating story, full of patterns you can learn and watch out for in real life. The best sources are:

Bad blood (book)
The dropout (series) is an 8 episode Hulu series about Theranos starring Amanda Seyfried as Elizabeth Holmes. The series is done so well, it's worth watching even if you've read the book.
Coffee and Cults (Youtube) is a Youtube channel which focuses on Theranos and WeWork content
Theranos and Elizabeth Holmes on Wikipedia

WeWork

The WeWork story is similar, but not identical to Theranos. In the case of Theranos, the CxOs misled investors by telling them they had a working machine that can run multiple tests on a drop of blood, when they didn't. Furthermore, their lab returned unreliable test results to millions of patients. In the case of WeWork, it was a case of grand visions sold to investors leading to outsized valuations, accompanied by a huge burn rate. WeWork raised a total of \$21B, the peak valuation was \$47B, but the current, post-bubble valuation is approximately \$5B. At its peak, the company was reportedly burning \$100M per week. What's remarkable is that when WeWork crashed and 90% of (virtual) shareholder value was destroyed, the two co-founders Adam Neumann and Miguel McKelvey still walked away with a combined $2B cash. Former CEO Adam Neumann was never sued, in fact it was he who sued investor Softbank to get the full amount of his final payout. Although WeWork's valuation tanked and Adam was fired, the company still exists, was led through Covid times by new management, and their co-working spaces still operate today.

Similarly to Theranos this is also a fascinating story worth diving into:

WeWork (documentary) is a 90-minute Hulu documentary, definitely worth watching
WeCrashed (series) is an 8 episode Apple series about WeWork, starring Jared Leto and Anne Hathaway
Coffee and Cults (Youtube) is a Youtube channel which focuses on Theranos and WeWork content
The Cult of We (book) - I haven't read it yet, so cannot make a recommendation
Billion Dollar Loser (book) - I haven't read it yet, so cannot make a recommendation
WeWork and Adam Neumann on Wikipedia

In case you're wondering how Adam Neumann convinced his investors that re-selling office space is a $47B business, watch this 40 minute post-crash interview him. He's insanely likeable, well-spoken, a master of selling and public relations.

The startup hustle

The word "hustler" has many meanings:

a person who is hard-working, passionate and determined to succeed (positive)
a person who is good at selling (positive)
a person who achieves their goals by unethical means (negative)

Investors want co-founders to be the first two, but not the last. Determination is important because startups are hard, and being the CEO of your own startups is the hardest job you'll ever have. Selling is important because on the way you'll have to convince a lot of people that you're right: investors, employees, customers, press.

What's interesting is the gray area where these 3 definitions intersect, when founders sell their vision really hard. When selling a vision and creating a 5 year sales forecast, where is the limit between ethical but extreme optimism and making it up in an unethical way?

Conclusion

Even in my limited experience of working at ~10 companies in my life, out of which 5-6 were startups (4 failed, including my own), I've seen many cases where the co-founders sold their vision hard and didn't fully succeed, and investors didn't get the returns they hoped for. So, this is actually quite common in startups. VCs know that co-founders are selling their vision hard, and take this into account. They don't expect all investments to succeed. They assume only 1 in $N$ will be a home-run, and they don't really care why the other $N-1$ failed (as long as there was no fraud). The only company I ever worked at where the founders fully achieved their goals (and more) was Facebook, and companies like Facebook should be considered an outlier — which is exactly what investors are trying to find.

Finally, it's worth noting that good investors also pick bad apples, see uBiome, which was YCombinator and Andreessen Horowitz backed, roughly in the same space as Theranos. According to Wikipedia:

In 2021, the Securities and Exchange Commission charged two of the cofounders (Richman and Apte) with defrauding investors.

Python types for Data Scientists - Part III

2022-04-22T00:00:00+02:00

Introduction

In the first post I showed how to get started using Python static type checking in ipython notebooks. The second post looked at slightly more advanced uses of typing to further increase the safety and readability of code. Here I will continue, and look at some aspects of type hinting:

type check errors and runtime errors
where type hints don't work
Abstract Base Classes vs. Protocols
types for class variables vs instance variables

The ipython notebook is up on Github. The best reference is the official Python documentation of the typing module.

Type check errors and runtime errors

It's important to remember that in Python, type hints are optional and ignored (not enforced) by the Python runtime. Type hints are interpreted by external programs. In these example, I use nb_mypy, which actually runs mypy to do type checking. Then, irrespective of the result of type checking, the regular Python runtime runs (and ignores all type hints). In other contexts, when using an IDE, the IDE would run type checks in the background, and show errors, or use the type hint informations for code complete.

Given the roots of Python, I find this to be a good trade-off to introduce typing and get 80% of the benefits.

But, it leads to some weird behaviour, which cannot be changed with nb_mypy: even if there is a typecheck error in the current cell, the code code in the cell will still be run after the typecheck completed. This leads to some confusing outputs. For example:

def foo(i: int) -> None:
    print(i)

foo("hello")

Output:

error: Argument 1 to "foo" has incompatible type "str"; expected "int"
hello

First the nb_mypy type checker runs, finds the type error, prints the error, but then the code is executed anyway. And since the Python runtime ignores all typehints, the code runs just fine, since to Python the function foo() is equivalent to:

def foo(i):
    print(i)

Where type hints don't work

There are some cases where writing type hints does not work as we'd expect. The big ones are for and while loops:

for i: int in range(3):
    print(i)

Output:

SyntaxError: invalid syntax

This is not a type check error, it's a syntax error. We cannot put the type hint for i in the for loop itself. It has to go before:

i: int
for i in range(3):
    print(i)

However, at least in such simple cases, I would just skip the type hint since it's quite ugly. It's actually not required, the type checker can infer the int type from range(), so this will throw a type check error:

def f(s: str) -> None:
    pass

for i in range(3): # okay
    f(i)           # not okay

Output:

error: Argument 1 to "f" has incompatible type "int"; expected "str"

Abstract Base Classes vs. Protocols

In the previous post, there was the example of declaring a Protocol for addability:

class Addable(Protocol):
    def __add__(self, other): # anything that declares __add__() can stand in for an Addable
        raise NotImplementedError

T = TypeVar('T', bound=Addable)

def add(a: T, b: T):
    print(type(a), type(b))
    print(a+b) # checks for __add__(), uses __str__()

add(int(1), int(2)) # okay

In this example, what we pass to add() needs to declare an __add__(). So if we define our own class MyInt like this:

class MyInt():
    num: int
    def __init__(self, num: int):
        self.num = num

add(MyInt(1), MyInt(2)) # not okay

Output:

error: Value of type variable "T" of "add" cannot be "MyInt"

We can fix this by implementing __add__() in MyInt:

class MyInt(): # note that MyInt does not inherit Addable
    num: int
    def __init__(self, num: int):
        self.num = num
    def __add__(self, other):
        return MyInt(self.num + other.num)
    def __str__(self): # so print() works
        return str(self.num)

T = TypeVar('T', bound=Addable)

def add(a: T, b: T):
    print(type(a), type(b))
    print(a+b) # checks for __add__(), uses __str__()

add(MyInt(1), MyInt(2)) # okay
add(int(1), int(2))     # okay

Output:

<class '__main__.MyInt'> <class '__main__.MyInt'>
3
<class 'int'> <class 'int'>
3

This works, even though MyInt does not mention Addable in the class declaration at all!

What happens if we go back to the Addable declaration and change Protocol to ABC, like:

class Addable(ABC):
    def __add__(self, other):
        raise NotImplementedError

...

add(MyInt(1), MyInt(2)) # not okay
add(int(1), int(2))     # not okay

We get a type error from both lines:

error: Value of type variable "T" of "add" cannot be "MyInt"
error: Value of type variable "T" of "add" cannot be "int"

Neither MyInt or int can stand in for an Addable if it's an ABC. Only classes that derive can stand in for abstract base classes.

Let's change MyInt to derive from Addable:

class MyInt(Addable):
    ....

add(MyInt(1), MyInt(2)) # okay, MyInt derives from Addable
add(int(1), int(2))     # not okay

Output:

error: Value of type variable "T" of "add" cannot be "int"

MyInt is now fine, int still cannot stand in for an Addable. This shows the difference between Protocol and ABC. With Protocol, anything that implements the declared functions can stand-in for that type, irrespective of inheritance. With ABC, only types that inherit from the base class (in the example above, MyInt inherits from Addable) can stand in for that type.

Types for class variables vs instance variables

Let's try this code:

class Foo:
    num: int # num is NOT a class variable
    def __init__(self, num: int):
        self.num = num

f = Foo(1)
print(f.num)        # prints 1
g = Foo(2)
print(f.num, g.num) # prints 1 2
print(Foo.num)      # AttributeError: type object 'Foo' has no attribute 'num'

Output:

1
1 2
AttributeError: type object 'Foo' has no attribute 'num'

Here, we declare the num instance variable of class Foo to be of type int. We create 2 instances of Foo, and we see that each of them carries a separate num. Then we try to access Foo.num class variable, and we get an AttributeError, because it doesn't exist.

Let's make one minor modification of the code, and assign some initial value to num:

class Foo:
    num: int = 0 # num is now a class variable
    def __init__(self, num: int):
        self.num = num

f = Foo(1)
print(f.num)          # prints 1
g = Foo(2)
print(f.num, g.num)   # prints 1 2
print(Foo.num)        # prints 0

This one change creates num as a class variable, which can be accessed. Note that both the class variable and the instance variable carry the int type:

f = Foo(1)
print(Foo.num, f.num) # prints 0 1
f.num = "hello"       # not okay
Foo.num = "world"     # not okay

Output:

error: Incompatible types in assignment (expression has type "str", variable has type "int")
error: Incompatible types in assignment (expression has type "str", variable has type "int")

Conclusion

This post concludes this short series on Python typing for Data Scientists. I think the verdict is still out whether type hints are worth in in Data Science code (which tends to be short, linear and less structured than application software), but it's good to know that type hints exist and how it works.

Python types for Data Scientists - Part II

2022-04-17T00:00:00+02:00

Introduction

In the previous post I showed how to get started using Python static type checking in ipython notebooks. Here I will look at slightly more advanced uses of typing to further increase the safety and readability of code. The ipython notebook is up on Github. The best reference is the official Python documentation of the typing module.

`Optional` types and `Union`

Sometimes we want to declare that something can be of a certain type, or None. Imagine we don't know about numpy.random.random_sample and we're writing a function randoms() to return a random list[float] of length num:

def randoms(num: int) -> list[float]:
    if num >= 0:
        return [random() for _ in range(num)]
    else:
        return None # not okay, NoneType is not list[float]

We want to be good programmers and return None if a negative value for num is passed in, but None is not a list[float], so this won't work:

error: Incompatible return value type (got "None", expected "List[float]")

This is what Optional[T] is for, it declares that the type will be T or None:

def randoms(num: int) -> Optional[list[float]]:
    if num >= 0:
        return [random() for _ in range(num)]
    else:
        return None # okay, return type is Optional[...]

The same can also be achieved by using Union[]:

def randoms(num: int) -> Union[list[float], None]:
    if num >= 0:
        return [random() for _ in range(num)]
    else:
        return None # okay, return type is Union[..., None]

Note that None as a type hint is a special case and is replaced by type(None) by Python.

What if we're a different kind of programmer, and we want to raise an exception instead of returning None, like:

def randoms(num: int) -> list[float]:
    if num >= 0:
        return [random() for _ in range(num)]
    else:
        raise ValueError # okay, there is no typed way to communicate raised exceptions

This is okay, in Python there is no typed way to communicate raised exceptions.

Finally, what if we want to return just a float if the user is asking for one random, and None on negative input. Union is the solution:

def randoms(num: int) -> Union[float, list[float], None]:
    if num == 1:
        return random()                       # float
    elif num >= 0:
        return [random() for _ in range(num)] # list[float]
    else:
        return None                           # None

Note that as of Python 3.10, Union[X, Y] can be written as X | Y, but this does not work yet on Python 3.9:

def randoms(num: int) -> float | list[float] | None:
    if num == 1:
        return random()                       # float
    elif num >= 0:
        return [random() for _ in range(num)] # list[float]
    else:
        return None                           # None

Type aliases and `NewType`

Suppose we are building a library for machine learning and we are using list[float] for feature vectors. One way we can communicate this to the user of our library is by calling the arguments of our functions names like feature_vector. We can also accomplish this in our typing, by declaring an alias for list[float]:

FeatureVector = list[float] # type alias

We can now write FeatureVector interchangeably with list[float]:

def predict(fv: FeatureVector) -> float:
    return random()

fv: list[float] = [0.1, 0.2, 0.3]
predict(fv) # okay
fv: FeatureVector = [0.1, 0.2, 0.3]
predict(fv) # okay

Suppose that we want to declare our type classes like in type aliases, but we want to be more strict. We only want to accept list[float]s that were explicitly declared to be FeatureVector classes. We can achieve this by using NewType:

FeatureVector = NewType('FeatureVector', list[float])
# all FeatureVectors are list[float], but not all list[float] are FeatureVectors

def predict(fv: FeatureVector) -> float:
    return random()

fv: list[float] = [0.1, 0.2, 0.3]
predict(fv) # not okay
fv: FeatureVector = FeatureVector([0.1, 0.2, 0.3]) # explicit cast
predict(fv) # okay

Output:

error: Argument 1 to "predict" has incompatible type "List[float]"; expected "FeatureVector"

In the above example, all FeatureVectors are list[float], but not all list[float] are FeatureVectors. So any function that accepts a list[float] will accept a FeatureVector, but not the other way around.

Generics with `TypeVar`

Suppose we want to write a function first() which returns the first element of a list, and we want to declare that the list contains things of type T, and the return type will be the same type T. We can accomplish this with a TypeVar:

T = TypeVar('T') # declare type variable T to be used
def first(li: list[T]) -> T:
    return li[0]

We can also mix TypeVars with Optional to make first() more useful:

T = TypeVar('T') # declare type variable T to be used
def first(li: list[T]) -> Optional[T]:
    return li[0] if len(li) > 0 else None # okay

Note that we cannot bind a TypeVar by usage. In the example below, we cannot bind T to be str (there is no "type solver"), this will return an error:

T = TypeVar('T') # declare type variable T to be used
def first(li: list[T]) -> T:
    return "hello" # not okay

The error is:

error: Incompatible return value type (got "str", expected "T")

Protocols

Let's look at another example, where we want to add two things:

T = TypeVar('T') # declare type variable T to be used
def add(a: T, b: T) -> T:
    return a+b # checks for __add__()

This will result in a type-error, because Python doesn't know whether T implements __add__():

error: Unsupported left operand type for + ("T")

To achieve the desired typing, we have to use Protocols:

class Addable(Protocol):
  def __add__(self, other): # anything that declares __add__() can stand in for an Addable
    raise NotImplementedError

Here we are declaring a class Addable using typing.Protocol, which declares __add__(). Anything that declares __add__() can stand in for an Addable, even if it's not descended from Addable. For example, an int is an Addable. Examples:

def add(a: Addable, b: Addable) -> Addable:
    print(type(a), type(b))
    return a+b # checks for __add__()

add(str(1), str(2))   # okay
add(int(1), int(2))   # okay
add(int(1), float(2)) # okay
add(int(1), str(2))   # not a typecheck error, but a runtime error

Output:

<class 'str'> <class 'str'>
<class 'int'> <class 'int'>
<class 'int'> <class 'float'>
<class 'int'> <class 'str'>
TypeError: unsupported operand type(s) for +: 'int' and 'str' # coming from the last add()

All 4 of these pass the type checks, because str, int and float are all Addable, since they have __add__(). The last one will raise a run-time exceptions, since + doesn't work implicitly for int and str. Note that this is a runtime exception coming from running the code, not a type error — the type checker did not raise any errors.

There are 2 ways we can think about this mini-problem:

We want to allow adding of 2 different types (eg. int and float), but only if it makes sense. We want the type checker to raise an error for cases when a runtime exception would be raised, (eg. adding int and str)
We only want to allow adding of exactly the same types, eg. int, int, float, float, str, str. We will see that this is not achievable in Python with generic types.

Let's look at another version of this, where we declare a TypeVar and bind it to be Addable:

T = TypeVar('T', bound=Addable)

def add(a: T, b: T) -> T:
    print(type(a), type(b))
    return a+b # checks for __add__()

add(str(1), str(2))   # okay
add(int(1), int(2))   # okay
add(int(1), float(2)) # okay
add(int(1), str(2))  # typecheck error and runtime error

Here, the last line raises a type check error and a runtime error:

error: Value of type variable "T" of "add" cannot be "object"  # typecheck error coming from the last add()
...
TypeError: unsupported operand type(s) for +: 'int' and 'str'  # runtime error coming from the last add()

Not that the third int, float version still runs fine. So this version implements case 1. above, where different types can still be passed, as long as addition makes sense for them.

One last attempt could be to use a union'd TypeVar, where we limit ourselves to certain types that can stand in for T. But as before, in this case the type checker also doesn't enforce the instances of T to be the same:

T = TypeVar('T', int, float, str)

def add(a: T, b: T) -> T:
    return a+b # checks for __add__()

add(int(1), float(2)) # okay

It turns we cannot use binding to get case 2. above, ie. to force the type checker to make sure that both a and b arguments are actually the same type in add(a, b).

Conclusion

In the next article I will look at more uses of protocol and abstract base classes.

Python types for Data Scientists - Part I

2022-04-08T00:00:00+02:00

Introduction

Python supports types, and from 3.5, type annotation and typechecking. Let's see:

turn on type checking in ipython notebooks
how to write type hints
simple examples

The official documentation is here. The ipython notebook is up on Github.

Type and type expressions

Let's use the built-in type function to get the types of some common Python expressions:

type(42), type(int('42')), type('hello'), type([]), type({}), type({1, 2, 3}), type(None)

Output:

(int, int, str, list, dict, set, NoneType)

Let's get more abstract:

type(int), type(type(42)), type(str), type(42) == type(int), type(int) == type(str)

Output:

(type, type, type, False, True)

Dynamic vs static typing

Python, by default, is dynamically typed. This means that variables can start of as an int, can become str or a list or NoneType later:

age = 42
age = 'hello'
age = [42, 'hello']
age = None

However, we can switch Python to static typing:

age: int

By default, Python treats the : int as an annotation and ignores it:

age: int = 42
age = 'hello'       # no problem since type checking is off
age = [42, 'hello'] # no problem since type checking is off
age = None          # no problem since type checking is off

We can load a static type checking module, and the type annotation will be enforced. First we need to pip install nb_mypy, then we can achieve type checking in ipython notebooks:

%load_ext nb_mypy
%nb_mypy On

Once we do this, we can get type checking each time a cell is executed:

age:int
age = 42            # okay
age = 'hello'       # not okay
age = [42, 'hello'] # not okay
age = None          # not okay

The output will be:

error: Incompatible types in assignment (expression has type "str", variable has type "int")
error: Incompatible types in assignment (expression has type "List[object]", variable has type "int")
error: Incompatible types in assignment (expression has type "None", variable has type "int")

Container types

Containers can also be typed:

l: list[int]
l = [1, 2, 3]    # okay
l = 1            # not okay!
l = [1, "hello"] # not okay!

Output:

error: Incompatible types in assignment (expression has type "int", variable has type "List[int]")
error: List item 1 has incompatible type "str"; expected "int"

Or:

l: list[int] = []
l.append(1)       # okay
l.append("hello") # not okay!

Output:

error: Argument 1 to "append" of "list" has incompatible type "str"; expected "int"

Functions

Functions can also carry type annotations:

def f(i:int) -> list[str]:
    return ['hello']

f(42)          # okay
f('hello')     # not okay
r: int = f(42) # not okay

Output:

error: Argument 1 to "f" has incompatible type "str"; expected "int"
error: Incompatible types in assignment (expression has type "List[str]", variable has type "int")

Classes

Class member variables:

class A:
    memberVariable: int
    def __init__(self, i:int) -> None:
        self.memberVariable = i
    def increase(self) -> int:
        self.memberVariable += 1
        return self.memberVariable

Then:

a = A(42)           # okay
a.increase()        # okay
a = A('hello')      # not okay
None + a.increase() # not okay

Output:

error: Argument 1 to "A" has incompatible type "str"; expected "int"
error: Unsupported operand types for + ("None" and "int")

Conclusion

In the next part, I will look at more complicated type expressions, such as Optional.

Solving 5 algorithmic interview questions

2022-03-26T00:00:00+01:00

Introduction

Recently I was considering whether to introduce some CS style algorithmic interview questions into our Data Science hiring loop, since having an understanding of algorithms and data structures can be useful for Data Scientists. Not having done this soft of interview for a few years I picked up my copy of Daily Coding Problem and starting solving a few problems to refresh my feeling for what it feels like as a candidate, and whether it would give us any useful signals.

Arrays #1

Given an input array li of numbers, return an array of numbers where the ith element is the multiplication of all other elements except the ith from li.

# easy, with division
def simple(li):
    product = reduce((lambda x, y: x * y), li)
    return [int(product/i) for i in li]

# tricky, without divisions
def tricky(li):
    befs = deepcopy(li)
    for i in range(1, len(li)):
        befs[i] = befs[i-1] * li[i]
    afts = deepcopy(li)
    for i in range(len(li)-2, -1, -1):
        afts[i] = afts[i+1] * li[i]
    result = [befs[i-1]*afts[i+1] for i in range(1, len(li)-1)]
    result.insert(0, afts[1])
    result.append(befs[len(li)-2])
    return result

N = randint(5, 10)
li = [randint(1, 10) for _ in range(N)]
print(f'Input: {li}')
result = simple(li)
print(f'Result: {result}')
result = tricky(li)
print(f'Result: {result}')

Yields:

Input: [2, 5, 1, 5, 6, 5, 8, 7, 9, 7]
Result: [2646000, 1058400, 5292000, 1058400, 882000, 1058400, 661500, 756000, 588000, 756000]
Result: [2646000, 1058400, 5292000, 1058400, 882000, 1058400, 661500, 756000, 588000, 756000]

Quick and dirty large scale test:

num_tests = 1000*1000
for _ in range(num_tests):
    N = randint(2, 10)
    li = [randint(1, 10) for _ in range(N)]
    assert(simple(li) == tricky(li))
print(f'{num_tests} random tests passed')

Arrays #2

Given an input array li of ints, what is the minimum range that need to be sorted? Ie. elements outside of this range are already in their correct position in li.

# easy, sort and compare
def simple(li):
    sd = sorted(li)
    eq = [i for i in range(N) if li[i] != sd[i]]
    return (eq[0], eq[-1]) if len(eq) > 0 else (0, N-1)

# tricky
def tricky(li):
    mins = deepcopy(li)
    for i in range(len(li)-2, -1, -1):
        if mins[i+1] < mins[i]:
            mins[i] = mins[i+1]
    maxs = deepcopy(li)
    for i in range(1, len(li)):
        if maxs[i-1] > maxs[i]:
            maxs[i] = maxs[i-1]
    mins = [i for i in range(len(li)) if
            mins[i] != li[i]]
    maxs = [i for i in range(len(li)) if maxs[i] != li[i]]
    mn = mins[0] if len(mins) > 0 else 0
    mx = maxs[-1] if len(mins) > 0 else len(li)-1
    return mn, mx

Yields:

Input: [2, 4, 1, 10, 3, 8, 7]
Range to be sorted: [0, 6]
Range to be sorted: [0, 6]

Quick and dirty large scale test:

num_tests = 1000*1000
for _ in range(num_tests):
    N = randint(1, 10)
    li = [randint(1, 10) for _ in range(N)]
    assert(simple(li) == tricky(li))
print(f'{num_tests} random tests passed')

Arrays #3

Given an array of integers li, what is largest contiguous sub-array sum?

def simple(li):
    max_sum, max_range = 0, []
    for mn in range(len(li)):
        for mx in range(mn, len(li)):
            s = sum(li[mn:mx+1])
            if s > max_sum:
                max_sum = s
                max_range = li[mn:mx+1]
    return sum(max_range)

def tricky(li):
    best, mx = 0, 0
    for i in range(len(li)):
        mx = max(li[i], mx+li[i])
        best = max(best, mx)
    return best

N = randint(1, 10)
li = [randint(-10, 10) for _ in range(N)]
print(f'Input: {li}')
max_range = simple(li)
print(f'Max sum: {max_range}')
max_range = tricky(li)
print(f'Max sum: {max_range}')

Yields:

Input: [7, 7, -10, 9, -1, -2, 10, 10, -9]
Max sum: 30
Max sum: 30

Quick and dirty large scale test:

num_tests = 1000*1000
for _ in range(num_tests):
    N = randint(1, 10)
    li = [randint(-10, 10) for _ in range(N)]
    assert(simple(li) == tricky(li))
print(f'{num_tests} random tests passed')

Strings #1

Given a word w and a string s, find starting locations of s that are anagrams of w (anagram = same length, same letters, different order).

# simple, continuously re-compute the per-character sums
def simple(w, s):
    return [i for i in range(len(s)) if Counter(w) == Counter(s[i:i+len(w)])]

# tricky, maintain the running sum
def tricky(w, s):
    ref = defaultdict(int)
    for c in w: ref[c] += 1
    inds, run = [], deepcopy(ref)
    for i in range(0, len(s)):
        if i >= len(w):
            run[s[i-len(w)]] += 1
        run[s[i]] -= 1
        if sum([abs(x) for x in run.values()]) == 0:
            inds.append(i-len(w)+1)
    return inds

w, s = 'bb', 'abbxbbbaabb'
print(f'w: {w}, s: {s}')
result = simple(w, s)
print(f'Result: {result}')
result = tricky(w, s)
print(f'Result: {result}')

Yields:

w: bb, s: abbxbbbaabb
Result: [1, 4, 5, 9]
Result: [1, 4, 5, 9]

Quick and dirty large scale test:

num_tests = 1000*1000
for _ in range(num_tests):
    w = str(randint(1, 999))
    s = str(randint(1000, 9999999999))
    assert(simple(w, s) == tricky(w, s))
print(f'{num_tests} random tests passed')

Strings #2

Given a list of words wl, find all pairs of words so that the concatenation of the words is a palindrome (eg. 'ab'+'a' is a palindrome).

def reverse(s):
    return s[::-1]

def is_palindrome(s):
    return s == reverse(s)

def simple(li):
    return sorted([(i, j) for i, j in permutations(range(len(li)), 2) if is_palindrome(li[i] + li[j])])

def tricky(wl):
    d = {w:i for i, w in enumerate(wl)}
    inds = []
    for i, s in enumerate(wl):
        for j in range(len(s)+1):
            pre = s[:j]
            suf = s[j:]
            if reverse(pre) in d.keys() and is_palindrome(suf) and i != d[reverse(pre)]:
                inds.append((i, d[reverse(pre)]))
            if reverse(suf) in d.keys() and is_palindrome(pre) and i != d[reverse(suf)]:
                inds.append((d[reverse(suf)], i))
    return sorted(list(set(inds)))

wl = ['ab', 'bba', 'xx', 'x', 'bab', 'a']
print(f'Input: {wl}')
result = simple(wl)
print(f'Result sum: {result}')
result = tricky(wl)
print(f'Result sum: {result}')

Yields:

Input: ['ab', 'bba', 'xx', 'x', 'bab', 'a']
Result sum: [(0, 1), (0, 5), (2, 3), (3, 2), (4, 0), (5, 1)]
Result sum: [(0, 1), (0, 5), (2, 3), (3, 2), (4, 0), (5, 1)]

Quick and dirty large scale test:

num_tests = 1000*1000
for _ in range(num_tests):
    l = randint(5, 10)
    wl = [str(randint(1, 9999)) for _ in range(l)]
    assert(simple(wl) == tricky(wl))
print(f'{num_tests} random tests passed')

Conclusion

My opinion, from having done many Software/Data Engineering loops (both as a candidate and an interviewer), is that I don't like these types of questions, because:

Candidates who tend to stress in such situations (like me, I get high blood pressure) will do much worse than in real life
The specific questions/problems are not relevant to the job (whether SWE or DS)
These interview situations are a competition against time, but being able to solve a 5-10 line programming puzzle in 10 minutes is like saying: "we want to hire people who can run fast, so they will spend less time walking to and from lunch, so they'll have more time left to work"
These sort of problems usually come from a question pool (like a book), and the questions tend to be alike, so they can be prepared for. Candidates can prepare for such interviews by solving a lot of similar problems, and if they're lucky, they'll have seen the problem on the interview, or it will follow a usual pattern (eg. use a hashmap). These sort of questions favor more junior candidates, who studied algorithms in school more recently, and who have more time to prepare (no family, more hungry).

Realistically, in a live interview situation, I would probably fail on 3-5 of these 5 questions [in a 10-15 minute limit].

I think the flaw is to expect candidates to come up with the more tricky efficient solution. I think for Data Scientists, more signal can be gained from telling them to come up with the simple, less efficient solution (and instead asking them to be as pythonic as possible in their solution). Data Scientists tend to use libraies (and SQL query engines) to do the heavy lifting, almost all the code they write is essentially "driver code"; being able to understand the structure of the problem and writing a clean Python solution is good enough.

Fair coin from biased coin

2022-03-22T00:00:00+01:00

Introduction

Recently I saw a fun puzzle on Hacker News: given a biased coin, construct a fair coin. Ie. you're allowed to toss a biased coin repeatedly and record the sequence of results (like 001...1, and the goal is to return 0/1 with 50%-50% probability, so use the biased coin to simulate a fair coin. I quickly scanned the article and comments to get some ideas, but I wanted to work through solution(s) myself. The code is up on Github.

Let's assume the biased coin comes up 1 with $p$ probability and 0 with $q=1-p$ probability.

First, some helper functions:

def coin_flip(bias=0.5):
    if random() < bias:
        return 1
    else:
        return 0

def make_biased_coin(bias=None):
    if bias is None:
        bias = random()
    print(f'Bias = {bias:.3f}')
    return lambda: coin_flip(bias)

von Neumann's probabilistic solution

The most elegant solution is from von Neumann: toss the biased coin twice. The 4 possible outcomes:

00: probability $P=p^2$
11: probability $P=q^2$
01: probability $P=pq$
10: probability $P=pq$

Observe that the last 2 probabilities are the same. Algorithm: if the result is 00 or 11, repeat the procedure. If the result is 01, return 0, if the result is 10, return 1.

In code:

def probabilistic(biased_coin):
    while True:
        s, w = biased_coin(), biased_coin()
        if s == 0 and w == 1:
            return 1
        elif s == 1 and w == 0:
            return 0

Statistical solution

I wanted to come up with a solution of my own for this puzzle. I thought, first let's measure (estimate) the bias of the biased coin, and then somehow correct for it. Let's flip the coin $N/2$ times first, take the mean of the flips, let's call this $\hat{p_1}$, our first estimate of $p$. Then, do another $N/2$ flips (for a total of $N$ flips) to get $\hat{p_2}$, our second estimate of $p$. We can pretend that the first measurement gets the true value of $p$, and then the second measurement varies around $p$ in a symmetrical guassian way, so $P(\hat{p_2} > \hat{p_1}) = P(\hat{p_2} < \hat{p_1})$.

What if get unlucky, and $\hat{p_2} = \hat{p_1}$? We repeat the process and try again. In code:

def statistical(biased_coin, N):
    while True:
        num_flips = int(N/2.0)
        s = sum([biased_coin() for _ in range(num_flips)])/num_flips
        num_flips = N - num_flips
        w = sum([biased_coin() for _ in range(num_flips)])/num_flips
        if s < w:
            return 1
        elif s > w:
            return 0

Permutations solution

This is based on a comment I saw on Hacker News. Flip the biased coin $N$ times to get an $N$ long sequence of bits. Count the number of 1s in the sequence, let's say it's $k$. The probability of getting this specific sequence is $P=p^k q^{N-k}$. We notice that all sequences with $k$ 1s have this probability. So if we can somehow enumerate all such sequences, we can order them lexicographically, and we can divide them into two sets of equal cardinality: pick the middle bit sequence, and take the ones smaller than the middle and the ones bigger than the middle bit sequence.

Implementing this is trivial with one more restriction: let's only consider bit sequences such that the bit sequence is not equal to its reverse (ie. if the bit sequence is 01011 then it's reverse is 11010). In this case each bit sequence has exactly one pair, its reverse, and we can create a trivial ordering between these pairs. If we get unlucky, and the bit sequence is the same as its reverse, we repeat the process and try again. In code:

def permutations(biased_coin, N):
    while True:
        inst = ''.join([str(biased_coin()) for _ in range(N)])
        revr = inst[::-1]
        if inst > revr:
            return 1
        elif inst < revr:
            return 0

All 3 solutions are the same

For the second and third solution, what should $N$ be? For the statistical solution, first I thought that the higher $N$, the more accurately it approximates a fair coin. But, this is not the case. All of the solutions are perfectly accurate at all $N$:

def experiment(num_flips, num_experiments, coin_type, coin_func):
    start = time()
    means = [mean([coin_func() for _ in range(num_flips)]) for _ in range(num_experiments)]
    m, s = mean(means), stdev(means)
    elapsed = (time() - start) * 1000
    print(f'{coin_type}: mean = {m:.3f}, stdev = {s:.3f}, elapsed = {elapsed:.0f} msec')

num_experiments = 1000
num_flips = 1000
biased_coin = make_biased_coin(0.2) # fix bias at 0.2 for measurement

experiment(num_flips=num_flips, num_experiments=num_experiments, coin_type='Fair coin',
           coin_func=lambda: coin_flip())

experiment(num_flips=num_flips, num_experiments=num_experiments,
           coin_type=f'Probabilistic coin',
           coin_func=lambda: probabilistic(biased_coin))

for N in [2, 50, 100]:
    experiment(num_flips=num_flips, num_experiments=num_experiments,
               coin_type=f'Statistical coin (N={N})',
               coin_func=lambda: statistical(biased_coin, N=N))

for N in [2, 50, 100]:
    experiment(num_flips=num_flips, num_experiments=num_experiments,
               coin_type=f'Permutations coin (N={N})',
               coin_func=lambda: permutations(biased_coin, N=N))

Yields something like:

Bias = 0.200
Fair coin:                 mean = 0.501, stdev = 0.016, elapsed = 629   msec
Probabilistic coin:        mean = 0.500, stdev = 0.016, elapsed = 1815  msec
Statistical coin (N=2):    mean = 0.500, stdev = 0.016, elapsed = 5293  msec
Statistical coin (N=50):   mean = 0.500, stdev = 0.016, elapsed = 13198 msec
Statistical coin (N=100):  mean = 0.500, stdev = 0.015, elapsed = 24275 msec
Permutations coin (N=2):   mean = 0.500, stdev = 0.016, elapsed = 4569  msec
Permutations coin (N=50):  mean = 0.499, stdev = 0.016, elapsed = 18403 msec
Permutations coin (N=100): mean = 0.499, stdev = 0.016, elapsed = 35645 msec

Given that, we should run them at the minimal $N$, which is $N=2$ coin flips. Then, if we think about it, the statistical and permutations based solutions trivially reduce to the von Neumann solution at $N=2$ coin flips:

statistical: flip the coin once, it's either 0 or 1, so the first mean $\hat{p_1}$ is either 0 or 1. Flip it a second time to get $\hat{p_2}$. The conditions of $\hat{p_1} < \hat{p_2}$ and $\hat{p_1} > \hat{p_2}$ reduce to the 01 and 10 cases (probability $P=pq$), and we flip again in the 00 and 11 case.
permuations: there are 4 possible outcomes of 2 flips. 00 and 11 are their own reverse, so we flip again, in the other two cases of 01 and 10 we return 0 or 1 (probability $P=pq$).

Conclusion

I had great fun thinking through this trivial puzzle.

The german tank problem in World War II

2022-03-12T00:00:00+01:00

Introduction

One of the challenges the Allies faced in World War II is knowing how many tanks the Germans had. Mathematically, the problem is the following: assume $N$ tanks are produced and numbered sequentially: $1, 2, 3... N$. $k < N$ tanks are destroyed, and their serial numbers are read off by soldiers and sent to a central intelligence agency. Based on these collected serial numbers, what is our best estimate for $N$? For the estimate to work, we assume that the destroyed tanks are a random sample from all the tanks, which may or may not be true in real life.

Per Wikipedia:

In the statistical theory of estimation, the German tank problem consists of estimating the maximum of a discrete uniform distribution from sampling without replacement. In simple terms, suppose there exists an unknown number of items which are sequentially numbered from 1 to N. A random sample of these items is taken and their sequence numbers observed; the problem is to estimate N from these observed numbers.

The problem can be approached using either frequentist inference or Bayesian inference, leading to different results. Estimating the population maximum based on a single sample yields divergent results, whereas estimation based on multiple samples is a practical estimation question whose answer is simple (especially in the frequentist setting) but not obvious (especially in the Bayesian setting).

The problem is named after its historical application by Allied forces in World War II to the estimation of the monthly rate of German tank production from very limited data. This exploited the manufacturing practice of assigning and attaching ascending sequences of serial numbers to tank components (chassis, gearbox, engine, wheels), with some of the tanks eventually being captured in battle by Allied forces.

Monte Carlo simulation

The solution is simple: we just have to order the serial numbers and compute the average gap between them. Then, we take the maximum and add the average gap. The intuition can be illustrated with a very small sample size. Imagine there are 100 tanks produced, and 3 are destroyed. If we randomly select 3 numbers, and order them, then the average of the smaller will be ~25, the middle one ~50, and the average of the bigger will be ~75, and the average gap will be ~25. The estimate for the number of tanks is $75+25=100$.

We can check our thinking with a simple Monte Carlo simulation:

num_tanks = 100
num_destroyed = 3
num_experiments = 1000*1000

destroyed_tanks = []
for _ in range(num_experiments):
    destroyed_tanks.append(sorted(sample(range(1, num_tanks+1), num_destroyed)))

print([mean([s[i] for s in destroyed_tanks]) for i in range(num_destroyed)])
estimate = mean([max(s) for s in destroyed_tanks]) + mean([(max(s)-min(s))/(len(s)-1) for s in destroyed_tanks])
print('Average estimate for num_tanks = %.2f' % estimate)

Prints something like:

[25.287, 50.517, 75.771]
Average estimate for num_tanks = 101.01

There is a small bug in our thinking. We expected the sorted averages to be [25, 50, 75], but they are approx. [25.25, 50.50, 75.75], the gap is 25.25 instead of 25, and the estimate is off by 1 (101 instead of 100). Why?

This is quite counter-intuitive (to me), it's because we're generating ints and not floats. Our intuition does work if we switch to floats:

num_tanks = 100
num_destroyed = 3
num_experiments = 1000*1000

destroyed_tanks = []
for _ in range(num_experiments):
    destroyed_tanks.append(sorted([uniform(1, num_tanks) for _ in range(num_destroyed)]))

print([mean([s[i] for s in destroyed_tanks]) for i in range(num_destroyed)])
estimate = mean([max(s) for s in destroyed_tanks]) + mean([(max(s)-min(s))/(len(s)-1) for s in destroyed_tanks])
print('Average estimate for num_tanks = %.2f' % estimate)

Prints something like:

[25.748, 50.498, 75.236]
Average estimate for num_tanks = 99.98

This is per our intuition, 50.5 is the midpoint between 1 and 100 (which is the interval we're uniformly sampling), and 25.75 is the midpoint between 1 and 50.5, and 75.25 is the midpoint between 50.5 and 100. Another way to see that the "quantization" by using integers matters is to think of the trivial example or 3 tanks and all 3 are destroyed: in this case the 3 numbers will [1, 2, 3], and clearly 1 is the not the midpoint between 1 and 2.

In any case, the actual formula for the least variance estimate for the number of tanks is:

$N = M + M/k - 1$

where $M$ is the maximum destroyed tank serial number and $k$ is the number of destroyed tanks. In code:

num_tanks = 100
num_destroyed = 3
num_experiments = 100*1000

destroyed_tanks = []
for _ in range(num_experiments):
    destroyed_tanks.append(sample(range(1, num_tanks+1), num_destroyed))

estimate = mean([max(s) + max(s)/len(s) - 1 for s in destroyed_tanks])
print('Average estimate for num_tanks = %.2f' % estimate)

Prints something like:

Average estimate for num_tanks = 100.01

Let's see how good this estimate is, and plot the standard deviation of for different $(N, k)$ combinations:

num_tanks = [100, 1000, 10*1000]
num_destroyed_ratio = [0.02, 0.05, 0.1, 0.2]
num_experiments = 10*1000

destroyed_tanks_dict = defaultdict(lambda: [])
for nt in num_tanks:
    for num_destroyed in [int(r*nt) for r in num_destroyed_ratio]:
        for _ in range(num_experiments):
            destroyed_tanks_dict[(nt, num_destroyed)].append(sample(range(1, nt+1), num_destroyed))

stdvs_dict = defaultdict(lambda: [])
for (nt, num_destroyed), destroyed_tanks in destroyed_tanks_dict.items():
    s = stdev([max(s) + max(s)/len(s) - 1 for s in destroyed_tanks])
    stdvs_dict[nt].append((num_destroyed, s))

plt.figure(figsize=(10,5))
plt.xlabel('k as a ratio of N')
plt.ylabel('stdev as ratio of N')
legends = []
for nt, stdvs in stdvs_dict.items():
    plt.plot([x[0]/nt for x in stdvs], [x[1]/nt for x in stdvs], marker='o')
    legends.append(f'N = {nt}')
plt.legend(legends)
plt.ylim((0, 0.5))
plt.show()

Shows:

This shows that even at $N=1000$, if you are able to observe about 10% of serial numbers, you can expect to make a very good estimate, you will essentially be on target to within a few % when estimating $N$!

Conclusion

From the Wikipedia article:

According to conventional Allied intelligence estimates, the Germans were producing around 1,400 tanks a month between June 1940 and September 1942. Applying the formula below to the serial numbers of captured tanks, the number was calculated to be 246 a month. After the war, captured German production figures from the ministry of Albert Speer showed the actual number to be 245.

Delightfully cynical half-truths about organizations

2022-02-12T00:00:00+01:00

Introduction

The author is in a (mid-level) leadership position.

Peter principle

Wikipedia: As long as a worker is good at their job, they will keep getting promoted, until they reach the first "level" where they're incompetent. Here they get stuck, and are no longer promoted. The corollary is that all higher level jobs in an organization is occupied by people who are incompetent at their jobs.

Dilbert principle

Wikipedia: Companies tend to systematically promote incompetent employees to management to get them out of the workflow.

Gervais principle

Ribbonfarm: Organizations are made up of Losers at the bottom, Clueless in the middle and Sociopaths (eg. CEOs and other CxOs) at the top. Sociopaths, to keep things running, promote over-performing Losers into middle-management, and to protect their own job, groom under-performing Losers into Sociopaths. The Clueless are the ones who lack the competence to circulate freely through the job market (unlike Sociopaths and Losers), and have long tenures at the organization.

Negative selection

Wikipedia: The person on the top of the hierarchy, wishing to remain in power forever, chooses his associates with the prime criterion of incompetence – they must not be competent enough to remove him from power. Since subordinates often mimic their leader, these associates do the same with those below them in the hierarchy, and the hierarchy becomes progressively filled over time with more and more incompetent people.

Dunning-Kruger effect

Wikipedia: The cognitive bias whereby people with low ability at a task overestimate their ability. Some researchers also include in their definition the opposite effect for high performers, their tendency to underestimate their skills.

Conclusion

The first four posit different mechanisms (promotion based on true merit, organizational-defensive promotion and self-defensive promotion) that result in incompetent people getting promoted to leadership position. The last one, the Dunning-Kruger effect, says that if promotion decisions are based on people's demeanor and self-promotion then less qualified will be promoted. These are half-truths, but there is some truth in them. What do high-performance organizations do to avoid such problems?

At a high-level, all these problems (apart from the Dunning-Kruger effect) have their roots in misaligned incentives. The employee's (whether management or individual contributor) and the organization's long-term interests are not aligned. A good tool for such alignment is stock options. However there are very few organizations that are on a growth curve where this influences employees to do the right thing at scale. For this to work, the majority of employees have to believe that doing "the right thing" will cause their (current) stock to rise more versus the benefits (higher salary, more stock) they can hope to get when doing "the wrong thing". Also see Tragedy of the Commons.
Another root cause is poor hiring. This is especially tricky when the organization is hiring for a role that is not within the core competency of the organization, such as when a non-tech company hires for technical roles, or when a tech startup hires for finance or sales roles. There are various ways around this, such as getting recommendations from investors, but in general this is not a solved problem in industry, even though this is a very costly mistake. If an incompetent manager is hired, then this person will build an incompetent organization. Once this happens, it takes years to solve rebuild, and this can even bring down the entire company before it's resolved.
Avoiding the Peter principle: instead of promoting to level $L+1$ when the person does a good job at level $L$, promote when the person already demonstrated competence at the $L+1$ level for some time. There are companies who do this, eg. Facebook did this in 2016-17 when I worked there. However, I found this unacceptable HR policy, because asking me to overperform at the next level for 18 months without compensation just to get promoted isn't fair. And this assumes that the employee's manager does not change, since usually a new manager will often not credit past performance. Another alternative is to promote based on performing at the $L$ level, but then demote if not performing at the $L+1$ level, and have a culture that accepts this and doesn't see it as major personal failure. This is also hard to accomplish at scale, since many people just aren't like this personally, won't accept demotion, and will just start looking for a new job. The industry solution seems to be just that: (i) promotions based on performance at the $L$ level, (ii) people leave their jobs for another $L+1$ level job if they don't perform at the $L+1$ level, (iii) but if they really are incompetent at this level, they will bounce around jobs and eventually slide down to lower tier companies.
Avoiding the Dilbert principle: some companies employ up-or-out management, so either get promoted or get managed out. At other companies it's not understood that promotion always happens every X years (on either IC or Management track). This is fundamentally a people management problem, people managers need to set realistic expectations: if the person's performance isn't good enough for promotion then promotion should not be used as a carrot-on-a-stick.
Negative selection: the best counter-measure is to try to align the management's long-term incentive with the company's.
Dunning-Kruger effect: organizations can avoid this by deploying tools such as OKRs with SMART outcomes across the organization hiararchy. This increases the probability that employees will be judged based on the objective outcomes they deliver, instead of their ability to market and promote themselves.

Probabilistic spin glass - Conclusion

2022-01-31T00:00:00+01:00

Introduction

I wrote 5 posts on probabilistic spin glasses:

Here I will summarize my main findings, misses, and reflect what I still don't have a good feel for.

Motivation

My motivation for studying these systems was to discuss a toy model similar to spin glasses, but without having to introduce the Hamiltonian, so non-physicist readers can also follow along. I'm not actually sure how well this "probabilistic spin glass" resembles spin glasses studied by physicists. (I stopped being a real physicist 10+ years ago.)

The idea was to take a $N \times N$ sized spin system, ie. each cell has two states, ↑ and ↓, or 0 and 1. If each spin is indepdendent of its neighbours, then the whole system is just a list of indepdendent random variables, the geometry doesn't matter. To make it interesting, I made the spins dependent on the neighbours: for spin $s$ and neighbour $n$: $P(s=↑ | n=↑) = P(s=↓ | n=↓) = p$. So the spins align with probability $p$ and are opposite with probability $1-p$.

The questions then are:

given $N$ and $p$, what is the probability of each possible $N \times N$ grid, of which there are $2^{N \times N}$?
given $N$ and $p$, how do we write a Monte Carlo simulation so that each grid out of the $2^{N \times N}$ options is generated with the correct possibility?
how do we verify our generation procedure?
what can we say about the entropy of the grid [ensemble]?
if we start with some grid, can we create a dynamic update rule given $N$ and $p$?
what happens to the system if we apply the above update rule?

Symmetry

In part I, I wrote a MC method to generate grids given $N$ and $p$. They looked right, passed the smell test:

However, later I noticed that the construction method is not symmetric. Ie. there are grids that are "the same" (grid A and B are the same if you can go from one to the other by some combination of flipping all bits at once, flipping the grid up-down or left-right, or rotating the grid in either direction).

I generated a large number of grids using the (broken) method and noticed that the frequencies don't match:

This was a pretty big miss. But, fixing this was relatively easy in the MC method, I just randomly applied a combination of the symmetry methods, and that fixed the frequencies:

At this point I wanted to be sure, so I verified that my construction process yields the correct $p$ probabilities within the grid (it does), so I used a MC simulation to plot the calibration curve:

Direct probabilities and entropy

I also wanted to plot the entropy as it varies with $N$ and $p$. In part I, I naively used the MC method, and generated $100 \times 2^{N \times N}$ grids in the hope of getting enough samples to get meaningful frequencies. Then in part II I realized that I can just compute the direct probabilility of each possible grid, and compute the exact entropy. Interestingly, the actual plot is the same, only high probabilility configurations seem to contribute:

This is interesting because it's the same shape as a random coin toss (ie. a $1 \times 1$ spin glass). If somebody would ask me how to "scale up" a random coin toss to $N > 2$ outcomes, while maintaining the concave shape of the entropy curve, I would be in trouble, but this is it.

Dynamic behaviour

In part III, I wanted to see what happens if I take a spin glass (grid), and start changing the spins one-by-one, by honoring the $p$ probabililty. To see what's going on, I rendered the process as a GIF:

Naively, by looking at the evolution, I assumed that if I generate a "typical" $(N,p)$ grid, and start evolving it, it will remain "close to the original", it will remain a "typical" $(N,p)$ grid. I believe this is naive for a number of reasons:

there are typical grids for a given $(N,p)$ because they have a relatively high probabilility, but all $2^{N \times N}$ possible grids have non-zero probability
in the same spirit, any evolution history has non-zero probability, including the one where all spins "spontanously" go to all ↑ or all ↓, or into a nice chessboard pattern, irrespective of the current configuration. So the above assumptions, which I put into quotes, are quite wishy-washy hand-wavy, and in fact I believe they are wrong (see below).

Fraction curve

I also plotted the fraction curve in part III, this shows the majority fraction $f$ of aligned spins (whichever of there are more, ↑ or ↓). This is interesting because it shows a non-linear relationship between $p$ and $f$:

However, I completely missed this plot's $N$-dependence in part III. In part IV I realized this:

The more spins there are (the larger the grid), the harder it is (the lower the probability) for more spins to align. Essentially, as $N$ gets larger, although islands of aligned spins form, eventually by chance the islands end, and an opposite spin island forms. Since the system is very large, and both spin directions are equally favored, overall the system will have roughly equal ↑ and ↓, and from far away, it will look like random noise (like TV noise). For large $N$, $p$ has to be very close to 1 for the majority fractions to move away from 0.5! This was a major insight for me, which came quite late, in part IV.

However, it is quite obvious. For example , drawing $500 \times 500$ grids, even the high $p$ ones are close to random noise. Contrast this with the first low $N$ plots at the beginning.

Attractors

In part V, I did a more thorough, but still simulation-based, investigation of $p$ and $N$-dependence of dynamic behaviour. Plotting the saturation trajectories for the different sized grids, for different $p$s (y-axis is % of majority aligned spins, x-axis is frame/steps, each trajectory is a line, 100 lines for 100 trajectories):

$10 \times 10$:

$20 \times 20$:

$50 \times 50$:

$100 \times 100$:

Plotting the saturation ratio (of the 100 trajectories), for different sized grids, at different $p$s:

This suggests:

higher $p$ leads to higher likelihood of saturation
higher $N$ makes it harder to saturate, ie. for a higher $N$ spin glass $p$ also has to be higher to saturate

However, this part of the investigation is inconclusive:

I suspect that majority fraction $f=0.5$ and $f=1$ attractors are "sticky", ie. it's hard to get away from them. For low $p$s, it's hard to get away from $f=0.5$ and it seems it practically "never" happens, then at some critical $p$ it becomes "possible" or "likely", and in those cases I believe it creeps up to $f=1$, where it gets stuck (sticky ceiling)
as I wrote earlier, probabilistically every configuration can go into any other configuration (assuming $0 < p < 1$), but I don't know how to express the apparent stickyness of the $f$ ceiling
it's also not clear to me whether the critical $p$, if it exists, is the same for all grid sizes, and I just have to run the simulations longer to see the same behaviour, or it differs

Conclusion

This has been an interesting investigation, purely conducted out of intellectual curiousity. I may re-visit it once I have some new ideas, especially on how to quantitatively classify the long-term dynamic behaviour and its potential attractors.

Optimal coverage for Wordle with Monte Carlo methods - Part III

2022-01-22T00:00:00+01:00

Introduction

In the previous article, we saw that a relatively simple Monte Carlo method can find a lot of 24-letter-unique wordlists, about 1 per second running at 16x parallelism. Here I use a crucial insights from the english language to make the search more efficient, and find a 25-letter-unique Worlde wordlist. The ipython notebook is up on Github.

Vowels

The key insight came from my friend and collegaue Rameez Kakodker, who introduced me to Wordle. We were discussing the methods for solving this problems, when he said that vowels are likely to duplicate. This led me to the following idea: to get a 25-unique-solution, we need 5 words with 25 unqiue letters. But if a word has two (or more) vowels, like BRAVE, it is very unlikely that this word would be in such a wordlist, because it contains both A and E. But there are only 5+1 vowels in english: AIEUO+Y, so ignoring Y, if a word uses up 2 vowels, like BRAVE, then we won't have enough vowels left for all the 5 words.

So the idea is:

Like before, prune the vowel wordlist, and keep only words which have 5 unique letters.
Further prune the wordlist, and keep only words that have 1 vowel. So eg. BRAVE and ALIEN is pruned, but STICK is kept.
When searching for the 5 words, always pick 5 words (from the above pruned words) which contain one of the 5 vowels AIEUO. In other words, treat the 5 word list like 5 slots, and always pick an A-word for the first slot, always pick an I-word for the second slot, and so on, eg. ['bawty', 'crimp', 'fjeld', 'vughy', 'zonks']

Step 1 and 2 in code:

vowels = sorted(['a', 'e', 'i', 'o', 'u'])

ltw = defaultdict(set) # letter-to-word index
for word in words:
    if len(set(list(word))) == len(word): # make sure all 5 letters are unique
        if len(set(vowels) & set(list(word))) == 1: # make sure there is exactly 1 vowel in the word
            for letter in word:
                ltw[letter].add(word)
len(words), sum([len(v) for _, v in ltw.items()])

For step 3, we will use the same method as before: randomly generate wordlists, and if we find one that is promising (22 or more unique letters), then stick to that and try a few variations:

def solve_wordle(num_tests=1000, num_improve_attempts=1000, random_seed=0):
    random.seed(random_seed * 131071)
    solutions = []
    for _ in range(num_tests):
        wordlist = [random.sample(ltw[vowel], 1)[0] for vowel in vowels]
        letters = set(list(''.join(wordlist)))
        if len(letters) >= 22:
            for _ in range(num_improve_attempts):
                v = random.randint(0, len(vowels) - 1) # pick out one of the vowels randomly
                word = random.sample(ltw[vowels[v]], 1)[0] # pick a new word which contains this vowel
                new_wordlist = deepcopy(wordlist)
                new_wordlist[v] = word
                new_letters = set(list(''.join(new_wordlist)))
                if len(new_letters) > len(letters):
                    wordlist, letters = new_wordlist, new_letters
            if len(letters) >= 24:
                print(len(letters), wordlist)
                solutions.append(wordlist)
    return solutions

The rest of the code is as before, using joblib to run in parallel:

def flatten_list(li):
    return [item for sublist in li for item in sublist]

def clean_solutions(solutions):
    return [t.split(',') for t in sorted({','.join(sorted(c)) for c in solutions})]

# single-threaded:
# solve_wordle(num_tests=10*1000*1000, num_improve_attempts=1*1000*1000)

# multi-threaded:
n_jobs = 16
num_tests = 10*1000*1000
num_improve_attempts = 100*1000
solutions = Parallel(n_jobs=n_jobs)(delayed(solve_wordle)
                                    (num_tests=num_tests,
                                     num_improve_attempts=num_improve_attempts,
                                     random_seed=i*random.randint(0, 100*1000))
                                    for i in range(n_jobs))
solutions = clean_solutions(flatten_list(solutions))
for wordlist in solutions:
    num_unique_letters = len(set(list(''.join(wordlist))))
    print(f'{num_unique_letters}-unique-letter solution: {wordlist}')
    if num_unique_letters == 25:
        print('*** JACKPOT! ***')
print(f'Found {len(solutions)} solutions...')

On my 12-core AMD system, after running it for 13 minutes, I got:

24-unique-letter solution: ['backs', 'flint', 'grews', 'jumpy', 'vozhd']
24-unique-letter solution: ['balky', 'cinqs', 'grump', 'vozhd', 'wheft']
...
24-unique-letter solution: ['brick', 'flews', 'jumpy', 'thanx', 'vozhd']
25-unique-letter solution: ['brick', 'glent', 'jumpy', 'vozhd', 'waqfs']
*** JACKPOT! ***
24-unique-letter solution: ['brick', 'gulpy', 'meynt', 'vozhd', 'waqfs']
24-unique-letter solution: ['bring', 'chomp', 'junky', 'veldt', 'waqfs']
...
24-unique-letter solution: ['jumby', 'knelt', 'pyric', 'vozhd', 'waqfs']
24-unique-letter solution: ['jumby', 'prong', 'veldt', 'wakfs', 'zilch']
Found 249 solutions...

Jackpot!

The following wordlist has 25 unique letters: ['brick', 'glent', 'jumpy', 'vozhd', 'waqfs']. The english alphabet has 26 letters, the one missing letter is x. Playing these 5 words in Wordle will usually reveral a lot of letters from the solutions, and probably a few positions. For example, for today's Wordle:

So, without any thinking, we already know the 5 letters (ICENW), and 2 positions (W**C*). The word is WINCE:

Words

3 of the 5 words are unknown to me, 2 of them are not really english:

glent: to move quickly especially in an oblique direction
vozhd: a Russian leader (the word is russian, and means "to lead")
waqfs: plural of waqf (probably from arabic), a Muslim religious or charitable foundation created by an endowed trust fund

Conclusion

I find it pleasing that hard problems can be solved on a modern desktop computer by brute-forcing with some minimal insights and minor optimizations.

Optimal coverage for Wordle with Monte Carlo methods - Part II

2022-01-21T00:00:00+01:00

Introduction

In the previous article, we saw that the simplest brute-force approach, when run in parallel, will yield 21-22-letter-unique solutions to Wordle. Let's look at simple ways to get 23-24-letter-unique solutions. The ipython notebook is up on Github.

Prune the dictionary

If we want to get a lot of unique letters, we need unique letters in the dictionary words. For example, HELLO is wasteful in the sense that it already has a duplicate. So let's only keep words where all 5 letters are unique. While we do this, let's a build a letter-to-word ltw index, which contains a list of words which contain each letter:

ltw = defaultdict(set) # letter-to-word index
for word in words:
    if len(set(list(word))) == len(word): # make sure all 5 letters are unique
        for letter in word:
            ltw[letter].add(word)

Two-stage Monte-Carlo

Let's take the following approach:

Select 5 unique random letters.
For each letter, take a random word to generate a 5-word candidate wordlist.
If the number of unique letters is less than 22, go to 1.
Else, this looks promising, go to stage 2, then go to 1.
Stage 2: Given the promising wordlist, try to improve it. Select a letter which is not currently in the wordlist, and replace one of the words with a word that contains this missing letter. If the new wordlist is better than the old, continue stage 2 with that. Perform this stage 2 search a million (or more) times.

In code:

def solve_wordle(num_tests, random_seed, num_loops):
    # stage 1:
    seed_solutions = generate_seed_solutions(num_tests, random_seed)
    # stage 2:
    solutions = run_improve_loops(deepcopy(seed_solutions), num_loops)
    return solutions

def generate_seed_solutions(num_tests, random_seed):
    random.seed(random_seed * 131071)
    seed_solutions = []
    for _ in range(num_tests):
        random_letters = random.sample(ltw.keys(), 5) # 5 random letters
        wordlist = sorted([random.sample(ltw[letter], 1)[0] for letter in random_letters])
        letters = set(list(''.join(wordlist)))
        if len(letters) >= 22 and wordlist not in seed_solutions:
            seed_solutions.append(wordlist)
    return seed_solutions

def run_improve_loops(seed_solutions, num_loops):
    loop_wordlist, solutions = seed_solutions, []
    for i in range(num_loops):
        additionals = []
        for j, wordlist in enumerate(loop_wordlist):
            new_wordlists, sub_solutions = generate_wordlists(wordlist)
            additionals.extend(new_wordlists)
            solutions.extend(sub_solutions)
        loop_wordlist = clean_solutions(additionals)
    return clean_solutions(solutions)

def generate_wordlists(wordlist):
    new_wordlists, solutions = [], []
    all_letters = set(list(string.ascii_lowercase))
    letters = set(list(''.join(wordlist)))
    remaining_letters = all_letters - letters
    for letter in remaining_letters:
        for new_word in ltw[letter]:
            for old_word in wordlist:
                new_wordlist = sorted(list(set(wordlist) - set([old_word]) | set([new_word])))
                new_letters = set(list(''.join(new_wordlist)))
                if len(new_letters) >= len(letters):
                    new_wordlists.append(new_wordlist)
                if len(new_letters) >= 24 and new_wordlist not in solutions:
                    solutions.append(new_wordlist)
    return new_wordlists, solutions

We can run this independently in parallel:

n_jobs = 16
num_tests = 10*1000*1000
solutions = Parallel(n_jobs=n_jobs)(delayed(solve_wordle)
                        (num_tests=num_tests, random_seed=i, num_loops=2)
                        for i in range(n_jobs))
solutions = clean_solutions(flatten_list(solutions))

Yields in 33 minutes of runtime:

24-unique-letter solution: ['ablet', 'crunk', 'jimpy', 'vozhd', 'waqfs']
24-unique-letter solution: ['absit', 'fling', 'jumpy', 'vozhd', 'wreck']
...
24-unique-letter solution: ['jumps', 'qubit', 'vozhd', 'wreck', 'xylan']
24-unique-letter solution: ['micky', 'pelfs', 'twang', 'urbex', 'vozhd']
Found 1863 solutions...

So this approach yields about one 24-letter-unique solution per second.

Conclusion

In the next article, I will show a Monte-Carlo approach which finds the jackpot, 25-unique-letter solution(s) in a few minutes.

Optimal coverage for Wordle with Monte Carlo methods - Part I

2022-01-19T00:00:00+01:00

Introduction

Wordle is a simple 5x5 word guessing game. It's free, a new puzzle is posted every day. You have to guess an unknown 5 letter word, like PROXY. You have 5 guesses, and after each guess, the game tells you if you got one of the letters, but it's not at the right position (brown), or it's the right letter in the right position (green). Only words from the Wordle dictionary are accepted as guesses, ie. HELLO and WORLD are valid guesses, but ABCDE is not an accepted guess.

The list of Wordle dictionary words is known, it's a set of 12,972 words.

Optimal coverage challenge

One strategy is to analyze the set of words, frequencies of letters, and given what we know so far, pick the word [from the Wordle dictionary] to guess that maximizes the average information gain of the next guess, assuming that the target word is randomly selected [from the Wordle dictionary]. One of the inputs to this is an assumption how the player values knowing the correct position of the letters (green) versus just knowing the set of letters making up the word (brown).

There is a coincidence (?) in the structure in the game: the english alphabet is 26 words, and we can guess a total of 25 characters. If we can find 5 words which share no identical characters, then we can always use these 5 words as our 5 guesses, and assuming the target word has N unique letters, we will always no at least N-1. Eg. if the word is HELLO, and after our 5 guesses we know that H E L and O are in the words, then:

either the word has a duplicate letter, one of those 4
the word has a fifth unique letter, the 26th letter that was not included in our 5 guesses (which covered 25)

We don't know which of the above is the case, but most of the time we can assume a reasonable player will be able to guess the target word at this point, also taking into account that on average a few letters' position will be known.

So the question is: from the Wordle dictionary, can we find 5 words such that all 25 letters are unique? If not, what is the maximum achievable unique-count?

Monte Carlo approach

The simplest Monte Carlo approach is to just randomly pick 5 words from the dictionary, and check how many unique letters we have:

words = ["cigar", "rebut", "sissy", ... ] # the Wordle dictionary
num_tests = 1000*1000
for _ in range(num_tests):
    wordlist = random.sample(words, 5) # 5 random words
    letters = set(list(''.join(wordlist)))
    if len(letters) >= 21:
        print(len(letters), wordlist)

It will print something like:

21 ['lotsa', 'finks', 'judge', 'macho', 'warby']
21 ['muzak', 'vales', 'letch', 'bilgy', 'fjord']
21 ['fagot', 'cushy', 'pawks', 'melon', 'diver']
21 ['miter', 'munch', 'pawky', 'bodge', 'lifes']
22 ['kight', 'flyby', 'roven', 'clasp', 'dwaum']

Let's try to be a little smarter.

Parallelization

The simplest, still brute force improvement is to run the same thing, but in parallel. I have a 12-core (24 threads) CPU, so I can run 16x parallelism without impact the usability of my computer:

def solve_wordle(num_tests=1000, random_seed=0):
    random.seed(random_seed * 131071)
    solutions = []
    for _ in range(num_tests):
        wordlist = random.sample(words, 5) # 5 random words
        letters = set(list(''.join(wordlist)))
        if len(letters) >= 21:
            solutions.append(wordlist)
    return solutions

def flatten_list(li):
    return [item for sublist in li for item in sublist]

def clean_solutions(solutions):
    return [t.split(',') for t in sorted({','.join(sorted(c)) for c in solutions})]

n_jobs = 16
num_tests = 1*1000*1000
solutions = Parallel(n_jobs=n_jobs)(delayed(solve_wordle)
                        (num_tests=num_tests, random_seed=i)
                        for i in range(n_jobs))
solutions = clean_solutions(flatten_list(solutions))
for wordlist in solutions:
    num_unique_letters = len(set(list(''.join(wordlist))))
    print(f'{num_unique_letters}-unique-letter solution: {wordlist}')
    if num_unique_letters == 25:
        print('*** JACKPOT! ***')
print(f'Found {len(solutions)} solutions...')

The ipython notebook is on Github. Prints something like:

21-unique-letter solution: ['admix', 'itchy', 'ovels', 'tupik', 'wrung']
21-unique-letter solution: ['adzed', 'boxty', 'brugh', 'vinyl', 'wacks']
 ...
21-unique-letter solution: ['fyces', 'goban', 'lurve', 'mewed', 'piths']
21-unique-letter solution: ['light', 'packs', 'vends', 'wauff', 'womby']
Found 47 solutions...

Conclusion

In the next part, I will show an improved Monte Carlo approach, which finds about one 24-unique-letter solution every second.

Probabilistic spin glass - Part V

2022-01-06T00:00:00+01:00

Introduction

In the previous articles I looked at various properties of probabilistic spin glasses by simulating ensembles of many samples and computing various statistics, while in the case of entropy I computed probabilities directly. Then I let grids evolve dynamically over "time" by changing spins one by one. In the previous article I ran simulations to understand the ensemble scaling behaviour for large spin glasses:

Now I will revisit the dynamic behaviour and see how the saturation behaviour scales with $p$ and $N$. The ipython notebook is on Github.

Saturation behaviour

Let's take a periodic $N \times N$ spin glass with $P(s=1|n=1)=p$ and let it evolve over time. One evolution step is taking a random spin, ignoring its current state, and based on the surrounding four states probabilistically setting a new state. Let's define a frame to be $N \times N$ such steps, so that on average each spin gets one chance to change per frame.

What I am interested here is the saturation behaviour of the spin glass. Saturation is when the majority of the spins align in one direction, whether that direction is ↑ or ↓. Since we're running probabilistic simulations, the ratio of aligned spins will not be exactly 1.0; we define a saturation threshold, above which we say that "essentially" all spins are aligned. For the smallest considered $10 \times 10$ spin glass let's use satuation_limit=0.90, for larger ones saturation_limit=0.95.

Let's take a $N \times N$ spin glass, and let it evolve for 1000 frames, using different $p$s. At each frame, let's plot the saturation, ie. the majority spin ratio. Let's repeat this for 100 different trajectories, and record the ratio of trajectories where the spin glass saturated (this will be shown in the plot titles like sat=0.8 meaning 80% of trajectories saturated ie. reached the saturation_limit of aligned spins). The code for this:

def get_saturation(N, p0, starting_ps, ps, num_frames,
                   n_jobs, num_trajectories_per_job, saturation_limit, draw_plots=True):
    if draw_plots: fig, axs = plt.subplots(1, len(ps), figsize=(20, 4))
    results = []
    for i, (starting_p, p) in enumerate(zip(starting_ps, ps)):
        trajectories = Parallel(n_jobs=n_jobs)(delayed(get_trajectories_periodic)
                                (rows=N, cols=N, p0=p0, starting_p=p, p=p,
                                 num_trajectories=num_trajectories_per_job,
                                 num_frames=num_frames, random_seed=i)
                                for i in range(n_jobs))
        trajectories = list(itertools.chain(*trajectories))
        saturation = np.mean([np.mean(t[-10:]) > saturation_limit for t in trajectories])
        results.append((N, p0, starting_p, p, saturation, trajectories))
        for trajectory in trajectories:
            if draw_plots: axs[i].plot(trajectory)
        if draw_plots: axs[i].set_ylim((0.49, 1.0))
        if draw_plots: axs[i].set_title(f'N={N}, start_p={starting_p:.3f}, p={p:.3f}, sat={saturation:.2f}')
    if draw_plots: plt.show()
    return results

Note the use of Parallel() to use all available cores for Monte Carlo simulation. I ran this with 20x parallelism for more sample size.

We can run this for different sized spin glasses. In each of the trajectories, let's start with a random noise starting_p=0.5 spin glass:

# fixed starting_p=0.5
p0, starting_p, num_frames, n_jobs, num_trajectories_per_job = 0.5, 0.5, 1000, 20, 5
ps = [0.50, 0.75, 0.90, 0.975]
Ns = [10, 20, 50, 100]
saturation_limits = [0.9, 0.95, 0.95, 0.95]
for N, saturation_limit in zip(Ns, saturation_limits):
    r = get_saturation(N, p0, [starting_p] * len(ps), ps, num_frames,
                         n_jobs, num_trajectories_per_job, saturation_limit)
    results.extend(r)

Plotting the saturation trajectories for the $10 \times 10$ case, for different $p$s (y-axis is % of majority aligned spins, x-axis is frame/steps, each trajectory is a line, 100 lines for 100 trajectories):

This shows that for the $10 \times 10$ spin glass (notice the sat in the plot titles):

at $p=0.5$, none of the trajectories saturate within 1000 frames (first, left-most plot)
at $p=0.75$, 67% of trajectories saturate within 1000 frames (second plot)
at $p=0.9$ and higher, all trajectories saturate

Repeating this for the $20 \times 20$ spin glass:

$50 \times 50$:

$100 \times 100$:

What this suggest is that:

Higher $p$ leads to higher likelihood of saturation.
Higher $N$ makes it harder to saturate, ie. for a higher $N$ spin glass $p$ also has to be higher to saturate.
The saturation ceiling is sticky. For all cases considered, it seems that once a system saturates, it's very hard to escape, ie. come back down from the saturation ceiling; of the trajectories shown, there is not a single case where this happens.

I will not show it here, but the notebook shows it: this behaviour is not dependent on starting_p, the $p$ used to generate the initial grid in the trajectories. Whether starting_p=p or starting_p=0.5 (random noise), the behaviour is the same (after 1000 frames).

Let's stop looking at individual trajectories, and plot the saturation ratio (of the 100 trajectories), for different sized grids, at different $p$s:

This confirms what we saw on the trajectory plots:

higher $p$ leads to higher likelihood of saturation
higher $N$ makes it harder to saturate, ie. for a higher $N$ spin glass $p$ also has to be higher to saturate

Conclusion

The lower $N$ curves suggest that there is a critical $p$, above which trajectories are very likely to saturate. I suspect that this is the case for larger spin glasses too, but 1000 frames is not enough to see this; however, this is not clear, since the definition of a frame is $N$-dependent. Subsequent experiments to run:

zoom in on the critical $p$ at varous spin sizes
see if the critical "floor to ceiling" behavior is there for large $N$, at higher frame count
as the frame count goes to infinity, would all spin glasses at all $p$s saturate eventually?

Probabilistic spin glass - Part IV

2021-12-31T00:00:00+01:00

Introduction

In the previous articles (Part I, Part II, Part III) I looked at various properties of probabilistic spin glasses by simulating ensembles of many samples and computing various statistics, while in the case of entropy I computed probabilities directly. Then I let grids evolve over "time" by changing spins one by one.

In this final article I will run simulations to understand the scaling behaviour for large spin glasses. The ipython notebook is up on Github.

Scaling behaviour

Let's see what different size spin glasses look like at different $P(s=1|n=1)=p$ values. Here are $50 \times 50$ grids:

Here are $500 \times 500$ grids, so each of these grids is $10 \times 10 = 100$ times larger than the previous ones:

These look quite different! In the $500 \times 500$ case, the $p=0.7$ and even the $p=0.8$ look like random noise, whereas for the smaller grid, we see the deviation from random noise much earlier.

This behaviour is quite interesting. Actually, each $50 \times 50$ part of the larger $500 \times 500$ grid looks about the same as the corresponding smaller grid, if we were to zoom in. But with a larger grid, there are more chances for the spins to mis-align, so even if there is a block of ↑ spins, eventually there will come a block of ↓ spins. For a smaller grid, this also happens, but there is less space for it to happen, so the grid is more likely to show a pattern. And for a large enough grid, in the zoomed out view, these will look like random specks of noise.

We can quantify the above by drawaing the majority fraction of the spins. This is just the fraction of the spins that all ↑ or ↓, whichever is more. In the previous article, we already looked at this, with the standard deviation of this fraction:

Let's draw the same curve, but for different sized grids. Let's look at the average and standard deviation seperately using our usual Monte Carlo measurement method:

# Monte Carlo measurement
p0, num_samples = 0.5, 100
measurements = []
for p in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.975]:
    for N in [50, 100, 200, 500]:
        rs = [fraction(create_grid_symmetric(rows=N, cols=N, p0=p0, p=p)) for _ in range(num_samples)]
        measurements.append((N, p, np.mean(rs), np.std(rs)))
# plot
fig, axs = plt.subplots(1, 2, figsize=(15, 6))
Ns = sorted(list(set([x[0] for x in measurements])))
for N in Ns:
    pts = sorted([(x[1], x[2], x[3]) for x in measurements if x[0] == N])
    axs[0].plot([x[0] for x in pts], [x[1] for x in pts], marker='o')
    axs[0].set_xlabel('p')
    axs[0].set_ylabel('majority fraction')
    axs[1].plot([x[0] for x in pts], [x[2] for x in pts], marker='o')
    axs[1].set_xlabel('p')
    axs[1].set_ylabel('stdev. of majority fraction')
    axs[1].legend([f'{N}x{N}' for N in Ns])

Yields:

This confirms the above. As the grid size gets larger, the (mean) majority fraction gets pushed down, as does its standard deviation. So in the limiting case, for an infinitely large grid, the majority fraction curve would get pushed down to 0.5, and then jump to 1 at $p=1$ (if $p=1$, all the spins have to align).

Conclusion

In retrospect, the above scaling behaviour makes sense, but it was not expected. I thought it follows the same fraction-curve at all grid sizes.

Probabilistic spin glass - Part III

2021-12-25T00:00:00+01:00

Introduction

In the previous articles (Part I, Part II), I looked at various properties of probabilistic spin glasses by simulating ensembles of many samples and computing various statistics, while in the case of entropy I computed probabilities directly. Here I will take a different route, and instead start with a starting grid, and let it evolve over "time" by changing spins one by one, and see how it behaves. The ipython notebook is on Github.

Evolution rule

Let's see what happens if we apply a dynamical probabilistic evolution operator to a probabilistic spin glass. The approach we will follow is simple:

Generate an initial grid per the (symmetrized) method described in the previous articles.
Pick a random (non-edge) spin, and given the four neighbours, pick a new alignment with the appropriate probability.

First, we have to arrive at the evolution probabilities: for a spin glass defined by $P(s=1 | n=1)=p$, what is the correct evolution probability given 4, already set neighbours? Instead of actually deriving the proability (which seems non-trivial to me), I cheat, and generate a large number of spin glasses and count frequencies to arrive at the probabilities. Since for 4 neighbours there are only $2^4=16$ possibilities, it's relatively easy to get good statistics.

def make_star_conditional(rows, cols, p0, p, num_simulations=1000):
    joint_frequencies, total = defaultdict(int), 0
    for simulation in range(num_simulations):
        pct = int((simulation / num_simulations) * 10000) / 100
        print(f'Computing conditionals for the {rows}x{cols} spin glass, progress {pct}% ', end='\r')
        sys.stdout.flush()
        grid = create_grid(rows, cols, p0, p)
        for i in range(1, rows-1):
            for j in range(1, cols-1):
                pattern = '%d%d%d%d%d' % (grid[i, j], grid[i, j-1], grid[i, j+1], grid[i-1, j], grid[i+1, j])
                joint_frequencies[pattern] += 1
                total += 1
    joint_probabilities = {} # joint_probabilities
    for pattern, freq in joint_frequencies.items():
        joint_probabilities[pattern] = freq/total
    return lambda up, down, left, right: ...

Visualization

Now we can run this probabilistic evolution, and on every steps_per_frame step, draw the frame on the screen:

def save_animation(filename, rows, cols, p0, starting_p, p, seconds=100, frames_per_second=5, steps_per_frame=6*1000):
    conditional_set_four_neighbours = make_star_conditional(rows, cols, p0, p)
    num_steps = seconds * frames_per_second * steps_per_frame
    grid, frames = create_grid(rows, cols, p0, starting_p), []
    for step in range(num_steps):
        if step % steps_per_frame == 0:
            frames.append(grid.copy())
        i, j = randint(1, rows-2), randint(1, cols-2)
        grid[i, j] = conditional_set_four_neighbours(grid[i, j-1], grid[i, j+1], grid[i-1, j], grid[i+1, j])
    fig = plt.figure()
    fig.suptitle(f'starting_p={starting_p:.2f}, p={p:0.2f}')
    im = plt.imshow(frames[0], cmap='Greys', vmin=0, vmax=1)
    plt.axis('off')
    def animate_func(i):
        im.set_array(frames[i])
        return [im]
    anim = animation.FuncAnimation(fig, animate_func,
        frames=seconds*frames_per_second,
        interval=1000/frames_per_second)
    anim.save(filename, fps=frames_per_second)

Previously, spin glasses had 4 parameters: rows, cols, p0, p. Here there's a fifth one, starting_p. This is in case we want the initial starting grid to be generated with a different p then in the evolution steps. This is interesting because we can check whether and how quickly the system forgets its initial configuration.

This is what a rows=50, cols=50, p0=0.5, starting_p=0.9, p=0.9 spin glass looks like:

And here is the same, but with starting_p=0.5, so it starts from random noise:

Note: the spins at the edges of the grid are not changed in the simulation.

The main takeaways:

On the first simulation, the overall pattern doesn't change much. This is because at p=0.9, if all 4 neighbours are the same spin alignment, the spin is very likely to align. So the spins that tend to change are the ones on the edges of the patterns, where at least 1 or 2 of the spins are not aligned.
On the second simulation with starting_p=0.5 we start out with a p=0.5 grid, which is just random noise. But then, very quickly, since we're evolving with the p=0.9 probabilities, out of the original randomnesss, a typical p=0.9 patterned grid forms, which then seemingly behaves like described in the previous point.

Visualizations of other parameters:

Same, but starting from starting_p=0.5 noise:

Convergence behaviour

We can check the convergence behaviour more systematically, ie. how quickly does the system forget the starting_p. Let's look at multiple trajectories, with different starting_ps, and let's use the majority fraction of spins that are aligned as the measure of order (so this is the y-axis):

def fraction(grid):
    r = np.sum(grid)/grid.size
    r = max(r, 1-r)
    return r

def get_trajectories(rows, cols, p0, starting_p, p, num_trajectories=3, seconds=100, frames_per_second = 5, steps_per_frame=6*1000):
    cache_key = str((rows, cols, p0, p))
    if cache_key not in conditional_set_four_neighbours_cached:
        conditional_set_four_neighbours_cached[cache_key] = make_star_conditional(rows, cols, p0, p)
    conditional_set_four_neighbours = conditional_set_four_neighbours_cached[cache_key]
    num_steps = seconds * frames_per_second * steps_per_frame
    trajectories = []
    starting_grid = create_grid(rows, cols, p0, starting_p)
    for t in range(num_trajectories):
        fs = []
        grid = starting_grid.copy()
        fs.append(fraction(grid))
        for step in range(num_steps):
            i, j = randint(1, rows-2), randint(1, cols-2)
            grid[i, j] = conditional_set_four_neighbours(grid[i, j-1], grid[i, j+1], grid[i-1, j], grid[i+1, j])
            if step % steps_per_frame == 0:
                fs.append(fraction(grid))
                pct = int((step / num_steps) * 10000) / 100
                print(f'Doing trajectory {t+1}/{num_trajectories} on the {rows}x{cols} spin glass, progress {pct}% ', end='\r')
                sys.stdout.flush()
        trajectories.append(fs)
    return trajectories

rows, cols, p0 = 50, 50, 0.5
mts = []
for p in [0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.975]:
    for starting_p in [0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.975]:
        trajectories = get_trajectories(
            rows=rows, cols=cols, p0=p0, starting_p=starting_p, p=p, num_trajectories=1)
        mts.append((p0, starting_p, p, trajectories[0]))

Yields:

What this suggests is that it doesn't matter what the starting grid is, this initial condition is quickly washed out, and the grid behaves as if it always was running at its evoluationary p.

But why is the fraction so different in different simulation runs at higher p? To make sense of this, let's look at the fraction-curve from Part I again, but this time let's also plot the standard deviation, specifically for a $50 \times 50$ grids:

This is in agreement with what we see in the simulation runs: at p higher than 0.7, the standard deviation of the majority spin alignment is quite high, so there is a lot of deviation from the average fraction.

Conclusion

In the next and final piece I will make the system periodic and look at how behaviour changes with grid size.

Probabilistic spin glass - Part II

2021-12-18T00:00:00+01:00

Introduction

This is a continuation of the previous article on probabilistic spin glasses, with improvements to the simulation code and improved entropy computation. The ipython notebook is up on Github.

Symmetry

In the previous article I generated probabilistic spin glasses such as neighbouring spins are aligned with probability $p$. The code used started in the top left corner, first generated the first spin with probability $p_0$, and then used $P(s=1 | n=1) = p$ to generate the spins in the first row. The first element in the second row is again trivial to generate using $p$, then each spin has 2 neighbours, so I derived the conditional formula for 2 neighbours. To be sure the construction is correct, I ran Monte Carlo simulations: I generated a large number of spin glasses, and made sure that neighbouring spins are aligned with probability $p$.

However, the construction still has a shortcoming. We would expect spin glasses that are "the same" to be generated with equal probability. Ie. if X and Y are spin glasses and X is an up-down or left-right mirror or for square grids a 90° rotation (or an arbitrary combination of the previous 3) of Y, then X and Y are really "the same", and should be generated with the same probability in a Monte Carlo ensemble.

Let's check this for the code from the previous article. Let's generate a large number of grids, and count the frequencies:

num_simulations = 10*1000*1000
rows, cols, p0, p, d = 2, 2, 0.5, 0.9, 4
frequencies = Counter()
for i in range(num_simulations):
    if i % 100 == 0:
        pct = int((i / num_simulations) * 10000) / 100
        print(f'Computing grid probabilities for the {rows}x{cols} spin glass, progress {pct}% ', end='\r')
        sys.stdout.flush()    
    grid = create_grid(rows, cols, p0, p)
    frequencies[json.dumps(grid.tolist())] += 1

fig, axs = plt.subplots(d, d, figsize=(10, 10))
for i, (grid, f) in enumerate(frequencies.most_common()):
    ax = axs[i // d, i % d]
    ax.imshow(np.array(json.loads(grid)), cmap='hot', vmin=0, vmax=1)
    ax.axes.get_xaxis().set_visible(False)
    ax.axes.get_yaxis().set_visible(False)
    ax.set_title(f'p=%0.4f' % (f/num_simulations))
plt.show()

Yields:

The symmetry does not hold! The first grid in the 2nd row and the first grid in the last row are the same after a left-right flip, but their probabilities are not the same! There are two ways to fix the construction:

When generating the grid, randomly pick the starting point (top or bottom, left or right) and randomly pick the starting direction (horizontal or vertical). Implementing it yields hard to read code.
Generate the grid using the simple code from the previous article, then randomly apply one of the 8 symmetry transformations.

Let's do the second:

def create_grid(rows, cols, p0, p):
    # ...
    # code from previous article generates grid
    # ...
    # make the ensemble symmetric
    if random() < 0.5:
        grid = np.fliplr(grid)
    if random() < 0.5:
        grid = np.flipud(grid)
    if rows == cols and random() < 0.5:
        grid = np.rot90(grid)
    return grid.astype(int)

After this, the probabilities for grids in the same symmetry groups are equal:

Probabilities instead of Statistics

In the above example, I generated 10,000,000 grids to get good sample size. But this is not really required. For a $ N \times K $ spin glass, there are $ 2^{N \times K} $ up-down spin combinations. We can turn the construction process on its head: instead of using the probabilities to generate random grids, we can just enumerate all possible grids, and compute the probability of generating that. The basic logic is very similar to the Monte Carlo simulation:

def spin_glass_probability_inner(grid, p0, p):
    rows, cols = grid.shape
    cp2 = calculate_conditional_probs_for_1(p0, p)
    grid_probability = 1
    # first element
    grid_probability *= cp(grid[0, 0], p0)
    # first row
    for j in range(1, cols):
        grid_probability *= cp(grid[0, j-1] == grid[0, j], p)
    # remaining rows
    for i in range(1, rows):
        grid_probability *= cp(grid[i-1, 0] == grid[i, 0], p)
        for j in range(1, cols):
            grid_probability *= cp(grid[i, j], cp2['%d%d' % (grid[i-1, j], grid[i, j-1])])   
    return grid_probability

However, this has the same symmetry problem, let's fix that:

def spin_glass_probability(grid, p0, p):
    grid_probability = 0
    # create all possible symmetry transformations of grid
    # all combinations of left-right, up-down mirroring and 90 degree rotation
    # total of 2^3 = 8 transformations, including identity, ie. no transformation
    # the symmetric probability of the grid is the average probability
    grid_probability += spin_glass_probability_inner(grid, p0, p)
    grid_probability += spin_glass_probability_inner(np.fliplr(grid), p0, p)
    grid_probability += spin_glass_probability_inner(np.flipud(grid), p0, p)
    grid_probability += spin_glass_probability_inner(np.fliplr(np.flipud(grid)), p0, p)
    normalization = 4
    if rows == cols:
        grid_probability += spin_glass_probability_inner(np.rot90(grid), p0, p)
        grid_probability += spin_glass_probability_inner(np.fliplr(np.rot90(grid)), p0, p)
        grid_probability += spin_glass_probability_inner(np.flipud(np.rot90(grid)), p0, p)
        grid_probability += spin_glass_probability_inner(np.fliplr(np.flipud(np.rot90(grid))), p0, p)
        normalization += 4
    return grid_probability / normalization
    #
    # without honoring symmetries:
    # return spin_glass_probability_inner(grid, p0, p)

We can now compare this to the Monte Carlo simulation to verify that we compute the same probabilities for each $2 \times 2$ grid:

rows, cols, p0, p, d = 2, 2, 0.5, 0.9, 4
probabilities = Counter()
for li in list(itertools.product([0, 1], repeat=rows*cols)):
    grid = np.asarray(li).reshape(rows, cols).astype(int)
    grid_probability = spin_glass_probability(grid, p0, p)
    probabilities[json.dumps(grid.tolist())] = grid_probability

fig, axs = plt.subplots(d, d, figsize=(10, 10))
for i, (grid, grid_probability) in enumerate(probabilities.most_common()):
    ax = axs[i // d, i % d]
    ax.imshow(np.array(json.loads(grid)), cmap='hot', vmin=0, vmax=1)
    ax.axes.get_xaxis().set_visible(False)
    ax.axes.get_yaxis().set_visible(False)
    ax.set_title(f'p=%0.4f' % grid_probability)
plt.show()

With this approach we don't waste CPU time generating 10,000,000 samples. The Monte Carlo version took 203 seconds, this took just a few miliseconds!

There is still value in the Monte Carlo method, since computing all $ 2^{N \times K} $ combinations is not feasable for large spins, eg. a $ 100 \times 100 $ spin glass has $2^{10,000}$ spin combinations. By the time we go through that, the Sun has exploded and the Universe has ended. Here the MC method's importance sampling is still preferable. Let's use MC to draw the calibration curve for a $ 100 \times 100 $ grid using 100 samples, to make sure the generated grids obey the desired probabilities:

# consistency check
num_simulations = 100
rows, cols, p0 = 100, 100, 0.5
ps = []
for p in [ZERO, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, ONE]:
    same, total = 0, 0
    for _ in range(num_simulations):
        grid = create_grid(rows, cols, p0, p)
        for i in range(0, rows-1):
            for j in range(0, cols-1):
                if grid[i, j] == grid[i+1, j]:
                    same += 1
                if grid[i, j] == grid[i, j+1]:
                    same += 1
                total += 2
    measured_p = same/total
    ps.append((p, measured_p))

plt.plot([x[0] for x in ps], [x[1] for x in ps], marker='o')
plt.xlabel('actual p')
plt.ylabel('measured p')

Yields:

Entropy

Next, let's see how entropy varies with grid size for square grids. Since we can directly compute the probability of each grid, we enumerate the grids, get the probabilities, and use the standard formula for entropy:

def spin_glass_entropy(rows, cols, p0, p, base=2):
    entropy = 0
    for i, li in enumerate(list(itertools.product([0, 1], repeat=rows*cols))):
        grid = np.asarray(li).reshape(rows, cols).astype(int)
        grid_probability = spin_glass_probability(grid, p0, p)
        entropy -= grid_probability * log(grid_probability, base)
    return entropy

grid_entropies = []
for n in [1, 2, 3, 4]:
    for p in [ZERO, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, ONE]:
        h = spin_glass_entropy(rows=n, cols=n, p0=p0, p=p)
        grid_entropies.append((n, p, h))

for n in sorted(list(set([x[0] for x in grid_entropies]))):
    xs = [x[1] for x in grid_entropies if x[0] == n]
    ys = [x[2] for x in grid_entropies if x[0] == n]
    plt.plot(xs, ys, marker='o')
plt.xlabel('p')
plt.ylabel('entropy (bits)')
plt.legend(sorted(list(set(['%d x %d' % (x[0], x[0]) for x in grid_entropies]))))
plt.ylim((0, 17))

Yields:

We can plot the same another way, drawing lines for indentical proabilities $p$:

For a given grid size, the entropy $H(p)$ is concave. For a given probability $p$, the entropy $H(N)$ is convex and scales like $O(N^2)$ for a $N \times N$ spin glass (for $p=0.5$, entropy is exactly $H(N)=N^2$).

Conclusion

The main lesson here was that it's not always necessary to perform Monte Carlo simulations. Sometimes it's possible to enumerate all possibilities with the appropriate proabilities. When computing the entropy, this is preferred, since importance sampling does not necessarily yield a good approximation of entropy.

Probabilistic spin glass - Part I

2021-12-11T00:00:00+01:00

Introduction

In the previous articles I talked about entropy, and as an example of entropy in Physics, I derived the Sackur-Tetrode entropy of a monatomic ideal glass. Here I will discuss what I call probabilistic spin glass, a simple mathematical model of magnetized matter with short range interactions. In this article I introduce the concept and run some Monte Carlo simulations to get a feel for the model, and use entropy to characterize its order-disorder transition. The code is up on Github.

Probabilistic spin glass

Let's take an $N \times N$ sized spin system, ie. each cell has two states, ↑ and ↓, or 0 and 1. If each spin is indepdendent of its neighbours, then the whole system is just a list of indepdendent random variables, the geometry doesn't matter. To make it interesting, let's make the spins dependent on the neighbours: for spin $s$ and neighbour $n$: $P(s=↑ | n=↑) = P(s=↓ | n=↓) = p$. So the spins align with probability $p$ and are opposite with probability $1-p$.

Monte Carlo simulations

Let's take the above and run Monte Carlo simulations to see what happens as we vary $p$. We will take an array of size $N \times N$ and start filling out the spins according to the probabilities above. First we have to introduce an additional probability $p_0$, which is the probability of the first (let's say top left) spin being ↑. Once we set the first spin, we can set the rest of the spins in the first row. Once we get to the second row, we run into a difficulty. The spins above it have already been set, so these spins have 2 set neighbours. We need to compute the conditional probability $P(s=↑ | n_1=., n_2=.)$ given 2 neighbours $n_1$ and $n_2$ in a way that is consistent with the basic definition of $p$. Using Bayes' theorem, for the specific case of $P(s=↑ | n_1=↑, n_2=↑)$:

$P(s=↑ | n_1=↑, n_2=↑) = \frac{P(s=↑, n_1=↑, n_2=↑)}{P(s=↑, n_1=↑, n_2=↑) + P(s=↓, n_1=↑, n_2=↑)} $

The joint triplet probabilities are easy to compute, we can think of setting $s$ first with probability $p_0$, and then the left and right neighbour with probability $p$:

$P(s=↑, n_1=↑, n_2=↑) = p_0 p^2$

Expressed as Python code calculate_conditional_probs_for_1() returns the conditional probabilities for $P(s=↑, n_1=., n_2=.)$:

def calculate_conditional_probs_for_1(p0, p):
    q = 1 - p
    joint_probs = {
        '000': p0 * p * p,
        '111': p0 * p * p,
        '010': p0 * q * q,
        '101': p0 * q * q,
        '110': p0 * p * q,
        '011': p0 * p * q,
        '001': p0 * p * q,
        '100': p0 * p * q,
    }
    return {
        '00': joint_probs['010']/(joint_probs['010']+joint_probs['000']),
        '01': joint_probs['011']/(joint_probs['011']+joint_probs['001']),
        '10': joint_probs['110']/(joint_probs['110']+joint_probs['100']),
        '11': joint_probs['111']/(joint_probs['111']+joint_probs['101']),
    }

The rest of the system:

def make_unconditional_set(p0):
    return lambda: random() < p0

def make_conditional_set_one_neighbour(p):
    return lambda n: (n) if (random() < p) else (1 - n)

def make_conditional_set_two_neighbours(p0, p):
    conditional_probs_for_1 = calculate_conditional_probs_for_1(p0, p)
    return lambda n1, n2: random() < conditional_probs_for_1[str(int(n1))+str(int(n2))]

def create_grid(rows, cols, p0, p):
    unconditional_set = make_unconditional_set(p0)
    conditional_set_one_neighbour = make_conditional_set_one_neighbour(p)
    conditional_set_two_neighbours = make_conditional_set_two_neighbours(p0, p)
    grid = np.zeros([rows, cols], dtype=np.bool)    
    # first element
    grid[0, 0] = unconditional_set()
    # first row
    for j in range(1, cols):
        grid[0, j] = conditional_set_one_neighbour(grid[0, j-1])
    # remaining rows
    for i in range(1, rows):
        grid[i, 0] = conditional_set_one_neighbour(grid[i-1, 0])
        for j in range(1, cols):
            grid[i, j] = conditional_set_two_neighbours(grid[i-1, j], grid[i, j-1])
    return grid

def show(grid):
    plt.imshow(grid, cmap='hot', interpolation='nearest')
    plt.show()

Let's run it with $p_0=0.5$ and $p=0.7$ and see a realization:

show(create_grid(rows=10, cols=10, p0=0.5, p=0.7))

Validation

First, let's make sure the math and logic is correct. Let's simulate a large $100 \times 100$ spin glass with the above rules, then count all pairs, and make sure that the original conditional probability holds:

# consistency check
num_simulations = 100
rows, cols, p0, p = 100, 100, 0.5, 0.70
same, total = 0, 0
for _ in range(num_simulations):
    grid = create_grid(rows, cols, p0, p)
    for i in range(0, rows-1):
        for j in range(0, cols-1):
            if grid[i, j] == grid[i+1, j]:
                same += 1
            if grid[i, j] == grid[i, j+1]:
                same += 1
            total += 2
measured_p = same/total
print(f'Ran {num_simulations} Monte Carlo simulations for grid size {rows}x{cols} with p={p}, measured p was {measured_p:.2}')

Prints:

Ran 100 Monte Carlo simulations for grid size 100x100 with p=0.7, measured p was 0.7

Disorder and domains

First, let's look at the special case of $p=0$. In this case the spin glass looks always looks like this (or inverted):

This is because $p=0$ forces the spins to always be inverted compared to the neighbour.

Let's look at what happens as we vary $p$ from 0.1 to 1.0:

The same, on $100 \times 100$ spin glass systems:

As $p$ increases, disorder increases. Note that at $p=0.5$ the neighboring spins are independent. At $p > 0.5$ domains form, because spins prefer to align with each other, but sometimes randomly an inversion occurs, and a patch is formed.

Let's check how the fraction $f$ of aligned spins varies with $p$:

rows, cols, p0 = 100, 100, 0.5
num_samples = 100
pairs = []
for p in [0.001, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.975, 0.999]:
    rs = []
    for _ in range(num_samples):
        grid = create_grid(rows, cols, p0, p)
        r = np.sum(grid)/(rows*cols)
        r = max(r, 1-r)
        rs.append(r)
    pairs.append((p, np.mean(rs)))
plt.plot([x[0] for x in pairs], [x[1] for x in pairs], marker='o')
plt.xlabel('p')
plt.ylabel('f')
plt.ylim((0.48, 1.0))

Perhaps unintuitively, the dependence is not linear. The plot has two parts:

for $0 \leq p \leq \frac{1}{2}, f=\frac{1}{2}$. In this region, on average always half of the spins is ↑, half is ↓
- at $p=0$, the spins are completely ordered like a chess board
- at $p=\frac{1}{2}$, the spins are completely independent
at $p > \frac{1}{2}$, one of the spin directions starts to dominate as $f$ breaks away from $\frac{1}{2}$, but it's very slow
- $p$ has to get very close to 1 for $f$ to approach 1

Disorder and entropy

$f$ is constant between $0 \leq p \leq \frac{1}{2}$, but visually we can see that the system goes from order (chessboard) to complete disorder (independent spins). We can use entropy to characterize the order-disorder property of the model. Let's do brute-force Monte Carlo simulations to compute the entropy of a small $4 \times 4$ spin glass:

def entropy(frequencies, base=2):
    return -1 * sum([f/sum(frequencies) * log(f/sum(frequencies), base) for f in frequencies])

def spin_glass_entropy(rows, cols, p0, p, num_samples):
    frequencies = Counter()
    for _ in range(num_samples):
        grid = create_grid(rows, cols, p0, p)    
        frequencies[str(np.packbits(grid))] += 1
    return entropy(frequencies.values())

pairs = []
for p in [0.001, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.999]:
    h = spin_glass_entropy(rows=4, cols=4, p0=0.5, p=p, num_samples=100*1000)
    pairs.append((p, h))
plt.plot([x[0] for x in pairs], [x[1] for x in pairs], marker='o')
plt.xlabel('p')
plt.ylabel('entropy (bits)')
plt.ylim((0, 18))

Note: the entropy shown on the y-axis is specific to this $4 \times 4$ spin glass, it would be higher for a larger system.

The entropy plot above proves our intuition: at $p=0$, the entropy is just 1 bit, which is coming from $p_0=\frac{1}{2}$, which gives two (inverted) ways to get a chess board. It's also 1 at $p=1$, where the entire spin glass is either all ↑ or all ↓. In the middle, at $p=\frac{1}{2}$, the entropy is 16 bits, since $4 \times 4 = 16$, and all the spins are independent.

The concave shape of the entropy plot is the same as that of a $p$-biased coin with entropy function $H(p)=-(p) log[p] - (1-p)log[1-p]$, so with respect to entropy the spin glass behaves like a biased coin, where the biasing factor $p$ is the correlation strength between neighboring spins.

Conclusion

Spin glasses are a widely studied models of magnetism in Physics. Here I used the toy probabilistic spin glass model to avoid having to introduce too much physical formalism (eg. Hamiltonian). We saw how the behaviour of the spin glass strongly depends on $p$, the correlation probability, and that entropy describes the order-disorder aspect of the system that we observe visually.

The physical Sackur-Tetrode entropy of an ideal gas

2021-11-29T00:00:00+01:00

Introduction

In the previous article, I derived a coarse-grained formula for the entropy of an ideal gas, taking into account only the uncertainty in position of the particles. The derived expression for entropy cannot be physical, since it depends on how we as observer's choose to coarse grain the volume of the gas ($M$ bins). Also, it doesn't take into account the velocity of the particles. The next step, coming in this article, is to take make physical arguments and compute an expression for $W$ that takes into account the velocities, parametrized using the kinetic energy of the system and derive the Sackur–Tetrode equation for a monatomic ideal gas of indistinguishible particles. Once we have that, we can also derive the ideal gas equation of state $pV = NkT$.

Macroscopic state

When defining the entropy of an ideal gas, we first need to define some of its macroscopic properties. For example, are we talking about 1 liter of gas or 2 liters? What is the density of the gas? The parameters we need to define are:

$V$ volume
$N$ number of particles (specifying the number of particles $N$ is the same as specifying density $\rho$, since $\rho = N/V$)
$E$ total kinetic energy of the particles making up the gas

The first 2 make sense, but what about the 3rd one? We know from real-world experience that at the human, macroscopic scale, objects have a temperature. We will see that the total kinetic energy of particles is related to the temperature of the gas (slower movement is colder, faster movement is hotter).

Quantum mechanics

In classical physics, both position $x$ and velocity $v$ can be measured at arbitrary accuracy. So, in a thought experiment, we could measure $x$ and $v$ at time $t_0$, and we would know everything about the system at any other time. All we have to do is run a simulation, either backward or forward in time, and we could always tell $x_i(t)$ and $v_i(t)$ for all particles $i=1...N$. So, where does the uncertainty come from? Does it come from our unwillingness or inability to measure all the positions and velocities? If that were the case, entropy would be an observer or measurement dependent quantity, similar to the expression derived in the previous article.

The way out is to remember Heisenberg's uncertainty principle from quantum mechanics, which states that although a particle's position or velocity (momentum) can each be measured to arbitrary precision, both cannot be at the same time. There is a joint uncertainty in the measurement:

$\Delta x_i \Delta p_i \ge h$

Here $x_i$ is the position of the $i$th particle, $p_i=m v_i$ is the $i$th particle's momentum, $m$ is the common particle mass, and $h$ is a physical constant called Planck constant. In other words, the uncertainty principle introduces a natural coarse graining on the combined $(x, p)$ space!

Phase space

For an ideal gas, the microscopic state corresponds to specifying the position $x_i$ and momenta $p_i$ of all $i=1...N$ particles. For our treatment, we will take all the these and build one large phase vector $(x_1, p_1, x_2, p_2, ... x_N, p_N)$. Here each $x$ and $p$ is itself an (x, y, z) Cartesian 3D vector, so this vector lives in the $6N$-dimensional so-called phase space. Each point in phase space defines all the positions and momenta of the particles making up the gas and corresponds to a possible microscopic state of the system.

However, we must put additional constraints on this allowable points in the phase space:

the positions $x_i$ have to fall within the volume of the gas
the kinetic energies of the particles have to add up to the total kinetic energy $E$ of the gas

Marbles, bins and indistinguishible particles

In the previous article, we did not use a phase space formalism. Instead, we assumed we coarse grain the available volume into $M$ bins, and compute how many $W(M, N)$ ways we can put $N$ particles into the $M$ bins. In the phase space formalism, each point (and after quantization into $h$-sided hypercubes, each hypercube) in phase space already corresponds to a complete specification of the system. In other words, in this formalism, we are not computing $M$, the number of bins, we are directly computing $W$, the number of microscopic states!

How do we take into account the indistinguishibility of particles in the phase space formalism? If we pick a phase vector (or surrounding $h$-sided hypercube), where $(x_1, p_1)=A$ and $(x_2, p_2)=B$, that is really the same phase vector as when $(x_1, p_1)=B$ and $(x_2, p_2)=A$, since we can't tell particles 1 and 2 apart. So in the phase space formalism, if we have indistinguishible particles, we are double (multi) counting microscopic states, so we have to devide $W_{phase}$ by some factor to get the real $W$. What is the factor? We have to use some physical intuition and hand-waving. Because $h$ is a very small number, dividing the phase space into $h$-sided hypercubes yields a very large number of hypercubes, even compared to the number of particles $N,$ so the phase space will have a lot more hypercubes than $N$. This means that most of the time, no two particles will be in the same hypercube, ie. we can ignore cases like $(x_1, p_1)=A$ and $(x_2, p_2)=A$ (where $=A$ denotes "falls into hypercube A"). In that case, if all $N$ particles are in a different hypercube, the degree of overcounting is just $N!$, the number of permutations of N distinct letters.

Sackur-Tetrode equation

Based on the above considerations, we can now derive the physical entropy of a monatomic ideal gas. Let's assume the gas is in a cube of volume $V=L^3$, and the total kinetic energy of particles is $E= \frac{m}{2} \sum_i v_i^2 $. Let's switch to using momenta $p_i = m v_i$ and $v_i^2 = \frac{1}{m^2} p_i^2$, then the energy in terms of momenta is $E = \frac{1}{2m} \sum_i p_i^2$. So the phase space $(x_1, p_1, x_2, p_2, ..., x_N, p_N)$ is constrained to the manifold:

$ 0 < x_i < L$ — this contraint for $x_i$s defines a hypercube of volume $V_{hyper}=V^N$, of dimensionality $3N$ (note that each $x_i$ here has 3 internal coordinates $(x, y, z)$)
$E = \frac{1}{2m} \sum_i p_i^2$ — this contraint for $p_i$s defines the $(3N-1)$-dimensional shell of a $3N$-dimensional hypersphere of radius $r^2 = 2mE$ (note that each $p_i$ here has 3 internal coordinates $(p_x, p_y, p_z)$). This shell has area $ A_{hyper} = \frac{2 \pi ^ {\frac{3N}{2}} (2mE)^{\frac{3N-1}{2}} }{\Gamma(\frac{3N}{2})} $

In lower dimensions if we take a circle (with circumference $C$) and a square (with area $A$), they form a cylinder with wall area $C A$. This is also what happens in higher dimensions, a hypersphere with shell area $A_{hyper}$ and a hypercube with volume $V_{hyper}$ form a hypercylinder with wall area $ A_{hyper} V_{hyper} $. So the phase space volume $ P = V_{hyper} A_{hyper} $ is:

$ P = V^N \frac{2 \pi ^ {\frac{3N}{2}} (2mE)^{\frac{3N-1}{2}} }{\Gamma(\frac{3N}{2})} $

Here $\Gamma(x)$ is the Gamma function. Let's assume that $N$ is even, so $\frac{3N}{2}$ is integer, in which case the $\Gamma$ function is $\Gamma(x)=(x-1)!$. With this:

$ P = V^N \frac{2 \pi ^ {\frac{3N}{2}} (2mE)^{\frac{3N-1}{2}} }{(\frac{3N}{2}-1)!} $

Now we face a problem, or problems. The phase space itself is of 6N-dimensions ($N$ particles, each has 3 position and 3 momenta coordinates), but the manifold is actually a "surface" of 6N-1 dimensions, because the momenta contraints defined a $(3N-1)$-dimensional shell of a $3N$-dimensional hypersphere. To turn the area into a volume, we need to give the shell some thickness. So we will assume that the shell has an infinitesimal thickness $\Delta p$. Since $E=\frac{m}{2} v^2=\frac{1}{2m}p^2$, so $p^2=2mE$ and $p(E)=\sqrt{2mE}$. This means that $dP = \sqrt{\frac{m}{2E}} dE$, so the thickness is $ \Delta p = \sqrt{\frac{m}{2E}} \Delta E $. This term will be approximately ignored later, so it's not that important.

At this point, the fixed phase space volume is:

$ P = V^N \frac{2 \pi ^ {\frac{3N}{2}} (2mE)^{\frac{3N-1}{2}} }{(\frac{3N}{2}-1)!} \sqrt{\frac{m}{2E}} \Delta E $

Next, we use the uncertainty principle to coarse-grain this volume into hypercubes. Each $x-p$ pair (3 per particle, for $(x, y, z)$ physical 3D space) contributes an $h$, so the hypercubes have volume $h^{3N}$. We divide $P$ by this to get $W_{phase}$:

$ W_{phase} = \frac{P}{h^{3N}} = \frac{V^N}{h^{3N}} \frac{2 \pi ^ {\frac{3N}{2}} (2mE)^{\frac{3N-1}{2}} }{(\frac{3N}{2}-1)!} \sqrt{\frac{m}{2E}} \Delta E $

Now we just need to divide by the factor $N!$ to take into account the indistinguishibility of particles and get rid of the overcounting to get the final expression:

$ W = \frac{W_{phase}}{N!} = \frac{V^N}{N! h^{3N}} \frac{2 \pi ^ {\frac{3N}{2}} (2mE)^{\frac{3N-1}{2}} }{(\frac{3N}{2}-1)!} \sqrt{\frac{m}{2E}} \Delta E $

The entropy $S$ is then (in units of $k$, the Boltzmann constant):

$ \frac{S}{k} = ln[W] = ln[ \frac{V^N}{N! h^{3N}} \frac{2 \pi ^ {\frac{3N}{2}} (2mE)^{\frac{3N-1}{2}} }{(\frac{3N}{2}-1)!} \sqrt{\frac{m}{2E}} \Delta E ] $

This looks quite complicated, but can be handled quickly. The trick is to use the properties of the logarithm ($ln[ab]=ln[a]+ln[b]$, $ln[\frac{a}{b}]=ln[a]-ln[b]$ and $ln[a^k]=k ln[a]$) and Stirling's approxmiation $ln[n!]=n ln[n]-n$ to create a long list of log-terms. Then, put it in a form like $W=N A + B$, where $A$ is all the terms that have an $N$ multiplier outside of the logarithm. In the $A$ part, approximate $ln[\frac{3N-1}{2}] \approx ln[\frac{3N}{2}]$, and the entire $B$ part can be approximated to be small compared to the large $N$ multiplier $A$ part, so we are only left with the $N A$ part (I did it with pen and paper..). This leaves us with the famous Sackur-Tetrode equation for entropy of a monatomic ideal gas:

$\frac{S}{kN} = ln[\frac{V}{N}(\frac{4 \pi m E}{3 h^2 N})^\frac{3}{2}] + \frac{5}{2}$

Note that the entropy $S$ is a function of $S(V, N, E)$, and the Planck constant $h$ also appears.

Entropy is extensive

The above expression is obviously extensive. If we double our system, which takes $V \rightarrow 2V, N \rightarrow 2N, E \rightarrow 2E$, then due to the $\frac{V}{N}$ and $\frac{E}{N}$ term the logarithm value does not change, however the $N$ outside introduces the 2 multiplier, so $S(2V, 2N, 2E) = 2S(V, N, E)$.

Temperature, pressure and the equation of state

For a closed system, the 1st law of thermodynamics for an infinitesimal process where N does not change can be written as:

$dE = T dS - P dV$

Now let's take the Sackur-Tetrode equation, and rewrite it like:

$ S = k N ( ln[V] + \frac{3}{2} ln[E] ) + f(N) $

We kept just the $V$ and $E$ terms, and put all the rest into the function $f$, which is only a function of $N$. Now take the differential:

$ dS = k N ( \frac{1}{V} dV + \frac{3}{2E} dE ) + f'(N) dN$

Since we're realing with a process where $N$ does not change, $dN = 0$, we can ignore the last term. We can rewrite the remaining terms like:

$ dE = \frac{2E}{3kN} dS - \frac{2E}{3V} dV $

Comparing this with the 1st law, we can read off the expression for temprate $T=\frac{2E}{3kN}$ and pressure $P=\frac{2E}{3V}$.

Multiplying $P$ by $V$ yields:

$PV = \frac{2E}{3}$

Multiplying $T$ by $N k$ yeilds:

$NkT = \frac{2E}{3}$

We can read off the famous ideal gas equatation of state $PV=NkT$. We derived this from computing the entropy from physical considerations and using the 1st law of thermodynamics, which for our purposes effectively defined $T$ and $P$.

$h$ but no $c$

In Physics, when a formula contains the Plank constant $h$ it means that some aspect of quantum mechanics was taken into account. The Sackur-Tetrode equation contains $h$ because we used the uncertainty principle from quantum mechanics to quantize the phase space. Similarly, if a formula contains the speed of light $c$ it means that special relativity is taken into account. The Sackur-Tetrode equation does not contain $c$, it is not a relativistically correct. There is no contraint on the maximum speed any given particle can obtain, there is just the global contraint $E = \frac{1}{2m} \sum_i p_i^2$, so a microscopic state where one of the particles has a very high speed exceeding $c$ is allowed, even though it is not physically possible.

Conclusion

The article used the example of the physical entropy of a monatomic ideal gas to show the similarities and differences between entropy as it appears in phsyics and math (data science). In math, we have well-defined random variables and a simple defintion of entropy on top of that. In physics, we concept of entropy is more nuanced, and involves lots of idealizations and approximations (eg. we silently assumed all microscopic states we considered have the same probability).

Entropy of an ideal gas with coarse-graining

2021-11-19T00:00:00+01:00

Introduction

An ideal gas is a model of a real physical gas where we make some simplifications. We model the gas as hard billiard balls, bouncing off the walls of the container and each other in perfectly elastic collisions. In classical physics, momentum is always conserved, in elastic collisions kinetic energy is also conserved (ie. kinetic energy never gets transfered to some other form of energy [1]). Here I will consider a monoatomic ideal gas, ie. all the gas is made up of one type of billiard ball.

[1] In inelastic collisions, kinetic energy is lost as energy is converted to heat energy. This wouldn't make sense here, since we are using the billiard balls at the micro level to model heat and temperature at the macro level.

Entropy is the function of the probabilities in a random variable. If we assume that all outcomes are equally likely, so $ p_i = p = 1/N $, then the expression for the entropy becomes $ H = log[N] $.

Let's start with a naive (and wrong) thought experiment to get to entropy. Let's assume we have a container, with 1 particle in it. The particle has some velocity and is bouncing around the container. Let's compute the entropy due to the position of the particle.

Let's start with the simplest possible coarse-graining, and split the container into 2 bins (A and B), so the ball is either in A or B. With this coarse-graining, the entropy of the system is 1 bit. Now, if we imagine 2 of these systems sitting next to each other, but separated by a thin wall, the entropy will be 2 bits. As long as the wall is there, the systems like independent random variables, so the joint entropy H(X, Y) = H(X) + H(Y). What if we now remove the wall? We will have 2 balls in the system, and keeping the physical size of the coarse graining, there are now 4 parts. In total, we can put 2 identical balls into 4 bins $4+3+2+1=10$ ways, so the entropy is $log_2[10]=3.3$ bits. So the entropy of the combined system with the wall removed is greater than the entropy of the sum of the individual systems, because the balls can now occupy just the left side or just the right side of the system, a state that was not present before.

From a physical point of view, this is where our thought experiment fails. If we put two identical containers of monatomic gas (identical atoms, pressure, temperature, density) next to each other, remove the barrier, and let them mix, then effectively nothing happens, so intensive properties (like pressure, temperature, chemical potential) remain the same and extensive properties (like volume, entropy and particle number) are added (they are additive). In the above example, we'd expect the combined system to have 2 bits of entropy (at this level of coarse-graining).

The problem is simple: the ideal gas model of thermodynamics only works if $N$, the number of particles is very large, and $N=1$ is not large enough. So let's consider a lot of particles.

Notation

In physics, when we deal with entropy, there is some minor differences in notation:

we use $S$ instead of $H$ to denote entropy
after Ludwig Boltzmann, we use $W$ instead of $N$ to denote the total number of possible outcomes (W as in "ways things can happen")
we the $ln=log_e$ instead of $log_2$

Marbles and bins

Let's generalize the above naive approach, and imagine $N$ particles and a coarse-graining of volume so that we have $M$ bins of equal volume. Given $N$ indistinguishible marbles, how many $W$ ways can we put them into $M$ distinguishible bins? The answer, using combinatorics, is:

$ W(N, M) = \frac{(N + M - 1)!}{N!(M-1)!} $

You can double-check on paper that in the case of the above toy model, $W(2, 4) = 10$.

In a physical system, assuming all microscopic states are equally likely, entropy scales like $ S \approx log[W] $. With this, we can revisit the above situation of putting two containers of ideal gas next to each other. The question is:

$ S(W(2N, 2M)) \stackrel{?}{=} 2 S(W(N, M)) $

Stirling's approximation

The last tool we need is Stirling's approximation. For large $N$:

$ ln[N!] \approx N ln[N] - N $

Using basic properties of the logarithm ($log[ab] = log[a] + log[b]$ and $log[\frac{a}{b}] = log[a] - log[b]$):

$ S(W(2N, 2M)) = ln[\frac{(2N + 2M - 1)!}{(2N)!(2M-1)!}] = ln[(2N + 2M - 1)!] - ln[2N!] - ln[(2M-1)!] $.

Then, using Stirling's approximation:

$ S(W(2N, 2M)) = (2N + 2M - 1) ln[2N + 2M - 1] - (2N + 2M - 1) - 2N ln[2N] + 2N - (2M-1) ln[2M-1] + (2M-1) $

The non-log parts cancel out, so:

$ S(W(2N, 2M)) = (2N + 2M - 1) ln[2N + 2M - 1] - 2N ln[2N] - (2M-1) ln[2M-1] $.

For large N, we can approximate $ (N-1) ln[N-1] \approx N ln[N] $, so:

$ S(W(2N, 2M)) \approx (2N + 2M) ln[2(N + M)] - 2N ln[2N] - 2M ln[2M] $

Now, we can break the logs out, like $ln[2N] = ln[2] + ln[N]$. The terms multiplied with $ln[2]$ just cancel out, so:

$ S(W(2N, 2M)) \approx 2(N + M) ln[N + M] - 2N ln[N] - 2M ln[M] = 2 ((N + M) ln[N + M] - N ln[N] - M ln[M]) \approx 2 S(W(N, M))$.

So, for large N, the coarse-grained entropies are additive (approximately)! It we consider something like a liter of gas, it will have $ N \approx 10^{22} $, so these approximations hold to very high precision.

Numerical checks

The code shown below is up on Github. First, let's check Stirling's approximation for large N: $ ln[N!] \approx N ln[N] - N $

# numerical verification of Stirling's approximation for large N

def log_factorial(N):
    # compute log(N!) as a sum, so we can compute if for large N
    # using the identity log(a*b) = log(a) + log(b)
    # this is not an approximation, aside from floating point issues
    sum = 0
    for i in range(1, N+1):
        sum += log(i)
    return sum

def stirling(N):
    return N*log(N)-N

xs, ys = [], []
for N in range(10, 1000, 10):
    xs.append(N)
    ys.append(stirling(N)/log_factorial(N))
plot(xs, ys, title='Ratio of log[N!] to its Stirling-approximation', xlabel='N')

Next, let's verify that $ (N-1) ln[N-1] \approx N ln[N] $:

# numerical verification that (N-1)*log(N-1) ~ N*log(N) for large N

def f(N):
    return N*log(N)

xs, ys = [], []
for N in range(10, 1000, 10):
    xs.append(N)
    ys.append(f(N-1)/f(N))
plot(xs, ys, title='The ratio f(N-1)/f(N) for f(N)=N*log(N)', xlabel='N')

Next, let's verify that the exact entropy S and its Stirling-approximated are the same for large N:

# numerical verification that the exact entropy S and its
# Stirling-approximated are the same for large N

def W(N, M):
    return factorial(N+M-1) / (factorial(N)*factorial(M-1))

def S(N, M):
    return log(W(N,M))

def S_stirling(N, M):
    return (N+M-1)*log(N+M-1) - (M-1)*log(M-1) - N*log(N)

xs, ys = [], []
for N in range(10, 500, 10):
    xs.append(N)
    ys.append(S(N, N)/S_stirling(N, N))
plot(xs, ys, title='The ratio the exact entropy and its Stirling-approximation', xlabel='N')

Finally, let's verify our final result that the entropy $S$ is extensive:

# numerical verification that entropy is extensive

def S_stirling(N, M):
    return (N+M-1)*log(N+M-1) - (M-1)*log(M-1) - N*log(N)

xs, ys = [], []
for N in range(3, 500, 10):
    xs.append(N)
    ys.append(2*S_stirling(N, N) / S_stirling(2*N, 2*N))
plot(xs, ys, title='The ratio 2*S(N, N) / S(2*N, 2*N) showing that entropy is extensive', xlabel='N')

Next steps

The above formula for entropy cannot be physical, since it depends on how to decide to coarse grain the volume of the gas ($M$). Also, it doesn't take into account the velocity of the particles. The next step, coming in the next article, is to take make physical arguments and compute an expression for $W$ that takes into account the velocities, parametrize using the kinetic energy of the system and derive the Sackur–Tetrode equation. Once we have that, we can also derive the ideal gas equation of state $pV = NkT$.

WeToddle retrospective

2021-10-29T00:00:00+02:00

The idea

The idea behind WeToddle came from the Baby Fanclub group we have on Messenger, which has most of our family in it. It’s essentially a mini-social network centered around The Baby. I noticed that some of our older parents effectively use it as The Social Network, and start posting content unrelated to the baby in there. Many friends confirmed they have something similar going on, with similar behaviour.

I thought, shouldn’t this be its own app?

I didn’t do any research, I just wanted to write some code and see what happens.

Code

I started writing code for a generic groups app (authentication, posts, pictures, comments, likes). Writing the server-side code was easy and fun. Writing the client side code was not fun at all, but I got a simple Android app working. One of the things I learned is that most of the pain is in the client side development. And I was not even doing iOS, just Android. Towards the end I found Flutter, which potentially takes some of this pain away, but I didn't try it.

The first commit is from May 21, the last one from August 11. I started working on it a bit earlier, probably May 11. So I spent 3 calendar months on this. There are 23 commits, I spent around 100 work hours on this (\$10k billable assuming \$100/hr), mostly during vacation days and weekends. In terms of actual cost, there was none, I didn’t spend any extra money on this.

Cocoon

Then I accidently saw this article: Substack acqui-hires team behind subscription social app Cocoon. It turns out some ex-Facebook people had a similar idea in 2019, raised $3M, spent a 2 years on it, and then gave up because it didn’t go anywhere (presumably).

Reading Coccon’s copy and looking at their app, they had the same exact motivation and built the exact same app I had in mind. So I immediately knew this means I shouldn’t invest any more time into this.

Following the links on Techcrunch, this is the announcement from 2019: Cocoon’s social app for close friends gets VC backing to chase Path’s dream. Per Techcrunch:

You may have heard the pitch before, Facebook, Twitter and Instagram aren’t homes for your real friends anymore because they’re too big, too commercial and too influencer-y, the result is that your most important relationships have been relegated to the lowest common denominator tool on your phones: your texting app. Cocoon, a startup from a couple of ex-Facebook employees that went through YC earlier this year, is hoping to create the dedicated software that you use for that most important group chat in your life. The iOS-only app is a bit of a cross between Life360, Slack and Path.

Later they also released an Android app. They even supported voice calls! Here are some screenshots from their Google Play page:

Cocoon was a YC startup. From their fact sheet:

Co-founders:
- Alex Cornell
- Sachin Monga
When: 2019 winter batch
Location: San Francisco
Employees: 6
Tagline: Cocoon actively brings your most important groups closer together. It provides a private space to gather, the tools you need to keep each other close, and helps you keep writing your ongoing story as a unit.

Conclusion

I didn’t do any research, I just wanted to write some code and see what happens. However, once I figured out the idea doesn’t work, I stopped writing code. So even for a toy project I need to believe it could end up as something successful, otherwise I don’t continue writing code, even as a self-education project.

I didn’t do enough research on existing startups when I started to work on this. If I would have found Cocoon in May, even though they have not given up at that point, I would probably not have started to work on it at all, since:

Somebody with a very similar background (ex-Facebook), $3m funding and a 2 year headstart is already working on this idea. Great, no problem! I would have run with another idea.
I would have seen that Cocoon is not showing signs of success.

Still, this was a success. I only spent 3 months / 50-100 hours on this, and I learned that this idea doesn’t work (at least today, with the set of ideas I had, which were a 100% match with Cocoon). Meanwhile, I learned a lot about client-side app development.

My #1 learning is that I need to spend more time researching existing solutions, even if I’m just doing it for fun (as a coding exercise). I should probably “learn” how to search on Techcrunch and Crunchbase more effectively. The other, less valuable learning is technical: I spent a lot of time blocking on app development. I can move very fast on the server-side, but I repeatedly hit brick walls on the client side, I could not move fast. I need to find a way so I can quickly get crappy versions of app out, in a few weeks.

I don’t exactly know why the Cocoon gave up, but I can take an educated guess: like me, they thought that making the app free would result in the same business model (and eventually, product) as Facebook, but then what’s the point? So they tried to charge users $5/month. I assume the problem is, Facebook/Messenger/Whatsapp are really polished products, everybody is already there, so the value proposition of a few, relatively minor differentiating features are just not enough to make people use another app and even pay for it.

Entropy in Data Science

2021-10-24T00:00:00+02:00

Introduction

I first wrote about entropy in What's the entropy of a fair coin toss? followed by Cross entropy, joint entropy, conditional entropy and relative entropy, which explained these different entropy-related concepts. Now I will discuss 4 uses of entropy in Data Science:

Cross entropy as a loss function for training neural network classifiers
Entropy as a splitting criterion for building decision trees
Entropy for evaluating clustering algorithms
Entropy for understanding relationships in tabular data

The ipython notebook is here.

Cross entropy as a loss function

When building a supervised neural network classifier, cross entropy or relative entropy (usually called Kullback–Leibler divergence in Deep Learning frameworks) are common loss functions. For example, suppose we're building an MNIST digit classifier, so there are 10 output classes, corresponding to digits 0..9. Since there are 10 classes, the neural network has 10 outputs. When the network is being trained, for each input image we know the actual digits it's showing, which we encode in a vector like $v_{actual}=[0,0,..,1]$. Since the last number is 1, this is a digit 9. Then we run the input image through the neural network, and read out the outputs for the 10 classes, which may look like $v_{predicted}=[0.1, 0, 0, ... 0, 0.1, 0.8]$. To make sure the output probabilities some to 1, we always put a softmax layer as the last layer of the network. So in this case, the cross entropy loss is $H(p=actual, q=predicted) = -\sum_i p_i log [ q_i ] = - log [ 0.8 ] $. In this expression the only $p_i$ that's non-zero is the one for the actual class, and for that $p_i = 1$, so cross entropy loss really only contains that one term. Note that during actual training, cross entropy loss is computed (summed) for an entire minibatch, ie. multiple training points.

It would be tempting to say that cross entropy loss doesn't care what probabilities the model assigns to the incorrect classes, but this is misleading. As mentioned before, to get the probabilities of the output layer to sum to 1, the last layer is always a softmax, and each of the output probabilities depends on all the outputs of the previous layer. In the MNIST case, we have 10 outputs $z_i$, which are then softmax'd like $q_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$, the divident contains all $z_i$s, so $q_i = q_i(z_0 ... z_9)$.

What's the difference between cross entropy and relative entropy as a loss function? In neural networks, the training method is usually some form of gradient descent, which means that the loss function is differentiated by the weights of the neural network. Per the previous article, relative entropy and cross entropy are related like $ D_{KL}(actual | predicted) = H(actual, predicted) - H(actual) $. Here, $H(actual)$ is a constant with respect to the network weights [but it changes for each mini-batch], so it's derivative wrt to the network weights is 0. Because of this, although the two loss values are different, their derivatives wrt network weights are identical ( $ \frac{\partial D_{KL}(actual | predicted)}{\partial w_i} = \frac{\partial H(actual, predicted)}{\partial w_i} $ because $ \frac{\partial H(actual)}{\partial w_i} = 0$ ), so their usage is equivalent, since the computed gradients propagated back into the network will be exactly the same!

See the Bytepawn article Solving MNIST with Pytorch and SKL for an example Pytorch deep neural network classifier using cross entropy.

Entropy in decision tree classifiers

When building a decision tree, the algorithm starts with just one root nodes which contains all samples. Then, at each step, leaf nodes are split into 2 child nodes until a stopping condition is met. Given a set $S$ of samples in the current node to be split, what is the criterion to divide it into two children $S_L$ and $S_R$? If we look at Scikit Learn's DecisionTreeClassifier, the first parameter is called criterion and can either be gini or entropy. Let's look at entropy: in this case, the algorithm will try to split to maximize Information Gain, which is defined as $I = H_{parent} - H_{children}$. Here $H_{parent}$ is the entropy of all samples in the parent node, where probability is just the frequency. So if the parent node contains 4 samples for class A and 6 samples for class B, then $H_{parent} = H([4,6]) = -\frac{4}{10}log\frac{4}{10} - \frac{6}{10}log\frac{6}{10} = 0.97$ bits. Suppose the algorithm is evaluating a split where the left side contains $[A,B]=[3, 1]$ samples and the right side contains $[A,B]=[1, 5]$ samples. So $H_{left} = H([3, 1]) = -\frac{3}{4}log\frac{3}{4} - \frac{1}{4}log\frac{1}{4} = 0.81$ bits. For the right side, $H_{right} = H([1, 5]) = -\frac{1}{6}log\frac{1}{6} - \frac{5}{6}log\frac{5}{6} = 0.65$ bits. Then, $H_{children}$ is sum of these two entropies, weighted by sample size: $H_{children} = \frac{4}{10} H_{left} + \frac{6}{10} H_{right} = 0.71$ bits. So the information gain is $I = H_{parent} - H_{children} = 0.26$ bits. The algorithm looks at multiple possible splits, and selects the one with the highest information gain. Note that if it finds a way to put all As into the left node and all Bs into right node (or vica versa), then the child entropies will be 0, this is the maximum achievable $I_{max}=H_{parent}$, the best possible outcome of the split. Note that Gini impurity is also related to the concept of entropy.

Entropy in clustering

Suppose we want to evaluate some unsupervised clustering algorithms, or the same algorithm with different feature vectors. We have some labeled data, so we can compare the cluster classification to the real classes. So we have for each sample point $i$ a feature vector $x_i$, actual class $A_i$ and the clustering algorithm's cluster identity ie. the modeled class $M_i$. Note that the $M_i$s will not be labeled, since a clustering algorithm just says that a subset of points belong to a cluster, but it doesn't know what that cluster is, there is no name given to it. The $A_i$s are usually labeled, like car, boat, airplane. Because the $A_i$s and $M_i$s are different, cross entropy and relative entropy don't make sense here, since there it's assumed that the two random variables take on the same values.

One approach is to take each cluster (so set of points where $M_i$ is the same), and calculate the entropy using the $A_i$s. We want this to be low, since that corresponds to a cluster with lots of points with the same actual class. Do this for each cluster, and then take the weighted sum of per-cluster entropies. What I described is just the conditional entropy $H(A|M)$! The downside of using conditional entropy is that it can be gamed by creating a lot of little clusters; in the worst case, each point is its own cluster, and all per-cluster entropies are 0. This is best controlled by limiting the number of clusters the clustering algorithm is allowed to create to the number of actual classes.

A better alternative is to use mutual information $I(A, C)$, which will be zero if $A$ and $M$ are completely independent (worst case), and maximal if $A$ and $M$ completely detemine each other, ie. if there are an exactly equal number of classes, and knowing a modeled class $M_i$ always corresponds to the same $A_i$, and vica versa. For more, see this article.

The implementation of the above clustering metrics are straightforward in Python. To demonstrate, let's use the standard Iris flower dataset, which has 4 float numbers describing the length and the width of the sepals and petals in centimeters, and a label for the type of iris (setosa, versicolor, virginica). Let's use k-means clustering from Scikit learn and have it return 3 clusters. Then, let's compute both the conditional entropy and information gain between the modeled and actual classes. Let's see what happens if we run the clustering with just 1, 2, 3 or all 4 of the available feature vector elements:

from math import log
from sklearn import datasets
from sklearn.cluster import KMeans
from collections import defaultdict, Counter

def iris_clusters(use_columns=None):
    iris = datasets.load_iris()
    model = KMeans(n_clusters=len(iris.target_names))
    if use_columns is None:
        data = iris.data
    else:
        data = iris.data[:, use_columns]
    model.fit(data)
    return iris.target, model.predict(data)

def entropy(frequencies, base=2):
    return -1 * sum([f/sum(frequencies) * log(f/sum(frequencies), base) for f in frequencies])

def conditional_entropy(y, x, base=2): # computes H(Y|X)
    clusters = defaultdict(lambda: [])
    for yy, xx in zip(y, x):
        clusters[xx].append(yy)
    return sum([len(cluster)/len(x) * entropy(Counter(cluster).values(), base) for cluster in clusters.values()])

def information_gain(p, q, base=2):
    return     entropy(Counter(p).values(), base) - conditional_entropy(p, q, base)
    # same as  entropy(Counter(q).values()) - conditional_entropy(q, p)

for use_columns in [[0], [0, 1], [0, 1, 2], [0, 1, 2, 3]]:
    actuals, modeled = iris_clusters(use_columns)
    print('Using %d columns in feature vector, conditional entropy = %.2f bits, information gain = %.2f bits' % 
          (len(use_columns), conditional_entropy(actuals, modeled), information_gain(actuals, modeled)))

Prints:

Using 1 columns in feature vector, conditional entropy = 0.99 bits, information gain = 0.60 bits
Using 2 columns in feature vector, conditional entropy = 0.56 bits, information gain = 1.02 bits
Using 3 columns in feature vector, conditional entropy = 0.44 bits, information gain = 1.14 bits
Using 4 columns in feature vector, conditional entropy = 0.39 bits, information gain = 1.19 bits

The result is as expected, allowing the clustering algorithm to see more of the feature vector yields lower relative entropy (better) and higher informationg gain (better).

Entropy for understanding columnar data

We can use entropy to see how much information is contained in a column. For example, suppose we have a table users, which has a column country. We know that most of our users are located in our home country, but there may be others. We can compute the per-column entropy with a bit of SQL:

WITH
counts AS
(
    SELECT
        country,
        COUNT(*) AS cnt
    FROM
        users
    WHERE
        country IS NOT NULL
    GROUP BY GROUPING SETS
        ((), (country))
    -- there will be one row with country = NULL, corresponding to the overall counts
    -- the rest of the rows are the country-wise counts
),
probs AS
(
    SELECT
        g.country,
        CAST(g.cnt AS FLOAT) / a.cnt AS p
    FROM
        (SELECT * FROM counts WHERE country IS NOT NULL) g
    CROSS JOIN
        (SELECT * FROM counts WHERE country IS NULL) a
)
SELECT
    -1 * SUM(p * log(2, p)) AS entropy_bits
FROM
    probs

This snippet can be extended to handle more columns, the simplest method is to concatenate columns like '(' || col1 || ',' || col2 || ... ')'. Note that if entropy_bits is 0, it means all values in the column are the same. Interestingly, the entropy (in bits) we compute is also the number of bits a database engine would need to efficiently encode these values in a row-wise manner.

We can also check if another column contains additional information. Suppose there is another column in users called locale, which is the language locale of their browser, like en_us or fr_fr. For this we can use conditional entropy H(locale|country):

WITH
counts AS
(
    SELECT
        country,
        locale,
        COUNT(*) AS cnt
    FROM
        users
    WHERE
        country IS NOT NULL AND locale IS NOT NULL
    GROUP BY GROUPING SETS
        ((), (country), (country, locale))
),
marginal_probs AS
(
    SELECT
        g.country,
        CAST(g.cnt AS FLOAT) / a.cnt AS p
    FROM
        (SELECT * FROM counts WHERE country IS NOT NULL AND locale IS NULL) g
    CROSS JOIN
        (SELECT * FROM counts WHERE country IS NULL AND locale IS NULL) a
),
joint_probs AS
(
    SELECT
        g.country,
        g.locale,
        CAST(g.cnt AS FLOAT) / a.cnt AS p
    FROM
        (SELECT * FROM counts WHERE country IS NOT NULL AND locale IS NOT NULL) g
    CROSS JOIN
        (SELECT * FROM counts WHERE country IS NULL AND locale IS NULL) a
)
SELECT
    -1 * SUM(xy.p * log(2, xy.p/x.p) AS entropy_bits
FROM
    marginal_probs AS x
INNER JOIN
    joint_probs AS xy
USING
    (country)

For example, if country -> locale is a deterministic mapping, ie. USA always maps to en_us, then conditional entropy will be 0. As in earlier examples, information gain also makes sense for tabular data.

Conclusion

The above examples show that entropy is a common and useful concept in Data Science. I plan to write a follow-up article about interesting examples of entropy in Physics.

100 articles

2021-10-18T00:00:00+02:00

Introduction

The previous article, Cross entropy, joint entropy, conditional entropy and relative entropy was the 💯th post on Bytepawn! I took a few hours and reviewed my writing so far.

First article

I wrote the first article, Cargo Cult Data, in 2015 January because I saw that many companies are not getting the most out of their data and data teams due to organizational and cultural discipline issues. At that time, the issues I saw were flaky logs, no standardized metrics, or constant and pointless arguments around data (eg. correlation vs causation, trustwortyness). The topic of cargo culting is still relevant, but today I would write a different article. Since then I've worked at a lot more companies, and what I called "cargo culting" is the way things are at most companies — the only exception I've seen is Facebook (the company). Since then Data Science and AI have exploded and moved the topic of conversation away from traditional analytics topics like logging and A/B testing. DS/ML/AI have their own set of problems which are not best described by the "cargo cult" analogy. The final reason I would write a different article today, being almost 7 years wiser, is that today I believe it's not constructive to be confrontative on such issues. It's better to educate people, leading by showing, which is why later I wrote so many posts on data and A/B testing related topics here.

Airflow

When I left my job at Prezi I took a month off before joining Facebook (the company), and I spent some time reviewing open source ETL tools. At Prezi we spent a lot of time building hand-rolled ETL tools, one in bash, a second one in Haskell and a third one in golang. By the end of my tenure I realized this is crazy, there must be (or should be) a good enough open source solution. I reviewed Airflow, Luigi and Pinball and finally wrote a 3-way comparison piece. To this day, this gets a lot of daily long-tail hits — people are stil curious about what ETL tool to use. My takeaway then was that Airflow is the way forward. Since then Airflow has become the de-facto open source ETL tool. At Facebook I didn't use Airflow, we had something called Dataswarm, but I learned that Airflow is actually based on Dataswarm (written by an ex-Facebook engineer who left the company). So after leaving Facebook, I've set up Airflow and made it the standard ETL tool at all subsequent jobs, also teaching team members how to use it (building an Airflow job is always part of onboarding). Over the years I've also written a few more articles on how we use Airflow in production. The biggest "trick" — which I hope every team figures out — is that we always write helper functions that internally construct DAGs, so my less technical Data Scientist teammates can just ignore the technicalities and use a template for building DAGs. We currently use Airflow for both SQL transform jobs as well as ML model building (for which it is less ideal), and I plan to continue to use Airflow in the future.

Fetchr

My first job in Dubai, after I left Facebook and London, was at a last mile delivery startup called Fetchr. One of the upsides about working at Fetchr was that, unlike at Facebook, I could write about work related topics freely, nobody cared. I wrote about 10 articles about interesting Data Science work we did at Fetchr. I'm especially fond of the article Automating a Call Center with Machine Learning, this service is probably the most impactful piece of software I've ever written: it ended up saving about $3-5m/yr for Fetchr, which was about 3-5% of expenses at that time. It's been 2 years since I left, unfortunately Fethcr is not doing too well.

Pytorch

The following are all true for the Deep (and Reinforcement) Learning revolution: (i) the productization opportunities are genuinely exciting (ii) everybody's talking about it (iii) it's very interesting technically (iv) yet, very few companies are doing it in production. Big tech does use DL in production, but in my experience almost all other companies don't, because 80-90% of tabular business problems can be solved by SQL, scripting and dashboarding. For the remaining 10-20% there is some benefit in using Scikit-Learn models (tree-based models, gradient boosting, etc) or building FBProphet forecasts (or similar libraries like XGBoost or LGBM). Whenever I actually use a neural network at work, it's usually when I'm building a classifier, and I'm playing around with Scikit-Learn's LogisticRegression, DecisionTreeClassifier, RandomForestClassifier, GradientBoostingClassifier, etc. models and I also try and see how MLPClassifier performs with 2-3 hidden layers. Usually I decide against it, since performance is either not superior or within 1-2% of the second best, but training is slow and convergence is not always deterministic.

I still had to learn Deep Learning! It's so interesting, exciting, and you first have to know what it is before you can make a good call whether to (not) use it for a given problem. So I used this blog to drive my learning, writing an article on each mini-project I built. I cast my vote for Pytorch (vs Tensorflow), because of it's native integration with Python which makes learning, experimentation and debugging easy (vs the Tensorflow runtime model). Some of the articles took a lot of time (also runtime), up to 20-40 hours sometimes. All in all I wrote 18 posts tagged pytorch. The approach worked, I know the basics (and a bit more) about Deep (and Reinforcement) Learning. My favorites articles are the ones where I go beyond getting something to work, such as seeing how many pixels I have to change on MNIST digits to break a deep neural network classifier or the one where I play around with the information storage capacity of autoencoders.

Craftmanship

I love thinking and writing about craftmanship. Craftmanship are the little things such as indenting SQL code properly, formatting numbers and charts so they're readable, or writing readable regular expressions. Articles from this theme:

A/B testing

The most frequent topic I wrote about is A/B testing, there are 20 articles tagged A/B testing. In my experience writing clean SQL code and the concepts of A/B testing are the two most important skills in Data Science and Analytics — but very few people are clear on them. 4 out of 5 people I interview cannot explain to me what a p-value is. This is why I wrote articles such as Building intuition for p-values and statistical significance. I had a lot of fun with running Monte Carlo simulations of A/B testing on random graphs (A/B testing on social networks and A/B testing and networks effects.

A/B testing was also my greatest blunder on Bytepawn. In this article I incorrectly claimed that it's a fallacy to look at historic (before the A/B test) data. This post made it to the Hacker News front page (most don't), and people called me out on this incorrect claim and pointed me to something called CUPED. This was painful, because in all my A/B testing articles I run Monte Carlo simulations to check my thinking, and here I got lazy and did not. This was a good reminder for me that although A/B testing seems like an easy subject, it's easy to mislead yourself, even for somebody who spends a lot of time thinking about it. It's always better to check your thinking with simple Monte Carlo simulations. But, this way I learned about CUPED, and wrote 5 articles on it — with Monte Carlo simulations, of course!

Pelican

Pelican is the static site generator I use for Bytepawn. I wrote an article about it, what's changed since then is now I host it myself (with nginx and Let's Encrypt) to avoid the ~10 second delays when changing content with Github Pages. The combination of writing articles in Markdown, not worrying too much about formatting and Pelican has worked well in the past 5 years, I don't plan to change it. I don't want to switch to something like Medium, because this way I have and feel more ownership for the blog.

Conclusion

Writing the first 100 articles has been great fun, I hope and plan to write another 100! I write this blog for fun, it's not monetized, I don't regularly push the links to broadcast-social-media like Hackers News, only to my local-social-media (my Facebook and Linkedin). It used to get around 50-100 readers a day, this year it went up to 100-150 per day, probably because I write regularly: one of my yearly goals is to write 40 blog posts in 2021 (I'm on track_. Much more is not doable next to work and family, but I'll try to push myself to write 50 in 2022, and it'd be nice to have ~250 readers a day. A big question to myself is whether I should start writing articles about Physics — time will tell!

Onward to the next 💯!

Cross entropy, joint entropy, conditional entropy and relative entropy

2021-10-09T00:00:00+02:00

Introduction

In the previous article I discussed entropy. As a follow-up, I revisit more advanced variations of entropy and related concepts:

Joint entropy
Conditional entropy
Cross entropy
Relative entropy (also known as Kullback–Leibler divergence)
Mutual information (also known as Information gain)

Everything I cover here is elementary information theory, mostly found in the first chapter of the classic Cover, Thomas: Elements of Information Theory or the wikipedia pages linked above.

Entropy

Entropy is the amount of uncertainty of a random variable, expressed in bits. Imagine Alice has a random variable and she needs to communicate the outcome over a digital binary channel to Bob. What is a good encoding to minimizes the average amount of bits she sends? In the previous article I discussed the case of a fair coin, so let's make it a bit more complicated here, and use a 4-sided dice, ie. a tetrahedron. Let's say it's a fair tetrahedron, so each side comes up with $ p = \frac{1}{4} $ probability.

There are 4 outcomes, so she needs to use 2 bits:

1 -> 00
2 -> 01
3 -> 10
4 -> 11

Since all outcomes are equally likely, there is no reason for her to deviate from this trivial encoding. So, on average she will use 2 bits per message.

What if one of the sides is more likely to come up? Suppose the distribution is $ p = [\frac{1}{2}, \frac{1}{6}, \frac{1}{6}, \frac{1}{6}] $. In this case, it makes sense to use less bits for the first side, since it will come up more, so she sends less bits on average. Eg. she can try:

1 -> 0
2 -> 10
3 -> 110
4 -> 111

With this encoding, on average she's sending $[\frac{1}{2}, \frac{1}{6}, \frac{1}{6}, \frac{1}{6}] \times [1, 2, 3, 3] = \frac{11}{6} = 1.83$ bits per message.

The formula for entropy is $H(X) = - \sum_i p(x_i) log[p(x_i)]$. For the fair tetrahedron, $H(X)=2$, which means that the trivial encoding is optimal. For the biased tetrahedron above, $H(X)=1.79$ bits, which is less than the encoding above. This means that there are better encodings, we can still save about 0.04 bits per message. To do this, we need to do block encoding, encode several messages, like:

11 -> 0
12 -> ...
14 -> ...
14 -> ...
21 -> ...
...

By creating larger and larger block encodings, we can arbitrarily approach the theoretical limit of $H(X)$, but we can never do better than that.

The formula $H(X) = - \sum_i p(x_i) log[p(x_i)]$ for entropy can be made intuitive: the number of bits we should use to encode an outcome with probability $p(x_i)$ is of length $ - log[p(x_i)] $ bits. Eg. for $ p(x_i) = \frac{1}{2} $ this is 1 bit, for $ p(x_i)= \frac{1}{4} $ this is 2 bits. Considering the examples above, this is intuitive. And the $p(x_i)$ multiplier is just how often this outcome will occur. So it makes sense that this is the average number of bits sent per message.

Also note the entropy of a random variable that has just one outcomes is 0, we don't need any bits if we already know what the outcome is going to be.

Joint entropy

Just as there is probability $p(x)$, $p(y)$ and joint probability $p(x, y)$, we can define entropy $H(X)$ and joint entropy $H(X, Y)$.

$ H(X, Y) = - \sum_{ij} p(x_i, y_j)log[p(x_i, y_j)]$

If $X$ and $Y$ are independent, then $H(X, Y) = H(X) + H(Y)$. This is easy to see, because then $p(x, y) = p(x) p(y)$, so:

$ H(X, Y) = - \sum_{ij} p(x_i, y_j)log[p(x_i, y_j)] = - \sum_{ij} p(x_i)p(y_j)log[p(x_i)p(y_j)] $

Now we use $log(ab)=log(a)+log(b)$:

$ H(X, Y) = - \sum_{ij} p(x_i, y_j)log[p(x_i, y_j)] = - \sum_{ij} p(x_i)p(y_j)[log[p(x_i)] + log[p(y_j)]] $

$ H(X, Y) = - \sum_j p(y_j) \sum_i p(x_i)log[p(x_i)] - \sum_i p(x_i) \sum_y p(y_j)log[p(y_j)] $

$ H(X, Y) = - \sum_j p(y_j) H(X) - \sum_i p(x_i) H(Y) = H(X) \sum_j p(y_j) + H(Y) \sum_i p(x_i) = H(X) + H(Y) $.

If $X$ and $Y$ are not indepedent, then:

$ H(X, Y) < H(X) + H(Y) $.

A simple example of the above is the 4 sided dice. Suppose $X$ is the random variable for each side coming up (4 outcomes each with probability $\frac{1}{4}$, $Y$ is the random variable for even or odd numbered sides coming up (2 outcomes, like a toin coss), referring to the same toss. $X$ and $Y$ are not independent, eg. $P(X=1,Y=even)=0$ but $P(X=1)=\frac{1}{4}$ and $P(Y=even)=\frac{1}{2}$. Obviously, $H(X)=2 $ bits and $H(Y)=1 $ bits, but $H(X, Y)=H(X)=2 $ bits, because knowing $X$ already tells us everything about $Y$.

Conditional entropy

Suppose we have two random variables $X$ and $Y$. Suppose that we know the outcome of $X$, and the question is, how much entropy is "left" in $Y$? Another mental model: suppose we have Alice and Bob communicating over a digital channel. Alice needs to encode the outcome of $Y$ in bits and send it over to Bob (many times), but there is a second random variable $X$, whose outcome both Alice and Bob have access to. For example, Alice has a thermometer ($Y$) and is encoding and sending the reading to Bob who is a mile away, but both Alice and Bob experience the same weather ($X$). Can Alice use less bits to encode $Y$?

We define conditional entropy like:

$ H(Y | X) = \sum_i p(x_i) H(Y | X = x_i) $

We go through all the values $X$ can take, calculate the entropy of $H(Y | X = x_i)$ of $Y$, and we average this over the outcomes of $X$. Note that this is similar to the formula for conditional expectation. $H(Y | X = x_i)$ is just the entropy over the conditional probabilities:

$ H(Y | X = x_i) = - \sum_j p(y_j | x_i) log[p(y_j | x_i)] $.

Since $ p(y_j | x_i) p(x_i) = p(y_j, x_i) $ conditional entropy can also be written as:

$ H(Y | X) = - \sum_{ij} p(y_j, x_i) log[p(y_j | x_i)] = - \sum_{ij} p(y_j, x_i) log\frac{p(y_j, x_i)}{p(x_i)} $.

Let's look at conditional entropy in the above example of the tetrahedron. $H(Y | X)$ is easy, if we know $X$, the outcome of the toss, then we don't need any additional bits to communicate whether it's even or odd, so $H(Y | X) = 0$ bits. $H(X | Y)$ is also easy, if we know whether the side landed on an even or odd number, in each case we have 2 options left (if it landed odd, it can be 1 or 3, if it landed even, it can be 2 or 4), all have the same $p = \frac{1}{2}$, so it's like a coin toss, so $H(X | Y) = 1$ bits.

If $X$ and $Y$ are independent, $ H(Y | X) = H(Y) $, since knowing $X$ doesn't tell us anything about $Y$.

Finally, for all random variables: $ H(Y | X) = H(X, Y) - H(X)$ (chain rule). We can check this again with the tetrahedron example: $ H(Y | X) = 0 = H(X, Y) - H(X) = H(X) - H(X) $ and $ H(X | Y) = H(X, Y) - H(Y) = 1 = 2 - 1 = 1$.

Cross entropy

Suppose Alice and Bob are communicating again, but this time there is no out-of-bounds communication. Alice wants to encode the outcomes of a random variable $X$ and send it to Bob, with the least number of average bits per outcome. However, let's assume Alice has incomplete or incorrect information about $X$: she mistakenly thinks that the distribution is per another random variable $Y$, and she constructs her encoding per $Y$. How many bits will Alice use on average?

Let's look at the tetrahedron again, but this time it's uneven, let the probabilities be $X ~ [\frac{4}{8}, \frac{2}{8}, \frac{1}{8}, \frac{1}{8}]$. It would make sense for Alice to encode it with a prefix code like [0, 10, 110, 111], to save bits on the most likely outcomes.

However, if she mistakenly thinks that the probabilities are $Y ~ [\frac{1}{8}, \frac{1}{8}, \frac{2}{8}, \frac{4}{8}]$, she will choose the encoding [110, 111, 10, 0]. She will send 3 bits in the most common case and 1 bit in the least common case.

Let $p_i$ be the probabilities for $X$, $q_i$ be the probabilities for $Y$, then the cross entropy is:

$ H(p, q) = - \sum_i p_i log[q_i] $.

Note that this is not symmetric, ie. in general $ H(p, q) = - \sum_i p_i log[q_i] \neq H(q, p) = - \sum_i q_i log[p_i] $. Obviously, if $p=q$, then $H(p, q) = H(p) = H(q)$, ie. if Alice has the right distribution, she can construct an encoding that is maximally optimal and approaches the entropy.

The definition is very intuitive: as mentioned for regular entropy, if an outcome has probability $p_i$, it's best to encode it with a message of length $-log[p_i]$ bits. For example, if $p_i=\frac{1}{2}$, then $-log[p_i]=1$ bit, and in that case we should encode that as a 0 (or 1). On average, this outcome occurs $p_i$ times, so this outcome contributes $-p_i log[p_i]$ bits on average. However, if Alice believes the probabilities are $q_i$, she will pick an encoding where this outcomes in encoded as $-log[q_i]$ bits, but in reality this occurs $p_i$ times, so it contributes $-p_i log[q_i]$ bits on average.

Relative entropy (Kullback–Leibler divergence)

In the above example, Alice thought that events arrive per $Y \sim q$, but in reality the distribution is $X \sim p$. She uses $ H(p, q) $ bits on average to communicate her outcomes, instead of using the $ H(p) $ bits she'd use if she knew the correct distribution. The difference between the two is called relative entropy, also known as Kullback–Leibler divergence:

$ D_{KL}(p | q) = H(p, q) - H(p) $.

Writing out the two definitions for $H(p, q)$ and $H(p)$, we get:

$ D_{KL}(p | q) = H(p, q) - H(p) = - \sum_i p_i log \frac{q_i}{p_i} $.

So, relative entropy $ D_{KL}(p | q) $ is the extra bits that Alice wastes. Like cross entropy, relative entropy is also not symmetric.

Mutual information (Information gain)

Mutual information is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" obtained about one random variable by observing the other random variable. For two random variables $X$ and $Y$, the mutual information is the relative entropy between their joint distribution and the product of their individual distributions:

$ I(X, Y) = - \sum p(x, y) log \frac{p(x)p(y)}{p(x, y)} $

If you compare this to the relative entropy formula above, it's the same with $p = p(x, y)$ and $q = p(x)p(y)$. It follows trivially from the definition that mutual information is symmetric, $I(X, Y) = I(Y, X)$. What if $X$ and $Y$ are independent? In that case, $ I(X, Y) = 0 $ because $ p(x, y) = p(x) p(y) $ and $ log[1] = 0 $. If $X$ and $Y$ completely determine another, eg. $ X = Y + 1 $, then one contains all the information about the other, so $ I(X, Y) = H(X) = H(Y) $. This is because in such a case, certain $p(x, y)$ combinations will be non-zero (eg. when $ x = y + 1 $), and for these non-zero cases $ p(x,y) = p(x) = p(y) $, so $ I(X, Y) = - \sum p(x) log[p(x)] = H(X) $.

The following image explains the relationship between entropy, conditional entropy, join entropy and mutual information.

So, for example $ H(X, Y) = H(X|Y) + I(X,Y) + H(Y|X) $.

Mutual information does not have a useful interpretation in terms of channel coding.

Conclusion

Everything I cover here is introductory information theory, mostly found in the first chapter of the classic Cover, Thomas: Elements of Information Theory or the wikipedia pages linked above. I plan to write a follow-up post to give examples of using these metrics in Data Science and Machine Learning.

What's the entropy of a fair coin toss?

2021-09-25T00:00:00+02:00

Introduction

I use a set of 20 screening questions for Data Science interviews. The questions are asked by my (non-technical) recruiter on the phone, so I wrote them so they have objective answers, like yes/no/0/1/42.

One of the questions goes like this:

Q: What is the entropy of a fair coin toss?
A: 1 bit

Then there is a follow-up:

Q: What is the entropy of a coin that is almost always heads?
A: 0 bits (or approximately 0 bits)

My recruiter reports that from a sample size of ~50 candidates, very few can answer these entropy related questions. So, why did I think these are good and relevant screening questions?

Entropy is a common and useful concept in Data Science. Our current datascience-main git repo has 7 occurences of the string entropy in our own code (not library code).
I learned about entropy in multiple classes during my Comp.Sci. degree (20 years ago).
I learned about entropy in multiple classes during my Physics degree (20 years ago).
I find entropy to be one of the most intellectually pleasing concepts, partially because it occurs across a wide array of disciplines.

So I'm writing this post to remind and excite ourselves about entropy.

What is entropy?

Per the Wikipedia page on Entropy:

Entropy is a scientific concept, as well as a measurable physical property that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in diverse fields, from classical thermodynamics, where it was first recognized, to the microscopic description of nature in statistical physics, and to the principles of information theory.

If you then jump over to https://en.wikipedia.org/wiki/Entropy_(information_theory), it even has the answers:

.. and to communicate the outcome of a coin flip (2 possible values) will require an average of at most 1 bit (exactly 1 bit for a fair coin).

and

.. a double-headed coin that never comes up tails, or a double-tailed coin that never results in a head. Then there is no uncertainty. The entropy is zero: each toss of the coin delivers no new information as the outcome of each coin toss is always certain.

I'm not sure if I knew this when I wrote the questions, but the opening Wikipedia page has the answers.

What is the entropy of a fair coin toss?

Let's look at two ways to get the answers:

(1) Use the formula for information entropy (Shannon entropy), which is $ H(X) = - \sum_i P(X=i) log_2 P(X=i) $. For a coin where the probability of H is p, the shape of the entropy function is:

For a fair coin, we have two outcomes, both have $ P(X=H)=P(X=T)=\frac{1}{2} $, so $ H(X) = - [ \frac{1}{2} log_2\frac{1}{2} + \frac{1}{2} log_2\frac{1}{2} ] = 1 $. Since we're using base 2 logarithm, the result is in bits. For the coin that is almost always H or almost always T, both terms are zero, so the entropy is 0.

(2) Use intuition: the most untuitive way to think about entropy is this. Suppose there you have to transmit messages using 0 and 1 bits. The number of message types is the number of outcomes of the random variable $X$. So, for a random coin toss, there are 2 messages, H and T. Then, there is a daemon who randomly generates these messages according to the probabilities of $X$. The question is, what is the average minimum number of bits per message we can use to transmit messages coming from $X$? The answer is just $H(X)$, the entropy. So for a fair coin toss, we can encode H as 0, T as 1, and since both occur with equal chance, we can't get more efficient than that, so we need 1 bit. And if the coin always lands on either H or T, there is no randomness, we don't need to transmit anything, the other side already knows.

Let's strengthen our intuition by first deconstructing it and reconstructing it: the above 2 cases (1 bit and 0 bit entropy) are edge-cases, let's look at the "middle." What if H occurs $\frac{3}{4}$ of the time, and T occurs $\frac{1}{4}$ of the time? Per the formula above, $H(X) = 0.81 $. How can we go below 1 bit to transfer two possible outcomes? For the case of the "almost always H" coin, the answer was zero bits, but that was kindof cheating, because we don't need to send anything at all. How do we send 0.81 bits?

The answer is, we can encode multiple outcomes: since H is more likely than T, it makes sense to use less bits to encode HH than TT or even HT. We can do the following:

HH -> P(HH) = 9/16 -> encode as 0   (length = 1 bit )
HT -> P(HT) = 3/16 -> encode as 10  (length = 2 bits)
TH -> P(TH) = 3/16 -> encode as 110 (length = 3 bits)
TT -> P(TT) = 1/16 -> encode as 111 (length = 3 bits)

This code is a prefix code, which means that if $C_i$ is a codeword (like 0 or 10 in the above example), then $C_i$ never occurs as a prefix of another codeword $C_j$ (eg. above 0 is a code, and none of the other codes start with 0, 10 is also a code, and none of the others start with 10). So on the receiver end, if this code is used, it is always possible to unambigiously decode the series of 0s and 1s and get back the series of Hs and Ts. With the above code, we're using as average of $ 1\frac{9}{16} + 2\frac{3}{16} + 3\frac{3}{16} + 3\frac{1}{16} = \frac{27}{16} = 1.68 $ bits to transfer 2 messages, so 0.84 bits per message. So on average we're below 1 bit, although this coding is still about 0.03 bits higher than the entropy itself. We can further approach the theoretical limit set by the entropy by encoding longer sequences (eg. HHH, HHHH, etc). Note that this is not Huffman-coding, Huffman-coding would just assign 0 and 1 to H and T.

Entropy in Data Science

I will mention two areas where entropy pops up in everyday Data Science work:

(1) Splitting nodes during Decision Tree building: The first parameter for Scikit-learn's DecisionTreeClassifier is:

criterion {gini, entropy}, default=gini
> The function to measure the quality of a split. Supported criteria are
> gini for the Gini impurity and entropy for the information gain.

One of the ways to build Decision Trees is to look for splits such that the 2 child nodes have on average maximally lower entropy than the parent node. So, if for example the parent node has 10 data points labeled HHHHHTTTTT and splitting along attribute A yields two nodes with labels HHTTT and HHHTT, while splitting along attribute B yields two nodes with labels HTTTT and THHHH, then the second one is better, because it has lower average entropy (and hence higher information gain). See this post on TowardsDataScience for a full example.

(2) Cross-entropy as a loss function when training Deep Neural Networks: From the Wikipedia page for Cross entropy:

In information theory, the cross-entropy between two probability distributions p and q over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution q, rather than the true distribution p... Cross-entropy can be used to define a loss function in machine learning and optimization. The true probability p is the true label, and the given distribution q is the predicted value of the current model.

Not to be confused with:

Entropy in Physics

Entropy was originally invented/discovered in Physics in the 1800s, before Shannon re-discovered it for Information Theory 100 years later. Today it pops up all over physics: Thermodynamics, Statistical mechanics, Quantum physics, even in General relativity for black holes. The most famous occurence is the Second law of thermodynamics:

The second law of thermodynamics establishes the concept of entropy as a physical property of a thermodynamic system. Entropy predicts the direction of spontaneous processes, and determines whether they are irreversible or impossible, despite obeying the requirement of conservation of energy, which is established in the first law of thermodynamics. The second law may be formulated by the observation that the entropy of isolated systems left to spontaneous evolution cannot decrease, as they always arrive at a state of thermodynamic equilibrium, where the entropy is highest. If all processes in the system are reversible, the entropy is constant. An increase in entropy accounts for the irreversibility of natural processes, often referred to in the concept of the arrow of time.

The classic explanation goes like this: suppose we have a cylinder split in half with a thin divider. The left side has gas in it, the right side has no gas, it's a vacuum. If we remove the divider, the gas will quickly fill up the whole cylinder. The configuration with the gas constrained to the left side has lower entropy, since certain possibilities (eg. positions of molecules on the right cide) are not possible; this is like the coin that is heads $\frac{3}{4}$ of the time. The final spread out gas has higher entropy, this is like the fair coin. On a microscopic level, gas molecules can be thought of as billiard balls constantly colliding with each other and the wall. Under certain conditions we can assume that the collisions are elastic, which means everything is reversible. This means that if we record a video of the gas, and then it backwards, all the microscopic colisions that we see (happening backwards) are allowed by physics to happen [even if time is going forward]. So, if somebody shows us microscopic, zoomed in cideo footage of this gas, we can't tell if the tape is playing forward or backwards. However, if we are allowed to see the whole tape zoomed out (macroscopic), and we see that the gas is going from a low-entropy state to a high-entropy state, we know the tape is playing forward, while if we see the gas going from a high-entropy state to a low-entropy state, we know that the tape is playing backwards.

A stylized but more every-day version goes like this: imagine all the air molecules (oxygen, nitrogen, etc) in the closed room you're sitting in. Why doesn't all the gas spontaneously collect in the bottom left corner of your room [which would lead you to suffocate]? The other way it works: if we somehow collect all the air in one corner, and then let it go, it will spread out, but the opposite never occurs. This is essentially what the second law is saying: the possible set of configurations of air molecules in just one corner of the room has lower entropy than if the gas is spread out all over the room. So, spreading out naturally occurs all the time, while the reverse never happens. You can actually compute this by dividing the room into grids, and placing the air molecules in grids (also velocities), counting configurations, and approximating the entropy ratio.

Note: a lovely intellectual rabbit hole to go down: what about gravity? Gravity tends to pull things together, so does gravity decrease entropy?

Conclusion

Hopefully this post is a good reminder that entropy is a useful concept, or enough to get people interested. A good short refresher written by physicist Edward Witten is A Mini-Introduction To Information Theory. If you need a textbook refresher on entropy and information theory, the standard reference is: Cover, Thomas: Elements of Information Theory. I plan to write a follow-up post about Joint entropy, Conditional entropy, Kullback–Leibler divergence and Mutual information.

Five ways to reduce variance in A/B testing

2021-09-19T00:00:00+02:00

Introduction

When performing A/B testing, we're measuring the mean of a metric (such as spend or conversion) on two distinct subsets, and then compare the means to each other. Because of the Central Limit Theorem, the measurement of the two means follows a normal distribution, as does the difference of the two (if A and B are normal, so is A - B).

Relevant Wikipedia articles:

Relevant Bytepawn articles:

Per the above, the variance of the lift (defined here as the difference) is $ s^2 = s_A^2 + s_B^2 $, where $s_A^2$ and $s_B^2$ are the variances of the A and B measurements, respectively, and $ s_i^2 = \frac{ \sigma^2 }{ N_i } $. So $ s^2 \approx \frac{\sigma^2}{N} $, where $\sigma^2$ is is the variance of the metric itself over the entire population.

I will use toy Monte Carlo simulations to illustrate 5 ways to reduce variance in A/B testing:

increase sample size
move towards an even split
reduce variance in the metric definition
stratification
CUPED

The ipython notebook for this post is up on Github.

Increase sample size

The simplest way to reduce variance is to collect more samples, if possible. For every 4x increase of $N$, we get a 2x decrement of $s$. In code:

def get_AB_samples(mu, sigma, treatment_lift, N):
    A = list(normal(loc=mu                 , scale=sigma, size=N))
    B = list(normal(loc=mu + treatment_lift, scale=sigma, size=N))
    return A, B

N = 1000
N_multiplier = 4
mu = 100
sigma = 10
treatment_lift = 2
num_simulations = 1000

print('Simulating %s A/B tests, true treatment lift is %d...' % (num_simulations, treatment_lift))

n1_lifts, n4_lifts = [], []
for i in range(num_simulations):
    print('%d/%d' % (i, num_simulations), end='\r')
    A, B = get_AB_samples(mu, sigma, treatment_lift, N)
    n1_lifts.append(lift(A, B))
    A, B = get_AB_samples(mu, sigma, treatment_lift, N_multiplier*N)
    n4_lifts.append(lift(A, B))

print('N samples  A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(n1_lifts), cov(n1_lifts)))
print('4N samples A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(n4_lifts), cov(n4_lifts)))
print('Raio of lift variance = %.2f (expected = %.2f)' % (cov(n4_lifts)/cov(n1_lifts), 1/N_multiplier))

bins = linspace(-2, 6, 100)
plt.figure(figsize=(14, 7))
plt.hist(n1_lifts, bins, alpha=0.5, label='N samples')
plt.hist(n4_lifts, bins, alpha=0.5, label=f'{N_multiplier}N samples')
plt.xlabel('lift')
plt.ylabel('count')
plt.legend(loc='upper right')
plt.title('lift histogram')
plt.show()

Prints:

Simulating 1000 A/B tests, true treatment lift is 2...
N samples  A/B testing, mean lift = 2.00, variance of lift = 0.18
4N samples A/B testing, mean lift = 2.01, variance of lift = 0.05
Raio of lift variance = 0.26 (expected = 0.25)

Note: in real life, collecting more samples is not always possible, since the amount of samples may be limited, or there may be a time limit by which we want to conclude the experiment. Also, in many situations, the relationship of the length of time $t$ that the experiment runs and sample size $N$ is sub-linear, because over time users tend to return. In other words, if we run an experiment for 2 weeks, and get $2N$ users, running it for another 2 weeks will probably yield less than $2N$ additional users.

Move towards an even split

If we have $N$ samples available to use for A/B testing, one question is, what percent to put into A (control) and what percent to put into B (treatment). Almost always there are risk-management concerns, but it's worth knowing that in terms of the best lift measurement (lowest variance), the most optimal split is the even split of 50%-50%, and a 20%-80% split is better than a 10%-90%. We can see this with simple math. Suppose our $N$ total samples are split into $pN$ and $(1-p)N$ for A and B variants, then the variance is:

$ s^2 = \frac{ \sigma^2 }{ pN } + \frac{ \sigma^2 }{ (1-p)N } = \frac{ \sigma^2 }{ N } ( \frac{1}{p} + \frac{1}{1-p} ) $.

So, $ s^2(p) = C ( \frac{1}{p} + \frac{1}{1-p} ) $, which has a minimum at $p=\frac{1}{2}$:

The more the split is away from even, the higher the variance compared to the optimal case:

def get_AB_samples(mu, sigma, treatment_lift, N_A, N_B):
    A = list(normal(loc=mu                 , scale=sigma, size=N_A))
    B = list(normal(loc=mu + treatment_lift, scale=sigma, size=N_B))
    return A, B

def expected_ratio(p, q):
    top = 1/p + 1/(1-p)
    bot = 1/q + 1/(1-q)
    return top/bot

N = 4000
mu = 100
sigma = 10
treatment_lift = 2
even_ratio = 0.5
uneven_ratio = 0.9
num_simulations = 1000

print('Simulating %s A/B tests, true treatment lift is %d...' % (num_simulations, treatment_lift))

even_lifts, uneven_lifts = [], []
for i in range(num_simulations):
    print('%d/%d' % (i, num_simulations), end='\r')
    A, B = get_AB_samples(mu, sigma, treatment_lift, int(N*even_ratio), int(N*(1-even_ratio)))
    even_lifts.append(lift(A, B))
    A, B = get_AB_samples(mu, sigma, treatment_lift, int(N*uneven_ratio), int(N*(1-uneven_ratio)))
    uneven_lifts.append(lift(A, B))

print('Even split   A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(even_lifts),   cov(even_lifts)))
print('Uneven split A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(uneven_lifts), cov(uneven_lifts)))
print('Raio of lift variance = %.2f (expected = %.2f)' % ((cov(even_lifts)/cov(uneven_lifts)), expected_ratio(even_ratio, uneven_ratio)))

Prints:

Simulating 1000 A/B tests, true treatment lift is 2...
Even split   A/B testing, mean lift = 1.98, variance of lift = 0.10
Uneven split A/B testing, mean lift = 1.98, variance of lift = 0.29
Raio of lift variance = 0.36 (expected = 0.36)

Reduce variance in the metric definition

In the equation $ s^2 = \frac{ \sigma^2 }{ N } $, the previous two methods were increasing the sample size $N$ to reduce $s^2$. Another way to reduce $s^2$ is to reduce the variance $ \sigma^2 $ of the underlying metric itself. I will mention two approaches here:

1. Winsorizing, ie. cutting or normalizing outliers. For example, if the underlying metric is spend per head, there may be outliers, such as high-value B2B customers among the overall population of B2C customers, or bots appearing to spend a lot of time on a site compared to real users. If we cut-off these values at some $\sigma$ (technically, we can either discard these values or set them to the highest acceptable value), we will get a better measurement of our "real" population:

def get_AB_samples(scale, treatment_lift, N):
    A = list(exponential(scale=scale                 , size=N))
    B = list(exponential(scale=scale + treatment_lift, size=N))
    # add outliers
    for i in range(int(N*0.001)):
        A.append(random()*scale*100)
        B.append(random()*scale*100 + treatment_lift)
    return A, B

N = 10*1000
scale = 100
treatment_lift = 2
num_simulations = 1000

orig_lifts, trunc_lifts = [], []
for i in range(num_simulations):
    print('%d/%d' % (i, num_simulations), end='\r')
    A, B = get_AB_samples(scale, treatment_lift, N)
    orig_lifts.append(lift(A, B))
    A, B = [x for x in A if x < 5*scale], [x for x in B if x < 5*scale]
    trunc_lifts.append(lift(A, B))

Another method is to measure the median instead of the mean. Note that with these approaches we are changing the definition of the metric itself. So, for example, Finance's "spend per head" definition may no longer match the winsorized or median-based definition of the experimentation framework!

2. Measuring a metric further down the funnel. Suppose we are measuring a metric such as spend per head. We can split this metric into two parts:

(spend per head, overall) = (ratio of users who spend more than 0, conversion) x (spend per head of those who spend more than zero, conditional)

In real-life, the ratio is usually low, less than 10%. In such cases, measuring the two terms (the conversion and the conditional) separately yields lower variance:

def get_AB_samples(p, p_lift, mu, sigma, treatment_lift, N_A, N_B):
    A = list(normal(loc=mu                 , scale=sigma, size=N_A))
    B = list(normal(loc=mu + treatment_lift, scale=sigma, size=N_B))
    A = [x if random() < p        else 0 for x in A]
    B = [x if random() < p+p_lift else 0 for x in B]
    return A, B

N = 10*1000
p = 0.1
p_lift = 0.01
mu = 100
sigma = 25
treatment_lift = 2
num_simulations = 1000

print('Simulating %s A/B tests, true treatment lift is %d...' % (num_simulations, treatment_lift))

cont_lifts, cond_lifts, conv_lifts = [], [], []
for i in range(num_simulations):
    print('%d/%d' % (i, num_simulations), end='\r')
    A, B = get_AB_samples(p, p_lift, mu, sigma, treatment_lift, int(N/2), int(N/2))
    cont_lifts.append(lift(A, B)/p)
    A_, B_ = [x for x in A if x > 0], [x for x in B if x > 0]
    cond_lifts.append(lift(A_, B_))
    A_, B_ = [1 if x > 0 else 0 for x in A], [1 if x > 0 else 0 for x in B]
    conv_lifts.append(lift(A_, B_))

bins = linspace(-8, 12, 100)
plt.figure(figsize=(14, 7))
plt.hist(cont_lifts, bins, alpha=0.5, label='overall')
plt.hist(cond_lifts, bins, alpha=0.5, label='conditional')
plt.xlabel('lift')
plt.ylabel('count')
plt.legend(loc='upper right')
plt.title('lift histogram')
plt.show()

bins = linspace(-0.1, 0.1, 100)
plt.figure(figsize=(14, 7))
plt.hist(conv_lifts, bins, alpha=0.5, label='conversion')
plt.xlabel('lift')
plt.ylabel('count')
plt.legend(loc='upper right')
plt.title('lift histogram')
plt.show()

Stratification

Stratification lowers variance by making sure that each sub-population is sampled according to its split in the overall population. As a toy example, assume there are two sub-populations, M and F, with different distributions. For simplicity let's assume that the population is exactly 50% M and 50% F. In an A/B test, if we randomly sample the overall population and assign them to A and B, we will on approximately get 50% Ms and 50% Fs in both A and B, but not exactly. This small difference will impact our measurement variance. However, if we make sure that both A and B have exactly 50% Ms and 50% Fs, the variance of the measurement is decreased:

def get_AB_samples_str(
    pop1_mu, pop1_sigma,
    pop2_mu, pop2_sigma,
    pop1_ratio,
    treatment_lift,
    N,
):
    A =  list(normal(loc=pop1_mu, scale=pop1_sigma, size=int(N*pop1_ratio)))
    A += list(normal(loc=pop2_mu, scale=pop2_sigma, size=int(N*(1-pop1_ratio))))
    B =  list(normal(loc=pop1_mu + treatment_lift, scale=pop1_sigma, size=int(N*pop1_ratio)))
    B += list(normal(loc=pop2_mu + treatment_lift, scale=pop2_sigma, size=int(N*(1-pop1_ratio))))
    return A, B

def get_AB_samples_rnd(
    pop1_mu, pop1_sigma,
    pop2_mu, pop2_sigma,
    pop1_ratio,
    treatment_lift,
    N,
):
    A = []
    for _ in range(N):
        if random() < pop1_ratio:
            A.append(normal(loc=pop1_mu, scale=pop1_sigma))
        else:
            A.append(normal(loc=pop2_mu, scale=pop2_sigma))
    B = []
    for _ in range(N):
        if random() < pop1_ratio:
            B.append(normal(loc=pop1_mu + treatment_lift, scale=pop1_sigma))
        else:
            B.append(normal(loc=pop2_mu + treatment_lift, scale=pop2_sigma))
    return A, B


N = 4000
pop1_ratio = 0.5
pop1_mu = 100
pop2_mu = 200
pop1_sigma = pop2_sigma = 10
treatment_lift = 2
num_simulations = 1000

print('Simulating %s A/B tests, true treatment lift is %d...' % (num_simulations, treatment_lift))

str_lifts, rnd_lifts = [], []
for i in range(num_simulations):
    print('%d/%d' % (i, num_simulations), end='\r')
    A, B = get_AB_samples_str(pop1_mu, pop1_sigma, pop2_mu, pop2_sigma, pop1_ratio, treatment_lift, N)
    str_lifts.append(lift(A, B))
    A, B = get_AB_samples_rnd(pop1_mu, pop1_sigma, pop2_mu, pop2_sigma, pop1_ratio, treatment_lift, N)
    rnd_lifts.append(lift(A, B))

print('Stratified sampling   A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(str_lifts), cov(str_lifts)))
print('Random sampling       A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(rnd_lifts), cov(rnd_lifts)))
print('Raio of lift variance = %.2f' % (cov(str_lifts)/cov(rnd_lifts)))

Prints:

Simulating 1000 A/B tests, true treatment lift is 2...
Stratified sampling   A/B testing, mean lift = 2.01, variance of lift = 0.05
Random sampling       A/B testing, mean lift = 1.97, variance of lift = 1.28
Raio of lift variance = 0.04

If we cannot control the sampling of the population, we can also re-weigh samples after data collection is over. Stratification and CUPED are closely related, check section 3.1 and 3.3 of the CUPED paper.

CUPED

CUPED is a way to reduce variance in A/B testing if the past historic values of the metric are correlated wit the current values we measure in the experiment. In other words, CUPED works if eg. high spenders before the experiment also tend to be high spenders during the experiment, and the same for low spenders. I have written four articles on CUPED, so I will just link to those:

Conclusion

The post used toy examples to show some of the fundamental methods and levers to reduce variance in experiments. It's worth noting that I used Monte Carlo simulations over many experiments (typically 1,000) to show the decrease in variance, by visualizing the spread of the lift histogram. The point is, in one specific experiment evaluation, the decrease is not visible, since one measurement does not have variance-of-lift by itself (it's just one specific lift we measure). The rewards of lower variance in A/B testing are reaped in the long term, by having more accurate measurements, making better decisions and releasing better products more quickly.

Correlations, seasonality, lift and CUPED

2021-09-05T00:00:00+02:00

Introduction

This is the fourth post about CUPED. The previous three were:

In this final post, I will address some questions relating to CUPED:

what if some units are missing "before" data?
what if "before" and "after" is not correlated?
what if "before" and "after" is correlated, but the treatment lift is 0?
what if our measurement is not continuous (like USD spend per user), but binary conversion data?
what if we run an A/B test, and evaluate using both traditional and CUPED methodology, and pick the "more favorable"?
is correlation between "before" and "after" the same as seasonality?
what are other ways to reduce variance in A/B testing?

What if some units are missing "before" data?

With CUPED, we can reduce the variance of our A/B test measurement, assuming we have "before" data, and it is correlated with the data collected during the experiment, which I referred to as "after" data in the last posts. But what if for some experimental units, the "before" data is missing? For example, we can imagine that the experiment unit is users, and for some users, we are missing the "before" data. In this case, all we have to do is: use the un-adjusted "after" metric. So, including this modification in the "CUPED recipe" from the first post:

Let $Y_i$ be the ith user's spend in the "after" period, and $X_i$ be their spend in the "before" period, both for A and B combined. We compute an adjusted "after" spend $Y'_i$ with the following process:

Compute the covariance $cov(X, Y)$ of X and Y.
Compute the variance $var(X)$ of X.
Compute the mean $\mu_X$ of X.
For the i'th user, if $X_i$ is missing: $ Y'_i := Y_i $.
For the i'th user, if $X_i$ is not missing:$ Y'_i := Y_i - (X_i - \mu_X) \frac{cov(X, Y)}{var(X)} $.
Evaluate the A/B test using $Y'$ instead of $Y$.

What if "before" and "after" are not correlated?

If "before" and "after" are not correlated, and we use CUPED, are we making a mistake? No. This question was answered in the first first post. If there is no correlation, then in the above equation $ cov(X, Y) = 0 $, so $ Y_i' = Y_i $. In actual experiment instances $ cov(X, Y) $ will not be exactly 0, but will be a small number, so $ cov(X, Y) \approx 0 $, so $ Y_i' \approx Y_i $. Running Monte Carlo simulations of many experiments showed that over many experiments this is equivalent to traditional A/B testing.

What if "before" and "after" is correlated, but the treatment lift is 0?

This was explored in the third post. If the treatment lift is 0, then in an A/B test the null hypothesis is actually true, and we're really running an A/A test. In this case, we still get the benefit of CUPED, we're just getting a more reliable measurement of the 0 lift. CUPED doesn't care what the lift is, it reduces the variance of the lift measurement in all cases, if there is correlation. See the plot below of an A/A test, where the variance is reduced for 0 lift (the histogram is centered on 0):

What if our measurement is not continuous, but binary conversion data?

This was explored in the second post for the binary 0/1 conversion case, and it is not a problem, CUPED can be applied. But it is somewhat counter-intuitive, since the CUPED-adjusted values are no longer 0 and 1, but become continuous. Since the CUPED formula has 2 variables, before and after, and in the binary case each can take on the value 0 and 1, there are 4 possible CUPED-adjusted values. In one experiment realization from that post, these values were:

Possible mappings:
(before=1, after=0) -> adjusted=-0.722
(before=0, after=1) -> adjusted=1.081
(before=0, after=0) -> adjusted=0.081
(before=1, after=1) -> adjusted=0.278

This illustrates that the binary data loses its binary nature with CUPED.

What if we run an A/B test, and pick the "more favorable" outcome?

What if we run an A/B test, evaluate using both traditional (without using "before" data) and CUPED A/B testing, and pick the more favorable outcome, the one with higher lift / lower p-value? This was explored in the third post, and was shown to be a fallacy. We showed with Monte Carlo simulations that this behavior skews the lift distribution to the right and overestimates the lift:

Note that this error is similar to other "p-hacking" fallacies:

peeking at experiment results before the experiment finishes (and stopping early in case of favorable results)...
evaluating multiple metrics (and using the most favorable one to report results)...
checking multiple subsets like countries for favorable results...

... without controlling the p-value for multiple checks.

Is correlation between "before" and "after" the same as seasonality?

No, they are different things. In the first post, I used this toy model to generate correlated "before" and "after" data:

def get_AB_samples(before_mean, before_sigma, eps_sigma, treatment_lift, N):
    A_before = list(normal(loc=before_mean, scale=before_sigma, size=N))
    B_before = list(normal(loc=before_mean, scale=before_sigma, size=N))
    A_after  = [x + normal(loc=0, scale=eps_sigma)                  for x in A_before]
    B_after  = [x + normal(loc=0, scale=eps_sigma) + treatment_lift for x in B_before]
    return A_before, B_before, A_after, B_after

Notice that here, A_before and A_after both have the same mean, before_mean, since A_after is just A_before with some (symmetric) noise added (the same is true for the Bs if treatment_lift = 0). So here we have a correlated experiment, but the means are the same before and after, there is no seasonal change. Now look at this:

def get_AB_samples(before_mean, before_sigma, seasonal_lift, treatment_lift, N):
    A_before = list(normal(loc=before_mean, scale=before_sigma, size=N))
    B_before = list(normal(loc=before_mean, scale=before_sigma, size=N))
    A_after  = list(normal(loc=before_mean + seasonal_lift                   scale=before_sigma, size=N))
    B_after  = list(normal(loc=before_mean + seasonal_lift + treatment_lift, scale=before_sigma, size=N))
    return A_before, B_before, A_after, B_after

In this setup, A_after has a seasonal_lift compared to A_before (and B has an optional treatment_lift), but A and B are uncorrelated.

These examples show that seasonality and correlation are not the same thing in A/B testing. Seasonality is a shift in the mean between before and after, correlation is correlation. All 9 combinations between {no correlation, correlation} x {no seasonal lift, seasonal lift} x {no treatment lift, treatment lift} are possible and occur in real life. CUPED only helps in reducing variabce if there is correlation.

What are other ways to reduce the variance of the lift measurement?

In these Monte Carlo simulations, I used the difference in means for lift:

def lift(A, B):
    return mean(B) - mean(A)

Because of the Central Limit Theorem, the distribution of the error of the mean measurement is a normal, and the difference of two normals is also normal. This is the bell curve we've been seeing on all the Monte Carlo histograms. The standard deviation of this normal is called the standard error, and its formula is $ s = \frac{\sigma}{\sqrt{N}} $. The important part here is the $ \sqrt{N} $, which says that for every 4x sample size, variance of the lift goes down 2x. So, another way to reduce variance of the lift measurements is to collect more N samples, if possible.

The other important method for reducing variance is stratification, where we identify sub-populations of our overall population and calibrate our sampling accordingly, or re-weight sub-population metrics after sampling.

Conclusion

This is the fourth and final post on CUPED. In the future I plan to write about stratification, another method to reduce variance in A/B testing.

A/A testing and false positives with CUPED

2021-08-15T00:00:00+02:00

Introduction

In the previous posts, Reducing variance in A/B testing with CUPED and Reducing variance in conversion A/B testing with CUPED, I ran Monte Carlo simulation to get a feel for how CUPED works in continuous (like $ spend per customer) and then binary (conversion) experiments. In both cases, as long as historic "before" and the experiment's "after" data is correlated, CUPED yields lower variance measurements when compared to traditional A/B testing, where only "after" data is used. One of the interesting aspects of CUPED is that although CUPED reduces variance on average, there is no such guarantee for each individual experiment outcome. We saw that on a given experiment outcome, it is possible that the statistics computed from the transformed CUPED variables can be worse than evaluating using just the "after" data using traditional A/B testing:

the lift computed with CUPED can be lower (or higher), irrespective of what the true lift is
the p-value computed with CUPED can be lower (or higher), irrespective of what the true lift is

In real-life, Data Scientists are often under pressure to achieve positive results. This poses the potential danger of hacking the experiment, when a Data Scientist computes outcomes with both CUPED and traditional A/B testing and reports the more favorable results, ie. the one with higher lifts and lower p-values. Here I will run Monte Carlo simulations to show (to myself and the reader) that such practice results in incorrect lift measurements and incorrect p-value measurements.

TLDR

The TLDR learning of this post is: as a Data Scientist, we have to pick whether we use traditional A/B testing evaluation (using just "after" data) or CUPED before we evaluate the experiment data, preferably before we even run the experiment. The best case is if there is an experimentation platform which makes this choice for us. As demonstrated below, evaluating both and picking and reporting more favorable results is not statistically sound.

The ipython notebook is up on Github.

Correlated A/A tests

First, let's run an A/A test where "before" and "after" is the same; A/A means there is no difference between A and B, ie. the true lift is 0. I will use the same code as in previous posts, setting treatment_lift=0:

N = 1000
before_mean = 100 
before_sigma = 50
eps_sigma = 20
treatment_lift = 0
num_simulations = 1000
...

Prints:

Simulating 1000 A/B tests, true treatment lift is 0...
Traditional    A/B testing, mean lift = 0.00, variance of lift = 6.09
CUPED adjusted A/B testing, mean lift = 0.03, variance of lift = 0.78
CUPED lift variance / tradititional lift variance = 0.13 (expected = 0.14)

We see that both traditional and CUPED correctly estimate the true lift of 0, but CUPED has lower variance. Plotting the histograms, we see the familiar shape of a narrower CUPED, but this time centered on 0 lift:

Lift hacking

Let's simulate our Data Scientist under pressure in code: at the end of each experiment, we pick and report the higher lift (between traditional and CUPED):

N = 1000
before_mean = 100 
before_sigma = 50
eps_sigma = 20
treatment_lift = 0
num_simulations = 10*1000

print('Simulating %s A/B tests, true treatment lift is %d...' % (num_simulations, treatment_lift))

traditional_lifts, adjusted_lifts, hacked_lifts = [], [], []
for i in range(num_simulations):
    print('%d/%d' % (i, num_simulations), end='\r')
    A_before, B_before, A_after, B_after = get_AB_samples(before_mean, before_sigma, eps_sigma, treatment_lift, N)
    A_after_adjusted, B_after_adjusted = get_cuped_adjusted(A_before, B_before, A_after, B_after)
    traditional_lifts.append(lift(A_after, B_after))
    adjusted_lifts.append(lift(A_after_adjusted, B_after_adjusted))
    hacked_lifts.append(max(
        lift(A_after, B_after),
        lift(A_after_adjusted, B_after_adjusted)
    ))

print('Traditional    A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(traditional_lifts), cov(traditional_lifts)))
print('CUPED adjusted A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(adjusted_lifts), cov(adjusted_lifts)))
print('Hacked         A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(hacked_lifts), cov(hacked_lifts)))

Prints:

Simulating 10000 A/B tests, true treatment lift is 0...
Traditional    A/B testing, mean lift = 0.00, variance of lift = 5.82
CUPED adjusted A/B testing, mean lift = 0.00, variance of lift = 0.80
Hacked         A/B testing, mean lift = 0.89, variance of lift = 2.50

We can see that:

traditional and CUPED correctly estimate a mean lift of 0
the hacked result overestimates the lift

We can plot all three on a histogram:

It's a bit hard to see, let's just see the "hacked" histogram:

The histogram seems to have its maximum at 0, but it's skewed towards positive values, so on average our imaginary Data Scientist overestimates the lift.

False positive rate

We can now repeat the above logic, and also record the p-value. Our imaginary Data Scientist picks traditional or CUPED adjusted, depending on which has higher lift, and also records that p-value. Let's assume the they use a critical p-value of p_crit = 0.05, so they reject the null hypothesis that A and B are the same, and accept the action hypothesis that B is better. Since A and B are actually the same (since treatment_lift = 0, since we are running A/A tests), these are all false positives.

Let's remember the definition of the p-value. This is the probability of incorrectly rejecting the null hypothesis and accepting the action hypothesis, even though the null hypothesis is true. The null hypothesis is that A and B are the same, ie. an A/A test, which is what we're simulating. So by setting p_crit = 0.05, we are saying we accept a false positive rate (FPR) of 0.05. So if our statistical methodology is sound, repeating our experiment many times (num_simulations), we should get a false positive rate of 0.05. Let's see what happens.

In code:

N = 1000
before_mean = 100 
before_sigma = 50
eps_sigma = 20
treatment_lift = 0
num_simulations = 10*1000
p_crit = 0.05
traditional_fps, cuped_fps, hacked_fps = 0, 0, 0

print('Simulating %s A/B tests, true treatment lift is %d...' % (num_simulations, treatment_lift))

traditional_lifts, adjusted_lifts = [], []
traditional_pvalues, adjusted_pvalues = [], []
for i in range(num_simulations):
    print('%d/%d' % (i, num_simulations), end='\r')
    A_before, B_before, A_after, B_after = get_AB_samples(before_mean, before_sigma, eps_sigma, treatment_lift, N)
    A_after_adjusted, B_after_adjusted = get_cuped_adjusted(A_before, B_before, A_after, B_after)
    adjusted_pvalue = p_value(A_after_adjusted, B_after_adjusted)
    if p_value(A_after, B_after) < p_crit:
        traditional_fps += 1
    if p_value(A_after_adjusted, B_after_adjusted) < p_crit:
        cuped_fps += 1
    if lift(A_after, B_after) < lift(A_after_adjusted, B_after_adjusted):
        if p_value(A_after, B_after) < p_crit:
            hacked_fps += 1
    else:
        if p_value(A_after_adjusted, B_after_adjusted) < p_crit:
            hacked_fps += 1

print('False positive rate (expected: %.3f):' % p_crit)
print('Traditional: %.3f' % (traditional_fps/num_simulations))
print('CUPED:       %.3f' % (cuped_fps/num_simulations))
print('Hacked:      %.3f' % (hacked_fps/num_simulations))

Prints:

Simulating 10000 A/B tests, true treatment lift is 0...
False positive rate (expected: 0.050):
Traditional: 0.049
CUPED:       0.047
Hacked:      0.073

We see that both with traditional and CUPED, we get around the expected 0.05. However, when we hack our experiment evaluation and pick and choose, our FPR is significantly higher. This demonstrates that pick and choose is not statistically sound: we both overestimate the lift and have a higher false positive rate!

No correlation

In the above case, we were running A/A tests on data where the "before" and "after" was correlated. What happens if we do the same when there's no correlation? Recall that the transformation equation for CUPED is:

$ Y'_i = Y_i - (X_i - \mu_X) \frac{cov(X, Y)}{var(X)} $

If there is no correlation, $cov(X, Y) \approx 0$, so $Y'_i \approx Y_i$. I write $\approx$ for approximate equality instead of equality, because even if there is no correlation, numerically $cov(X, Y)$ won't exactly equal 0, it will be some small number, like 0.001. But still, the transformed $Y_i'$ values will be very close to the original $Y_i$ values, so the lifts (traditional vs. CUPED) will also be very close, as will the p-values. So the above pick and choose hacking will not "work" here, since the two choices (lifts) will be very close, almost the same. We can see this if we run many experiments, and for each one we plot both the traditional and CUPED computed lifts. It's a tight fit to the $y=x$ line:

If we repeat the A/A tests above, but with uncorrelated data:

Simulating 10000 A/B tests, true treatment lift is 0...
False positive rate (expected: 0.050):
Traditional: 0.050
CUPED:       0.049
Hacked:      0.050

We see that hacking doesn't work.

Conclusion

The Monte Carlo simulations show explicitly that we have to pick whether we use traditional A/B testing evaluation (using just "after" data) or CUPED before we evaluate the experiment data, preferably before we even run the experiment. As demonstrated in this post, evaluating using both and picking the one with more favorable results is not statistically sound.

Reducing variance in conversion A/B testing with CUPED

2021-08-07T00:00:00+02:00

Introduction

In the previous post, Reducing variance in A/B testing with CUPED, I ran Monte Carlo simulations to demonstrate CUPED, a variance reduction technique in A/B testing on continuous data, like $ spend per customer. Here I will repeat the same Monte Carlo simulations, but with binary 0/1 conversion data.

The experiment setup is almost entirely the same as in the last post. The only difference is in the get_AB_samples() and get_AB_samples_nocorr() functions, since these now have to generate 0/1 conversion data. The jupyter notebook for this post is up on Github.

Generating correlated conversion data

The approach is to assume a base conversion before_p, say 10%. For each user, do a random 0/1 draw with this probability. This "before" outcome is either 0 or 1. Then, generate the "after" data:

if the "before" outcome was 0, do a random 0/1 draw with conversion 0 + offset_p, say 10%. This way, most 0s remain 0s.
if the "before" outcome was 1, do a random 0/1 draw with conversion 1 - offset_p, say 90%. This way, most 1s remain 1s.
in the B variant, add treatment_lift (say 1%) to the above probabilities.

This is just one possible scheme, but it results in "before" and "after" data being correlated. In code:

def draw(p):
    if random() < p:
        return 1
    else:
        return 0

def lmd(x):
    return list(map(draw, x))

def get_AB_samples(before_p, offset_p, treatment_lift, N):
    A_before = [int(random() < before_p) for _ in range(N)]
    B_before = [int(random() < before_p) for _ in range(N)]
    A_after  = [int(random() < abs(x - offset_p))                  for x in A_before]
    B_after  = [int(random() < abs(x - offset_p) + treatment_lift) for x in B_before]
    return lmd(A_before), lmd(B_before), lmd(A_after), lmd(B_after)

We can validate by computing conditional probabilities:

counts = defaultdict(int)
for b, a in zip(A_before, A_after):
    counts[(b, a)] += 1
    counts[(b)] += 1

print('(after = 1 | before = 0) = %.2f' % (counts[(0, 1)]/counts[(0)]))
print('(after = 0 | before = 0) = %.2f' % (counts[(0, 0)]/counts[(0)]))
print('(after = 1 | before = 1) = %.2f' % (counts[(1, 1)]/counts[(1)]))
print('(after = 0 | before = 1) = %.2f' % (counts[(1, 0)]/counts[(1)]))

Prints:

(after = 1 | before = 0) = 0.10
(after = 0 | before = 0) = 0.90 # if it was 0, it's likely to be 0 again
(after = 1 | before = 1) = 0.90 # if it was 1, it's likely to be 1 again
(after = 0 | before = 1) = 0.10

Simulating one experiment

The driver code is, apart from parametrization, the same as before:

N = 10*1000
before_p = 0.1
offset_p = 0.1
treatment_lift = 0.01

A_before, B_before, A_after, B_after = get_AB_samples(before_p, offset_p, treatment_lift, N)
A_after_adjusted, B_after_adjusted = get_cuped_adjusted(A_before, B_before, A_after, B_after)

print('A mean before = %.3f, A mean after = %.3f, A mean after adjusted = %.3f' % (mean(A_before), mean(A_after), mean(A_after_adjusted)))
print('B mean be|fore = %.3f, B mean after = %.3f, B mean after adjusted = %.3f' % (mean(B_before), mean(B_after), mean(B_after_adjusted)))
print('Traditional    A/B test evaluation, lift = %.3f, p-value = %.3f' % (lift(A_after, B_after), p_value(A_after, B_after)))
print('CUPED adjusted A/B test evaluation, lift = %.3f, p-value = %.3f' % (lift(A_after_adjusted, B_after_adjusted), p_value(A_after_adjusted, B_after_adjusted)))

Prints (not deterministic):

A mean before = 0.097, A mean after = 0.179, A mean after adjusted = 0.180
B mean be|fore = 0.099, B mean after = 0.191, B mean after adjusted = 0.190
Traditional    A/B test evaluation, lift = 0.012, p-value = 0.028
CUPED adjusted A/B test evaluation, lift = 0.010, p-value = 0.021

In this particular run, CUPED did a better job approximating the lift with a lower p-value. However, as we saw before, this is not always the case. Although on average CUPED is a better estimator with lower variance, there are experiment runs where CUPED makes the lift measurement worse. For example, after a few re-runs:

A mean before = 0.098, A mean after = 0.178, A mean after adjusted = 0.180
B mean be|fore = 0.103, B mean after = 0.189, B mean after adjusted = 0.187
Traditional    A/B test evaluation, lift = 0.011, p-value = 0.053
CUPED adjusted A/B test evaluation, lift = 0.006, p-value = 0.133

Next, it's interesting to print out the CUPED-transformed variables:

print('Possible mappings:')
mappings = set(['(before=%d, after=%d) -> adjusted=%.3f' % (b, a, p) for
  b, a, p in zip(A_before+B_before, A_after+B_after, A_after_adjusted+B_after_adjusted)])
for m in mappings:
    print(m)

Prints:

Possible mappings:
(before=1, after=0) -> adjusted=-0.722
(before=0, after=1) -> adjusted=1.081
(before=0, after=0) -> adjusted=0.081
(before=1, after=1) -> adjusted=0.278

Remember the CUPED transformation equation was $ Y'_i := Y_i - (X_i - \mu_X) \frac{cov(X, Y)}{var(X)} $. In this equation $\mu_X$, $cov(X, Y)$, and $var(X)$ are constants computed from the experiment results. Both $X_i$ and $Y_i$ can take on two values, 0 or 1, so there are 4 possible combinations, hence $Y'_i$ can take on 4 values. In the above run, the values were -0.722, 1.081, 0.081 and 0.278. It's a bit counter-intuitive, since now the conversion experiment has these weird, even negative values, instead of just 0 and 1.

Simulating many experiments

As with continuous variables, CUPED measures the same lift [in conversion], but with lower variance:

Simulating 10,000 A/B tests, true treatment lift is 0.010...
Traditional    A/B testing, mean lift = 0.010, variance of lift = 0.00030
CUPED adjusted A/B testing, mean lift = 0.010, variance of lift = 0.00019
CUPED lift variance / tradititional lift variance = 0.626 (expected = 0.668)

We can observe the tighter lifts on a histogram:

As with continuous variables, the p-values decrease:

As illustrated above, CUPED lifts and p-values are a better estimate with respect to variance, but not in all cases:

No correlation

The simplest way to generate uncorrelated conversion data is to use random draws independently. In code:

def get_AB_samples_nocorr(before_p, treatment_lift, N):
    A_before = [before_p] * N
    B_before = [before_p] * N
    A_after = [before_p] * N
    B_after = [before_p + treatment_lift] * N
    return lmd(A_before), lmd(B_before), lmd(A_after), lmd(B_after)

Checking conditional probabilities:

P(after = 1 | before = 0) = 0.10
P(after = 0 | before = 0) = 0.90
P(after = 1 | before = 1) = 0.10
P(after = 0 | before = 1) = 0.90

It's uncorrelated, because P(after = 1) = 0.10 in both 0 and 1 before cases, so "after" is independent of "before" (and same for P(after = 0)).

With this generator function, running num_experiments=10,000, we can observe no variance reduction (since "before" and "after" is uncorrelated) on the lift and p-value histograms:

Conclusion

CUPED works for both continuous and binary experimentation outcomes, and reduces variance if "before" and "after" are correlated.

Reducing variance in A/B testing with CUPED

2021-07-31T00:00:00+02:00

Introduction

CUPED is a variance reduction technique introduced by Deng, Xu, Kohavi and Walker. Using CUPED we can achieve lower variance in A/B testing by computing an adjusted evaluation metric using historic "before" data. Based on blog posts, it seems that many big Internet companies are using CUPED, such as Bing, Facebook, Netflix, Booking, Etsy.

In this post I will use Monte Carlo simulations of A/B tests to demonstrate (to myself and readers) how CUPED works. The Jupyter notebook is up on Github.

Experiment setup

Assume an A/B testing setup where we're measuring a metric M, eg. $ spend per user. We have N users, randomly split into A and B. A is control, B is treatment. We have metric M for each user for the "before" time period, when treatment and control was the same, and the "after" period, when treatment had some treatment applied, which we hope increased their spend.

Let $Y_i$ be the ith user's spend in the "after" period, and $X_i$ be their spend in the "before" period, both for A and B combined. We compute an adjusted "after" spend $Y'_i$.

The CUPED recipe:

Compute the covariance $cov(X, Y)$ of X and Y.
Compute the variance $var(X)$ of X.
Compute the mean $\mu_X$ of X.
Compute the adjusted $ Y'_i = Y_i - (X_i - \mu_X) \frac{cov(X, Y)}{var(X)} $ for each user.
Evaluate the A/B test using $Y'$ instead of $Y$.

The point of CUPED is, when using $Y'$ instead of $Y$, the estimate of the treatment lift will have lower variance, ie. on average we will be better at estimating the lift.

Mathematically, the decrease in variance will be $ \frac{var(Y')}{var(Y)} = 1 - corr(X, Y)^2 $ where $ corr(X, Y) = \frac{cov(X, Y)}{\sqrt{var(X)var(Y)}} $ is the correlation between X and Y.

Note: in some of the blog posts about CUPED, there seems to be a typo and the squared is missing from the $corr(X, Y)^2$. It's there in the original paper.

CUPED implementation

Implementing CUPED is straightforward. Assuming A_before and B_before contain the $X_i$ values for each customer in groups A and B, and A_after and B_after contain the $Y_i$ values, A_before[i] is the ith customer spend in the "before" period, A_after[i] is the same customer's "after" spend, same for the Bs. So we're indexing the users in A and B separately, the ith user in the A list (who is in group A) is a different user than the ith user in the B list (who is in group B). In code:

def get_cuped_adjusted(A_before, B_before, A_after, B_after):
    cv = cov([A_after + B_after, A_before + B_before])
    theta = cv[0, 1] / cv[1, 1]
    mean_before = mean(A_before + B_before)
    A_after_adjusted = [after - (before - mean_before) * theta for after, before in zip(A_after, A_before)]
    B_after_adjusted = [after - (before - mean_before) * theta for after, before in zip(B_after, B_before)]
    return A_after_adjusted, B_after_adjusted

Simulating one experiment

We need some code to generate the A_before, B_before, A_after, B_after lists, and in a way so that "before" and "after" are correlated. For this experiment, I chose:

$X$ follows a normal distribution like N(before_mean, before_sigma)
The A control $Y_i$ is the the same as $X_i$ plus a normal random variable like N(0, eps_sigma)
The B treatment $Y_i$ is the the same as $X_i$ plus a normal random variable like N(0, eps_sigma) plus a fixed treatment_lift

In code:

def get_AB_samples(before_mean, before_sigma, eps_sigma, treatment_lift, N):
    A_before = list(normal(loc=before_mean, scale=before_sigma, size=N))
    B_before = list(normal(loc=before_mean, scale=before_sigma, size=N))
    A_after  = [x + normal(loc=0, scale=eps_sigma) for x in A_before]
    B_after  = [x + normal(loc=0, scale=eps_sigma) + treatment_lift for x in B_before]
    return A_before, B_before, A_after, B_after

With this, we can now run a single experiment, to see what happens with CUPED:

N = 1000
before_mean = 100 
before_sigma = 50
eps_sigma = 20
treatment_lift = 2

A_before, B_before, A_after, B_after = get_AB_samples(before_mean, before_sigma, eps_sigma, treatment_lift, N)
A_after_adjusted, B_after_adjusted = get_cuped_adjusted(A_before, B_before, A_after, B_after)

print('A mean before = %05.1f, A mean after = %05.1f' % (mean(A_before), mean(A_after)))
print('B mean before = %05.1f, B mean after = %05.1f' % (mean(B_before), mean(B_after)))
print('Traditional    A/B test evaluation, lift = %.3f, p-value = %.3f' % (lift(A_after, B_after), p_value(A_after, B_after)))
print('CUPED adjusted A/B test evaluation, lift = %.3f, p-value = %.3f' % (lift(A_after_adjusted, B_after_adjusted), p_value(A_after_adjusted, B_after_adjusted)))

Yields, for example:

A mean before = 099.4, A mean after = 099.9, A mean after adjusted = 100.6
B mean before = 100.7, B mean after = 102.9, B mean after adjusted = 102.2
Traditional    A/B test evaluation, lift = 2.941, p-value = 0.214
CUPED adjusted A/B test evaluation, lift = 1.688, p-value = 0.052

So this shows that in this particular instance, the CUPED adjusted lift computed from $Y'$ (lift = 1.688) was closer to the actual lift of 2 than the traditional lift computed from $Y$ (lift = 2.941), and the p-value was lower.

If I run it a couple of times, we can also get:

A mean before = 101.8, A mean after = 102.5, A mean after adjusted = 101.9
B mean before = 100.5, B mean after = 101.7, B mean after adjusted = 102.3
Traditional    A/B test evaluation, lift = -0.854, p-value = 0.724
CUPED adjusted A/B test evaluation, lift = 0.380, p-value = 0.670

This one is quite counter-intuitive, here CUPED flipped the lift. Without CUPED, A was better than B (lift = -0.854), but with the CUPED adjustment, B is better than A (lift = 0.380). Running it a couple more times, we can also get:

A mean before = 099.5, A mean after = 099.7, A mean after adjusted = 101.6
B mean before = 103.2, B mean after = 103.4, B mean after adjusted = 101.6
Traditional    A/B test evaluation, lift = 3.743, p-value = 0.120
CUPED adjusted A/B test evaluation, lift = 0.006, p-value = 0.995

Here, CUPED adjustment made the lift estimate worse (lift = 0.006 is farther from 2 than lift = 3.743), and had a higher p-value. This is because CUPED reduces the variance on average, but not in every particular experiment instance.

We can also visualize the original $X$ and $Y$ and adjusted $Y'$ on a scatterplot:

plt.figure(figsize=(14, 7))
plt.scatter(A_before, A_after, marker='.')
plt.scatter(B_before, B_after, marker='.')
plt.scatter(A_before, A_after_adjusted, marker='.')
plt.scatter(B_before, B_after_adjusted, marker='.')
plt.xlabel('before')
plt.xlabel('after')
plt.legend(['A without adjustment', 'B without adjustment', 'A with adjustment', 'B with adjustment'])
plt.show()

On this scatterplot, the x-axis is the "before" spend, y-axis is the "after" spend, and it's showing both $Y$ vs $X$ and $Y'$ vs $X$, for A and B, so 4 groups of points. The "linear trend" is not seasonality, it's the correlation baked in the experiment setup: somebody who spent less "before", is also likely to spend less "after", which then gets removed by CUPED:

Simulating many experiments

In order to demonstrate the variance decrease $ \frac{var(Y')}{var(Y)} = 1 - corr(X, Y)^2 $, we need to run many A/B tests, store the lifts computed using both traditional A/B testing without CUPED adjustment and with CUPED adjustment, and then compute the variances. Let's run 1,000 A/B tests and record the variances:

num_simulations = 1000
traditional_lifts, adjusted_lifts = [], []
traditional_pvalues, adjusted_pvalues = [], []
for i in range(num_simulations):
    A_before, B_before, A_after, B_after = get_AB_samples(before_mean, before_sigma, eps_sigma, treatment_lift, N)
    A_after_adjusted, B_after_adjusted = get_cuped_adjusted(A_before, B_before, A_after, B_after)
    traditional_lifts.append(lift(A_after, B_after))
    adjusted_lifts.append(lift(A_after_adjusted, B_after_adjusted))
    traditional_pvalues.append(p_value(A_after, B_after))
    adjusted_pvalues.append(p_value(A_after_adjusted, B_after_adjusted))

print('Traditional    A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(traditional_lifts), cov(traditional_lifts)))
print('CUPED adjusted A/B testing, mean lift = %.2f, variance of lift = %.2f' % (mean(adjusted_lifts), cov(adjusted_lifts)))
print('CUPED lift variance / tradititional lift variance = %.2f' % (cov(adjusted_lifts)/cov(traditional_lifts)))

Prints:

Simulating 1000 A/B tests, true treatment lift is 2...
Traditional    A/B testing, mean lift = 2.13, variance of lift = 5.66
CUPED adjusted A/B testing, mean lift = 2.06, variance of lift = 0.80
CUPED lift variance / tradititional lift variance = 0.14

Over 1,000 A/B tests, the mean lift using traditional A/B testing (without using historic "before" data) was mean lift = 2.13 and the variance of the individual lifts about this mean was variance of lift = 5.66. The CUPED adjusted method was closer to the true mean of 2 with mean lift = 2.06 and had significantly lower variance of variance of lift = 0.80, a ratio of CUPED lift variance / tradititional lift variance = 0.14. I seperately verified that this ratio matches the theoretical exptected $ \frac{var(Y')}{var(Y)} = 1 - corr(X, Y)^2 $:

def expected_lift_ratio(A_before, B_before, A_after, B_after):
    cv = cov([A_after + B_after, A_before + B_before])
    corr = cv[0, 1] / (sqrt(cv[0, 0]) * sqrt(cv[1, 1]))
    return 1 - corr**2

large_N = 1000*1000
A_before, B_before, A_after, B_after = get_AB_samples(before_mean, before_sigma, eps_sigma, treatment_lift, large_N)
elr = expected_lift_ratio(A_before, B_before, A_after, B_after)
print('CUPED lift variance / tradititional lift variance = %.2f (expected = %.2f)' % (cov(adjusted_lifts)/cov(traditional_lifts), elr))

Prints:

CUPED lift variance / tradititional lift variance = 0.14 (expected = 0.14)

It does match, this is a good way to make sure there is no bug in the transformation code.

We can also plot the histogram of lifts to visualize the decrease in variance, ie. the decrease in spread:

We can also visualize the histogram of p-values to see that reduced variance goes along with lower p-values:

Counter-intuitive aspects

Whether something is counter-intuitive is subjective. Below I show what I personally find counter-intuitive.

As shown in the simulation above, CUPED reduces variance on average if "before" and "after" are correlated. So in the long-run, applying CUPED is worth it. But sometimes the adjustment yields a worse measurement, which could be confusing in practice. This can be visualized by putting the traditional $Y$ and adjusted $Y'$ lift on a scatterplot, with lines showing the true lifts:

This shows that sometimes, CUPED moved the lift from better to worse, sometimes it flips it from positive to negative or negative to positive. This is not a problem, but when you get an experiment realization like this, it could be confusing.

The traditional vs CUPED adjusted p-values can also be shown on a scatterplot:

The earlier p-value histogram shows that most of the time CUPED reduces the p-value, but this shows that sometimes it increases it (experiments above the y=x line).

No correlation

Finally, let's check what happens if "before" and "after" is not correlated at all:

def get_AB_samples_nocorr(mu, sigma, treatment_lift, N):
    A_before = list(normal(loc=mu, scale=sigma, size=N))
    B_before = list(normal(loc=mu, scale=sigma, size=N))
    A_after  = list(normal(loc=mu, scale=sigma, size=N))
    B_after  = list(normal(loc=mu, scale=sigma, size=N) + treatment_lift)
    return A_before, B_before, A_after, B_after

Per the formulas, if $cov(X, Y)=0$ then CUPED does "almost nothing" in $ Y'_i = Y_i - (X_i - \mu_X) \frac{cov(X, Y)}{var(X)} $. I say "almost", because in an experiment realization the covariance will not be exactly 0, but some small number. Simulating 1,000 A/B tests confirms this:

Simulating 1000 A/B tests, true treatment lift is 2...
Traditional    A/B testing, mean lift = 2.16, variance of lift = 5.26
CUPED adjusted A/B testing, mean lift = 2.16, variance of lift = 5.25
CUPED lift variance / tradititional lift variance = 1.00

Visually:

Conclusion

CUPED is a win. It allows us to more accurately measure experimental lift if there is correlation between "before" and "after", without degrading measurements if there is not. I plan to do more posts on CUPED:

Repeating these simulations for conversion experiments.
Exploring how trends relate to correlation.
Check what happens if a Data Scientist evaluates using both traditional and CUPED adjusted, and picks the more favorable outcome — I suspect this is not correct, I suspect it's "p-hacking".

Finally, some meta-commentary: in almost all the posts about A/B testing here on Bytepawn, I run Monte Carlo simulations to make sure I don't fool myself. The last post was an exception, I didn't run MC simulations, I argued "intuitively", and (mis)led myself down a rabbit hole. This is a great lesson: it's best to double-check my intuition with simulations, they are relatively quick to implement and run, worth it.

A/B testing and the historic lift paradox [redacted]

2021-07-25T00:00:00+02:00

This post has received very useful comments on Hacker News. Multiple commenters pointed to CUPED (by Deng, Xu, Kohavi, Walker), a method to compute an adjusted evaluation metric using historic "before" data. If "before" and "after" are correlated, which is often the case, this adjusted metric has lower variance than the original metric. In practical A/B testing terms, this means that if we apply CUPED we will on average see lower p-values, which is desirable. So the explanation in the post below, which was that "before" should never be used, is not correct! I redacted the misleading explanation to avoid confusing readers. Read this follow-up post about CUPED, which shows how to use correlations in historic data to reduce variance.

Introduction

Recently I encountered a new paradox in A/B testing. It goes something like this:

Randomly split the set of people into treatment and control and apply the treatment to the treatment group.
Measure the target metric M (eg. \$ sales per person) for control and treatment for T days: $ \alpha_C $ and $ \alpha_T $.
Using historic data, for the same groups, measure metric M for the T days preceeding the application of the treatment: $ \beta_C $ and $ \beta_T $.
Compute the "control lift" and "target lift": $ L_C = \frac{\alpha_C}{\beta_C} - 1 $ and $ L_T = \frac{\alpha_T}{\beta_T} - 1 $
The "effect lift" is then $ L = L_T - L_C $

Let's put in some example numbers: We send some marketing materials to the treatment group, with the intention of increasing their spend. Measuring on a period of 28 days, the average spend in the treatment group is \$110 per person. For the same period, it's \$105 per person in the control group. Looking at historic data, we see that in the 28 days preceeding the experiment, the spends are \$101 per person and \$100 per person for treatment and control. So the "control lift" is \$105 / \$100 - 1 = 5% and the "treatment lift" is \$110 / \$101 - 1 = 8.9%, and the "effect lift" is 8.9% - 5% = 3.9%.

Note: red means no treatment, blue means treatment applied.

What's the problem here? The simplest way to show that the logic above is wrong is to play around with the numbers a little bit. Let's change just one number, the top left number, from \$100 to \$95. This changes percentages like:

So now the "effect lift" is -1.6%, which means treatment didn't work. Right? But wait! If we look at the second row, we see that during the experiment, treatment spent \$110 per person and control spent \$105 per person, so the "experimental lift", equal to $ \frac{\alpha_T}{\alpha_C} -1 $, is 4.7%. It's a paradox! Is treatment better or is control better? What's going on?

Note that this is not just a technical nuance. If we accept -1.6%, we say treatment is worse than control, and stop it going forward. If we accept 4.7%, we say treatment is better than control, the exact opposite. So the real-life business outcomes are very different!

The Full Stack Data Scientist

2021-07-23T00:00:00+02:00

Introduction

What is a full-stack data scientist? I would argue it's a data scientist who can sustainably achieve bottom-line impact, without blocking on external help from other roles [1]. Having said that, in many situations external help from other roles is available in the form of tools, platforms or actual help, and it's perfectly fine to rely on that. But, in other situations, such as an early startup environment, or when working for a company with no established technology culture, help is not always available.

The term Data Science covers a lot of ground. Being a Data Scientist is very different in a 10 person startup versus a 100,000 person company like Google or Facebook. What I write here is based on my personal experience at small to medium sized San Francisco style startups, big tech companies like Facebook, and non-tech companies in the delivery and retail space.

In terms of tooling, fortunately there is a bewildering amount of open source and Saas offerings for Data Science and Machine Learning. No single individual can learn all the tools, that is not what "full-stack" means. I believe the right approach is judgmental vertical cutting: using your experience, judgement and taste, we have to pick a vertical slice of tooling (eg. Scala or Python, AWS or Azure, etc), stop thinking about the other 100 options [2], and get good at the tools we picked.

What are the core skills a data scientist needs to sustainably achieve bottom-line impact, without blocking on external help from other roles?

Core skills

Reading and learning. I put this first because it is a building block for the rest. Because the field is moving so fast, reading and learning is a core skill. If we also want to keep up-to-date with latest results in neural networks or reinforcement learning, we also need to be able to scan academic articles to extract relevant information [3].

Product sense. (or business sense) We need to understand the product or business, both qualitatively and quantitatively (metrics), so we can formulate ideas/hypothesis/experiments on how to improve them, eg. get more engagement, more active users, sell more tickets. Without this, however good technically, we won't know what's worth working on.

SQL. Hopefully your data already lives in an SQL database (a "data warehouse"). If not, first order of business is moving it there. In my experience, we need to spend a significant amount of our time deep-diving data in SQL, so we understands what datasets we have, data quality, delays and landing times, what the core metrics values are, etc. Also, it's a good practice to feed back results from our own work (A/B tests, tabular ML outputs) into the database, then it's easy to work on it (SQL is better than awk or pandas), share it with others, put dashboards on top.

Dashboarding. To understand data, we have to visualize it. It's just worth it to do this on dashboards, that way the visualization automatically updates when the underlying data changes. This makes sense even if we're the only ones looking at the dashboard (eg. a model perfomance dashboard).

ETL. ETL is just techno speak for "data job automation". The point is, there are some jobs, involving data, which have to be triggered once an hour or day or week. Examples:

Usually, when we're visualizing data, we're not working off the raw tables, but make smaller, derived summary tables. These have to be continually re-computed.
Machine Learning models have to be re-trained and re-deployed, usually daily or weekly.

Setting up and running an ETL scheduler like Airflow is quick and easy, it's just worth it. At tech companies, this is one of the jobs of data engineers.

Modeling. This is what people usually think about when they think Data Science: building random forests, timeseries forecasting models, training neural networks for image recognition, etc. But I suspect that for most of us, they only spend 10-20% of their time doing modeling. At big companies, there are lots of different roles (infra engineers, data engineers, ML engineers, analysts, data scientists), so each role ends up more focused on their own turf, and can safely assume the other roles will do ther part. But most companies are not big tech companies, so we have to be much more full-stack and be able to step outside of just modeling and statistics.

Experimentation and A/B testing. Experimentation is a core activity of Data Science. Whether we're running an A/B test with a product team to determine which version of a signup funnel works best, or comparing a new version of a model to the current one in production. A/B testing is a beautiful topic with unexpected depth, check out the many articles on this blog on A/B testing.

Math, probability theory, statistics. Without this, we can not make sense of the inputs and outputs of our models, run A/B tests. It's painful when people make mistakes and don't notice that correlation should be between -1 and 1, a z score of 100 is suspect, as is a MAPE of 0.1%, confuse accuracy and AUC, or don't know what a p-value means.

Programming and systems. In my experience, a full stack Data Scientist needs to have a strong software engineering background. Eg. Python is necessary to do basic modeling work in notebooks, but also to productionalize models later. Some companies will have dedicated people for productionalizing, but not at smaller shops. We have to be able to install packages and use tools such as git and Github. Also, in our daily work we inevitably hit on issues such as exceptions, out-of-memory issues. We have to be able to find and examine logs if our production model crashes.

Productionalizing, MLOps. To sustainably achieve impact, at a high velocity, we need to build an infrastructure that allows developing, deploying, training and hyperparameter tuning (possibly with AutoML) monitoring, experimenting and logging of models. MLOps is a relatively new field (outside of big tech), there are lots of options, so it's especially important to be judicious and apply taste.

Communication. Whether working with a product team or supporting other teams with analytics, insights, experimentation and decision science, we need to be credible and communicate in a clear and concise way [4].

Conclusion

In my experience, in Data Science it's best to have a software engineering background combined with a quantitative background like mathematics, physics or economics [5]. That, combined with curiousity, and never saying "that's not my job", over an extended period of time, is the way to success in Data Science.

Footnotes:

[1] Google, Stackoverflow, etc is fine of course...

[2] I will give an example what the right amount of judgmental is. If we're currently using X, and somebody comes along and shows us that Y is 10x faster or accomplishes the same thing in 10x less code, then we should switch to Y and thank the person for having taught us something. But in general, getting bogged down in "why are we using X and not Y" type discussions is a waste of time, because there are just too many Ys. If X works reasonably well, the burder of proof is on the other side to show a 10x improvement. The 10x rule I stole from the database engineering community, where it's common wisdom that if a challenger database technology wants to displace the current king, it needs to be 10x better in some relevant dimension. Otherwise, people just won't go through the trouble of replacing their core database systems.

[3] Academic articles are not read "cover to cover", especially not by practicioners. To get an article published, academics are forced to surround a core result or argument with 80% fluff. A good rule of thumb is to read the abstract, the introduction, then skip sections where they talk about previous work, and look at the core result, usually in section 3-5. Benchmarks can usually also be safely skipped, as can the conclusion, which is the same as the abstact and introduction.

[4] I don't believe in dumbing down analytics and data science and too much "storytelling". Partner roles in 2021 need to be analytics savvy.

[5] If I had to pick one, I would pick Computer Science. As part of a good computer science degree, we learn all the probability and statistics that's necessary.

Comparing NeuralProphet and Prophet for timeseries forecasting

2021-07-20T00:00:00+02:00

Introduction

In the last past I ran forcasting experiments with Prophet. In the conclusion I mentioned NeuralProphet, a Pytorch and neural network based alternative to Prophet.

An overview of the NeuralProphet architecture from the documentation:

NeuralProphet is a decomposable time series model with the components, trend, seasonality, auto-regression, special events, future regressors and lagged regressors. Future regressors are external variables which have known future values for the forecast period whereas the lagged regressors are those external variables which only have values for the observed period. Trend can be modelled either as a linear or a piece-wise linear trend by using changepoints. Seasonality is modelled using fourier terms and thus can handle multiple seasonalities for high-frequency data. Auto-regression is handled using an implementation of AR-Net, an Auto-Regressive Feed-Forward Neural Network for time series. Lagged regressors are also modelled using separate Feed-Forward Neural Networks. Future regressors and special events are both modelled as covariates of the model with dedicated coefficients.

Here I will compare Prophet and NeuralProphet forecast and runtime performance. As in the previous post, let's use a sample timeseries dataset which contains hourly energy usage data for the major US energy company American Electric Power(AEP), in megawatts. We expect this timeseries to have daily and weekly seasonality, so it's an ideal candidate for forecasting.

The notebook for this post is on Github.

Getting started with NeuralProphet

Getting started with NeuralProphet is easy, the library interfaces are similar to Prophet, though, unfortunately, not identical:

# download data
df = pd.read_csv('https://github.com/khsieh18/Time-Series/raw/master/AEP_hourly.csv')
# rename columns, NeuralProphet expects ds and y
df.columns = ['ds', 'y']
df['ds'] = df['ds'].astype('datetime64[ns]')
# keep training data
training_days = 2*365
forecast_days = 30
df = df.sort_values(['ds']).head(training_days * 24)
df.index = np.arange(0, len(df))
# train model
model = NeuralProphet(yearly_seasonality=True)
model.fit(df, freq="H")
# forecast
df_predict = model.make_future_dataframe(df, periods=forecast_days * 24)
df_predict = model.predict(df_predict)
fig = model.plot(df_predict)

Yields:

NeuralProphet vs Prophet forecast performance

Let's compare NeuralProphet vs Prophet forecast performance, in terms of Mean Absolute Percentage Error (MAPE) and running time (in seconds, training and forecasting time combined, on an 8-core 64GB Intel Macbook Pro). For both MAPE and runtime, lower is better. The following helper functions have the core logic:

def compute_mape(df, df_predict, forecast_days):
    df_cross = df.tail(forecast_days*24).merge(right=df_predict, on='ds', suffixes=['', '_predict'])
    df_cross = df_cross[['ds', 'gt', 'yhat']]
    mape = mean([2 * abs((row['gt'] - row['yhat']) / (row['gt'] + row['yhat'])) for _, row in df_cross.iterrows()])
    return mape

def prepare_dfs(csv_path, training_days, forecast_days, drop_ratio=0.0):
    df = pd.read_csv(csv_path)
    # rename columns, Prophet expects ds and y
    df.columns = ['ds', 'y']
    df['ds'] = df['ds'].astype('datetime64[ns]')
    df = df.sort_values(['ds']).head((training_days + forecast_days) * 24)
    df.index = np.arange(0, len(df))
    # save ground truth
    df['gt'] = df['y']
    # wipe target variable y for to-be-forecasted section
    for i, row in df.iterrows():
        if i >= training_days * 24:
            df.at[i, 'y'] = None
    drop_inds = [i for i in range(len(df)) if random() < drop_ratio and i < len(df) - forecast_days * 24]
    df = df.drop(df.index[drop_inds])
    df_train = df.dropna()[['ds', 'y']]
    return df, df_train

def fbprophet_test(model, csv_path, training_days, forecast_days, drop_ratio=0.0):
    df, df_train = prepare_dfs(csv_path, training_days, forecast_days, drop_ratio)
    model.fit(df_train)
    df_predict = df[['ds']]
    df_predict = model.predict(df_predict)[['ds', 'yhat']]
    return compute_mape(df, df_predict, forecast_days)

def neuralprophet_test(model, csv_path, training_days, forecast_days, drop_ratio=0.0):
    df, df_train = prepare_dfs(csv_path, training_days, forecast_days, drop_ratio)
    model.fit(df_train, freq="H")
    df_predict = model.make_future_dataframe(df_train, periods=forecast_days * 24)
    df_predict = model.predict(df_predict)[['ds', 'yhat1']].rename(columns={'yhat1': 'yhat'})
    return compute_mape(df, df_predict, forecast_days)

The differences between fbprophet_test() and neuralprophet_test() show the minor differences between the two forecasting APIs.

Let's compare training on 1, 2, 3, 4, 5 years of hourly data and forecasting on 1, 3, 6, 9, 12 months of hourly data:

df = pd.read_csv('https://github.com/khsieh18/Time-Series/raw/master/AEP_hourly.csv')
training_years = [1, 2, 3, 4, 5]
forecast_months = [1, 2, 3, 6, 9, 12]
models = [
    (lambda: Prophet(yearly_seasonality=True), fbprophet_test),
    (lambda: NeuralProphet(yearly_seasonality=True), neuralprophet_test),
]
results = []
for ty in training_years:
    for fm in forecast_months:
        for funcs in models:
            training_days = int(ty * 365)
            forecast_days = int(fm * 30)
            start = time.time()
            model = funcs[0]()
            test_func = funcs[1]
            mape = test_func(model, csv_path, training_days, forecast_days)
            elapsed = time.time() - start
            print('%s, training years=%d, forecast months=%d, MAPE = %.2f, elapsed secs = %.2f' % (model.__class__.__name__, ty, fm, mape, elapsed))
            results.append((model.__class__.__name__, ty, fm, mape, elapsed))

Plotting the results, both in terms of MAPE and runtime (seconds), lower is better for both. On all plots, the x-axis is forecasting months (from 1 to 12), on the left side plots the y-axis is MAPE (all axes go from 0 to 0.25), the right side plots show runtime seconds (not 0 grounded):

training_years = 1

training_years = 2

training_years = 3

training_years = 4

training_years = 5

Takeaways:

NeuralProphet is significantly faster than Prophet, even running on a Macbook Pro without GPU support for Pytorch
NeuralProphet has lower (better) MAPE in most cases

Handling missing data

Let's fix training years at 5 and forecast months at 6, but vary the drop ratio between 0.0 and 0.5. meaning that at most 50% of all training rows are dropped, and let's see how MAPE varies:

df = pd.read_csv('https://github.com/khsieh18/Time-Series/raw/master/AEP_hourly.csv')
ty = 5
fm = 6
drop_ratios = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5]
models = [
    (lambda: Prophet(yearly_seasonality=True), fbprophet_test),
    (lambda: NeuralProphet(yearly_seasonality=True), neuralprophet_test),
]
results2 = []
for drop_ratio in drop_ratios:
    for funcs in models:
        training_days = int(ty * 365)
        forecast_days = int(fm * 30)
        start = time.time()
        model = funcs[0]()
        test_func = funcs[1]
        mape = test_func(model, csv_path, training_days, forecast_days, drop_ratio)
        elapsed = time.time() - start
        print('%s, training years=%d, forecast months=%d, MAPE = %.2f, elapsed secs = %.2f, drop_ratio=%.1f' 
            % (model.__class__.__name__, ty, fm, mape, elapsed, drop_ratio))
        results2.append((model.__class__.__name__, ty, fm, mape, elapsed, drop_ratio))
print('\nDone!')

Yields:

Interestingly, neither Prophet or NeuralProphet is affected by dropping up to 50% of rows. NeuralProphet does a slightly better job at higher drop_ratios.

Conclusion

This toy benchmark is not conclusive, but it indicates that NeuralProphet is competitive with Prophet on MAPE, and much faster in terms of runtime. Since NeuralProphet is quite similar to Prophet and easy to use, it's worth checking out for real life production use-cases to save time and possibly gain a few MAPE points.

Timeseries forecasting with Prophet

2021-07-18T00:00:00+02:00

Introduction

Prophet, formely FBProphet, is a best-of-class timeseries forecasting library from Facebook. It is open source, released by Facebook's Core Data Science Team. At Majid Al Futtaim we use it on a regular basis. It is a "one-shot" forecasting solution, because it gives close to optimal forecasts with default arguments, without extensive parameter tweaking or feature engineering.

Here I will show how to use Prophet without going into too much detail about its architecture. For that, check out the extensive documentation and other resources:

The Jupyter notebook for this post is up on Github.

Getting started

Getting started with Prophet is easy. Let's use a sample timeseries dataset which contains hourly energy usage data for the major US energy company American Electric Power (AEP), in megawatts. We expect this timeseries to have daily and weekly seasonality, so it's an ideal candidate for forecasting:

# download data
df = pd.read_csv('https://github.com/khsieh18/Time-Series/raw/master/AEP_hourly.csv')
# rename columns, Prophet expects ds and y
df.columns = ['ds', 'y']
df['ds'] = df['ds'].astype('datetime64[ns]')
# keep training data
training_days = 2*365
forecast_days = 30
df = df.sort_values(['ds']).head((training_days + forecast_days) * 24)
df.index = np.arange(0, len(df))
# wipe target variable y for to-be-forecasted section
for i, row in df.iterrows():
    if i >= training_days * 24:
        df.at[i, 'y'] = None
df_train = df.dropna()
# train model
model = Prophet(yearly_seasonality=True)
model.fit(df_train)
# forecast
df_predict = df[['ds']]
df_predict = model.predict(df_predict)[['ds', 'yhat', 'yhat_lower', 'yhat_upper']]
fig = model.plot(df_predict)

Prophet also allows us to visually examine the forecast:

In my experience, if a "default" run of Prophet, like above, has some Mean Absolute Percentage Error (MAPE), then if we invest 1-2 weeks of custom modeling, we can find improvements of 2-3%, maybe 5% if we're lucky, but not more. So, Prophet yields very good forecasts out of the box.

Speaking about MAPE, let's adjust the code above to compare the forecast to actuals and compute the MAPE explicitly. Training on 2 years worth of data and forecasting 30 days:

# download data
df = pd.read_csv('https://github.com/khsieh18/Time-Series/raw/master/AEP_hourly.csv')
# rename columns, Prophet expects ds and y
df.columns = ['ds', 'y']
df['ds'] = df['ds'].astype('datetime64[ns]')
# keep training data
training_days = 2*365
forecast_days = 30
df = df.sort_values(['ds']).head((training_days + forecast_days) * 24)
df.index = np.arange(0, len(df))
# save ground truth
df['gt'] = df['y']
# wipe target variable y for to-be-forecasted section
for i, row in df.iterrows():
    if i >= training_days * 24:
        df.at[i, 'y'] = None
df_train = df.dropna()
# train model
model = Prophet(yearly_seasonality=True)
model.fit(df_train)
# forecast
df_predict = df[['ds']]
df_predict = model.predict(df_predict)[['ds', 'yhat', 'yhat_lower', 'yhat_upper']]
fig = model.plot(df_predict)
# join train and predict
df_cross = df.tail(forecast_days*24).merge(right=df_predict, on='ds', suffixes=['', '_predict'])
df_cross = df_cross[['ds', 'gt', 'yhat']]
mape = mean([2 * abs((row['gt'] - row['yhat']) / (row['gt'] + row['yhat']))
             for _, row in df_cross.iterrows()])
print('MAPE = %.3f' % mape)

Yields a MAPE=0.12 or 12%, which is pretty typical for real-life multi-day forecasts.

Holidays and periodicity

Internally, Prophet models the timeseries as:

$ y(t)= g(t) + s(t) + h(t) + ε(t) $

where:

$ g(t) $: growth, piecewise linear or logistic
$ s(t) $: periodic changes (e.g. daily, weekly, yearly seasonality)
$ h(t) $: holiday effects
$ ε(t) $: error term

Adding holidays is as simple as:

model.add_country_holidays(country_name='AE') # for UAE holidays

Regarding the $ h(t) $ term, most countries' holidays are correctly included out of the box, or you can add your own custom holidays.

Let's test how the periodic modeling term $ s(t) $ performs. As written above, out of the box Prophet assumes daily, weekly and yearly periodicity. Let's construct a sine wave time series which is perfectly periodic every 24 hours and fit Prophet:

forecast_days = 7
df = pd.DataFrame({'ds': pd.date_range('2020-01-01', '2020-01-31', freq='1H', closed='left')})
df['y'] = 2.0

spectrum = [1]

def f(i, row, period_day=1):
    c = 2 * pi / (period_day * 24)
    return sin(c * i)

for i, row in df.iterrows():
    gt = row['y'] + sum([f(i, row, period) for period in spectrum])
    if i < len(df) - forecast_days * 24:
        y = gt
    else:
        y = None
    df.at[i, 'y'] = y
    df.at[i, 'gt'] = gt

df_train = df.dropna()

model = Prophet()
model.fit(df_train)

df_predict = df[['ds']]
df_predict = model.predict(df_predict)[['ds', 'yhat', 'yhat_lower', 'yhat_upper']]

df_cross = df.tail(forecast_days*24).merge(right=df_predict, on='ds', suffixes=['', '_predict'])
df_cross = df_cross[['ds', 'gt', 'yhat']]

mape = mean([2 * abs((row['gt'] - row['yhat']) / (row['gt'] + row['yhat']))
             for _, row in df_cross.iterrows()])
print('MAPE = %.3f' % mape)

fig = model.plot(df_predict)

The resulting MAPE is 0, which means it's a perfect fit:

By changing the spectrum of the function, we can superimpose several sine waves. For example, we can combine a 1-day periodic wave and a 7-day periodic wave by changing one line in the above code:

spectrum = [1, 7]

This now looks more funky, but it's still modeled with MAPE=0 by Prophet:

However, Prophet does not assume other periodicities. So if we add something else to the spectrum, the model's fit will no longer be perfect:

spectrum = [1, 3, 7]

This yields MAPE=0.962, which means it's way off.

However, we can tell Prophet that we suspect our data some specific periodicity:

model.add_seasonality(name='monthly', period=3, fourier_order=3)

After inserting this line, MAPE becomes 0 again.

Random noise

To test the error term $ ε(t) $ let's add some random noise of amplitude 20% compared to the original signal of the training data:

sigma = 0.2
for i, row in df_train.iterrows():
    gt = row['y'] + sum([f(i, row, period) for period in spectrum])
    if i < len(df_train) - forecast_days * 24:
        y = gt + gauss(mu=0, sigma=sigma)
    else:
        y = None
    df_train.at[i, 'y'] = y
    df_train.at[i, 'gt'] = gt

Since Prophet explicitly models the error term, the MAPE is only 1-3% depending on the amount of training data.

Missing data

Prophet handles missing data out of the box. As a user, we don't have to write any interpolation logic. Let's continue with the sine wave toy problem, but drop 20% of the rows:

drop_ratio = 0.2
drop_inds = [i for i in range(len(df)) if random() < drop_ratio and i < len(df) - forecast_days * 24]
df = df.drop(df.index[drop_inds])

The MAPE remains 0! What's interesting is, even if I drop 90% of the rows, the MAPE is still 0. This is because the model only needs 3 points to find the daily sine wave, which is what it's assuming internally.

Other features

Other useful features of Prophet, not explored further in this post, include:

External regressors: tell Prophet that a portion of the history had some external effect (such as Covid-19) that it can regress on.
Floors and ceilings: Prophet is domain agnostic, so it doesn't know about natural or reasonable limits. For example, if the tail end of the historic data is sloping downwards, Prophet may pick up on that and create a decreasing timeseries that goes to $ - \infty $, even though that may not make sense, eg. the megawatt usage per hour cannot be less than 0. By setting floors and ceilings, we inform Prophet of these "natural" bounds.
Changepoints: Sometimes there is a distinct changepoint in the timeseries. For example, a SaaS company's daily active user (DAU) data may be flat until viral growth kicks in.

Conclusion

Prophet is a highly useful, one-shot timeseries forecasting tool which comes with defaults tuned to real-life product and business use-cases, such as daily, weekly and yearly seasonality and built-in holidays. Prophet is highly recommended for everyday production use.

If you are interested in timeseries forecasting, also check out NeuralProphet, which is inspired by Prophet but uses Pytorch and deep learning (not from Facebook), and Kats (also from Facebook), which provides a one-stop shop for time series analysis, including detection, forecasting, feature extraction and multivariate analysis.

YOLO object detection architecture

2021-07-10T00:00:00+02:00

Introduction

In the previous post I conducted simple experiments using YOLOv5. I checked how the model responds to rotation, scaling, stretching and blurring, and found that it's reasonably robust against these transformations. I briefly touched on the architecture in the Limitations section of the post:

There is also a local maximum of objects that it can detect, due to the architecture of the neural network. So if there is a large image, with a smaller section containing 100s of objects to be detected, the user will run into this local maximum constraint.

Here I will try to explain the architecture in more detail:

input-output considerations of the neural network
bounding boxes
loss function
training algorithm

Sources:

Input-output considerations of the neural network

Let's zoom out for a moment and think about object detection as an input-output problem. Let's assume the input image is is W pixels wide and H pixels high. If it's grayscale, it can be encoded as a W x H matrix of floats between 0 and 1 representing pixel intensities. If it's an RGB image, it's a 3 channel tensor, 3 x W x H, each layer encoding one color intensity. But, what about the output?

Ideally, the output should be a list of variable length, like:

object 1: confidence Z1, center X1, center Y1, width Q1, height T1, class C1 (like car, dog, cat)
object 2: confidence Z2, center X2, center Y2, width Q2, height T2, class C2
and so on, depending on how many objects are on the image...

However, neural networks cannot return a variable number of outputs. The output of a neural network, like the input, is a tensor, ie. a block of float values.

Notice that each object's information (confidence Z, center X, center Y, width Q, height T, class C) can easily be encoded as a vector, with each value between 0 and 1:

confidence as a probability
center X / image width W is a value between 0 and 1
center Y / image height H is a value between 0 and 1
object width Q / image width W is a value between 0 and 1
object height T / image height H is a value between 0 and 1
each C class gets its own float between 0 and 1, encoding the probability of the object belonging to that class (sum to 1)

So if we have a total of C classes, each detection can be encoded in a vector of 5 + C floats.

One thing we could do is set an M maximum number of objects the network can detect, like M=100 or M=1000, and have the output of the network be a tensor (matrix) of dimensions M x (5 + C), each row encoding a possible object detection. This sounds pretty simple, and similar to the limitations of this scheme, YOLO also has a global limit on number of objects. However, this is not how YOLO works.

The problem with the simple scheme above is that it would be hard to train. Imagine a training image which contains 5 cars. Which of the M rows in the output tensor do we want to give us the detection? There's a danger that certain rows get specialized, so it's always detecting cars of dogs, or detecting objects in certain regions. Then, during usage, if the input image has some other configuration, for example all cars (M cars), then some of the output rows won't be useful.

So YOLO puts a simple twist on the above scheme. It divides the image into an S x S regions, and each region is its own detector. Each region is responsible for detecting images whose bounding box is centered in that region. As before, tensors have to be of a fixed size, so each region can detect up to B objects. This way, the global maximum of objects that can be detected is S x S x B. This way, it's more clear which row is responsible for detecting which object, although there is still ambiguity, but it's greatly reduced, from S x S x B to B. In principle, by increasing S, B could be set to 1, but this is not the case:

in the original YOLOv1 architecture, S = 7 and B = 2, so a total of 7 x 7 x 2 = 98 objects can be detected. C = 20 classes of objects can be detected.
in YOLOv2, S = 13, B = 9.
in later YOLO versions, all these parameters have increased to larger numbers, and can be changed if you train your own network (S, B, C classes)

The center floats are normalized to the region dimensions, so (0, 0) corresponds to the top left corner of the region (not the entire image), (0.5, 0.5) corresponds to the center of the region, and so on.

One more change is that YOLOv1 does not detect a class for each object, it detects a class for each region. So each region can detect B objects, but each region only has one set of C floats for detecting the classes, so the B objects are all of the same class. So in the end, the output tensor of YOLOv1 is S x S x (B x 5 + C) floats. Later, in YOLOv2 this limitation has been removed, and each detection has its own class, making the output tensor S x S x (B x (5 + C)).

Note that the price we pay with this architecture is that if an input image has big blank regions, and lots of small objects clustered in a small region, the network will miss a lot of them due to the local B limit. This is what the quote from the last post was referring to.

One final step is pruning the output. Since a neural network is a "numerical beast", when using the model in production, all output floats will be non-zero, even though there is nothing there. To get rid of this noise, YOLO cuts off object detection confidence at 0.3, so anything less than 0.3 is thrown out as a non-detection.

The image above is from the original YOLO paper. Note how each region correspnds to one class (depicted by the colors). This was changed in YOLOv2. The top image shows all 98 bounding boxes, the right side shows the 3 where confidence was higher than 0.3.

Loss function and bounding box scoring during training

Imagine we're training the neural network described above. One data point in the training set is an input picture, along with a set of objects that are on the picture. Each object has a bounding box (center X, center Y, width Q , height T and class C). Suppose we then run this through the neural network. Given the architecture of the network, there is one specific region which contains the center coordinates, which has B detection boxes. So we want that region to detect the box. However, it's possible that:

multiple boxes (of the B) in that region start "finding" the object
boxes in other regions also "find" the object (with shifted bounding boxes)

The question is, how do we "score" these detections, to force the neural network to learn to detect each object once, and in the appropriate region only? In a neural network, this scoring happens through the loss function, so we have to encode our logic / intention in the loss function.

The way the YOLO loss function is set up, detections outside the correct region are deemed incorrect and ignored (zero multiplier in the loss function). Inside the correct region, the detector with the highest overlap gets scored, the rest is ignored (zero multiplier in the loss function). Another trick is to "tell" the loss function that errors in bounding box width and height matter more if the bounding box itself is small (ie. a 10 pixel width error in a 20 pixel wide box matters more than a 10 pixel width error in a 200 pixel wide box). This is accomplished by square-rooting the widths. The overall loss function from YOLOv1 is of a quasi $ L^2$ form:

The $ \lambda $ multipliers are set to balance between localization and classification error.

Training method

There are 2 main points about the training method I will point out:

The CNN portion on the deep neural network is pre-trained on ImageNet classification, so it already has some "memory" about what features to pick up. Obviously, it can also be trained on other datasets for other use-cases.
The (C)NNs are not naturally resistant to rotation, scaling, stretching and blurring. Also, they may pick up spurious features, such as a toothbrush always occuring near a face. To make the model more general, the training includes rotation, scaling, blurring, superimposing different parts of images, and so on. Starting in YOLOv4, images are also "mosaiced" together to make the detection less context-dependent.

Conclusion

The YOLO models are deep neural networks, so they're fundamentally black boxes. Like all neural networks, as of today they are more art than science. We don't exactly know why one thing works better than the other. The field is very experimental, and when something works, others copy it and start trying out further variations. YOLO is no different, the CNN portion of YOLOv1 is a variation of the earlier GoogLeNet. I say this because "understanding" a neural network is a misnomer, the best we can do is get a feeling for why, combined with the given training regimen, it works. Here I tried to explore the aspects of YOLO that are special to it, ie. the unified bounding box and classification training. I didn't talk about eg. activation functions, since in that respect there is nothing new in YOLO.

YOLOv5 object detection experiments

2021-07-02T00:00:00+02:00

Introduction

YOLOv5 is a set of pre-trained PyTorch neural network based image detection and classification models. With YOLOv5, performing image detection and classification is a couple of lines of code, whether the source is a local image, an image URL, a video or a live stream, such as a laptop's webcam. For example:

import torch
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)
results = model(['https://www.brianhonigman.com/wp-content/uploads/2015/10/Large-crowd-of-people-014.jpg'])
results.show()

Yields:

What's happening in the above code:

YOLOv5s, the Small version of the YOLOv5 pre-trained neural network is downloaded from PyTorch Hub
The image is retrieved from the URL
Object detection is run on the image
Image with multiple bounding boxes and confidences is shown

If we git clone the full YOLOv5 repo, we get helper scripts which make running the models on the laptop's webcam feed a one-liner:

$ git clone https://github.com/ultralytics/yolov5/
$ cd yolov5
$ python3 detect.py --source 0

Here is YOLOv5s correctly detecting me and some objects in the background, under low light conditions which make the image grainy:

Here I put my sunglasses close to my mouth, which fools the model into thinking it's toothbrush:

The above is running S (small) version of the network, which runs at real-time FPS. Bigger models may not be able to handle real-time depending on the hardware.

It's worth noting the difference between object classification and object detection:

In object classification, the entire image is assigned one category. For example, with MNIST digits, each image of a hand-written digit belongs to one of 10 classes. The task is not to locate the digit on the image, just to figure out which digit (if any) is most likely on the image. See this earlier post about MNIST digit detection with Pytorch.
Object detection is much harder. Here, the image potentially contains multiple objects, each belonging to a different class (eg. person, car, dog). The task is to locate each object (bounding box or pixels) and for each located object, return the most likely class.

Previous computer vision approaches consisted of multiple independent steps, for example in step 1 detect objects (for example using features such as edges and contours), whatever the object may be, then in step 2 classify the detected objects. The downside of these approaches is that the two sub-modules are trained independently, so errors cannot propagate from one to the other. Another possibility is to just take an image classifier, and run it repeatedly on sections of the image. This can be very slow as the model is run repeatedly at different coordinates for different window sizes, since this is essentially a brute-force approach.

YOLO stands for You Only Look Once. In the YOLO architecture, a deep neural network is trained to perform multiple object detection and classification in one go: find both the bounding boxes of objects on the image, and their probable class (eg. person, car, dog). The best reference for the YOLO architecture is the original 2015 paper. I will not discuss the details of the architecture in this post, just highlight that the YOLO, specifically YOLOv5 architecture is state-of-the-art as of the writing of this post:

YOLO is very popular, so there are many good tutorial and dicussions of the architecture. Good starting points are:

There are 2 option if one wants to use YOLO:

Download a pre-trained model.
Train a model yourself: the Github repo contains documentation on how to. You will need a set of labeled images (bounding boxes and classification per bounding box).

I was curious about the performance of the pre-trained models. Models, because there are difference sizes of YOLOv5 models available, from S for Small to X for Xlarge, the biggest. The smallest S model runs about 7x faster than the biggest X model, but finds less objects and is less certain of the findings. Here I will show results for S and X.

Limitations

The YOLO architecture is fundamentally a deep convolutional neural network (CNN), as illustrated by this high-level diagram from the original YOLO paper:

Although being able to download a pre-trained neural network and get pretty good object detection in 3 lines of Python is machine-learning-magic, it's worth remembering that:

The model will only detect objects it was trained to detect: for example, if the training set didn't include any turtles, then the model will never return turtles. YOLOv5 is pre-trained on the COCO dataset and knows about 80 classes.
CNNs such as YOLOv5 are not naturally scale or rotation invariant. This has to be taught at training time, for example by scaling and rotating the training images and passing them in as separate images during training.
There is a global maximum number of object YOLOv5 can detect, 300. There are also local maximums that it can detect, due to the architecture of the neural network. So if there is a large image, wit a smaller section containing 100s of objects to be detected, the user will run into this local maximum constraint.

Below I run experiments to understand how the pre-trained YOLOv5 models react to rotation, scaling, stretching and blurring. The ipython notebook is up on Github. Note that YOLOv5 cuts of confidence at 0.3, so if an object detection's confidence is less than 0.3, the object is not returned. This is why confidences lower than 0.3 ever occur on the plots below. For the experiments I use a picture of a yellow Porsche 911:

Rotation

First, I test object detection as a function of image rotation. I rotate the car 0, 1 ... 90 degrees using the Pillow image manipulation library, and run YOLOv5s and YOLOv5x on the modified image:

ms = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)
mx = torch.hub.load('ultralytics/yolov5', 'yolov5x', pretrained=True)
for angle in range(0, 91, 1):
    image_mod = image.rotate(angle)
    detections = ms([image_mod]), mx([image_mod])
    ...

Detection confidences:

It's really interesting how at some "critical" angles the S model fails (the confidence drops below 0.3), but then at higher angles it's back up again. The X model is able to detect the car at angles up to 65 degrees.

Scaling

Next, I scale the image by 0.1, 0.2 ... 1.0, ie. I make it smaller:

for scale in np.arange(0.1, 1.1, 0.1):
    image_mod = image.resize((int(width*scale), int(height*scale)), Image.ANTIALIAS)
    detections = ms([image_mod]), mx([image_mod])
    ...

Detection confidences:

At least on this clean picture, scaling is not a problem for YOLOv5, even the smallest 0.1x image is confidently detected.

Stretching

Next, I stretch the image, first in the vertical direction, then in the horizontal direction. The code is similar to the stretching case, so I just show the results:

Here, YOLOv5 also performs very well. From a stretch factor of 0.2-0.3, it's able to recognize the car.

Blurring

Next, I apply a gaussian blur of increasing stregth:

for blur in range(0, 15, 1):
    image_mod = image.filter(ImageFilter.GaussianBlur(blur))
    detections = ms([image_mod]), mx([image_mod])
    ...

Finally, I also try blurring on the picture of crowd of people shown at the beginning of the post. Since the picture contains 50+ people, instead of plotting a single confidence, I plot the number of people detected:

On the simple car picture, the model breaks down at a blur factor of about 10-12, whereas on the more realistic crown picture the starts to miss people much earlier, at lower blur factors.

Conclusion

In Dubai, privacy laws prohibit the storage of videos taken of people. Since we cannot store CCTV video feeds, we cannot label it and train our own (YOLOv5 or other) models on our own labeled images. Hence, using pre-trained models such as YOLOv5 is our only option. This is true irrespective of the intended usage of the model, even if it is privacy-neutral, for example count people entering and leaving through the door.

The above experiments show that YOLOv5 does have limitations, it is not magic in the sense of being equivalent to a human. But overall I find the performance of the pre-trained YOLOv5 models quite impressive.

Predicting party affiliation of US politicians using fasttext

2021-06-20T00:00:00+02:00

Introduction

Recently a colleague pointed me to the excellent fasttext text classification library from Facebook. The promise of fasttext is that we don't have to bother with regularizing the text, tokenizing, hand-engineering the tokens. We just feed the raw text and labels, and get a working classifier model within seconds.

There is a short 4 page paper describing the topline architecture of fasttext here. As of 2016, when the paper was written, fasttext was on-par with deep neural networks for text classification tasks. FastText: Under the Hood from Towards Data Science explains some of the internals of fasttext, I will not discuss it here.

The ipython notebook is up on Github.

Dataset

I wanted to use an interesting and relatable toy dataset, so I went with a dump of 1.2M tweets from U.S. politicians, such as Donald Trump and Adam Schiff. The dataset contains a total of 1,243,370 tweets from a total of 548 politicians.

I tried two toy exercises with this dataset:

build a binary classification model to predict the political affiliation (Democrat or Republican) of the author (ignoring Independents and Libertarians)
build a model to predict the author of a tweet (harder)

Enriching the dataset

To run the above exercises, I need to transform the tweets into a list of tuples like:

(screen_name, party, text)

Like:

(screen_name='RepAdamSchiff', party='Democrat', text='Admin review should be accompanied by thorough investigation by both House & Senate Intel Committees, perhaps meeting in joint session.')

screen_name and text is in the json files, but party is not. For that, I wrote a script to scrape each politician's Wikipedia page, which contains the party in a structured format in the page's infobox:

The function is called with the poitician's name attached to the Twitter account, which is also contained in the data dump:

def fetch_politician_info(name):
    info = {}
    info['name'] = name
    info['wiki_page'] = ''
    info['party'] = ''
    try:
        normalized_name = normalize_name(name)
        search_url = 'https://en.wikipedia.org/w/api.php?action=query&list=search&format=json&srsearch=' + normalized_name + ' american politician'
        wiki_page = requests.get(search_url).json()['query']['search'][0]['title']
        info['wiki_page'] = 'https://en.wikipedia.org/wiki/' + wiki_page.replace(' ', '_')
        wiki_info_url = 'https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&rvsection=0&redirects=1&format=json&titles=' + wiki_page
        result = requests.get(wiki_info_url).json()
        pages = result['query']['pages']
        for _, v in pages.items():
            lines = v['revisions'][0]['*'].split('\n')
            for line in lines:
                if line.startswith('| party') or line.startswith('|party'):
                    for party in parties:
                        if party.lower() in line.lower():
                            info['party'] = party
                            return info
    except:
        pass
    finally:
        return info

For example:

fetch_politician_info('Adam Schiff')
> {
>   'name':      'Adam Schiff',
>   'wiki_page': 'https://en.wikipedia.org/wiki/Adam_Schiff',
>   'party':     'Democratic'
> }

This method works for 537 out of 548 of the politicians (~98%). The remaining few I looked up manually. The final counts for the political affiliations:

Democratic:  242
Republican:  301
Independent: 4
Libertarian: 1

Limiting information content of tweets

Examining the tweets I found that some signals may make classification too simple. For example, authors often @mention each other. I didn't check this, but I assume Dems are more likely to mention other Dems, and so on. Also, tweets often contains (shorted) URLs, which may also give away the author's identity or affiliation. So I tried the classification tasks both with the full tweet text, and with the @mentions and URLs removed.

Fasttext input format

Fasttext does not take dataframes or Python lists as input. The input needs to be a file, 1 line per data point. The target label is just part of the text, with a special prefix to designate it. The default prefix is __label__.

For predicting the author, I emit lines like:

Admin review should be accompanied by thorough investigation by both House & Senate Intel Committees,
perhaps meeting in joint session __label__RepAdamSchiff

For predicting the party, I emit lines like:

Admin review should be accompanied by thorough investigation by both House & Senate Intel Committees,
perhaps meeting in joint session __label__Democratic

Train-test split

I sort the tweets by create time, and use the first 1,000,000 for training and the chronological tail, 243,370 tweets for test. This way there is no data leakage from train to test (other than at the boundary, which I ignore for this toy exercise), eg. a test tweet that is just a retweet of a training tweet.

Using fasttext

This is where fasttext shines. Ones I build the training file per the above format, it's as simple as:

model = fasttext.train_supervised(file_path)

Training the model on 1M data points takes about 5 seconds on an 8-core Intel Macbook Pro.

Once the model is trained, predicting on a piece of text is also simple:

# sample tweet from Trump
txt = """
Mike Pence didn’t have the courage to do what should have been done
to protect our Country and our Constitution, giving States a chance to
certify a corrected set of facts, not the fraudulent or inaccurate ones
which they were asked to previously certify. USA demands the truth!
""".replace('\n', ' ')
model.predict(txt)
(('__label__Republican',), array([0.79000771]))

Accuracy of the fasttext models

First, how well does fasttext do predicting the political party of the author? I only kept Dems and Reps, and re-balanced both the training and test set, so for this binary-classification exercise, the baseline accuracy is 50%. The balanced dataset has 881,460 training tweets (down from 1,000,000) and 195,804 test tweets (down from 243,370). With the original tweet texts:

Train accuracy: 92.9%
Test accuracy:  74.6%

After removing @atmentions and URLs:

Train accuracy: 85.2%
Test accuracy:  71.8%

Second, let's see how fasttext does predicting the author of the tweets? This is a C=548 classification problem, so the baseline random predictor would achieve 1/548=0.1% accuracy. Compared to this, fasttext achieves, on the original tweet texts:

Train accuracy: 51.2%
Test accuracy:  22.9%

After removing @atmentions and URLs:

Train accuracy: 41.1%
Test accuracy:  18.4%

Conclusion

Fasttext is very convenient to use, and saves a lot of time that would be spent manually regularizing, normalizing and tokenizing the text and then manually building feature vectors. In terms of runtime, it's very fast, both for training and prediction. Although I have no other baseline available for the above toy example, the accuracy fasttext achieves out of the box is promising. Predicting the full author is a hard problem, but fasttext does ~200x better than the random baseline out-of-the-box, which is impressive.

I'm a bit surprised fasttext only achieves ~75% accuracy on the binary Dem-or-Rep classification problem, I expected this to be a tractable problem, where ~90%+ accuracy is achievable. By default, fasttext does not build multi-word n-words, so I tried running it with wordNgrams=2 and 3, but it didn't improve accuracy. I may re-visit this problem in the future with a hand-built classifier to get a comparison baseline.

Random digits and Benford's law

2021-05-29T00:00:00+02:00

Introduction

This is a post exploring the distribution of digits of random and non-random numbers:

Given a set of uniform random numbers, what is the distribution of digits?
How about non-random numbers, amounts like $4.95 from receipts? How close are they to random?
Can we reproduce Benford's law with receipt digits?

Random numbers

Let's start with random numbers. First, let's generate uniform random numbers between 0 and 1000, and look at the density plot of the numbers themselves. Left side is the histogram, right side is the cumulative histogram:

This just confirms that the code used to generate random numbers is bug-free:

num_numbers = 100*1000
upper_limit = 1000
numbers = [randint(0, upper_limit - 1) for _ in range(num_numbers)]

Now, the digits of these numbers. To get the digits, I will just use str(i), like:

plt.hist(list(map(int, list(''.join(str(i) for i in numbers)))),
         density=True, edgecolor='black', bins=list(range(0, 11)))
plt.title('Density of digits of random numbers')
plt.show()

There is a deficit of 0s. This makes sense, since leading 0s are not printed with str(i), so 56 is 56 and not 056. We can "fix" this, by using '{:03d}'.format(r):

This looks right. But formatting with leading 0s doesn't neccesarily mean that the digit distribution is uniform, just because the distribution of the number is uniform — it depends on the range of the numbers. Below is the distribution of digits for even numbers drawn from different ranges 200, 500, 1000:

Finally, let's see the distribution of first digits (without leading 0s):

Amounts from receipts

Let's leave random numbers behind, and look at amounts from real-world receipts from Dubai. Similarly as above, I cut off numbers at 1000. First, let's look at the distribution:

Next, let's look at the distribution of digits, with amounts formatted like 4.90, with 2 decimals, without leading 0s:

Unsurprisingly, 0 is the most frequent digit, because the decimals are often 00 or x0. What if we cut off the trailing 0s, so 4.90 becomes 4.9:

If we look at it like this, 0 is the least frequent digit, and 5 is the most frequent digit. To get a better feeling for the decimal part, we can plot the density of just the decimal numbers, so 90 from 4.90.

This shows that prices are likely to be multiples of 5s (like 4.00, 4.05, 4.10 ... 4.95), 00 and 50 being the most likely. Interestingly, prices are not likely to end in 9 or 99 for this dataset.

Finally, the density of digits of the decimal parts, confirming that 5 is the most likely non-zero digit:

Plotting just the last digit:

Benford's law

Next, let's check Benford's law:

Benford's law, also called the Newcomb–Benford law, the law of anomalous numbers, or the first-digit law, is an observation about the frequency distribution of leading digits in many real-life sets of numerical data... It has been shown that this result applies to a wide variety of data sets, including electricity bills, street addresses, stock prices, house prices, population numbers, death rates, lengths of rivers, and physical and mathematical constants. A set of numbers is said to satisfy Benford's law if the leading digit d occurs with probability:

$ P(d) = log_{10} (1 + 1/d) $ $ d ∈ {1, ..., 9} $

Plotting the distribution of first digits of our amounts, with Benford's law:

Note that first digits from uniform random numbers did not follow Benford's law, but our "real" numbers do. Interestingly, the same distributions also hold for the total amounts from receipts.

Conclusion

Digits from uniform random numbers follow a truncated-uniform distribution, depending on the formatting and upper bound. Digits from real sources like receipts and zip codes follow some artificial pattern that depends on the source, for example here the amounts tend to be multiples of 5s. Benford's law hold holds for 1st digits of "real" numbers.

Tricks vs implementation in coding interviews

2021-05-22T00:00:00+02:00

Recently I bought the book Daily Coding Problem. I bought it because:

I find these sorts of problems intellectually pleasing
it's a good way to keep myself sharp as a programmer
perhaps I can re-purpose some of the problems for Data Science interviews.

The problems and solutions in the book reminded me of my own interviewing experiences, as a candidate: I get nervous in interview situations and don't do well on coding interviews. Doctors call this the white coat effect:

The alerting reaction to the physician's visit is known to induce a blood pressure rise termed "white coat effect." This phenomenon has often been associated with a clinical condition characterized by a persistently high blood pressure in the doctor's office and a persistently normal blood pressure at other times.

In other words, the measurement ("how well can this person program") result is below the actual truth. Let's look at an example:

Given an array of numbers, find the maximum sum of any contiguous subarray of the array.

I think any candidate for a programming job can be expected to code up the $ O(n^3) $ brute force solution, but I'm not sure I would figure out the more efficient $ O(n) $ solution known as Kadane's algorithm, with my blood pressure at 160-170, instead of my normal 135, in 3-5 minutes flat:

def max_sum(arr):
    max_sum = float('-inf')
    cur_sum = 0
    for x in arr:
        cur_sum = max(0, cur_sum + x)
        max_sum = max(max_sum, cur_sum)
    return max_sum

So as a hiring manager, I think a lot about how to make hiring loops less painful and more fair. It occured to me that potentially a better interviewing approach would be to:

Ask the candidate to figure out the naive / brute force solution on their own, and code it up.
Then, tell them verbally what the optimal solution is, and have them code it up.

Why?

Removing the step of the candidate having to think up tricky solutions would remove significant pressure.
It still tests the most important think: ability to implement without errors and write clean code.

The example above is not a good example, because Kadane's algorithm, once written down, is so simple, it's hard to "tell it" without effectively giving the candidate the solution itself. A better example is:

Given an array of numbers, find the bounds of the smallest window that must be sorted in order for the whole array to be sorted. For example, for the input [3, 7, 5, 6, 9], return [1, 3].

So:

Candidate gives the naive / brute force solution: copy and sort the array, then compare with the original. Time complexity is $ O(n log(n) ) $.
Tell the candidate that: Traverse the array and note whether the element is less than the maximum up to that point. If it is, this element would have to be part of the sorting window, since it's out of order compared to that previous maximum. Same in the reverse direction. Time complexity is $ O(n) $.

Implementing the algorithm is still not trivial in an interview setting:

def min_window(arr):
    left, right = None, None
    max_seen, min_seen = float('-inf'), float('inf')

    for i in range(len(arr)):
        max_seen = max(max_seen, arr[i])
        if arr[i] < max_seen:
            right = i

    for i in range(n - 1, -1, -1):
        min_seen = min(min_seen, arr[i])
        if arr[i] > min_seen:
            left = i

    return left, right

This seems like a better interviewing approach, because in a work setting, thinking up tricky solutions in 3-5 minutes is not a requirement. Usually, there are days or weeks for that. But implementing an idea, once the idea is there, should be straightforward for a good programmer.

10 ways to iterate from 0 to 1 with deciles

2021-05-14T00:00:00+02:00

Suppose we want to iterate from 0 to 1, in steps of 0.1, like 0.0, 0.1, 0.2, ... 0.9. This happens regularly in Data Science notebooks.

Note that iteration like this is usually exclusive at the end, ie. we expect it to stop at 0.9, the same way range(0, 10) stops at 9.

Approach #1: Use range, divide manually:

for i in range(10):
    print(i/10)

> 0.0
> 0.1
> 0.2
> 0.3
> 0.4
> 0.5
> 0.6
> 0.7
> 0.8
> 0.9

The downside of this basic solution is that the iteration logic leaks into the print() code. Also what if we want to iterate with steps of 0.42:

for i in range(10):
    print(i*0.42)

In real code, it would be less clear that this is actually a clean iteration, and not some other reason we're multiplying.

Approach #2: Use arange from numpy

The overall best practice is to use arange from numpy:

import numpy as np

for i in np.arange(0, 1, 0.1):
    print(i)

> 0.0
> 0.1
> 0.2
> 0.30000000000000004
> 0.4
> 0.5
> 0.6000000000000001
> 0.7000000000000001
> 0.8
> 0.9

The only downside here is that arange creates the whole array in memory, instead of yielding a Python iterator. This is a non-issue in real-life, but let's geek out and see what the options are to avoid creating the whole list/array in memory.

Approach #3: Use count() from itertools

The closest built-in helper function Python has is count() from itertools. You supply start and step arguments, but it counts infinitely, so you have to take care of the stopping condition:

from itertools import count

for i in count(0, 0.1):  # yields 11 times, a logical error!
    if i >= 1: break
    print(i)

> 0
> 0.1
> 0.2
> 0.30000000000000004
> 0.4
> 0.5
> 0.6
> 0.7
> 0.7999999999999999
> 0.8999999999999999
> 0.9999999999999999 <- due to floating point precision issues, it also returns ~1

This is very ugly because of the if, but also, it has a big logical error: it yields a value 11 times instead of 10 times, because due to floating point precision issues, it also returns ~1, and doesn't stop at ~0.9. We'll return to floating point issues in a bit.

Approach #4: Use count() and islice() from itertools

We can use islice to return the first int((stop-start)/step) elements from the count() iterator. This saves us the ugly manual if.

from itertools import count, islice

def frange(start, stop, step):
    return islice(count(start, step), start, int((stop-start)/step))

for i in frange(0, 1, 0.1):
    print(i)

> 0
> 0.1
> 0.2
> 0.30000000000000004
> 0.4
> 0.5
> 0.6
> 0.7
> 0.7999999999999999
> 0.8999999999999999

In this case the floats work out in our favor and it yields 10 elements.

Approach #5: Use map()

Let's be functional and use map():

for i in map(lambda x: x / 10, range(10)):
    print(i)

> 0.0
> 0.1
> 0.2
> 0.3
> 0.4
> 0.5
> 0.6
> 0.7
> 0.8
> 0.9

Approach #6: Use map() in a cleaner way

We can put the map in a function to make it cleaner:

def frange(start, stop, step):
    return map(lambda x: start + x * step, range(int((stop-start)/step)))

for i in frange(0, 1, 0.1):
    print(i)

> 0.0
> 0.1
> 0.2
> 0.30000000000000004
> 0.4
> 0.5
> 0.6000000000000001
> 0.7000000000000001
> 0.8
> 0.9

Approach #7: Write it yourself

Or we can just implement frange ourselves:

def frange(start, stop, step):
    while start < stop:
        yield start
        start += step

for i in frange(0, 1, 0.1): # yields 11 times, a logical error!
    print(i)

> 0
> 0.1
> 0.2
> > 0.30000000000000004
> 0.4
> 0.5
> 0.6
> 0.7
> 0.7999999999999999
> 0.8999999999999999
> 0.9999999999999999 <- due to floating point precision issues, it also returns ~1

Approach #8: Epsilons

Some of the approaches had logical problems, because they yielded 11 times, also yielding ~1, due to floating point issues (the exact case when this is a problem is platform and argument dependent). One way to address this is to assume floating point precicion problems and explicitly handle ε differences to some user-specified precision. But this is such a bad idea, I will stop here.

Approach #9: Use decimal

The smart way to get rid of the floating point issues above is to use the Python standard decimal library:

from decimal import *

for i in frange(Decimal('0'), Decimal('1'), Decimal('0.1')):
    print(i)

> 0
> 0.1
> 0.2
> 0.3
> 0.4
> 0.5
> 0.6
> 0.7
> 0.8
> 0.9

Notice that the arguments to the Decimal constructor are strings, not numbers. Passing in numbers yields:

for i in frange(Decimal(0), Decimal(1), Decimal(0.1)):
    print(i)

> 0
> 0.1000000000000000055511151231
> 0.2000000000000000111022302462
> 0.3000000000000000166533453693
> 0.4000000000000000222044604924
> 0.5000000000000000277555756155
> 0.6000000000000000333066907386
> 0.7000000000000000388578058617
> 0.8000000000000000444089209848
> 0.9000000000000000499600361079

Note that you can set the precision for the decimal library to cut off the ugly trailing noise, and then you can avoid using ugly strings in the constructor:

getcontext().prec = 6

for i in frange(Decimal(0), Decimal(1), Decimal(0.1)):
    print(i)

> 0
> 0.100000
> 0.200000
> 0.300000
> 0.400000
> 0.500000
> 0.600000
> 0.700000
> 0.800000
> 0.900000

Approach #10: Alternative signatures

The floating point troubles point to the idea that frange(start, stop, step) is maybe not the best signature, frange(start, step, count) is cleaner:

def frange(start, step, num):
    for i in range(num):
        yield start
        start += step

This way, we never have to worry about floating point issues yielding an extra value, even without using decimal:

for i in frange(start=0, step=0.1, num=10):
    print(i)

> 0
> 0.1
> 0.2
> 0.30000000000000004
> 0.4
> 0.5
> 0.6
> 0.7
> 0.7999999999999999
> 0.8999999999999999

Also, here backward iteration works as expected, unlike in some of the solutions given before:

for i in frange(start=0, step=-0.1, num=10):
    print(i)

> 0
> -0.1
> -0.2
> -0.30000000000000004
> -0.4
> -0.5
> -0.6
> -0.7
> -0.7999999999999999
> -0.8999999999999999

Or:

def frange(start, step, num, backward=False):
    if backward:
        step *= -1
    for i in range(num):
        yield start
        start += step

The downside here is that if we want to switch stepsize from 0.1 to 0.05, we have to remember to also change num to get to 1.

There is also a third possible signature, linspace from numpy:

numpy.linspace(start, stop, num=50, endpoint=True, ...): return evenly spaced numbers over a specified interval.

Conclusion

In real-life, my recommendation:

for start, stop, step signature use numpy.arange()
for start, stop, num signature use numpy.linspace()
for start, step, num signature use the frange() above.

Sometimes brute forcing just works

2021-05-06T00:00:00+02:00

Introduction

Recently I was working on the problem of "parsing" store receipts. The customer takes a picture of the receipt using their smartphone and submits it, and it is our job to extract certain core fields from the receipt, such as the total amount or the date of the receipt. We had an existing solution which used a state of the art, but not receipt specific OCR tool to convert the image to a chunk of text, and then used various regular expressions to extract these fields --- however the extraction rate and accuracy only 70-80% for each extracted field. I started working on this as a weekend hack to see whether I can improve the accuracy of the existing solution.

I will describe how I cracked extracting the date field, where a surprisingly simple brute force search base approach worked, without any machine learning. It's a good lesson to remember that sometimes relatively simple searching and string manipulation works, and we don't need to go to gradient boosting or neural networks.

It's fair to ask: isn't this a solved problem? Aren't there out of the box solutions for receipt parsing? I thought the same thing, so as a zeroth step we looked at 3-4 different SaaS offerings, but they had very poor recognition rates (in the 50-60% range), possibly because of our arabic text, and because we don't currently enforce quality standards on the uploaded images (eg. many are blurry, etc).

Brute force search

Since I was hacking, I decided to first stick with the existing OCR approach, before moving on to more fancy deep learning on the image itself. First I did "data exploration", which in this case means looking at a lot of the receipt pictures and the corresponding OCR text to get a sense of the problem. I was able to do this, because the existing solution has been in production for a while, so we have more than a million labeled samples (there is a team of humans correcting and labeling low confidence results).

An additional difficulty here was that most of the invoices also had arabic text on them, which tripped off the OCR software, so sometimes it switched from left-to-right to right-to-left word ordering in the text output. Also, the OCR software is not specific to invoices, so sometimes it got confused and thought the invoice is like a column of text in a newspaper, and parsed out the line items ("t-shirt") on the left of the invoice as a block of text, and then all the amounts on the right. This means that the resulting OCR string was very "dirty", very hard to "read" and see patterns in it.

LANDMARK, RETAIL INVESTMENT CO. LLC مركة داند مارلا وپتيل للاستثمار ذ
م Hone Box Sharjah City Center Sharjah 800HOMEBOX الكاشي : 173157
:Cashier بیان :173157 :Sales ۲۲۰۱۶-۲۱/۱۴ ۹۶/۶۷۴۹/۳۰۹۰۳۰/۲۰۶/۱۱۱
23-04-21/12:45/4739/29030/204/111 : : : TRN: 100260641400003 ۱۰۰۳۶۰۶۱۹۰۰۰۰۳ 
: وتم تسجيل ضريبة القيمة هم ببیة فاتورة TAX INVOICE 29 0 3 0 2 0 4 4 7 3 9 2 0 2 1 0 4 2 2 
۹۰۳۰۳۰ ۷۳ ۲۱۰۳۲ الصنف العدل السعر Iten Quantity Rate Value-AED وو؟ ۳۹۹۱۹۹۷۰۳۹۹۳ ۴X 
6299169703493 2 X Knit Basket 3.3L w/o Lid , 4.00 8.00 سنة تخز بین 
HH Excl. VAT Unit Price = 3.91 غير شامل ضريبة القيمة المضافة ۳,۹۱ = سعر الوحدة مفرد مائدة - 
P -Blaza - Beaded Placetat ۶۴۹۹۱۹۹۹:۱۶ ۱۷ ۱۰,۰۰ 5299149955164 1X 10.00 P . 
Excl. VAT Unit Price = 9.52 10.00 غير شامل ضريبة القيمة المضاة ۹,۵۲ = سعر الوحدة إجمالي الكمية 
Total Quantity :: : الاجمالي Total : 18.00 نقدی Cash مجمل المناقصة 100.00 و ۱۰۰
Total Tender Change Due : 100.00 المتبقي المستحق : ۸۳,۰۰۰ -82.00 اجمالي الفواتي 
Tax Summary رمز الضرائب المعفاة الضريبة ضريبة الدخل Code غربیبة ه 
Ex Tax ۱۷,۱ 17.14 Tax ۰,۸۹ 0.86 ,۸۶ 0.86 Inc Tax ۱۸,۰۰ 18.00 AD ۱۴ ,۱۷ الإجمالي
Total 17.14 18.00 معدلات الضرائب Tax Rates = %: 5% 5: % 5 % :5%AD = ضري 5 % معدل تباسي من أن %5 AD :
Standard Rate of عفري معدل :أو 04_ :40 خارج النطاق : أو 00s : 
Out of Scope معني من الضرائب : أكسر EX : Sax Exempt Excl. 
VAT Unit Price : غير شامل فر بيبة القيمة المضافة سعر الوحدة

Looking at the invoices, I noticed:

Almost every one had the date in a different format, from 2020-12-24 to 12-24-2020 to 2020 Dec'24 to December 24, 2020.
Because the text was the result of OCR, and the pictures were not clear, little characters like , . ' were not reliably parsed.
However, putting aside the dirtyness of the little characters, once I found the date on the invoice image, I was usually able to find it in the OCR text.
A big "insight" was that the date of the receipt was always very close to the date of submission (=the date when the user took the picture and uploaded it), and the date of submission is known at parse time.

So I tried a simple thing. I made a list of 10-20 Python format strings for datetime.strftime(), like %Y-%m-%d, and tried the following logic:

def extract_date(text):
    formats = ['%Y-%m-%d', ...]
    previous_days = self.get_previous_days(submission_day, num_days)
    # previous_days is ordered to start at submission_day, and go back in time
    for dt in previous_days:
        for fmt in formats:
            ds = dt.strftime(fmt)
            if ds in text:
                return dt # found it!
    return None

To my big surprise, this worked very well. For the cases where it returned something (not None), it correctly found the date at 95%+ accuracy!

So I went further down this path, and started adding more and more date formats. I quickly realized that instead of manually generating the date formats, I should have the computer do that, too. So I wrote a simple loop to generate lots of possible date formats, with different orderings, dividers and y/m/d formats:

year_formats = ['%Y', '%y']
month_formats = ['%m', '%b', '%B']
day_formats = ['%d']
dividers = ['-', '/', '\\', ...]
orderings = ['Y-M-D', 'D-M-Y', 'M-D-Y']
# generate all possible format strings and test
formats = []
for fmt, divider, year_format, month_format, day_format in product(orderings, dividers, year_formats, month_formats, day_formats):
    fmt = fmt.replace('-', divider)
    fmt = fmt.replace('Y', year_format)
    fmt = fmt.replace('M', month_format)
    fmt = fmt.replace('D', day_format)
    formats.append(fmt)

This approach continued to work well. Eventually I was at ~80% recognition rate, still at ~95% accuracy. At this point I was finding receipts which had formats which I couldn't generate with datetime.strftime(), so I came up with a minimal wrapper around it, which let me add my own formattings. And finally, for the final few cases I found, I manually extended the generated date format list. Overall, in the end I had a list of 100s of date formats to try.

Finally, add to this some lower/uppercasing, logic to handle cases when multiple candidates are found, and some logic to handle cases when spaces and small characters are lost in the OCR process, I arrived at a very competetive solution.

Conclusion

This brute force, search based method makes a prediction for ~97% of cases, and gets ~95% accuracy, outperforming our current solution by a very wide margin. Because of the repeated string searches (Python substring in string in a loop) it's a bit slow, but it's not worth it to optimize, since it can still process 1000s of requests per second, but we only need to process around 1-10 per second — so optimizing the string searches or rewriting in C++ would be premature optimization.

Building intuition for p-values and statistical significance

2021-04-25T00:00:00+02:00

Introduction

This is the transcript of a talk I did on experimentation and A/B testing to give the audience an intuitive understanding of p-values and statistical significance.

The format of the talk was a short introduction, then live-coding an ipython notebook. The initial, short snippets I typed out to make it interesting, for the later parts I switched to an existing notebook to keep the talk's momentum going.

I opened the talk by saying that in experimentation Data Scientists have 3 jobs. To (1) design (2) run, and (3) evaluate experiments. This talk is about the evaluation phase, and can be summed up as "don’t get fooled by randomness".

Coin flips

Let's build our intuitive understanding of p-values and statistical significance by considering coin flips. Whenever I think about A/B testing, in my head I'm secretly thinking about coin flips.

Suppose somebody gives us a coin, and we suspect it's biased. What can we do? As an experimentalist, what we do is we start flipping it and record the outcomes.

Suppose we flip this coin 10 times, and we get 7 heads. What can we say about the coin?

To see how usual or unusual this is, let's conduct many coinflipping experiments, and count. However, instead of doing it for real, let's use the computer's built-in random number generator:

random()

> 0.6316696366339524

So if we call random(), it returns a random float number between 0 and 1. Using this, we can simulate a fair coin:

# let's simulate fair coin flips
def coinflip():
    if random() < 0.5:
        return 'H'
    else:
        return 'T'

Let's try it. If we call it repeatedly, it returns H or T, fifty-fifty:

coinflip()

> 'H'

Okay, now let's change one little thing. Let's agree to use 1 instead of H and 0 instead of T:

# let's simulate fair coin flips
def coinflip():
    if random() < 0.5:
        return 1
    else:
        return 0

This will make our life easier in subsequent steps. Now, let's flip our virtual coin 10 times:

# 10 coinflips
[coinflip() for _ in range(10)]

> [1, 0, 1, 0, 0, 0, 1, 0, 1, 1]

Since we're using 1 for heads, we can just add up the results to get how many heads we had:

sum([coinflip() for _ in range(10)])

> 5

Let's create a shorthand for this:

def coinflips(num_flips=10):
    return sum([coinflip() for _ in range(num_flips)])

coinflips(10)

> 5

Okay, now it gets interesting. Let'd conduct this 10-flip experiment 10 times:

[coinflips(10) for _ in range(10)]

> [6, 5, 2, 5, 6, 4, 4, 3, 6, 5]

So, the first time we got 6 heads out of 10, the second time 5, then just 2 out of 10, and so on.

Presentation note: this is where I switch from typing to running an existing notebook.

Let's count how many times we 1 heads out of 10, 2 heads out of 10, 3 heads out of 10, if we repeat the 10-flip experiment many times:

NUM_FLIPS = 10
NUM_EXPERIMENTS = 10
outcomes = [coinflips(NUM_FLIPS) for _ in range(NUM_EXPERIMENTS)]
count = defaultdict(lambda: 0)
for outcome in outcomes:
    count[outcome] += 1
for i in range(NUM_FLIPS+1):
    print('Out of {} experiments, we flipped {} heads {} times'.format(
        NUM_EXPERIMENTS, i, count[i]))

> Out of 10 experiments, we wlipped 0 heads  0 times
> Out of 10 experiments, we wlipped 1 heads  0 times
> Out of 10 experiments, we wlipped 2 heads  1 times
> Out of 10 experiments, we wlipped 3 heads  2 times
> Out of 10 experiments, we wlipped 4 heads  3 times
> Out of 10 experiments, we wlipped 5 heads  2 times
> Out of 10 experiments, we wlipped 6 heads  1 times
> Out of 10 experiments, we wlipped 7 heads  0 times
> Out of 10 experiments, we wlipped 8 heads  1 times
> Out of 10 experiments, we wlipped 9 heads  0 times
> Out of 10 experiments, we wlipped 10 heads 0 times

Let's increase the number of experiments to get more counts and more representative statistics. Since we're using our computer's random number generator, we can do it a million times in about a second:

NUM_FLIPS = 10
NUM_EXPERIMENTS = 1000*1000  <---
outcomes = [coinflips(NUM_FLIPS) for _ in range(NUM_EXPERIMENTS)]
count = defaultdict(lambda: 0)
for outcome in outcomes:
    count[outcome] += 1
for i in range(NUM_FLIPS+1):
    print('Out of {} experiments, we flipped {} heads {} times'.format(
        NUM_EXPERIMENTS, i, count[i]))

> Out of 1000000 experiments, we flipped 0 heads  1004 times
> Out of 1000000 experiments, we flipped 1 heads  9719 times
> Out of 1000000 experiments, we flipped 2 heads  44091 times
> Out of 1000000 experiments, we flipped 3 heads  116981 times
> Out of 1000000 experiments, we flipped 4 heads  206072 times
> Out of 1000000 experiments, we flipped 5 heads  245371 times
> Out of 1000000 experiments, we flipped 6 heads  205059 times
> Out of 1000000 experiments, we flipped 7 heads  117282 times  <---
> Out of 1000000 experiments, we flipped 8 heads  43764 times
> Out of 1000000 experiments, we flipped 9 heads  9675 times
> Out of 1000000 experiments, we flipped 10 heads 982 times

Going back to our initial question, what can we say about a coin if we get 7 heads out of 10? From the above, we can see that if repeat this 10-flip experiment with a fair coin a million times, we actually get 7 heads out of 10 about 117,000 times. Let's just do one more change, instead of showing raw counts, let's divide them by NUM_EXPERIMENTS = 1000*1000 to get percentages:

NUM_FLIPS = 10
NUM_EXPERIMENTS = 1000*1000
outcomes = [coinflips(NUM_FLIPS) for _ in range(NUM_EXPERIMENTS)]
count = defaultdict(lambda: 0)
for outcome in outcomes:
    count[outcome] += 1
for i in range(NUM_FLIPS+1):
    print('Out of {} experiments, we flipped {} heads {:.2f}% of the time'.format(
        NUM_EXPERIMENTS, i, 100*count[i]/NUM_EXPERIMENTS))

> Out of 1000000 experiments, we flipped 0 heads  0.10% of the time
> Out of 1000000 experiments, we flipped 1 heads  0.99% of the time
> Out of 1000000 experiments, we flipped 2 heads  4.40% of the time
> Out of 1000000 experiments, we flipped 3 heads  11.71% of the time
> Out of 1000000 experiments, we flipped 4 heads  20.53% of the time
> Out of 1000000 experiments, we flipped 5 heads  24.59% of the time
> Out of 1000000 experiments, we flipped 6 heads  20.51% of the time
> Out of 1000000 experiments, we flipped 7 heads  11.70% of the time  <---
> Out of 1000000 experiments, we flipped 8 heads  4.39% of the time
> Out of 1000000 experiments, we flipped 9 heads  0.97% of the time
> Out of 1000000 experiments, we flipped 10 heads 0.10% of the time

Just as with the counts, we see that if repeat this 10-flip experiment with a fair coin many times, we actually get 7 heads out of 10 about 11.7% of the time!

What is the p-value

Given an experimental outcome: "we flip 7 heads out of 10", assume the "boring", "non-action" case (the statistical term is "null hypothesis") = "the coin is fair, it's not biased".
Compute the probability of the experimental outcome (7 heads) in the "boring" case (=11.69%)
.. and also add to it the probabilities of even-more-extreme outcomes (8, 9, 10 heads, which is + 4.36% + 0.98% + 0.10%) = 17.13%

$ p = 0.17 $

In summary: in real life, we don't know if the coin is fair or not. We just know we flipped 7 heads out of 10. What we know from the above is, if we assume it's fair, we would see this outcome (7 heads), or more extreme outcomes (8, 9, 10 heads out of 10), a combined 17% of the time. That is not very unlikely, about 1 in 6 odds.

Comment: I've skipped 1-sided vs 2-sided testing, as it's not critical for building the core intuition.

What is statistical significance

Is the above statistically significant? What does "statistically significant" mean?

Statistical significance is a way to make a decision based on an experiment. It's a pre-agreed upon, set in protocol, $p_{critical}$ threshold for the p-value, so that if $p < p_{critical}$, we declare the result "statistically significant" and we reject the "boring" null hypothesis, and we accept the alternative, "action" hypothesis.

In our example, the "boring" hypothesis is that the "coin is fair", rejecting it would mean we conclude the "coin is biased". Since we calculated $p=0.17$, assuming we are working with $p_{critical} = 0.05$, we would not reject the "boring" null hypothesis. We would not reject the hypothesis that the coin is biased.

So statistical significance is just a human-agreement for decision-making. Usually $p_{critical} = 0.05$ or 5% or 1 in 20 odds, but this is completely arbitrary. There is nothing special or "right" about 5%. We can agree that we will use 1% or 10% or 20% in our experiments.

From the above we can see that for 10 coin flips, if our $p_{critical}$ is 5%, we'd need to get 0, 1 or 9, 10 heads to conclude that the coin is biased.

Notes:

At this point I explained that if we pick a p-value of 5%, that means on average, of the times when we reject the null hypothesis, we will be wrong 1 in 20 times. Depending on the audience, this may be too much and confuse them.
When I presented I had a question about how to pick $p_{critical}$. Here's a dedicated blog post I wrote about how to pick $p_{critical}$ and how to balance experimentation velocity vs being sure: A/B tests: Moving Fast vs Being Sure.

Sample size

We saw that getting 7 out of 10 flips, so 70% heads, is not that unusual. Is 70 heads out of 100 flips also not unusual? Let's check:

NUM_FLIPS = 100  <---
NUM_EXPERIMENTS = 100*1000
outcomes = [coinflips(NUM_FLIPS) for _ in range(NUM_EXPERIMENTS)]
count = defaultdict(lambda: 0)
for outcome in outcomes:
    count[outcome] += 1
for i in range(NUM_FLIPS+1):
    print('Out of {} experiments, we flipped {} heads {:.3f}% of the time'.format(
        NUM_EXPERIMENTS, i, 100*count[i]/NUM_EXPERIMENTS))

> ...
> Out of 100000 experiments, we flipped 70 heads 0.001% of the time
> ...

What we see here is that 70% only happens 0.001% of the time, very rarely! And if we add the %s of for 71, 72 .. 100, it's still a very small number. So with 100 coin flips, a 70% heads (or more) outcome yields a much-much smaller p-value than the 17% we had for 10 coin flips.

This is the effect of sample size! N=100 coin flips is a lot more than N=10 coin flips. Telling apart a biased and a fair coin is much easier if you flip more and more. If we flip a coin 100 times and get 70 heads, then it is very unlikely that this coin is fair.

Note: the notebook also has plots the bell curves, but I don't think it helped the audience.

A/B testing

Okay, now that we have built our intuition with coin flips, let's talk about a marketing A/B test. Suppose we have a control (A) and a treatment (B) version of an email. Maybe A is text-only, while B also has some colorful images. Let's say we send it out to N=1000 customers each. We wait one week, look at the logs, and the measurement outcome is that A converts at 10% and B converts at 12%.

What can we say? It's actually the same thing as coin flips! Except here, the "fair" base coin is not a 50-50 coin, it's the 10% conversion we see in our control group A.

Note: here I'm decreasing rigor to make the explanation flow better. The gloss over the fact that 10% conversion of control is also a sampled result, so it's not the same as the hypothetical fair coin, which has no uncertainty.

Let's build a "control (A)" coin:

def coinflip():
    if random() < 0.10: <--- 0.10 instead of 0.50
        return 1
    else:
        return 0

coinflips(100)

> 11

Now, let's see our experiment:

NUM_FLIPS = 1000
NUM_EXPERIMENTS = 10*1000
outcomes = [coinflips(NUM_FLIPS) for _ in range(NUM_EXPERIMENTS)]
count = defaultdict(lambda: 0)
for outcome in outcomes:
    count[outcome] += 1
for i in range(NUM_FLIPS+1):
    print('Out of {} experiments, we flipped {} heads {:.2f}% of the time'.format(
        NUM_EXPERIMENTS, i, 100*count[i]/NUM_EXPERIMENTS))

> ...
> Out of 10000 experiments, we flipped 120 heads 0.52% of the time
> Out of 10000 experiments, we flipped 121 heads 0.38% of the time
> Out of 10000 experiments, we flipped 122 heads 0.26% of the time
> Out of 10000 experiments, we flipped 123 heads 0.18% of the time
> Out of 10000 experiments, we flipped 124 heads 0.20% of the time
> Out of 10000 experiments, we flipped 125 heads 0.11% of the time
> Out of 10000 experiments, we flipped 126 heads 0.13% of the time
> Out of 10000 experiments, we flipped 127 heads 0.11% of the time
> Out of 10000 experiments, we flipped 128 heads 0.09% of the time
> Out of 10000 experiments, we flipped 129 heads 0.04% of the time
> Out of 10000 experiments, we flipped 130 heads 0.03% of the time
> ...

Let's change one thing, instead of saying "120 heads", let's change our vocabulary, divide by 1000, and talk about conversion, like 120/1000=12%:

NUM_FLIPS = 1000
NUM_EXPERIMENTS = 10*1000
outcomes = [coinflips(NUM_FLIPS) for _ in range(NUM_EXPERIMENTS)]
count = defaultdict(lambda: 0)
for outcome in outcomes:
    count[outcome] += 1
for i in range(NUM_FLIPS+1):
    print('Out of {} experiments, we had {:.2f}% conversion {:.2f}% of the time'.format(
        NUM_EXPERIMENTS, 100*i/NUM_FLIPS, 100*count[i]/NUM_EXPERIMENTS))

> ...
> Out of 10000 experiments, we had 12.00% conversion 0.48% of the time
> Out of 10000 experiments, we had 12.10% conversion 0.27% of the time
> Out of 10000 experiments, we had 12.20% conversion 0.41% of the time
> Out of 10000 experiments, we had 12.30% conversion 0.22% of the time
> Out of 10000 experiments, we had 12.40% conversion 0.18% of the time
> Out of 10000 experiments, we had 12.50% conversion 0.14% of the time
> Out of 10000 experiments, we had 12.60% conversion 0.11% of the time
> Out of 10000 experiments, we had 12.70% conversion 0.08% of the time
> Out of 10000 experiments, we had 12.80% conversion 0.07% of the time
> Out of 10000 experiments, we had 12.90% conversion 0.04% of the time
> Out of 10000 experiments, we had 13.00% conversion 0.03% of the time
> ...

Let's add up these "12% or more conversion cases":

print('p_value = ' + str(100*sum([count[i] for i in range(int(NUM_FLIPS*0.12), NUM_FLIPS)])/NUM_EXPERIMENTS) + '%')

> p_value = 2.13%

So if we assume that treatment's (B's) true, intrinsic conversion is also 10%, just like control's (A's) (=this is the "boring" null hypothesis), then we'd expect to see 12% or more conversion, just by chance, 2.13% of the time. If we were using $p_{critical} = 5%%$, we'd conclude that this is "statistically significant", we'd reject the "boring" hypothesis that A and B are the same, and we'd go with the "action" hypothesis: in the future, we will send the B variant to all users --- hence the name "action" hypothesis, it means we take some action.

Conclusion

I close the talk by talking briefly about the importance of having a culture of experimentation, that the most important thing is to actually run experiments, repeatedly, rigorously. I make the point of saying this is more important than statistical rigor itself. First is, run a lot of experiments, and then worry about p-values. I quote Jeff Bezos:

“Our success at Amazon is a function of how many experiments we run per year, per month, per week, per day.” - Jeff Bezos

A really good 5-minute pitch on experimentation culture is this Youtube video. Depending on how much time is left, some points can be lifted from here, like:

firsts organizations have to learn how to experiment
most experiments don't yield wins
eventual big wins pay for many losers.

Note: stressing experimentation culture may or may not be a good investment of time in a talk. For example, in big established companies, culture is usually top-down, so it only makes sense spending significant time on this if the audience includes top management.

Happy experimenting!

Random numbers, the natural logarithm and higher dimensional simplexes

2021-04-17T00:00:00+02:00

Introduction

This is a fun post to exercise our brain and practice college math.

Let's play a game: draw a uniform random number between 0 and 1. What is the expected value?

from random import random

def once(n=1000*1000):
    sum = 0.0
    for _ in range(n):
        sum += random()
    return sum/n
print(once())

> 0.50023

As expected, it's 0.5.

Okay, what about if we draw two numbers?

def twice(n=1000*1000):
    sum = 0.0
    for _ in range(n):
        sum += random()
        sum += random()
    return sum/n
print(twice())

>  1.00003

As expected, it's 1.0.

Okay, now let's turn it on its head: draw random numbers between 0 and 1, and stop when the sum exceeds 1. What is the expected number of draws?

def stop_at(z=1.0, n=1000*1000):
    total_draws = 0.0
    for _ in range(n):
        trial_sum = 0.0
        num_draws = 0
        while True:
            trial_sum += random()
            num_draws += 1
            if trial_sum >= z: break
        total_draws += num_draws
    return total_draws/n
print(stop_at(1))

> 2.71758

This returns $ e $, the base of the natural logarithm! Why does it show up here?

Practicing college math

As a reminder, the definition of $ e $:

$ e = 1 + \dfrac{1}{1!} + \dfrac{1}{2!} + \dfrac{1}{3!} + \dfrac{1}{4!} + ... = 1 + \sum\limits_{n=1}^{\infty} \dfrac{1}{n!} $

Why is this the expected value of the draws?

Let $ N $ denote the random variable which denotes the number of draws required to cross 1.0. The expected value:

$ E(N) = \sum\limits_{k=2}^{\infty}{ k \times P(N=k)} $

The sum starts at $ k = 2 $ because we need at least 2 draws, $ P(N=k) = 0 $ for $ k = 1 $.

The first step is to realize that $ P(N=k) $, the probability that we stop at $k$, is the probability that the sum is less than $1$ at $k-1$ draws, and exceeds $1$ after the $k$-th draw. Let $U_i$ denote the $i$-th draw, and let $S_k$ denote the sum up to $k$, $ S_k = \sum\limits_{i=1}^{k}{ U_i } $.

Then:

$ P(N=k) = P( S_{k-1} < 1 \land S_k \geq 1 ) $

Now some elementary probability theory:

$ P(A) = P(A \land B) + P( \land \neg B) $

With $ A = S_{k-1} < 1 $ and $ B = S_{k} \geq 1 $:

$ P(S_{k-1} < 1) = P( S_{k-1} < 1 \land S_k \geq 1 ) + P( S_{k-1} < 1 \land S_k < 1 ) $

Obviously, $ P( S_{k-1} < 1 \land S_k < 1 ) = P( S_k < 1 ) $, so:

$ P(S_{k-1} < 1) = P( S_{k-1} < 1 \land S_k \geq 1 ) + P( S_k < 1 ) $

Moving terms:

$ P( S_{k-1} < 1 \land S_k \geq 1 ) = P(S_{k-1} < 1) - P( S_k < 1 ) $

So:

$ P(N=k) = P( S_{k-1} < 1 \land S_k \geq 1 ) = P(S_{k-1} < 1) - P( S_k < 1 ) $

Now we just need to figure out the formula for $ P( S_k < 1 ) $.

This is easiest solved geometrically. For $k=2$, $ P( S_k < 1 ) $ is the triangle area inside the x-y unit square where $x+y < 1$, which is clearly $1/2$. For $k=3$, $ P( S_k < 1 ) $ is the simplex volume inside the x-y-z unit square where $ x+y+z < 1 $. The simplex has as its base the previous triangle defined by $x+y < 1$, and has unit height.

The general formula for the area of triangles (volumes of simplexes) with unit height in $k$ dimensions is $ V = A \times k $, where A is the area (volume) of the base, which is the same formula for $k-1$. Inductively, this means that:

$ P( S_k < 1 ) = \dfrac{1}{k!}$

We have the factorial from the definition of $e$! Now we apply a simple identity of the factorial, $ k! = k \times (k-1)! $ to our formula:

$ P(N=k) = P(S_{k-1} < 1) - P( S_k < 1 ) = \dfrac{1}{(k-1)!} - \dfrac{1}{k!} = \dfrac{k}{k \times (k-1)!} - \dfrac{1}{k!} = \dfrac{k}{k!} - \dfrac{1}{k!} = \dfrac{k-1}{k!} $

We're almost done:

$ E(N) = \sum\limits_{k=2}^{\infty}{ k \times P(N=k)} = \sum\limits_{k=2}^{\infty}{ k \times \dfrac{k-1}{k!} } = \sum\limits_{k=2}^{\infty}{ \dfrac{k-1}{(k-1)!} } $

Let $j = k-1$:

$ E(N) = \sum\limits_{j=1}^{\infty}{ \dfrac{j}{j!} } = 1 + \sum\limits_{j=2}^{\infty}{ \dfrac{j}{j!} } = 1 + \sum\limits_{j=2}^{\infty}{ \dfrac{1}{(j-1)!} } $

Let $n = j-1$:

$ E(N) = 1 + \sum\limits_{j=2}^{\infty}{ \dfrac{1}{(j-1)!} } = 1 + \sum\limits_{n=1}^{\infty}{ \dfrac{1}{n!} } = e $

Varying the stopping threshold

We saw above that if we stop at $z=1$, we need an average of $e$ draws. What if we want to stop at $z=2$ or $z=3.14$? What is the $E(N_z)$, the average number of draws required if we want to stop at $z$?

import matplotlib.pyplot as plt

x = [x/10 for x in list(range(10, 50, 1))]
y = [stop_at(i, n=100*1000) for i in x]
fig, ax = plt.subplots()
ax.plot(x, y, marker='o')
ax.set(xlabel='z', ylabel='average draws',
       title='E(N_z)')
ax.grid()
plt.show()

This is intuitive: if we need $e$ draws to reach $z$, and each additional draws averages $0.5$, so $E(N_z) = f(z) = e + 2(z-1)$.

Sometimes it's good to exercise our brain (and $\LaTeX$ skills) to stay sharp!

The ipython notebook is here.

Classification accuracy of quantized Autoencoders with Pytorch and MNIST

2021-04-09T00:00:00+02:00

Introduction

In the previous post I investigated the information contents, in bits, that Autoencoders store. I took MNIST digits, trained a simple Autoencoder neural network to first encode the pictures in 4..256 dimensions, where each vector element is a float32. Then I took the elements, and quantized them to 2..32 bits accuracy, and measured the reconstruction loss as a function of encoding dimension and quantization:

Based on the plot, my conclusion was:

... 512 bits --- which corresponds to 12x (lossy) compression --- is a good trade-off, or 1024 bits for 10% less loss. Loss does not decrease significantly after 1024 bits, that appears to be best the Autoencoder can accomplish.

However, this conclusion is just eye-balled based on the plots leveling off. The y-axis here is loss, which is related to pixel distance between original and reconstruction, but it's actually hard to know what a loss of ~300 really means. Also, this loss is not straightforward pixel distance, since it's standard practice to re-scale and normalize MNIST images to help the neural networks:

tv.datasets.MNIST(..., transform=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,)), <--- image pixels are re-scaled
]))

A better approach to quantify the "loss" is to run a classifier on the output and see how recognizable the digits are in terms of Accuracy %.

The notebook is up on Github.

Experiment setup

The MNIST dataset contains a total of 70,000 digits. Usually the first 60,000 are used for training, the remaining 10,000 for testing. Here, I will use:

30,000 for training a CNN classifier, as in this earlier post
30,000 for training a Autoencoder, like in the previous post
10,000 for testing, how the classifier performs on quantized digits coming from the Autoencoder

Then, I follow the same strategy as previously, but instead of using loss as a metric, I run the quantized Autoencoder's output through the classifier, and plot the accuracy as a as a function of encoding dimension and quantization.

Code

First, some helpers to split the dataset into 3 parts:

class MNISTPartialTrainDataset(Dataset):
    def __init__(self, root, num_samples=None, offset=0):
        self.base_dataset = tv.datasets.MNIST(root=root, train=True, transform=transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,)),
        ]))
        if num_samples is None:
            self.num_samples = len(self.base_dataset)
        else:
            self.num_samples = min(num_samples, len(self.base_dataset)-offset)
        self.offset = offset
    def __len__(self):
        return self.num_samples
    def __getitem__(self, idx):
        return self.base_dataset[self.offset+idx]

class MNISTPartialTestDataset(Dataset):
    def __init__(self, root, num_samples=None, offset=0):
        self.base_dataset = tv.datasets.MNIST(root=root, train=False, transform=transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307,), (0.3081,)),
        ]))
        if num_samples is None:
            self.num_samples = len(self.base_dataset)
        else:
            self.num_samples = min(num_samples, len(self.base_dataset)-offset)
        self.offset = offset
    def __len__(self):
        return self.num_samples
    def __getitem__(self, idx):
        return self.base_dataset[self.offset+idx]

The classifier is a vanilla CNN:

class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        self.model = nn.Sequential(
            nn.Conv2d(1, 20, 5, 1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),
            nn.Conv2d(20, 50, 5, 1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),
            nn.Flatten(),
            nn.Linear(4*4*50, 500),
            nn.ReLU(),
            nn.Linear(500, 10),
            nn.LogSoftmax(),
            )
    def forward(self, x):
        return self.model(x)

We train the classifier on the first 30,000 MNIST digits:

classifier = Classifier().to(device)
distance = nn.NLLLoss()
optimizer = torch.optim.SGD(classifier.parameters(), lr=0.01, momentum=0.5)
num_epochs = 20
for epoch in range(num_epochs):
    for data, target in classifier_dataloader:
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = classifier(data)
        loss = distance(output, target)
        loss.backward()
        optimizer.step()
    print('epoch {:d}/{:d}, loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()), end='\r')

correct = 0
classifier = classifier.to('cpu')
for data, target in classifier_dataloader:
    output = classifier(data)
    pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability
    correct += pred.eq(target.view_as(pred)).sum().item()
print('Classifier train accuracy: {:.1f}%'.format(100*correct/len(classifier_dataloader.dataset)))

The classifier achieves 99.8% train accuracy, as expected:

Classifier train accuracy: 99.8%

Now the Autoencoder, exactly the same as before:

class Autoencoder(nn.Module):
    def __init__(self, encoding_dims):
        super(Autoencoder,self).__init__()
        self.encoder = nn.Sequential(
            nn.Flatten(),
            nn.Linear(img_dims*img_dims, encoding_dims),
            nn.ReLU(),
            )
        self.decoder = nn.Sequential(
            nn.Linear(encoding_dims, img_dims*img_dims),
            nn.Unflatten(1, (1, img_dims, img_dims)),
            nn.Sigmoid(),
            )
    def forward(self, x, quantize_bits=None):
        x = self.encoder(x)
        x = torch.sigmoid(x)
        if quantize_bits is not None:
            x = round_bits(x, quantize_bits)
        x = torch.logit(x, eps=0.001)
        x = self.decoder(x)
        return x

The main loop is very similar to before, except there is an additional step, where the trained classifier is run on the Autoencoder's output digits:

results = []
for encoding_dims in [4, 8, 16, 32, 64, 128, 256]:
    # train autoencoder, code omitted
    ...
    # test
    for quantize_bits in [None, 2, 4, 8, 16, 32]:
        correct = 0
        autoencoder = autoencoder.to('cpu')
        for imgs, labels in autoencoder_test_dataloader:
            ae_imgs = autoencoder(imgs, quantize_bits=quantize_bits)
            output = classifier(ae_imgs)
            pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability
            correct += pred.eq(labels.view_as(pred)).sum().item()
        accuracy = correct/len(autoencoder_test_dataloader.dataset)
        results.append((encoding_dims, quantize_bits, accuracy))

Results

The results can be plotted to show the accuracy of the classifier per encoding_dims, per quantize_bits:

Alternatively we can plot total_bits = encoding_dims * quantize_bits on the x-axis:

The plots re-affirm what I read off the previous plots, that

each float32 in the encoding stores around 8 bits of useful information (out of 32), since all of the curves flatten out after 8 bits
128 dimensions is the maximum required, since the next jump to 256 yield no significant increase in accuracy
overall, based on these curves, encodim_dims = 64 and quantize_bits = 8 appears to be a good trade-off (total_bits = 64*8 = 512 bits)

However, now it's easier to quantify:

at encodim_dims = 64 and quantize_bits = 8 (total_bits = 64*8 = 512 bits), the classifier achieves 97% accuracy, which is still great
at higher total_bits, and with no quantization at all, the classifier achieves 98% on these unseen, encoded-than-decoded images, which is the same 98% we get without no quantization at all

Finally, to visually see what happens when we autoencode and quantize, a sample of the Autoencoder's output, for encoding_dims=64 and various quantization levels:

encodim_dims = 64, quantize_bits = 2 -> total_bits = 128, accuracy = 60%:

encodim_dims = 64, quantize_bits = 4, total_bits = 256, accuracy = 90%:

encodim_dims = 64, quantize_bits = 8, total_bits = 512, accuracy = 97%:

encodim_dims = 64, quantize_bits = 16, total_bits = 1024, accuracy = 98%:

encodim_dims = 64, quantize_bits = 32, total_bits = 2048, accuracy = 98%:

encodim_dims = 64, not quantized, accuracy = 98%:

More images here.

Investigating information storage in quantized Autoencoders with Pytorch and MNIST

2021-04-04T00:00:00+02:00

Introduction

In this experiment I wanted to understand the compression ratio of Autoencoders: how much information (how many bits) does an Autoencoder encode in the encoding dimensions? Let's say an autoencoder is able to encode a 28x28 grayscale MNIST image (28x28x8 bits = 6272 bits) in a 32 dimensional encoding space with acceptable reconstruction loss. What is the compression ratio? With a CUDA/GPU, those 32 dimensions are actually 32 float32's, so it's 32x32 = 1024 bits, which corresponds to 6.1x (lossy) compression. But are all those 1024 bits really needed? Intuitively the entire float32 space is probably not used. A related question is, what is the "right" number of encoding dimensions to pick for Autoencoders?

The notebook is up on Github.

Experiment setup

To answer these questions, I took a simple Autoencoder neural network with a Linear+ReLu encoder and a Linear+Sigmoid decoder layer. Since I will want to quantize the bits between the encoder and a decoder, I use the sigmoid() function to get the encoder's output to be between 0 and 1, and then the inverse, the logit() function before feeding back to the decoder.

Code

The code is a straightforward Autoencoder neural network implemented in Pytorch, with some additional transformations in the forward() function to implement quantization. The arrows mark the departure from a vanilla Autoencoder:

class Autoencoder(nn.Module):
    def __init__(self, encoding_dims):
        super(Autoencoder,self).__init__()
        self.encoder = nn.Sequential(
            nn.Flatten(),
            nn.Linear(img_dims*img_dims, encoding_dims),
            nn.ReLU(),
            )
        self.decoder = nn.Sequential(
            nn.Linear(encoding_dims, img_dims*img_dims),
            nn.Unflatten(1, (1, img_dims, img_dims)),
            nn.Sigmoid(),
            )
    def forward(self, x, quantize_bits=None):
        x = self.encoder(x)
        x = torch.sigmoid(x)                   <--- sigmoid
        if quantize_bits is not None:          <--- if not training
            x = round_bits(x, quantize_bits)   <--- .. then quantize the encoding
        x = torch.logit(x, eps=0.001)          <--- logit = inverse sigmoid
        x = self.decoder(x)
        return x

The function round_bits() quantizes the input number to 2**quantize_bits levels between 0 and 1:

def round_bits(x, quantize_bits):
    mul = 2**quantize_bits
    x = x * mul
    x = torch.floor(x)
    x = x / mul
    return x

The main training loop trains the Autoencoder for different encoding_dims, and then tests the reconstruction loss for various values of quantize_bits:

for encoding_dims in [4, 8, 16, 32, 64, 128, 256]:
    # train
    autoencoder = Autoencoder(encoding_dims=encoding_dims).to(device)
    distance = nn.BCELoss()
    optimizer = torch.optim.Adam(autoencoder.parameters(), lr=0.001)
    num_epochs = 50
    for epoch in range(num_epochs):
        for imgs, _ in autoencoder_train_dataloader:
            imgs = Variable(imgs).to(device)
            output = autoencoder(imgs)
            loss = distance(output, imgs)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
    # test
    distance = nn.MSELoss()
    for quantize_bits in [2, 4, 8, 16, 32]:
        loss = 0
        for imgs, _ in autoencoder_train_dataloader:
            imgs = Variable(imgs).to(device)
            with torch.no_grad():
                output = autoencoder(imgs, quantize_bits=quantize_bits)
            loss += distance(output, imgs)

Results

The results can be plotted to show the loss per encoding_dims, per quantize_bits:

The plot shows that:

each float32 in the encoding stores around 8 bits of useful information (out of 32), since all of the curves flatten out after 8 bits
128 dimensions is the maximum required, since the next jump to 256 yield no significant decrease in loss
overall, based on these curves, encodim_dims = 64 and quantize_bits = 8 appears to be a good trade-off (total_bits = 64*8 = 512 bits)

Alternatively we can plot total_bits = encoding_dims * quantize_bits on the x-axis:

This re-affirms that 512 bits --- which corresponds to 12x (lossy) compression --- is a good trade-off, or 1024 bits for 10% less loss. Loss does not decrease significantly after 1024 bits, that appears to be best the Autoencoder can accomplish. For reference, the entire MNIST training dataset, uncompressed is 28*28*8 * 60*1000 / 8 = 47,040,000 bytes. After gzip compression, the file size is 9,912,422 bytes, for a lossless compression ratio of 4.7x.

In the next post, I will explore what we lose with the Autoencoder's lossy compression in terms of recognizability of the digits.

Building a Pytorch Autoencoder for MNIST digits

2021-03-18T00:00:00+01:00

Introduction

In the previous posts I was training GANs to auto-generate synthetic MNIST digits:

Here I take a step back to a simpler idea from unsupervised learning, Autoencoders. The idea is simple: take the input, reduce the dimensionality toward the middle of the deep neural network in the encoder part, and then restore the original dimensionality in the second, decoder part. The network is then trained to attempt to restore the original input, which in this case is MNIST digits, with minimal loss.

The Wikipedia explanation:

An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. Along with the reduction side, a reconstructing side is learned, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input, hence its name.

In this post, I will try to build an Autoencoder in Pytorch, where the middle "encoded" layer is exactly 10 neurons wide. My assumption is that the best way to encode an MNIST digit is for the encoder to learn to classify digits, and then for the decoder to generate an average image of a digit for each. The ipython notebook is here.

Code

I will only show the relevant parts of the code. First, the Autoencoder model, split into an encoder and a decoder:

class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder,self).__init__()
        self.encoder = nn.Sequential(
            # 28 x 28
            nn.Conv2d(1, 4, kernel_size=5),
            # 4 x 24 x 24
            nn.ReLU(True),
            nn.Conv2d(4, 8, kernel_size=5),
            nn.ReLU(True),
            # 8 x 20 x 20 = 3200
            nn.Flatten(),
            nn.Linear(3200, 10),
            # 10
            nn.Softmax(),
            )
        self.decoder = nn.Sequential(
            # 10
            nn.Linear(10, 400),
            # 400
            nn.ReLU(True),
            nn.Linear(400, 4000),
            # 4000
            nn.ReLU(True),
            nn.Unflatten(1, (10, 20, 20)),
            # 10 x 20 x 20
            nn.ConvTranspose2d(10, 10, kernel_size=5),
            # 24 x 24
            nn.ConvTranspose2d(10, 1, kernel_size=5),
            # 28 x 28
            nn.Sigmoid(),
            )
    def forward(self, x):
        enc = self.encoder(x)
        dec = self.decoder(enc)
        return dec

Note that the forward() pass is decoder(encoder(x)), the only reason I separated the 2 parts of the network is that so I can peek at the the middle, encoded part, to see how it's encoding the digits.

The training loop:

distance = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

for epoch in range(num_epochs):
    for data in dataloader:
        img, _ = data
        img = Variable(img).cpu()
        output = model(img)
        loss = distance(output, img)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print('epoch [{}/{}], loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

Evaluation

Let's compare the activation of the 10 encoded neurons to the true labels of unseen MNIST test data:

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Autoencoder().to(device)
testset = tv.datasets.MNIST(root='./data',  train=False, download=True, transform=transform)
dataloader = torch.utils.data.DataLoader(testset, batch_size=10*1000, shuffle=True, num_workers=4)

confusion_matrix = np.zeros((10, 10))
for data in dataloader:
    img, labels = data[0].to(device), data[1].to(device) 
    imgs = Variable(imgs).cpu()
    encs = model.encoder(imgs).detach().numpy()
    for i in range(len(encs)):
        predicted = np.argmax(encs[i])
        actual = labels[i]
        confusion_matrix[actual][predicted] += 1

Note that the confusion matrix here is not a classical confusion matrix in the sense that it doesn't necessarily have to be an identity matrix, since the encoded neurons ordering is arbitrary. In the best case, the resulting matrix can be row-column-reordered and normalized to be an identity matrix (assuming balanced data).

Plotting the confusion matrix, where rows are the actual digits (0-9) and columns are the indices of the most active neuron (0-9):

For example, for row 6 (actuall the seventh, since numbering starts at 0), we can see that column 8 is the most active. And conversely, most of the time column 8 is active is for row 6. So the model learned to tell 6s apart from the rest, with some accuracy.

However, this unsupervised classifier is very far from perfect. For example, column 3 activates for actuals 3, 5, and 8. That's no surprise, since these digits actually look alike. To gauge how successful the encoding is, we can pick, for each endoding, the most likely actual value:

classifier = {
    # enc -> predict
    0: 0,
    1: 2,
    2: 2,
    3: 3,
    4: 2,
    5: 5,
    6: 9,
    7: 2,
    8: 6,
    9: 1,
}

If we run a prediction model on the encoder with this mapping, we get an accuracy of 40%. It's much worse than the 99% than is achievable with supervised learning, but it's much better than the 10% we'd expect from random chance.

Conclusion

This was an experiment in seeing whether an autoencoder can learn, without supervision, to recognize the 10 digits in MNIST, since that would probably be an optimal encoding --- assuming the images for each digit class are not too different in the MNIST data, so that encoding in that way is optimal. The result is worse than I expected, even after I asked for help on Stackoverflow and I merged those solutions with my own. I suspect with more tweaking of the network and the training loop, at least 2/3 accuracy could be achieved. If I can improve this model significantly, I will re-visit this topic in a future post.