Better be first 99% of the time than second 100% of the time

Introduction

This post is a review of the Donald MacKenzie's book Trading at the Speed of Light, which gives an excellent history and inside-peek of the world of High Frequency Trading, or HFT. Donald MacKenzie is a Professor of Sociology at the University of Edinburgh, Scotland, his main focus is the social studies of finance.

I've been interested in HFT for 20 years, and found this book interesting and a pleasure to read — highly recommended!

I took notes on the book and then used ChatGPT to generate most of the text in this article.

The Evolution of Trading Systems

The book delves deep into the history of trading systems, tracing the journey from pit trading to electronic platforms. Pit trading involved open outcry, where traders shouted orders and clerks recorded them in physical order books. Electronic trading emerged alongside, but the transition was far from smooth. It required overcoming cultural, technical, and regulatory hurdles, such as exchanges resisting algorithmic trading and early HFT firms resorting to robotic arms to punch keys as modern electronic API were not available.

Though HFT generates estimated revenues of \$3-5 billion in the U.S., it's a high-cost, slim-margin game. Even a large HFT firm making about \$1 billion in revenue faces extensive costs, making profitability precarious. The difference between success and failure often comes down to fractions of a cent per trade.

The books shows that today's HFT landscape is a culmination of history, physics, computation, politics, market structures and laws. Despite its efficiency, HFT has not significantly lowered the cost of financial transactions for retail investors. Regulatory bodies like the SEC and CFTC play vital roles but are often navigating uncharted waters.

Early Pioneers

One of the first electronic exchanges was Island ECN, introduced around 1998. Unlike its predecessors like Instinet, Island had an algorithmic matching engine. Early HFT firms like Automated Trading Desk (ATD) were among the first to realize the importance of milliseconds and started to move their servers closer to the Island's data centers to gain a speed advantage. ATD was bought by Citigroup for $680M in 2007. For a list of HFT firms today, see here.

Financial Evolution and Market Politics

The evolution of trading systems was politically fraught. Each step of automation led to job losses, inciting resistance from incumbents. For example, share prices initially quantized to 1/8 dollars to ensure market makers could maintain a wider bid-ask spread, thus securing profits: market makers offered to buy a stock at \$100, and offered to sell at \$100+1/8 (or more), since that was the minimum quantum of the market. When this changed to finer-grained price units of \$0.01, legacy exchanges and market makers lost business to more modern competitors.

Market makers create liquidity for other market sellers by always having buy and sell orders in the market book. Thus, if a seller $S$ wants to sell at time $T$, and a buyer $B$ wants to buy some later time $T'$, they don't have to wait for each other, since the market maker will buy from $S$ at time $T$, and later sell to $B$ at time $T'$, for a "small profit" related to the bid-ask spread, which itself is a function of the market quantum.

The Order Book and Matching Engine

A critical component of trading systems is the matching engine, responsible for pairing buy and sell orders. The evolution has led from human-operated matching engines to fully algorithmic systems. With algorithmic matching, an open order book became a possibility, allowing market participants to view buy and sell orders at various prices. An open order book could either be anonymous or not, affecting strategies and politics among trading entities.

The non-anonymity of some open order books presents a unique challenge for HFT firms. They need to be cautious not to make too much money off big firms like big banks. Banks, having the leverage to influence exchange policies, can block trades if they suspect an HFT firm is disproportionately benefiting from them. Therefore, HFT firms often limit their gains from a single identifiable firm.

High-Frequency Signals and Strategies

HFT relies heavily on signals like futures lead, where price changes in futures markets (the Chicago Mercantile Exchange, located in Chicago) often precede similar movements in the underlying asset markets (NYSE and NASDAQ, located in New York). Other signals include order book dynamics, market fragmentation, and correlated share price movement. HFT algorithms trade primarily with each other, as well as with execution algorithms that break down large institutional orders into smaller child orders.

HFT activities are concentrated in about 25 global data centers, with four major ones in the U.S., an Equities Triangle in New Jersey consisting of Nasdaq, NYSE, and NY4/5, and the Chicago Mercantile Exchange. Connections between these data centers, such as CME-NYC, become lifelines for HFT firms, who pay a premium for the fastest possible links.

Hugging the Geodesic

Connectivity has always been at the core of HFT. The first fiber optic Gold Line from the CME datacenter to New Jersey was a significant milestone, but companies like Spread Networks invested hundreds of millions to lay cables that more closely "hugged the geodesic" to reduce latency. Spread Networks pushed the limit to 6.65ms, a significant improvement from the previous 8ms but still above the Einstein limit of 3.94ms. AB Services revolutionized the game by introducing microwave links that could perform at 3.98ms, nearly meeting the theoretical limit.

Laying these microwave links wasn't easy. For example, one of the main challenges was crossing 80km across Lake Michigan. The solution was to shoot signals from skyscraper tops across the lake. Many firms switched to higher frequencies, up to 23Ghz, which are more vulnerable to environmental conditions like fog or rain. Yet, in HFT, it's "better to be first 99% of the time than second 100% of the time."

As HFT operations expanded, the race extended within cities like New Jersey. In these densely populated areas, millimeter wave technology replaced microwaves due to bandwidth requirements. AOptics introduced a dual-mode millimeter plus laser technology, each with its susceptibilities balanced by the other, to combat issues like rain and fog.

The Physicality of Computers in HFT

CME's datacenter in Chicago serves as a linchpin in the complex web of HFT. The facility moved from its original location in Cermak in 2012, which was originally a printing plant, to a more advanced, separate facility. Cermak could handle 100MW of electricity, all eventually transformed into heat — a concern when power density is a crucial metric in datacenter operations. Security is stringent, with most doors protected by biometrics.

In the world of HFT, even the physical attributes of a computer become crucial. Programmers must consider the physics of computation, focusing on elements like wires, photons and electrons, instructions, and data packets, to minimize latency. This brings in concepts from physics, emphasizing the literal speed of light as a limiting factor in trade execution times. Initially, firms relied on C++ programming for a speed advantage, but the bar has been raised significantly. To remain competitive today, HFT operations must be executed on Field-Programmable Gate Arrays (FPGAs). This has brought down response time to as low as 42 nanoseconds.

Two interesting strategies that try to beat others to be first are scout orders and speculative triggering. Small scout orders (a loss worth it) are used to detect price changes microseconds before the competiton, and then trade on it. Speculative triggering is a technique where HFT firms employ custom packet processing in FPGAs. This approach preemptively guesses the correct trading action based on incoming bits and initiates the sending of response bits even before the full incoming packet has been processed. If the guess turns out to be incorrect, the final bits of the response checksum are deliberately altered to ensure that the exchange's network stack discards the packet.

Market Structure, Regulation, and Effects

The datacenter houses key components such as trading firms' systems, the exchange's order gateway, and matching engine. When this engine finds a matching pair of orders from two firms, it sends a fill message. If there's no match, a confirm message is returned. Trading firms also have the flexibility to send cancel or modify messages to their existing orders.

Trading in HFT is a "cat and mouse game," where firms try to decode the logic behind other trading algorithms to anticipate their moves and trap them for profit. This involves offline analysis of massive amounts of anonymized market data to statistically identify algorithmic behaviors. One possible strategy is market impact trading, where the algorithm itself places orders that influence market prices. Firms anticipate the reactions of other algorithms to these price moves and exploit them before canceling their original orders. Sweeping involves selling against existing bids to lower prices, causing smaller firms to liquidate their holdings, which are then purchased by the sweeper at the lower price. Finally, spoofing entails sending large sell offers above the market price, along with a smaller buy order at a lower price. This temporary imbalance drives the price down, enabling the buy order to succeed; the original spoof orders are then canceled, and the market readjusts. These strategies often blur ethical lines and are subject to regulatory scrutiny, as they can manipulate market dynamics to the advantage of HFT firms.

Exchanges have attempted to level the playing field by introducing protective mechanisms like a minimum quote lifespan of 250ms to counteract spoofing. Large market participants often have the ability to lobby exchanges to block entities that engage in predatory trading activities against them.

Conclusion

The world of high-frequency trading is a high-stakes game that involves heavy investment in technology, innovation, and data analysis. From the architecture of data centers to the microsecond battles fought in the trading algorithms, HFT is a realm where the pursuit of speed and efficiency is relentless, and the financial implications are enormous.

The layers of complexity and the speed at which changes occur in this field underscore the need for ongoing scrutiny, ethical considerations, and potentially, further regulatory oversight. It's a world where microseconds can mean the difference between millions gained or lost, and where the technological arms race shows no signs of slowing down.