The distributed nature of git is wonderful: we are able to commit locally into the local repo, and later push to the main github repo. Cloning is a great way for everyone to create forks for experimenting around and adding smaller features. Although git in itself is pretty cool, Github is the "killer app": it makes source control management beatiful, gives us a nice clicky interface for repo admin tasks, and lets everyone clone easily. We pay $12 for the small plan, this gives us 10 private repos to keep stealth mode projects in stealth mode. I bought the Pro Git book by Scott Chacon on Amazon for $23, who incidentally works at Github.

Git is all about the command line switches. Over the last couple of months, I've created a nice cheatsheet for our most common (non-trivial) git commands so we don't always end up googling them, mostly based on the Pro Git book and StackOverflow QAs. It also contains the 2+1 major gotchas I've encountered:

1. Do not amend commits that you have pushed to a public repository!
2. Do not rebase commits that you have pushed to a public repository!
3. Always update to the latest git major version (currently at 1.7.x)

Always keep these in mind and your co-developers won't hate you.

Finally, here's the cheat sheet (PDF), and a preview:

The full blog post is at the Scalien Blog.

Here's my list of things to make Mysql safe(r) for transactional usage:

1. Server SQL Modes:
   • STRICT_ALL_TABLES: Enable strict mode for all storage engines. Invalid data values are rejected.
   • NO_ENGINE_SUBSTITUTION: Control automatic substitution of the default storage engine when a statement such as CREATE TABLE or ALTER TABLE specifies a storage engine that is disabled or not compiled in.
   • NO_ZERO_DATE: In strict mode, don't allow '0000-00-00' as a valid date. You can still insert zero dates with the IGNORE option. When not in strict mode, the date is accepted but a warning is generated.
   • NO_ZERO_IN_DATE: In strict mode, do not accept dates where the year part is nonzero but the month or day part is 0 (for example, '0000-00-00' is legal but '2010-00-01' and '2010-01-00' are not).
   • TRADITIONAL: Make MySQL behave like a “traditional” SQL database system. A simple description of this mode is “give an error instead of a warning” when inserting an incorrect value into a column.
2. Transaction isolation levels: set this to SERIALIZABLE, then go down if necessary. The InnoDB default is REPEATABLE READ. The Wikipedia page on isolation explains the difference between the levels (easy to forget) and gives some examples.
   • READ UNCOMMITTED: Barely transactional, this setting allows for so-called 'dirty reads', where queries inside one transaction are affected by uncommitted changes in another transaction.
   • READ COMMITTED: Committed updates are visible within another transaction. This means identical queries within a transaction can return differing results. This is the default in some DBMS's.
   • REPEATABLE READ: The default isolation level for InnoDB tables. Within a transaction, all reads are consistent.
   • SERIALIZABLE: Updates are not permitted in other transactions if a transaction has run an ordinary SELECT query, i.e. queries are treated as if they had a LOCK IN SHARE MODE, which we saw in action last month.
3. Set the default engine to InnoDB: you don't want to accidently end up with MyISAM tables. Unfortunately it's not possible to disable MyISAM altogether (there's no skip-myisam).
4. Set all defaults to UTF-8 to avoid character encoding issues. Use init-connect to force users' connections to use UTF-8, too.
5. Log all warnings: as far as I can tell, this is not possible in Mysql. This is a problem, for example if you try to rollback a transaction that wrote into a MyISAM table, that will only generate a warning. The best thing would be to have Mysql generate an error in this case (not possible), the second-best thing would be to log all warnings, but it seems there's no option for this. You can specify --show-warnings to the mysql client, but that won't help if you're trying to catch warning running in a stored procedure from a cronjob.
6. Make sure your hardware doesn't do extra chaching and doesn't reorder writes. The Mysql documentation says: On ATA/SATA disk drives, a command such hdparm -W0 /dev/hda may work to disable the write-back cache. Beware that some drives or disk controllers may be unable to disable the write-back cache.
7. Log slow queries which potentially block other transactions.
8. Use innotop to keep and eye on your transaction workload.

With that said, here's my my.cnf:

[client]
port			= 3306
socket			= /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket			= /var/run/mysqld/mysqld.sock
nice			= 0

[mysqld]
user			= mysql
pid-file		= /var/run/mysqld/mysqld.pid
socket			= /var/run/mysqld/mysqld.sock
port			= 3306
basedir			= /usr
datadir			= /var/lib/mysql
tmpdir			= /tmp
language		= /usr/share/mysql/english
bind-address		= 127.0.0.1
key_buffer		= 16M
max_allowed_packet	= 16M
thread_stack		= 128K
thread_cache_size	= 8
myisam-recover		= BACKUP
query_cache_limit       = 1M
query_cache_size        = 16M
max_binlog_size         = 100M

# the settings below make Mysql safe for transactional usage
transaction-isolation	= SERIALIZABLE
sql-mode		= STRICT_ALL_TABLES,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,TRADITIONAL
default-storage_engine	= innodb
init_connect		= 'SET collation_connection = utf8_general_ci; SET NAMES utf8;'
default-character-set	= utf8
character-set-server	= utf8
collation-server	= utf8_general_ci

log-warnings            = 2
log                     = /var/log/mysql.log
log-error		= /var/log/mysql.err
log_slow_queries        = /var/log/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes

skip-external-locking
skip-bdb

[mysqldump]
quick
quote-names
max_allowed_packet	= 16M

[isamchk]
key_buffer		= 16M

!includedir /etc/mysql/conf.d/

If you think I missed something or have any suggestions, please post a comment!

Unfortunately, the rubber started to come off it, so I went out to buy a new mouse. I don't like Microsoft mouse, their scrollwheel doesn't feel right, so I was pretty much looking for another Logitech model. It's surprising how hard it is to find a good mouse, but finally I found a Logitech M705 Marathon mouse. The salesman said it goes 3 years with a pair of AA batteries, and that is a feat of engineering I felt I should invest in! In comparison, my old Logitech MX had to be recharged every week. So my new mouse is a Logitech M705 Marathon, this is what it looks like:

I've been using it for about a week, and I can't say I'm terribly happy with it. It just doesn't feel right. First of all, it doesn't fit into my (abnormally large) hands like the old mouse did. But I could get over that. What's more annoying is a design flaw: in order for the back button to register that you have pushed it, you have to apply just enough force to it that you I inadvertadly move the mouse, sometimes the left side even lifts up. But I could get over that. What's really pushing my buttons is the acceleration curve. As we all know, the acceleration curve is as it should be on Windows, and sucks on Macs. Using the mouse is one of the few things I enjoy doing on Windows. That is unless you're using a Logitech M705 Marathon Mouse, which comes with some terrible acceleration curve by default. when you're using this mouse, you don't have to drink and get high to be all over the desktop and constantly overshoot the X to close a damned window. I had to install Logitech's crappy SetPoint driver thing just to get it to behave within reason.

So, overall my verdict to Logitech on the Marathon M705 is a gigantic FAIL!

PS: The heavy scroll wheel is really nice though!

EDIT: Also, the Logitech control panel thing forgets my mouse settings on reboot on Windows XP. Damn you Logitech, damn you!

This often happens even if we spend a great amount of time at the beginning specifying and designing the program. This is because by the end of 1.0, and later during the lifetime of the project requirements change as features are added, irrespective of the amount of planning that happens at the beginning. But no amount of planning will make a program designed to solve the original problem solve a different problem. That's just not what planning is about: planning is about fleshing out the original problem.

This is why, in many cases, redesigning a large program at 2.0 still results in a brittle program: the program is still specific to a single requirement, though by now a different one.

My conclusion — based on comparing my programs to better programmer's programs — is that one must write a domain-specific "framework" (*) that is more general than the original requirements, but admits the original requirements as a special case. This way, if requirements change — and they will — one has a chance of accomodating them. Even so, the original framework will certainly not be general enough, but it will be easier to adapt. The trick is, of course, to find just the right dimensions to generalize and just the right amount of generalizations.

(*) When I say domain-specific framework, I mean that the actual domain specific functionality is generalized in the program, versus just writing general functions for handling strings and linked lists while still encoding the domain specific functionality in a brittle manner.

1. Keyspace whitepaper: the distributed database which is currently Scalien's primary product, under heavy development!
2. PaxosLease whitepaper: describes the Paxos-varaint invented by my co-authors and myself for negotiating distributed leases (time expiring locks) in a consistent manner.
3. Paper on open-source distributed systems for the 2009 FSF conference (in hungarian)
4. The Exciting physics of an Excited universe: a literature review paper of Norman Glendenning's book Compact stars, talks about neutron stars, pulsars and strange stars.

You should of course use the Google Docs Preview Firefox addon for previewing PDF files!

Tower of Babel

"NoSQL" and scalability. The original inspiration for many open-source projects is that large players like Google and Amazon chose not to use Mysql or Oracle in certain cases and developed in-house systems for scalability reasons, meaning storage, availability and performance scalability (see my Readings in Distributed Systems for more on these). There is a number of open-source projects which follow in these steps (like our own Keyspace datastore and our PaxosLease algorithm), trying to build truly scalable software. These systems are "NoSQL" because distributing complex relational operations across nodes is complicated, and these systems are to be used in performance intensive applications anyways, where key-value datastores are a good basis for optimization in the application layer. Unfortunately, the space of truly distributed open-source software is contaminated by virtually every non-relational project claiming to be distributed and scalable.
Systems in this category: Keyspace, Apache Hadoop, Cassandra

"NoSQL" and SQL. Some have claimed that "NoSQL" is not so much about getting rid of SQL as it is about building something scalable; but this is not true in general. The primary use-case here is CouchDB, which is getting a lot of attention from web developers who like its simple, REST-based data model and query facilities. CouchDB is written in Erlang and is fairly slow, so it cannot be taken seriously as a scalable system in the sense above, and most CouchDB users are not scaling their installation. CouchDB is a light-weight solution for web apps, similar to Microsoft Access / Lotus Notes in the bussiness world. Many web developers are non-advanced users of SQL anyway, not using "advanced" features like foregin keys, JOINs, inner queries or stored procedures. These users choose CouchDB (over an ORM layer) to get rid of ugly embedded SQL code which doesn't do much for them anyway. I believe there is a portion of "NoSQL" users who are trying to get rid of SQL.
Systems in this category: Redis, CouchDB

"NoSQL" and UNIX. Many proponents like "NoSQL" data stores because they adhere to the UNIX philosophy of building small, lightweight tools for a specific purpose but of general applicability: "do one thing well". Uncontroversially, relational databases do not adhere to the UNIX philosophy: they are large multi-million line code bases of non-trivial complexity and runtime characteristics. (On the other hand, they have stood the test of time and become the most successful and lucrative data model in computer science). This is one of the reason why projects like Memcached are put under the "NoSQL" umbrella, even though it's not a persistent datastore and as such not competition for a relational database. Also noteworthy are BerkeleyDB and Tokyo, which are low-level key-value storage engines that can be used to build more complex datastores (Mysql used to have a BDB engine, Keyspace 1.x uses BDB).
Systems in this category: Memcached, BerkeleyDB, Tokyo

I believe it would be beneficial to seperate these use-cases and treat them differently (eg. call one NoSQL and the other DDS for Distributed Data Store). While some of them concentrate on implementing good distributed algorithms, others concentrate on making web developer's lifes easier. Both are valid use-cases and noble goals. Clearly, the former may have less trendy API facilities, while the latter may be easier to setup and administer but slower. This is not a put-down, though, it resembles comparing PHP to C++: one of them gives you easy entry and rapid development cycles, while the other gives you raw speed at the expense of compiled code and pointers. But treating them under the same umbrella, as many presentations do, is confusing. It gives the impression of "Pick ONE" or "ONE will survive", which goes against the basic tenet of specialization and UNIX, the starting point of "NoSQL" datastores. Also, people looking to scale terabytes of data may not be interested in CouchDB while people looking to replace Mysql for their simple in-house web apps may not be interested in Hadoop.

Based on these experiences, GW definitely qualifies as a disruptive technology. It's pretty clear that GW is going to decrease the importance of email and IM. Also, several use-cases which required special web-based software for convenience will move to GW, such as: event planning, todos, reminders, bookmarking, notetaking, collaborative draft editing, online meetings, mini-blogs. These are all use-cases which are pretty much possible with the default GW web interface, or require little additinal functionality.

Google Wave is a product which will markedly increase the productivity of large corporations. Novell is already working on a product dubbed Google Wave for the Enterprise, called Novell Pulse. I watched their video presentation, and it looks good. Also look at the SAP presentation on the bottom of this page, which is a bit more enterprisey, but still pretty nice. I wonder what Microsoft's answer will be. I hope they don't come up with an incompatible-but-equivalent implementation.

In conclusion, Google Wave is still relatively young and immature (even Google's client sucks), lots of space for movement and innovation, it will be interesting to see how the ecosystem and monetization game unfolds.

1. Anandtech's overview (must read)
2. In-Page Logging B-Tree for Flash Memory [pdf]
3. Overview slides (good) [ppt]
4. uFLIP: Understanding Flash IO Patterns (must read, CIDR best paper) [pdf]
5. Optimize write performance for DBMS on Solid State Drive [pdf]
6. Lazy-Adaptive Tree: An Optimized Index Structure for Flash Devices (VLDB) [pdf]
7. Tree Indexing on Solid State Drives (VLDB) [pdf]
8. Log-structured file systems: There's one in every SSD

Two notes: First, the uFLIP paper is particularly good. Second, it seems that the In-Page Logging (IPL) paper assumes direct control of read/write/erase cycles of the SSD, which is not realistic for off-the-shelf consumer drives like the Intel X25-M, which include a complex FTL.

Finally, while putting together this list I found a much longer list of SSD papers here, also worth taking a look.

Download the Google Docs Preview Firefox Addon here.

Google has exported their very cool web-based PDF viewer. You can use it by telling it what PDF file to open like this:

http://docs.google.com/gview?url=PDF_URL

For example, here is the Keyspace whitepaper shown in Google Docs Preview:

http://docs.google.com/gview?url=http://scalien.com/pdf/Keyspace.pdf

You can also embed it in an IFRAME like this:

<iframe src="http://docs.google.com/gview?embedded=true&url=PDF_URL"
style="width:800px; height:600px;" frameborder="0"></iframe>

For example, here is the Keypspace whitepaper shown embedded:

I hacked together a Firefox addon that lets you open any PDF file from Firefox using the Google Docs Preview: download here. It lets you specify whether to

1. Open in the same or a new tab
2. Whether to focus the new tab

Here's a screeshot:

This paper describes the design and architecture of Keyspace, a distributed key-value store offering strong consistency, fault-tolerance and high availability. The source code is released as free, open-source software under the GNU Affero General Public License (AGPL). Keyspace is a product of Scalien Software, available for download at http://scalien.com

I'm one of the developers of Keyspace, have a look at it!

Next up: finish up the paper on PaxosLease, a diskless Paxos variant (similar to Fatlease) we have developed while working on Keyspace.

In the early 2000s Damian built a website for fellow gamers interested in the magic game dungeons and dragons. The site was simple, as it listed various properties of magic items and featured a messageboard. Damian used the LAMP stack: data about items was stored in Mysql, and various PHP scripts let the visitors of the site view items and statistics.

Now fast forward to 2009. Damian's site has evolved to a web game for playing dungeons online, in your browser. Damian is still using LAMP. Data about types of games (including parameters such as monster strength), user data (including status information), and data about active games (including players, the monters's health) are still stored in Mysql. It's all fine until there are only a few games in session and only a couple hundred players, but as the site gets popular, Damian's server is starting to see high load numbers.

Damian is experiencing scalability issues --- his current setup cannot handle tens of thousands of users. He searches the web and buys some books about Mysql, and starts optimizing:

• tweaking Mysql parameters
• tweaking Apache parameters
• moving to a dedicated server
• buying more RAM
• denormalizing tables
• optimizing SQL queries
• optimizing indexes

Each of these tweaks buys him more time, but the users keep on coming, and the load factor keeps creeping up. Damian reads even more articles, buy some more books, and gets his hands dirty scaling his system:

• discovers Memcached, and starts caching some SQL queries
• discovers Lighttpd, uses it to serve static content like images
• buys a second dedicated server to spread Mysql reads
• buys more RAM for the second server
• ...

As even more users register, the load is becoming unbearable. Damian discovers Amazon EC2...

Hold on a minute. What's going on here? Let's think for a second.

Most of Damian's problems are related to Mysql. Suppose we were not living the glorious Web2.0 days and you were writing what's basically an application server. Would you store everything, including state information in a database? No, of course not. You would store some information in the database, read it out when the application server launches, perform computation for the clients, and write data meant to be persistent back to the database. This is what Damien really needs. Damian needs a fast racecar, but his initial design resulted in a bike, and now he's frantically trying to turn the bike into a car by welding two bikes together and putting rocket boosters in the back.

Damian does not need fundamentally new software. He does not need a custom application server, nor could you convince him to write one. What he needs is the insight to identify state, cached data and persistent data in his application. Application state goes into an in-memory key-value store like Tokyo Tyrant. Cache data goes into Memcached. Persistent data goes into a database. Note that the seperation of code and application state may be beneficial later, because it allows you to scale easily by adding new memory servers. Mysql will probably need to be left behind for a persistent key-value store that is more easily distributed and replicated. As co-founder of Scalien and one of the developers of our Keyspace replicated key-value store, I can only recommend it. It's devilishly fast for consistently replicated writes.

When the server launches, before client requests are served, a script initializes the system by loading the appropriate state data into memory. Client requests are served primarily by using state information (global and per-user, per-game, per-session, etc.), and sometimes persistent data is written back to disk. Data that isn't absolutely essential, like an in-game chat transcript are written to disk asynchronously in the background.

Let's call this the Code-State-Cache-Data (CSCD) pattern. What Damian originally had was a Code-Data (CD) pattern, and later he optimized to get a Code-Cache-Data (CCD) pattern. It is my opinion that writing applications in the CSCD pattern is not substantially harder than the standard CD or CCD pattern. But it is a smarter architecture because it identifies data for what it is: state, cache and persistent data. It is a faster architecture, as more clients can be served with a single server. Additionally, it is a more scalable architecture, especially if Mysql is replaced with a distributed key-value store. Finally, it is pragmatic, because it uses existing components and existing practices only change slightly.

Let's keep in mind that there is no silver bullet. With a CSCD pattern, Damian will still have to do optimizations like tweaking Apache or buying more RAM, but he is optimizing to make a smart architecture run fast, instead of caching the hell out of a slow architecture.

What I'm saying here isn't anything radical or even new, but most web application are still written in the LAMP+CD pattern. Hopefully this article will lead to more scalable designs where code, state, cache and data are seperated to yield better performance.

PNUTS is Yahoo's in-house distributed tablestore used for serving some of its web properties. The goal in this post is finding out the basics: how replication is managed, what kind of guarantees the system makes, can branching occur...

Note: this post may contain misunderstandings and it definitely contains speculations!

Reading the paper, it hits an interesting spot in design space. One of the major goals is geographic replication: reads and writes of data belonging to a user in India should be fast from India. But the same data is also accessed from different points on the globe, but these are most likely reads. It's acceptable if these "remote reads" see an older version of the data, as long as the versions are not branching (see below). Summing up: reads should be fast everywhere and may return older versions, writes should be fast locally (in the same datacenter).

The article states that they want to avoid branching versions of data. Paxos, however, is out of the question here, since they may have as many as 10 replicas all geographically seperated, with 50-100 ms latencys. Since they're not using Paxos, it seems it is possible for divergent versions to occur in Yahoo's system. It's not clear to me what they're doing about this, the following quote is a "future plan":

Under normal operation, if the master copy of a record fails, our system has protocols to fail over to another replica. However, if there are major outages, e.g. the entire region that had the master copy for a record becomes unreachable, updates cannot continue at another replica without potentially violating record-timeline consistency. We will allow applications to indicate, per-table, whether they want updates to continue in the presence of major outages, potentially branching the record timeline. If so, we will provide automatic conflict resolution and notifications thereof. The application will also be able to choose from several conflict resolution policies: e.g., discarding one branch, or merging updates from branches, etc.

To come as close as possible to a "no branching" behavior, PNUTS uses a a 3-tiered architecture: there are 1. clients who initiate read and write operations, 2. the Yahoo Message Broker (MB) which they use as a basic replication primitive, and 3. geographically replicated storage nodes. The storage nodes use a master mechanism, where the master node is usually the one geographically closest to where the writes originate (in the same datacenter).

Branching.

As far as I can tell, a write is managed like this: the client initiates the write operation at the master storage node. The node passes the write request on to the MB. The write is considered commited if the MB says it commited it. The master, if it doesn't go down in the meanwhile, performs the write on its local data. The MB guarantees that it will deliver the write operation to the other replicas. The nice property of this setup is that if the master and the MB are in the same datacenter as the client then they don't have to wait for the replicas in distant datacenters to commit to their disks --- this is handled and waited on later by the MB.

The MB is not discussed in this or any other paper. I think it may be made up of n=3 Paxos cells. They say that there are many message brokers, which if I'm right means many Paxos cells. Write operations stored within a MB cell are guaranteed to be delivered in that order to the replicas (this seems easy), but what about writes (of the same data) stored in different MB cells? Here the paper is somewhat confusing to me. They write:

We provide per-record timeline consistency: all replicas of a given record apply all updates to the record in the same order.

and

.. in order to provide timeline consistency, we have developed a per-record mastership mechanism, and the updates published by a record's master to a single YMB cluster are delivered in the published order to the other replicas.

The second quote above seems to imply that the first one is only true for writes coming from the same MB cell.

Basically, the trick is that write operations are handed over to a MB cell, which then acts as a single logical writer. What happens if there is a master-failover and the new master publishes to a different MB? Given that the MB cells themselves are not in some sort of higher level Paxos cell, and the storage nodes also are not in a Paxos cell, I think all bets are off at this point and branching versions can come in, as discussed at the beginning.

Of course, since you cannot put replicas in different datacenters in a Paxos cell, you are forced to do something like this! It's an interesting approach, as it's somewhere between a full-on Paxos like in Google's Chubby, and full-on branching like in Amazon's Dynamo. The architecture is interesting to say the least, but my question is this: if the user has to prepare for branching and has to write the conflict resolution code, wouldn't a Dynamo-like (with some basic logic to always choose local nodes to write to) system be simpler and possibly faster?

1. Professional and/or financial success

Professional and financial success.

2. Compete where your strengths are

Feynman, according to himself, was an average bongo drummer.

3. Focus on fundamentals

4. Charge money

Example:

http://www-db.cs.wisc.edu/cidr/cidr2009/program.html

The decision was inspired by Colin Percival:

As many people who know me will be aware, I tend towards minimalism in computer software. As a result, when I decided to start writing this, I looked around for the simplest blogging software I could find (it was NanoBlogger) and decided that it was far too complex for my taste. So I wrote my own.

I don't need most of the features of a full blown blog engine like WordPress.

• PrimoBlog generates a static index of all articles. I think grouping by year and month is mostly useless.
• For comments, I use disqus, which only involves placing some JavaScript at the end of the HTML. It's a free lunch, really.
• Using anything but Google for searching is stupid anyways.
• Nobody cares about categories and tags and nifty per-category icons. Contents over style.

Wanted. Men for hazardous journey.
Low wages. Bitter cold.
Long hours of complete darkness.
Safe return doubtful.
Honor and recognition in the event of success.

Would this kind of ad work on programmers?

The latest table of contents:

I. The Google Papers
1. The Google File System
2. Bigtable: A Distributed Storage System for Structured Data
3. The Chubby Lock Service for Loosely-Coupled Distributed Systems
4. Paxos Made Live - An Engineering Perspective
5. MapReduce: Simplified Data Processing on Large Clusters

II. Distributed Filesystems
1. Petal: Distributed Virtual Disks
2. Frangipani: A Scalable Distributed File System
3. GPFS: A shared-disk file system for large computing clusters
4. Replication in the Harp File System
5. Swift: Using distributed disk striping to provide high I/O data rates
6. Ceph: a scalable, high-performance distributed file system
7. XtreemFS - a case for object-based storage in Grid data management
8. Andrew FS
9. Lustre
10. GlusterFS
11. Sinfonia: A new paradigm for building scalable distributed systems

III. Non-relational Distributed Databases
1. Dynamo: Amazon’s Highly Available Key-value Store
2. Facebook’s Cassandra
3. CouchDB

IV. The Lamport Papers
1. Paxos made simple
2. Reaching Agreement in the Presence of Faults
3. Time, clocks, and the ordering of events in a distributed system
4. The Part-Time Parliament
5. Distributed snapshots: determining global states of distributed systems

V. Implementation Issues
1. A scalable and explicit event delivery mechanism for UNIX
2. A design framework for highly concurrent systems
3. Comparing and evaluating epoll, select, and poll event mechanisms
4. Asynchronous I/O support in Linux 2.5
5. Linux AIO performance and robustness for enterprise workloads
6. Books written by the late W. Richard Stevens

Currently, there's a section on

• the Google papers: GFS, Bigtable, MapReduce, etc.: these are classics
• Distributed Filesystems: this is what I'm working on right now, so it's the most complete section
• Non-relational Distributed Databases: very incomplete
• The Lamport papers: Leslie Lamport seems to be the godfather of distributed algorithms, these are some of his most important papers. It's worth looking at them to get an idea of the problems you could run into designing / implementing distributed systems.

If you think something important is missing, please email me.

Initially I wanted to spend mega bucks on a digital SLR camera. I looked at Nikon D40 - D90s, and even the 18-200 Nikkor VR lens. As I watched the BBC series, I found out that my favorite photographs were street photography, the ones that capture the decisive moment:

Street photography uses the techniques of straight photography in that it shows a pure vision of something, like holding up a mirror to society. This genre of photography is present in contemporary times and is usually done as black and white photographs. Street photography often tends to be ironic and can be distanced from its subject matter and often concentrates on a single human moment, caught at a decisive or poignant moment. On the other hand, much street photography takes the opposite approach and provides a very literal and extremely personal rendering of the subject matter, giving the audience a more visceral experience of walks of life they might only be passingly familiar with. In the 20th century, street photographers have provided an exemplary and detailed record of street culture in Europe and North America, and elsewhere to a somewhat lesser extent.

Street Photography by Joel Meyerowitz

I ended up not buying a DSLR, because I found them too clunky. I wanted something lighter that I can carry around all day. Finally I settled on the high-end compact Canon PowerShot G9.

After a few days of usage I haven't had time to change the camera from its default settings, and still, it takes great pictures. Here is some street amateur photography I took (processed, stylized):

Greenpeace Girl on Moricz Square (Budapest, 2008/10/13)

Homeless Men on Ferenciek Square (Budapest, 2008/10/13)

Laughing out Loud (Budapest, 2008/10/11)

Swim 2.4 miles.
Bike 112 miles.
Run 26.2 miles.
Brag for the rest of your life!

(That's 3.8, 180, 42 kilometers.)

For me at least, it was a different experience. The biggest gain is what you carry over to other aspects of your life.

Discipline. In the final months, I trained about 15 hours net per week. That's about 6km swimming, 200km biking and 50km running a week. Integrating this kind of workout into your life forces you to manage your time efficiently. You stop wasting time, and start getting up at 6AM for morning workouts. Maybe you'll even start drinking raw eggs...

Focus. At the beginning of the season, I ordered Joe Friel and Gordon Byrn's excellent Going Long book from Amazon.

This is a great book with lots of good technical help, but here I'd like to point out the advice that applies not just to triathlon (paraphrased):

As in all aspects of life,
focus on fundamentals:
endurance, strength,
efficiency, recovery.
It ain't rocket science.

Focus of fundamentals: these three words, which I read in a triathlon book a year ago, have since changed the way I approach programming, physics, pretty much everything.

Motivation. As you train for the Ironman, you start being very motivated about yourself. You start listening to the Rocky theme song. During workouts you visualize yourself during the race or finishing the race. This motivation drips over to other aspects of your life, too.

Fitness. Anybody who earns his money typing away at a computer needs a workout a day. What's the point of being successful and making lots of money if you're not healthy (or won't be). Note that health is strongly correlated with body weight. I've lost 15 kg (33 pounds) during the training, and since training is now part of my daily routine, it's easy to stay fit.

Recommendation. I can only recommend the Ironman, and triathlon in general. It takes a lot of time and money, but the gain is tremendous. Most people I know who claim that it takes too much time are like me, they could easily convert wasted time to workouts while not taking away much time from other essential activities like work. (The only person I would not recommend it to is a young parent.)

To wrap it up, here are some pics from my race. (Note that I was at the hungarian Ironman, called ExtremeMan.) Some more motivation here (warning: media-engineered). If you think you can't do it, wind this video to 0:50.

Have a great workout!

Show me you algorithm,
and I will remain puzzled,
but show me your data structure,
and I will be enlightened.

This is a statement about software and coding, but first and foremost it is about human cognition. The way my brain works is to first visualize the data and then imagine what the algorithm does to it.

Given this cognitive limitation of my brain, I favor static, strong, class-based typing (C/C++) over dynamic, weak, object-based typing (JavaScript).

To give a strong example and stir up some controversy, consider Chapter 9 of Beautiful Code: Top Down Operator Precedence by Douglas Crockford. You can read it here, and you should to really get a feeling for what's going on...

I think it's spaghetti code.

The problems with his approach are many:

• No complete declarations of complete types, even though it would be (kind of) possible with JavaScript. So I can't start by vizualizing the data structures.
• No type declarations, but lots of confusing variable names, so I'm constantly going back trying to figure out what's what.
• Dynamic modification of fields in objects. For example the function advance() adds the field arity to the token object, which was not declared anywhere, so you only find out about this by reading this function. This goes against everything sane in programming.
• Going completely overboard, the field arity is used to store whether a token is a "name", "literal" or "operator" but later, for operators, changes this field to "unary", "binary", etc. (Truly bad programming, but not related to my argument about typing.)
• To deliver the final blow to understandablity, in functional style, the member functions of the objects are dynamically added and modified. At this point, if I see an object, I don't know its "type", but even if somebody tells me that, I don't know what state that "type" is in: what the data fields are, what they are used for and what the member functions do.
• Given the cognitive difficulties above, how does one look over the code to verify that it's correct? It's hard enough with static languages, here it seems quixotic. How about finding bugs?

As I was trying to understand the code, I found myself writing out on a piece of paper the different "types" of objects that come up in the code --- in other words the classes. I was writing the header files for the code in order to untangle the mess!

To close this rant, I'll quote a thread Google turned up:

> A while ago there was one of those static-vs-dynamic debates (in the hackers
> and painters thread, I think), and somewhere (I'm sure) someone mentioned a
> quote along the lines of:
> 
> Early on in a project, as code is rapidly evolving, you want static typing
> to help you as much as possible. When the code has matured (it's well tested
> and not changing much), you want more dynamic typing so that the design can
> be more flexible.
> 
> Can anyone remember who said this?
> 
Not me.  But it's interesting.  Usually the argument goes the other way:
Early on in a project, as code is rapidly evolving, you want dynamic
typing to keep as many design options open as possible. When the code has
matured (it's well tested and not changing much), you want more static
typing for efficiency and to help root out obscure bugs that may be
waiting to manifest themselves.

Getting locking right in a non-trivial system is pretty hard. SPJ's list of pitfalls is almost exhaustive:

1. Taking too few locks
2. Taking too many locks
3. Taking the wrong locks
4. Taking the locks in the wrong order
5. Freeing locks on error

I would a zeroth point: figuring out what to lock. This precedes everything else you do with locks.

Software Transactional Memory is, per Wikipedia:

In computer science, software transactional memory (STM) is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing. It functions as an alternative to lock-based synchronization. A transaction in this context is a piece of code that executes a series of reads and writes to shared memory. These reads and writes logically occur at a single instant in time; intermediate states are not visible to other (successful) transactions.

An example is worth a thousand words:

// change a1 and a2 such that a1 + a2 = const
// like transferring money between bank accounts
 
// lock-based version
lock l;
double a1, a2;
void transfer(double amount)
{
 lock(l);
 a1 = a1 - amount; // suppose this is atomic
 a2 = a2 + amount;
 unlock(l);
 }
}
 
// STM
stm_double a1, a2;
void transfer(double amount)
{
 transaction_start(); // like 'atomic' in Haskell
  a1 = a1 - amount;
  a2 = a2 + amount;
 transaction_end();
}

No big difference. But wait, what if you have several a's, say several accounts. You want transfer(a1, a2, amount). With locks, your code just got a whole lot more complicated. With STM, it's a free lunch.

STM solves most of the problems listed by SPJ, but cannot solve the zeroth problem: you still have to figure out what the lockspots are, declare them to be transactional variables, and then guard them with transactions. But with STMs, all you have to do is guard with, you don't have to worry about locking policy.

But remember, there are no silver bullets. There are two problems with STM:

1. No transactional I/O. M stands for Memory, meaning that it can only guard memory areas, but not I/O. On some systems, like Haskell, the STM library does not even let you mix STM and I/O operations, so you can't have I/O operations inside atomically { } (it won't compile). The reason is, you don't want to execute the I/O code twice in case the transaction has to be re-run.

2. Library overhead. You just went from a few instructions to write a variable to including a whole library that manages reading and writing variables for you. If the code you are writing is not high-performance, this is probably acceptable. I'd consider using STM when working in Java, .NET or Python --- here, raw speed is not possible anyway, so trading some more CPU time for simpler and less buggy code is acceptable.

How much speed are you sacrifying when using STM? To find out, I downloaded and compiled TinySTM, a C library implementing the basic STM primitives like stm_start() and stm_commit().

It turns out that the transaction manager's design matters a whole lot. Some good papers are:

• Dynamic Performance Tuning of Word-Based Software Transactional Memory - a presentation by the TinySTM authors
• Performance Comparison of Software Transactional Memory Implementations - Riya Kariath's M.Sc. Thesis

To test performance I created the simplest program possible (source code), with just one long integer called counter, NTHREADS many threads each wanting to increment counter NLOOPS many times. (This test is by no means representative, but it's something. You have to see what happens for your real workload.)

For comparison, I measured

1. a naive, non-locking implementation which is not correct, ie. the final value of counter is not NTHREADS * NLOOPS.
2. pthread mutex based implementation with one lock for counter
3. TinySTM based implementation with transactions using Write-back + Encounter Time Locking (ETL)
4. TinySTM based implementation with transactions using Write-back + Commit Time Locking (CTL)

The results are in milliseconds, first number is for naive, second for lock-based, third for STM-based implementation:

	NTHREADS = 10	NTHREADS = 100	NTHREADS = 1000
NLOOPS = 10	0 | 0 | 40 | 10	7 | 7 | 1440 | 18	112 | 153 | 1148071 | 147
NLOOPS = 100	0 | 6 | 54 | 8	6 | 100 | 2065 | 20	93 | 1037 | 1204106 | 875
NLOOPS = 1000	0 | 95 | 54 | 11	6 | 898 | 1200 | 40	99 | 9937 | 566714 | 379

Note that I didn't repeat the experiment many times and take an average value, but the trend is quite clear (and repeatable). As you get more and more threads competing for the same resource, the STM ETL implementation becomes unfeasable (third number), while the STM CTL implementation (fourth number) beats my locking code (second number).

What's the difference between the two STM architectures? With ETL, as soon as the transaction reads a variable, that variable is locked and not released until the transaction finishes. With CTL, the variables are not locked until the transaction attempts to commit. The problem with ETL is that whenever a thread acquires a lock on counter, and all the other threads start aborting and re-executing, hogging down the CPU. It's not clear why the CTL version is faster than ETL, I assume it's because threads get lots of timeslices during the execution of one one loop, so the ETL version aborts many more times than the CTL version. Why is it faster then the pthread version? TinySTM uses atomic_ops, which contains machine-specific optimizations, which the pthread library probably does not.

The lesson is:

• STM can be very slow depending on your workload and STM library
• STM can be very fast depending on your workload and STM library
• Look into atomic_ops (or find bottleneck is pthread code)

Pictures from the race.

I am referring to the slew of outages the last day’s and weeks of some high profile infrastructures like Amazon S3 and Google. I always thought Googles infrastructure was untouchable. But it seemed that I had woken up to a different world last week. When I saw tweets and messages going around with "Gmail is down".
...
P2P Cloud computing in general will be the future of infrastructure.

At first, this sounds like a good idea. At least it did in 2000, when it was called grid computing. Unfortunately, it's not.

It's been tried; it failed. This Nature article from 2000 is optimistic:

Internet computing and Grid technologies promise to change the way we tackle complex problems. They will enable large-scale aggregation and sharing of computational, data and other resources across institutional boundaries. And harnessing these new technologies effectively will transform scientific disciplines ranging from high-energy physics to the life sciences.

At the end of the article is a list of companies trying to commercialize the concept: Entropia, United Devices, Parabon and Popular Power. Guess what --- most of them are no longer around. The only one that is is Parabon, which is hosting the ComputeAgainstCancer project. Looking at their site, I get the feeling that their customers are mostly universities and other government agencies

Incentive and economics. People download SETI@Home out of philantrophy and because they think it's cool. You can't count on people to host your distributed app platform on their computer for philantrophy, so you'd have to pay them. Unfortunately, the amount of money their computational power is worth is so low that it wouldn't work. Would you bother to download and run some a program in the background for ~$5 / month? Also, you can't put your computer to sleep from now on. E.g. in the case of Parabon, the company mentioned above, you can't download and run their "Frontier Compute Engine", which is their distributed node software. The reason probably is that nobody wants to, anyway. So if a company wants to use Parabon's system, they either run their own nodes, or buy the capacity from Parabon's marketplace.

Latency. Google serves search requests in 10-100 milliseconds, wherever you are. Can you do that with P2P? Remember, latency vs. bandwidth:

Years ago David Cheriton at Stanford taught me something that seemed very obvious at the time -- that if you have a network link with low bandwidth then it's an easy matter of putting several in parallel to make a combined link with higher bandwidth, but if you have a network link with bad latency then no amount of money can turn any number of them into a link with good latency.

It's not attacking the right problems. I'm not sure about the Gmail outage, but Amazon posted a short explanation of their problem:

Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types.

Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue.

Even in a P2P architecture, I imagine that the authentication service would be located in the grid provider's own network, so P2P wouldn't make a difference. Also note that today's clouds (Google, Amazon, etc.) are explicitly designed to be unaffected by disk, machine or switch failure. So the only way to bring them down is either through an allocation mismatch (human error?), which is basically what happened with Amazon, a software bug / protocol bug (human error), some extreme power outage that outlasts the local generators (extremely unlikely?), or some kind of DDoS attack against the entire data center. Even with a P2P architecture, you still have to have some components running in the grid provider's local datacenter, so a DDoS against that would still bring the whole thing down. I think that the Gmail and Amazon problems were not really scalability problems, so P2P wouldn't have helped.

To close the article, I'll include one last quote from Jim Gray's short Distributed Computing Economics (2003) paper (emphasis mine):

Computing economics are changing. Today there is rough price parity between (1) one database access, (2) ten bytes of network traffic, (3) 100,000 instructions, (4) 10 bytes of disk storage, and (5) a megabyte of disk bandwidth. This has implications for how one structures Internet-scale distributed computing: one puts computing as close to the data as possible in order to avoid expensive network traffic.
...
The ideal mobile task is stateless (no database or database access), has a tiny network input and output, and has huge computational demand. For example, a cryptographic search problem: given the encrypted text, the clear text, and a key search range. This kind of problem has a few kilobytes input and output, is stateless, and can compute for days. Computing zeros of the zeta function is a good example. Monte Carlo simulation for portfolio risk analysis is another good example. And of course, SETI@Home is a good example: it computes for 12 hours on half a megabyte of input.
...
Most web and data processing applications are network or state intensive and are not economically viable as mobile applications. An FTP server, an HTML web server, a mail server, and Online Transaction Processing (OLTP) server represent a spectrum of services with increasing database state and data access. A 100MB FTP task costs 10 cents, and is 99% network cost. An HTML web access costs 10 microdollars and is 88% network cost. A Hotmail transaction costs 10 microdollars and is more cpu intensive so that networking and cpu are approximately balanced. None of these applications fits the cpu-intensive stateless requirement.

One alternative to social recommendations is data-based recommendations, where some site or company records my habits and points me to interesting stuff. The champion here is without a doubt Amazon. Their recomendation engine simply rocks:

Amazon makes recommendations based on my browsing history, my shopping history, other user's browsing history, other user's shopping history, the category tree of their books or the assumption that if I liked something from the author I might also like his other books. Here's a short PDF describing their system in greater detail:

Summary: Recommendation algorithms are best known for their use on e-commerce Web sites, where they use input about a customer's interests to generate a list of recommended items. Many applications use only the items that customers purchase and explicitly rate to represent their interests, but they can also use other attributes, including items viewed, demographic data, subject interests, and favorite artists. At Amazon.com, we use recommendation algorithms to personalize the online store for each customer. The store radically changes based on customer interests, showing programming titles to a software engineer and baby toys to a new mother. There are three common approaches to solving the recommendation problem: traditional collaborative filtering, cluster models, and search-based methods. Here, we compare these methods with our algorithm, which we call item-to-item collaborative filtering. Unlike traditional collaborative filtering, our algorithm's online computation scales independently of the number of customers and number of items in the product catalog. Our algorithm produces recommendations in real-time, scales to massive data sets, and generates high quality recommendations.

In the case of Amazon, I don't have to buy the recommendation --- being on-topic is enough to get me to click on the book, kicking off a browsing session which may result in a purchase (or pushing books to my shopping list).

The real question is, why is Amazon the only site that gets it right?

I can name two sites that I visit many times a day where data-based recommendations would rock:

Internet Movie Database. I use IMDb all the time for checking movies before watching them and to see what other people thought after I saw a movie. IMDb should have massive amounts of data about my movie habits, yet their recommendation engine sucks. (Note that IMDb is owned by Amazon, so this is just weird.) For example, I open the page for the movie In the Valley of Elah, and at the bottom of the page I see 5 recommendations. These are not personal recommendations, my preferences / history have not been taken into account. It's simply a list of movies deemed similar to the movie I'm viewing. (You can verify this by logging off and visiting the page again.) And they suck too. They recommend Rambo, which is pretty pointless since everybody's seen Rambo.

Google Reader. Power-users use Google Reader to avoid having to visit all the blogs they read. I'm always looking for blogs that might be interesting to read. Google has a large database of data telling it what feeds users subscribe to and what articles users click. It should be possible to make good recommendations based on this data, not to mention all the other data Google has on us. Still, Google Reader's recommendations suck. They don't even attempt to recommend individual articles, only entire feeds / blogs.

For Amazon recommendations are a core of part of their business. Unfortunately for users, IMDb and Google Reader probably don't see a good way to monetize such a feature, hence the lack of effort.

The NetFlix challenge was a great way to draw attention to recommendation algorithms, but I'm not aware of any startups coming out of it. This should be a call-to-arms for entrepeneurs to figure out a way to make money off recommendations. The task is a difficult one, since --- apart from finding a viable business plan --- you also face the classic chicken-or-the-egg problem: you need a lot of data to make good recommendations.

O'Reilly's Beautiful Code is an excellent book with 33 short, bite-sized chapters about interesting code. After I read the book I thought it'd be a good retrospective exercise to write your own Chapter 34 to share insights with other programmers. So here it goes:

Some time ago I worked at a fairly large one-product software company. Roughly every 2 years they would come out with a new version of their software (let's call it Software X) to keep the cash flowing in. On the side, the company also sold customized versions of the software to the far-east, where it's customary for large companies to request custom versions of software tailored to their needs. I was part of the Customization Team.

As a matter of company policy, the changes the Customization Team produced never made it back to the base source tree of Software X. To put it in perspective, each customized version contained roughly 1% changes if measured in lines of code. There were several customized versions, on the order of 10. The customized version shared some customized features, so there was overlap in the code. As the Base Team working on Software X created new stable builds, we had to obtain a copy of their new source tree and merge in our custom code. We had to do this every time a major milestone was reached on Software X, roughly every 3-6 months. This is what we called migration: take the old customized version, call it Custom-old, the new base version, call it Base-new, and produce the new customized version Custom-new.

Unfortunately, our changes were not only in source code: we had a custom build system built on top of theirs, so we could build all our custom versions from the same source tree. This involved changing the source tree at certain points in a systematic way. For example, we had to keep files in different places, such as keeping a copy of all the original resource files in one place, and product specific resource files for each custom version (such as the splash screen), and our build system would manage these before calling the base build system. To sum it up, we not only changed source code, but moved files around, duplicated some, created some, etc.

Human Labor. Before the group started using a tool to aid migraiton, a human (one of the programmers) would manually migrate by copying files and directories, set up our custom directory structure and merge the source and resource files.

Imitation. The programmer given this problem before me implemented an imperative migrator in C++ which mimicked what previously the human would have done: copy this, rename that, merge these files, etc. This is what it looked like:

CopyFiles(from1, to1);
...
std::vector exceptions;
exceptions.push_back(filename1);
exceptions.push_back(filename2);
...
CopyFiles(from2, to2, exceptions);

Obviously, C++ is not the best language for such a tool. However, this was not the core problem with this implementation, had he used Perl I would still be writing this article. The problem was that the source tree was highly unstable from version to version, in the sense that files appeared, disappeared or moved around the directory tree. The tool had to be patched often, and even when we considered running it a success after manually checking and verifying the migration we always had to fix the source tree up by hand to get a working version.

It was obvious that the imperative program telling the computer to move around files and then present certain files for merging to the user was too complicated because the source tree was too complex (over a gigabyte). When given the chance, I convinced my manager that it had to be replaced.

Evolution. My first improvement over the original was evolutionary. In the last step of migration, the tool would present all source files that we modified for manual merging. This meant sitting there for hours and clicking CTRL + right-arrow in Araxis Merge to bring over our modifications. The idea here was simple: if given the base source file BaseFile, which we modified to get CustomFile-old, the only time it was neccesary to bring up a merge window is if BaseFile actually changed in the new base version, eg. if BaseFile-old != BaseFile-new. In terms of pseudo-code:

if (BaseFile-new == BaseFile-old)
  CustomFile-new := CustomFile-old;
else
  CustomFile-new := Merge(..);

This is, of course, trivial to anyone who has ever used a source control system like cvs. (In fact, source control systems are even smarter since they detect where in a file changes occurred.) In terms of our migration process, this involved bringing the old base version Base-old into the game. This improvement reduced the number of diffs presented to the human migrator by 90%, saving hours of work on each migration. Unfortuantely, the process was still highly unstable.

Revolution. After convincing my manager that the migration tool could be improved, I set out to design my version. I was guided by two main design principles:

• design special-purpose meta-languages for your problem domain
• use declarative syntax whenever possible
The first point is best illustrated by John Carmack’s QuakeC, an interpreted language used to script the Quake engine. My knowledge of declarative languages is due to several semesters of Prolog at University.

The beautiful idea was to think in terms of migration rules instead of concrete migration operations such as copying or merging files. The rules are declarative, and make up the meta-language of the first point.

The migrator’s central object is a multi-file iterator. (I cannot find a case of someone else doing this, so I may have invented something?). The migrator is told to iterate n directories in parallel by matching filenames. On each iteration, it calls the user supplied ruleset, which tells it how to create the file in the resulting source tree.

Suppose the following three directories (colors denote different file contents):

Directory1	Directory2	Directory3
Apple.txt	Apple.txt	Null
Null	Banana.txt	Banana.txt
Orange.txt	Orange.txt	Orange.txt
Peach.txt	Null	Null

In the example above, we are iterating three directories is parallel. Directory1 contains three files, Apple.txt, Orange.txt and Peach.txt, Directory2 contains three files, Apple.txt, Banana.txt and Orange.txt, Directory3 contains two files, Banana.txt and Orange.txt. The iterator would return four times. The first time for Apple.txt, which exists in Directory1 and 2 but not in 3, the second time for Banana.txt, and so on. Every time the iterator returns, the user supplied ruleset is called. The return value of the rule determines how the given file (Apple.txt, Banana.txt, Orange.txt, Peach.txt) is created in the target directory. Suppose the ruleset looks like this:

// rules for files
if (exists in Directory3)
  copy from Directory3
else if (exists in Directory1 and Directory2)
{
  if (same in Directory1 and Directory2)
    copy from Directory1
  else
    copy from Directory1 and Merge from Directory2
}
else if (exists in Directory1)
  copy from Directory1
else
  copy from Directory2

The ruleset would produce the following result:

Directory1	Directory2	Directory3	TargetDirectory
Apple.txt	Apple.txt	Null	Copy Apple.txt then merge with Apple.txt
Null	Banana.txt	Banana.txt	Banana.txt
Orange.txt	Orange.txt	Orange.txt	Orange.txt
Peach.txt	Null	Null	Peach.txt

Following the declarative approach, the ruleset itself doesn’t perform any operations (such as copy), instead it signals what its desired action is by returning an appropriate object to the framework. In other words, the rulesets are callback functions. The above example tells most of the story, with one exception: directories can contain other directories. Thus, the user actually supplies two rulesets, one for files and one for directories. The above ruleset is thus accompanied by another one for directories:

// rules for directories
if (exists in Directory1 and Directory2 and Directory3)
  create directory, descend and apply rulesets recursively
else if (exists in Directory1 and not in Directory2 and not in Directory3)
  copy from Directory1
else if (exists in Directory1 and Directory2 and not in Directory3)
  log path and do nothing
...

Without reproducing actual code these examples tell most of the story, with one exception: one ruleset does not work everywhere in the sourcetree. For our merging process, I created ~10 different rulesets, and applied them for different subtrees of our source tree. Also note that the different rulesets iterated a different number of source directories to figure out what to do, which was "free" with the framework, since the multi-file iterator is implemented for the general n-case.

The one aspect of this approach that is not beautiful is keeping nested subtrees apart. Imagine applying Ruleset1 to DirectoryA’s subtree, which contains DirectoryB, where we would like to use Ruleset2. This has to be included in the ruleset itself:

if (directory.path == “.../DirectoryB”)
  do not descend

Fortunately, these cases were rare.

Conclusion. The declarative migration tool worked out very well. With the new migration tool, the process went far quicker and smoother than before, while trivially supporting features such as logging of weird conditions (see previous example). The entire framework was about 1000 lines of code, but the rulesets themselves were in total about 100 – an order of magnitude less than the previous version. We went from having a special purpose “copy tool” to a general purpose migration tool with myriad of uses. Since the framework included an n-way multi-file iterator, we could have n-way rules for different parts of the tree, e.g. use a different number of source directories in each case, something we ended up doing routinely.

Implementation. I wrote the framework in .NET / C#, since the company was a Windows-shop. This was not emphasised in the article, since pretty much anything (Python, C++) could be used to implement this framework in no more than 1000 lines of code. (Note that the rules were given in C# using framework primitives, i.e. the framework did not include a parser.)

Table of contents:
1. Introduction 2. Interfaces and Implementations
3. Atoms 4. Exceptions and Assertions
5. Memory Management 6. More Memory Management
7. Lists 8. Tables 9. Sets 10. Dynamic Arrays 11. Sequences 12. Rings 13. Bit Vectors
14. Formatting 15. Low-Level Strings 16. High-Level Strings
17. Extended-Precision Arithmetic 18. Arbitrary-Precision Arithmetic 19. Multiple-Precision Arithmetic
20. Threads

Short version. This book is highly recommended for the padawan C programmer on his quest to become a master. For the experienced C programmer this will be an easy and enjoyable read with one or two tricks or useful idioms per chapter. It is often recommended to read other people's code, but reading an exceptionally well-written book about good code is much more fun.

The book is written in the literate programming style, meaning that the book is interspersed with code fragments. Although it requires page flipping at times and is odd at first, it is the best technique for presenting code with the accompanying description I have come across. The worst aspect here is the author's use of the += meta-operator for appending code to a previous fragment, which almost always requires page-flipping.

As the book's title suggests the most basic pattern put forth is the seperation of interface into a .h file and implementation into .c files. Beginner programmers will first learn the

#ifndef FOO_H
#define FOO_H
... interface for FOO
#endif

pattern used in C programs to avoid duplicate #includes.

The objects discussed in the book are every programmer's standard tools such as:

• memory managers
• lists, tables, sets, rings
• atoms, strings
• threads

Fortunately most programmers don't need to reimplement these and instead use standard libraries. In the case of C++ the Standard Template Library (STL) includes much of the functionality described by Hanson. Still, real programmers know what goes on under the hood of their libraries, so looking at the author's implementations is highly recommended. Since programmers working in high-performance, systems and database programming have to (re)implement custom versions of these modules, looking at Hanson's code is easily worth it for them.

C Interfaces and Implementations is one of the few books where the Further Reading and Exercises sections at the end of each chapter are worth reading. Unlike academic style enumerated bibliographies here Hanson gives a short one sentence overview about the referenced papers and books. The Exercises point to possible extensions and modifications of the modules described in each chapter. Also, if you end up using Hanson's code in your projects the Interface Summary at the end of the book will come in handy.

Experienced programmers will find some oddities to disagree with. Here I will mention two I have come across.

Typedef'd data types. Throughout the interfaces Hanson uses the convention that object type names are typedef'd to T, so instead of writing out append(List* l, void* obj) he would write append(T l, void* obj). In this case, the * would be in the typedef making the function declarations hard to interpret. This trick seems counterproductive, more is lost in readability than gained by typing less characters.

Exceptions. One of the first modules described is an exception-handling mechanism implemented with the aid of the C preprocessor, in other words macro magic. If C is a motorcycle and C++ is a car, then this is like welding two motorcycles together to get a car. If I couldn't live without exceptions, I'd just use C++.

These issues aside, the book is highly recommended for the beginning C programmer, as it will teach through repeated use the best-practice idioms of C programming style. These patterns are demonstrated using common objects such as lists and sets, so beginners will gain additional insight from their inner workings. For experienced programmers the book will mostly be a fun read. However, advanced concepts such as the implementation of green threads will most likely be new material even for the seasoned programmer.

Still not convinced? Visit the book's official homepage to download a sample chapter and all the C source code.

Given bloatware, what is the end-user to do?

Ben Kenobi would tell you to use an elegant weapon from a more civilized time.

1. Use an older version. In this case, oldversion.com leads the way. Whenever I do a reinstall, I use it to retrieve:

• Adobe Reader 6
• Nero 6
• WinAMP
• PowerDVD
• WinRAR
• ...

2. Use a replacement. This is where it gets interesting, and why this post is tagged as Entrepreneurship. When a software corporation gets too large, management types take over decision making and start relying on releasing a new version every 12-24 months for cashflow while minimizing risk. The result is software bloat, an opportunity for startups to create a lightweight, killer version of the software. This is what happened or is happening to the bloatware I mentioned in Part I:

• Adobe Reader superseded by FoxIt Reader
• Nero is superseded by DeepBurner
• Adobe Photoshop superseded by Paint.NET
• Norton AntiVirus superseded by AVG

Notice the word supersede:

Main Entry: su·per·sede Pronunciation: \ˌsü-pər-ˈsēd\ Function: transitive verb ... 1 a: to cause to be set aside b: to force out of use as inferior 2: to take the place or position of 3: to displace in favor of another

There are many markets where the older players have become bloatware but no good replacements have yet popped up, presenting a business opportunity for startups. Here I will only mention one example I have personal experience with as a software developer: architectural CAD software. The standard players all have old legacy codebases which are bloated, they are very slow to add features and are unable to improve the application in fundamental ways. The market might be ripe for a small, lean and mean CAD machine. (On the other hand, CAD is not mainstream software, so marketing and sales is hard.)

3. Use a web application. AJAX and Flash is where all the start-up activity is in Silicon Valley. When using a web application, users innately expect less functionality that works better as compared to a desktop application, making these type of markets ideal for a <12 month startup cycle. Again sticking to the four examples in Part I:

• Online PDF viewing: Scribd makes it possible, but it's a step backward since it's Flash, so there's still room to innovate; e.g. instead of rendering in Flash, render in HTML!
• Online backup software: the user's upstream bandwidth is the only limit here; once that is surpassed, doing local backups and burns will be superseded by a company doing it for you and snail mailing the media to your postal address
• Online image editing: there's SplashUp but there's lots of room to innovate here
• Online AntiVirus: since most users get viruses through the Internet there's probably a market for a scanning and blocking proxy service

The Web 2.0 motto is Do one thing and do it well. In the desktop software world you have to do more, but the idea is the same: be lean and mean with a clean user interface and workflow. Bloatware always looses in the end.

Software bloat, also known as bloatware or elephantware, is a term used in both a neutral and disparaging sense, to describe the tendency of newer computer programs to be larger, or to use larger amounts of system resources (mass storage space, processing power or memory) than necessary for the same or similar benefits from older versions to its users. Additionally, the term bloatware is used in common language for pre-installed, huge software bundles, mostly consisting of demos and trial ware.

My favorite example of bloatware is Adobe Reader, currently at version 9. This latest version is a 33MB download, but you can't just simply download it. You first have to get the Adobe Reader Download Manager, a plugin just for downloading Adobe Reader (!). After installation the thing takes up 210MB of your harddrive (!). Ben Hoyts' article is especially funny:

For starters, version 8 was a 22 MB download. Version 9 is a mere 33 MB — a whole major version up, and not even twice as big. The default download includes a 19 MB eBay Desktop program, for all your “faster searching, smarter bidding” needs. So tempting … but I opted out.

On my cable connection, it took about 5 minutes to download, nicely allowing me enough time to brew a decent cup of coffee while I waited.

But it wasn’t just a simple, ordinary download. First Adobe told me to download a Firefox plugin. I assumed the plugin would help me read PDFs in my browser. But oh no, this was a special plugin, an Adobe Reader Download Manager (TM) — a plugin specially designed to help Firefox download Adobe’s powerful PDF viewer.

After the plugin was installed and Firefox restarted, the download began. I couldn’t wait to try Adobe’s MUCH FASTER and SMARTER product.

Download done, I double-clicked the installer. It spent a long time unpacking and validating the installer, which gave me warm fuzzies about Adobe’s good grasp of stability, security, and enterprisey robustness.

After the unpacking, the install process itself took 10 minutes. I could only thank Adobe’s engineers, presuming they were filling up my hard drive with yummy icons, tasty DLLs, and amazing 3D JavaScript add-ons. No matter — the 210 MB it required was there to be used.

I had just slurped down the dregs of my coffee when the installer finished. I was so thankful when it told me I needed to restart my computer, welcoming the extra time to drink coffee, as well as the pure delight I knew I’d get from starting all my applications again.

I could finally try out this new software. I was impressed. It started in a minuscule 13 seconds, plus the time it took me to skim their poetic and beneficent license agreement.

My other favorite is Nero, the CD and DVD burning tool. The latest version is Nero Ultra Edition 8, a 185MB download, which includes: Nero Home, StartSmart, Burning ROM, WaveEditor, SoundTrax, Vision, Recode, ShowTime, MediaHome, PhotoSnap, Cover Designer, Toolkit, BackItUp, and SoundBox. The amount of bloat here is stunning. There's actually an RSS reader in there, so you can catch up while waiting for your CD to finish. Nero went from burning CDs to wanting to "Create and Manage Your Digital Life", this coming straight from their website. Nero is so bad, there's actually a project called Nero Lite, where they strip the bloat from the installer to bring it down to 20MB.

And then there is Adobe Photoshop. The latest version, Adobe Photoshop CS3 comes in at 1GB installed (that's 2GB on the Mac, probably because it includes both the PowerPC and Intel binaries) according to adobe.com. Photoshop is so bloated that one of the developers, John Nack wrote about it on his personal blog:

Photoshop has been accreting power & users for the better part of two decades. The once-little app has proven almost endlessly adaptable to new needs and workflows, but all that morphing has a price. In many cases we've traded simplicity for power, and not all the pieces look like part of a cohesive whole. In fact, I sometimes joke that looking at some parts of the app is like counting the rings in a tree: you can gauge when certain features arrived by the dimensions & style of the dialog. (Cue old-timey prospector voice: "Oh, Lighting Effects--you can see the scorch marks from the great fire of '43...")

This isn't exactly a news flash--far from it. So, the question is, What exactly are we gonna do about it? No one wants to work with--or work on--some shambling, bloated monster of a program. (Emphasis added)

The last example is Norton AntiVirus. According to symantec.com, the latest 2008 version takes up 300MB on your disk. But with Norton, it's not so much the hard disk space as it is the fact that is installs memory resident daemons that completely hijack your system. Most computers will crawl to a halt and become unusable if Norton is running. Norton AntiVirus has become the problem it was designed to eradicate: a virus. I personally would rather have viruses on my computer than run Norton AntiVirus. Others have come to a similar conclusion:

As I recall, Norton was the program that harassed her so badly that she decided to switch to Mac. I remember my experience with using a PC and having Norton. It was terrible. I finally came to the conclusion that I had to uninstall it, regardless of what perils I might be subjecting myself to. I was sure that Norton screwed with my computer’s performance far more than any virus would. (Emphasis added)

Continue to Part II.

McElroy believes the IT industry has much to learn from Hollywood's eventual transformation into an industry in which largely outsourced production companies primarily make films on a project-by-project basis.

What follows is my interpretation of the model.

In the Hollywood Model there is no continuous employment, instead, participants are handpicked and assembled on a project-by-project basis. This has some obvious benefits and drawbacks:

Bootstraping overhead. The producer has to re-assemble a team for each project. Depending on the size of the product, this might take several weeks or months. The model seems to be ill-suited when a quick response (e.g. to competition) is required.

Competition (participants). The producer can hire the best for each project. Participants who have proven themselves previously are re-hired. Poor performers are not.

Competition (projects). Participants may choose between different projects. High-performers will choose more interesting tasks.

Wages. Proven high-performers can demand higher wages. When compared to the Corporate Model, different wages introduce less friction.

Dynamics. There is constant change and rearrangement of projects and groups. Thus, high-performing combinations are more likely to form.

No job safety. Participants are likely to be left without projects for differing amounts of time. For people with families, this might be unacceptable.

Alternative management. In her article, Sue Bushell quotes others:

"I don't have to ramp up, I don't have to ramp down, I don't need a huge management layer to oversee the resources. And I don't have the cashflow crunches that you go through when you have a big staff, the work is over, and you must come up with something for everybody to do ... or let them go," says Seropian.

The bootstraping overhead alone is enough to warrant a Corporate Hollywood Model, which is used by some large corporations such as game developers (and Google?). Participants are employees assigned to a project within the "corporate matrix". This in effect voids most of the benefits (such as wages based on merit) for employees.

In other corporate sectors this model is counter-productive. In desktop software publishing, the corporation's revenue stream is tied to delivering and selling new versions of the software every 12-24 months. It doesn't make sense to constantly re-assemble and disband the teams.

The Hollywood Model is probably well-suited for Internet startups, which are characterized by an initial surge of creativity / development followed by periods of maintenance. Only if the site is successful will there be a second surge, making continuous employment impractical.

Smaller web development outfits naturally use the model and refer to it as contract-based work. In this case, projects are small so the bootstrapping overhead is negligible. In my experience, these types of projects usually do not involve cutting-edge development, so top talent avoids it.

Others on this topic:

• Sue Bushell: Replicating the Hollywood Model
• Matthew Reinbold: Exploring the Hollywood Model