Simon Willison’s Weblog

Open Mosix

I can’t remember how I stumbled across it, but Open Mosix looks like a really interesting project. It’s a Linux kernel extension that makes creating a Linux cluster is as simple as installing a kernel module on a number of machines and supplying each one with a shared config file.

Once the cluster is set up, any of the machines on the network has the ability to farm long running processes out to a different box. The clustering only kicks in when a process is running in the background takes more than a few seconds to execute. Once that happens, Open Mosix checks the load averages on the machines in the cluster and, if a better host is found, migrates the process over to the other machine. The whole process is completely transparent; as far as the end user is concerned the processes they run just keep on running until they terminate.

Since clustering only kicks off for longer running processes this woudn’t be much use for something like a web server farm, but could be ideal for tasks such as compilation or rendering where individual processes perform computationally intensive work for a long period of time.

Even more fascinating is ClusterKnoppix, a modified Knoppix distro that uses an Open Mosix enabled kernel. Burn a bunch of CDs, boot some standard networked PCs with them and you’ve got an instant cluster.

I’ve been doing some pretty intense log crunching today on a year’s worth of daily web server logs, each one between 15 and 20 MB in size. Since most of the work is in running a regular expression on each line of each file it’s likely a cluster would have speeded the whole thing up considerably. Unfortunately, I doubt my co-workers would have been overjoyed with me turning their machines in to cluster nodes for the day.