19th December 2003
I can’t remember how I stumbled across it, but Open Mosix looks like a really interesting project. It’s a Linux kernel extension that makes creating a Linux cluster is as simple as installing a kernel module on a number of machines and supplying each one with a shared config file.
Once the cluster is set up, any of the machines on the network has the ability to farm long running processes out to a different box. The clustering only kicks in when a process is running in the background takes more than a few seconds to execute. Once that happens, Open Mosix checks the load averages on the machines in the cluster and, if a better host is found, migrates the process over to the other machine. The whole process is completely transparent; as far as the end user is concerned the processes they run just keep on running until they terminate.
Since clustering only kicks off for longer running processes this woudn’t be much use for something like a web server farm, but could be ideal for tasks such as compilation or rendering where individual processes perform computationally intensive work for a long period of time.
Even more fascinating is ClusterKnoppix, a modified Knoppix distro that uses an Open Mosix enabled kernel. Burn a bunch of CDs, boot some standard networked PCs with them and you’ve got an instant cluster.
I’ve been doing some pretty intense log crunching today on a year’s worth of daily web server logs, each one between 15 and 20 MB in size. Since most of the work is in running a regular expression on each line of each file it’s likely a cluster would have speeded the whole thing up considerably. Unfortunately, I doubt my co-workers would have been overjoyed with me turning their machines in to cluster nodes for the day.
More recent articles
- Weeknotes: datasette-enrichments, datasette-comments, sqlite-chronicle - 8th December 2023
- Datasette Enrichments: a new plugin framework for augmenting your data - 1st December 2023
- llamafile is the new best way to run a LLM on your own computer - 29th November 2023
- Prompt injection explained, November 2023 edition - 27th November 2023
- I'm on the Newsroom Robots podcast, with thoughts on the OpenAI board - 25th November 2023
- Weeknotes: DevDay, GitHub Universe, OpenAI chaos - 22nd November 2023
- Deciphering clues in a news article to understand how it was reported - 22nd November 2023
- Exploring GPTs: ChatGPT in a trench coat? - 15th November 2023
- Financial sustainability for open source projects at GitHub Universe - 10th November 2023
- ospeak: a CLI tool for speaking text in the terminal via OpenAI - 7th November 2023