Open Mosix
19th December 2003
I can’t remember how I stumbled across it, but Open Mosix looks like a really interesting project. It’s a Linux kernel extension that makes creating a Linux cluster is as simple as installing a kernel module on a number of machines and supplying each one with a shared config file.
Once the cluster is set up, any of the machines on the network has the ability to farm long running processes out to a different box. The clustering only kicks in when a process is running in the background takes more than a few seconds to execute. Once that happens, Open Mosix checks the load averages on the machines in the cluster and, if a better host is found, migrates the process over to the other machine. The whole process is completely transparent; as far as the end user is concerned the processes they run just keep on running until they terminate.
Since clustering only kicks off for longer running processes this woudn’t be much use for something like a web server farm, but could be ideal for tasks such as compilation or rendering where individual processes perform computationally intensive work for a long period of time.
Even more fascinating is ClusterKnoppix, a modified Knoppix distro that uses an Open Mosix enabled kernel. Burn a bunch of CDs, boot some standard networked PCs with them and you’ve got an instant cluster.
I’ve been doing some pretty intense log crunching today on a year’s worth of daily web server logs, each one between 15 and 20 MB in size. Since most of the work is in running a regular expression on each line of each file it’s likely a cluster would have speeded the whole thing up considerably. Unfortunately, I doubt my co-workers would have been overjoyed with me turning their machines in to cluster nodes for the day.
More recent articles
- Weeknotes: Llama 3, AI for Data Journalism, llm-evals and datasette-secrets - 23rd April 2024
- Options for accessing Llama 3 from the terminal using LLM - 22nd April 2024
- AI for Data Journalism: demonstrating what we can do with this stuff right now - 17th April 2024
- Three major LLM releases in 24 hours (plus weeknotes) - 10th April 2024
- Building files-to-prompt entirely using Claude 3 Opus - 8th April 2024
- Running OCR against PDFs and images directly in your browser - 30th March 2024
- llm cmd undo last git commit - a new plugin for LLM - 26th March 2024
- Building and testing C extensions for SQLite with ChatGPT Code Interpreter - 23rd March 2024
- Claude and ChatGPT for ad-hoc sidequests - 22nd March 2024
- Weeknotes: the aftermath of NICAR - 16th March 2024