Simon Willison’s Weblog

How the wayback machine works

How the Wayback Machine Works is a must read for anyone geeky enough to be interested in cheap clustered databases on a huge scale. The interview includes some fascinating details on the cost effectiveness of Linux clusters:

What’s amazing to me is the fact that the hardware is free. For doing things even in the hundreds of terabytes, it costs in the hundreds of thousands of dollars. When you talk to most people in IT departments, they spend a couple hundred thousand dollars just on a CPU, much less a terabyte of disk storage. You buy from EMC a terabyte for maybe $300,000. That’s just the storage for 1 TB. We can buy 100 TBs with 250 CPUs to work on it, all on a high-speed switch with redundancy built in.