Simon Willison’s Weblog

Blogmarks tagged scaling in 2007

Filters: Type: blogmark × Year: 2007 × scaling ×


Eventually Consistent. Werner Vogels explains the trade-offs involved in building scalable, highly-available data stores such as Amazon’s SimpleDB. # 20th December 2007, 5:59 pm

Techniques for safely consuming external HTTP on demand? I asked this question on programming.reddit.com yesterday and got some really insightful answers, including Joe Stump from Digg describing how Digg Images uses Danga’s Gearman worker queue. # 15th December 2007, 12:29 pm

NginxMemcachedModule. nginx can be set up to directly serve a URL from memcache if the corresponding cache key is set, and fall back to a backend application server otherwise. Application servers can then write directly to memcache when content needs to be cached or goes stale. # 15th December 2007, 1:59 am

What You Need To Know About Amazon SimpleDB. Amazon have finally launched the database component of their web service suite. It fits a bunch of current trends: key/value pairs, schemaless, built on top of Erlang. “Eventual consistency” is an interesting characteristic. # 14th December 2007, 11:21 am

Client Side Load Balancing for Web 2.0 Applications (via) I recall that early versions of Netscape picked a random server from a hard-coded list each time a user clicked the “What’s New” button, back before server-side scaling techniques were well understood. # 5th October 2007, 11:29 pm

Scale rails from one box to three, four and five. Excellent, concise run-down of what it takes to scale a web application. Most of the advice is easily portable to other frameworks. # 30th July 2007, 1:40 pm

High Scalability (via) New blog about building scalable, reliable sites. # 26th July 2007, 8:15 pm

YouTube Scalability Talk. Kyle Cordes’ notes on a Google Tech Talk on scaling YouTube by Cuong Do. # 14th July 2007, 10:26 pm

SlideShare: Webapps scalability. Lots of great presentations on scaling, from Twitter, Digg, Vox, LiveJournal, Last.fm and more. # 4th July 2007, 12:53 am

SELECT * FROM everything, or why databases are awesome. I’m beginning to think that for scalable applications the thinner your ORM is the better—if you even use one at all. # 22nd June 2007, 12:40 am

iLike: Holy cow... 6mm users and growing 300k/day! (via) Facebook platform offers a viral distribution mechanism for free. Downside: you have to double your capacity every few days. # 13th June 2007, 9:02 am

Rapid development serving 500,000 pages/hour (via) Curse Gaming are getting impressive performance out of Django. # 24th May 2007, 4:11 pm

Wikipedia internals (PDF) (via) A gold mine of scaling tips. # 11th May 2007, 11:35 am

MintCache for Django. Caching scheme for Django that solves the dog-pile effect, where high traffic causes many processes to regenerate stale cached data at the same time. # 2nd May 2007, 8:49 am

The top 10 presentations on scaling websites: twitter, Flickr, Bloglines, Vox and more. I normally avoid linking to “top 10” lists on principle, but this one pulls together some great resources and adds extra context to each one. # 1st May 2007, 1:51 pm

Capacity Planning for LAMP (via) John Allspaw’s MySQL Conf 2007 talk on capacity planning (John is Operations Engineering Manager at Flickr). # 27th April 2007, 8:41 pm

Scaling Twitter (via) Slides from Blaine’s recent talk. # 23rd April 2007, 11:02 am

Rails and Scaling with Multiple Databases. Ryan Tomayko explains how his team spreads a high traffic Rails application across five separate PostgreSQL databases by giving each client their own schema—similar to how WordPress MU scales. # 14th April 2007, 2:32 am

Scaling Python for High-Load Web Sites. Slides from a talk at PyCon. Be sure to switch to the notes view (Ø in the bottom right)—a really nice overview of scaling up from a CGIs to load balanced, memcached Python application servers. # 4th March 2007, 9:14 pm

Data::ObjectDriver. Benjamin Trott’s Perl ORM, with built in support for both caching and data partitioning. I think this is what Six Apart uses for Vox. # 25th February 2007, 12:43 am

A brief update with some numbers for hardware load-balanced mongrels. 4000 requests/second on 48 mongrels behind a hardware load balancer. # 5th February 2007, 12:38 am

Inside MySpace.com. Case study of scaling against a network effect. Includes pretty honest coverage of the mistakes made along the way, although the article was put together second hand from conference presentations rather than from interviews. # 17th January 2007, 9:18 am