Simon Willison’s Weblog

How we use Redis at Bump. A couple of neat tricks I hadn’t seen before: using Redis to aggregate log files from multiple servers (they all push in to a Redis queue, then one process pulls from the queue and writes to disk), and using Redis blocking queues for RPC by specifying a different temporary queue to return the result. # 16th July 2011, 4:37 pm

The New Heroku (Part 4 of 4): Erosion-resistance & Explicit Contracts. I really like Adam’s description of Software Erosion—I’ve seen that happen to my projects a bunch of times, and it really is an important problem to solve. # 29th June 2011, 5:26 pm

Visualizing WebKit’s hardware acceleration. Command line flags for launching Safari (and the iOS simulator) in a way that highlights areas of the screen that are being hardware accelerated—particularly useful if you are using the “-webkit-transform: translate3d(0,0,0)” trick. # 27th June 2011, 10:31 am

On HTTP Load Testing. Mark Nottingham explains that running good HTTP benchmarks means understanding available network bandwidth, using dedicated physical hardware, testing at progressively higher loads and a whole lot more. # 18th May 2011, 10:17 am

We Need to Stop Google’s Exploitation of Open Communities. Mikel Maron from OpenStreetMap is justifiably angry about Google MapMaker, which copies OpenStreetMap’s model of crowdsourcing geographic data (even copying the OSM idea of Mapping Parties) but keeps the data under a much more restrictive license, and uses the Google brand to market itself to African governments. # 22nd April 2011, 10 am

Why Facebook open-sourced its datacenters. Jon Stokes speculates that Facebook plan to use open source hardware to compete with Google at datacenter efficiency . This isn’t a new pattern. Years ago when I worked at Yahoo! I was furiously jealous of the secret sauce technologies that allowed Google to build big applications faster than anyone else, such as BigTable and map/reduce. Today, the open source world has created better, free alternatives—sponsored in part by Facebook, Yahoo! and other Google competitors. # 9th April 2011, 7:54 am

Qwery—The Tiny Selector Engine. A quarter of the size of Sizzle (1K gzipped and minified) due to only supporting ID, class and attribute selectors. Could be useful for things like embeddable widgets and badges, where depending on a larger library is impolite. # 2nd April 2011, 8:27 am

Product design at GitHub. At GitHub, every employee is a product designer. # 2nd April 2011, 7:51 am

YC Is Not a School for Startups—It Is Marine Corp Boot Camp for Startup Founders. This is a great description of what it’s actually like to do YC. If you’ve been wondering why I haven’t blogged much over the past three months, this is why. # 20th March 2011, 5:27 pm

Your Web, Half a Second Sooner. Google AdSense now serves a tiny bit of JavaScript that loads everything else in a dynamically populated iframe, thus avoiding blocking the rest of the page load. It’s about time online advertising providers started taking page performance seriously. # 17th March 2011, 5:39 pm

DNS Prefetching Implications. deviantart use a subdomain per user, which meant the DNS prefetching feature in Firefox and Chrome was costing them an extra 10 billion DNS queries per month. Disabling it with a meta tag saves them $1600/month in DNS service charges. # 9th March 2011, 10:54 pm

CSRF: Flash + 307 redirect = Game Over. Here’s the exploit that Django and Rails both just released fixes for. It’s actually a flaw in the Flash player. Flash isn’t meant to be able to make cross-domain HTTP requests with custom HTTP headers unless the crossdomain.xml file on the other domain allows them to, but it turns out a 307 redirect (like a 302, but allows POST data to be forwarded) confuses the Flash player in to not checking the crossdomain.xml on the host it is being redirect to. # 10th February 2011, 10:07 pm

Going Postel. Jeremy points out that one of the many disadvantages of publishing JavaScript dependent content on the Web is that a single typo can render your entire site unusable. # 9th February 2011, 2:18 am

Breaking the Web with hash-bangs. Mike Davies explains why Gawker’s new Ajax fragment-tastic redesign is a web architecture error of colossal proportions. # 9th February 2011, 2:17 am

elasticsearch: Percolator. Another fascinating elasticsearch feature: Percolator lets you register searches with your elasticsearch cluster, then pass in a document and have the matching query IDs returned. It’s an upside down search engine. I’m sure there are some very neat things you could build with this, I just haven’t figured out what they are just yet. # 8th February 2011, 11:16 pm

UK Web Archive: WW2 People’s War. Good news: the British Library has already archived the BBC’s WW2 People’s War site (on 22nd May 2006). # 8th February 2011, 12:58 am

Erase and rewind. The BBC plan to delete 172 sites from the bbc.co.uk domain, including WW2 People’s War, described thus: “The BBC asked the public to contribute their memories of World War Two to a website between June 2003 and January 2006. This archive of 47,000 stories and 15,000 images is the result.” Appalling. # 7th February 2011, 4:24 pm

How we made an API for BoingBoing in an evening. Fluidinfo really is a fascinating piece of software. The team loaded in 11 years of BoingBoing content, allowing you to run structured queries against the data using their standard API, but also allowing users to attach their own information to the same corpus using Fluidinfo tags. Writable APIs are much less common than read-only APIs—Fluidinfo instantly provides both. # 28th January 2011, 10:56 pm

Get Lanyrd conference recommendations by email. This is the first time I’ve built a custom email subscription feature, and it’s been a very interesting ride. We’re trying to find the right balance between keeping people informed in a timely fashion with useful information while not overloading their inbox with too many messages. You can opt for daily, weekly, fortnightly or monthly emails and we’ll try to ensure you only hear about events you haven’t seen before. # 28th January 2011, 11:28 am

Google APIs & Developer Products. Presented as a sort-of-periodic table. There’s quite a bit of stuff on here I didn’t know about. # 28th January 2011, 11:25 am

37signals Product Blog: We’ll be retiring our support of OpenID on May 1. The support costs far outweighed the benefits to customers, especially now that 37signals have their own single sign in mechanism that works across all of their products. # 25th January 2011, 4:17 pm

Tip: Flickr standard photo response as slideshow. Neat trick—you can construct a URL to Flickr’s slideshow widget that includes the results of any API method, including the all-powerful flickr.photos.search. It’s a shame you can’t embed the resulting slideshow in an iframe. # 25th January 2011, 3:51 am

The code injected to steal passwords in Tunisia. Here’s the JavaScript that (presumably) the Tunisian government were injecting in to login pages that were served over HTTP. # 24th January 2011, 6:45 pm

The Inside Story of How Facebook Responded to Tunisian Hacks (via) “By January 5, it was clear that an entire country’s worth of passwords were in the process of being stolen right in the midst of the greatest political upheaval in two decades.”—which is why you shouldn’t serve your login form over HTTP even though it POSTs over HTTPS. # 24th January 2011, 6:06 pm

The science of the hashtag. Interesting analysis of how the #lessambitiousmovies hash tag took off thanks to retweets from a couple of key users with very creative followers. # 14th January 2011, 4:02 am

Display your events on your own website with Lanyrd Badges. We’ve launched badges for Lanyrd—JavaScript that lets you embed a top bar or a content “splat” showing events you plan to attend, talks you’ve given in the past and other various combinations. I’m quite pleased with the implementation—the badges are configured using classes on a link to your Lanyrd profile, and the badges themselves are served through a combination of Amazon CloudFront for the initial script and a Varnish cache for the badge data itself to keep things nice and snappy. # 13th January 2011, 8:38 pm

The Virtues of Monitoring. Fantastic guide to the various levels of monitoring required for a modern web application. # 13th January 2011, 4:26 am

The First Few Weeks—ep.io. Another take on managed Python Django/WSGI hosting, from Andrew Godwin and Ben Firshman. # 13th January 2011, 4:25 am

Hello from Gondor. “Effortless production Django hosting” from the Eldarion team. # 13th January 2011, 4:24 am

Introducing the FluidDB Explorer. Every good API deserves a dedicated API browser. # 13th January 2011, 4:19 am