Simon Willison’s Weblog

Introducing docent. Paul Mison’s clever little Flickr app for viewing galleries that have been added by your contacts. It runs in Python on App Engine and makes extensive use of the Task Queue API. 28th January 2010, 8:35 pm

BLPOP and BRPOP in Redis. Added over Christmas—Redis now has blocking list pop operations. This means you can use Redis to drive a queue server without the need for polling—simply BLPOP against a key and, if it’s empty, your client will block until another client pushes an item on to the list. Multiple clients can block against the same key and only the first client will return when an item becomes available. 7th January 2010, 10:50 pm

Announcing Alice and Wonderland. Continuing the RabbitMQ “stuff to do with rabbits” naming convention, Alice is a RESTful interface to RabbitMQ which exposes information about vhosts/queues/users/exchanges/etc as JSON. Wonderland is a web UI for RabbitMQ implemented as a pure Ajax application which calls Alice. 17th July 2009, 9:12 am

Rabbits and warrens. Handy tutorial introduction to using RabbitMQ and AMQP with Python. 7th April 2009, 9:13 am

New Gearman Server & Library in C, MySQL UDFs. Gearman, the job queue written for LiveJournal and now used by Digg and Yahoo!, has been rewritten in C. Looks like a good candidate for an easily configured lightweight message queue. Also includes hooks for writing MySQL functions that can interact with queues. 13th January 2009, 4:41 pm

Flickr Engineers Do It Offline. Flickr wrote their own queuing mechanism (in PHP), and currently run ten queue servers on dedicated hardware for tasks like pushing new photos in to indexes, denormalisation and “backfills” which move data between clusters and run bulk scripts against large numbers of existing rows. 28th September 2008, 1:24 am

Queue everything and delight everyone. Les Orchard explains why I’ve been getting interested in queues recently: “One of the problems it seems most modern web apps face is the tendency to want to do everything all at once, and all in the same code that responds directly to a user.” 4th July 2008, 10:38 pm

Standing in Line. Simon Wistow coins “CLAMP” for LAMP + Cache, and expresses the need for a dirt-simple, high performance open source queue system. 19th March 2008, 9:41 am