Simon Willison’s Weblog

Introducing Resque. A new background worker management queue developed at GitHub, using Redis for the persistence layer. The blog post explains both the design and the shortcomings of previous solutions at length. Within 24 hours of the release code an external developer, Adam Cooke, has completely reskinned the UI.

# 4th November 2009, 8:20 pm / resque, open-source, redis, github, queue, workers, ruby, sinatra

Beanstalkd. This is the light-weight cross-language queue I’ve been waiting for. Similar to Starling but your workers don’t need to poll for new jobs; you can call the blocking “reserve” call instead.

# 12th May 2008, 9:12 am / starling, beanstalkd, queue, lightweightqueue

RubyForge: Starling. “Starling is a light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter’s backend, and is in production across Twitter’s cluster.”

# 11th January 2008, 9:47 pm / memcached, twitter, ruby, messaging, queue, starling, blaine-cook, rubyforge, message-queues

Techniques for safely consuming external HTTP on demand? I asked this question on programming.reddit.com yesterday and got some really insightful answers, including Joe Stump from Digg describing how Digg Images uses Danga’s Gearman worker queue.

# 15th December 2007, 12:29 pm / http, queue, workers, gearman, reddit, askreddit, joe-stump, digg, danga, scaling

lwqueue. Lightweight cross-language message queue system, written in Perl with client libraries in Perl, Python and Ruby.

# 16th July 2007, 10:04 am / perl, python, ruby, queue, lightweight, lwqueue, peter-cooper