Simon Willison’s Weblog

How does Sidekiq really work? (via) I really like this category of blog post: Dan Svetlov took the time to explore the Sidekiq message queue’s implementation and then wrote it up in depth. # 5th February 2024, 5:20 pm

Transactionally Staged Job Drains in Postgres. Any time I see people argue that relational databases shouldn’t be used to implement job queues I think of this post by Brandur from 2017. If you write to a queue before committing a transaction you run the risk of a queue consumer trying to read from the database before the new row becomes visible. If you write to the queue after the transaction there’s a risk an error might result in your message never being written. So: write to a relational staging table as part of the transaction, then have a separate process read from that table and write to the queue. # 18th December 2021, 1:34 am

huey. Charles Leifer’s “little task queue for Python”. Similar to Celery, but it’s designed to work with Redis, SQLite or in the parent process using background greenlets. Worth checking out for the really neat design. The project is new to me, but it’s been under active development since 2011 and has a very healthy looking rate of releases. # 25th February 2019, 7:49 pm

Introducing Faktory. A brand new worker queue system from Mike Perham, the author of Sidekiq for Ruby. It’s written in Go on top of RocksDB and is explicitly designed to support clients and workers in multiple different languages. # 25th October 2017, 3:09 am

HotQueue. A super-simple Python work queue using Redis. The API is neat, and makes clever use of generators for blocking consumption of queue items. # 22nd December 2010, 11:51 am

Introducing docent. Paul Mison’s clever little Flickr app for viewing galleries that have been added by your contacts. It runs in Python on App Engine and makes extensive use of the Task Queue API. # 28th January 2010, 8:35 pm

BLPOP and BRPOP in Redis. Added over Christmas—Redis now has blocking list pop operations. This means you can use Redis to drive a queue server without the need for polling—simply BLPOP against a key and, if it’s empty, your client will block until another client pushes an item on to the list. Multiple clients can block against the same key and only the first client will return when an item becomes available. # 7th January 2010, 10:50 pm

Announcing Alice and Wonderland. Continuing the RabbitMQ “stuff to do with rabbits” naming convention, Alice is a RESTful interface to RabbitMQ which exposes information about vhosts/queues/users/exchanges/etc as JSON. Wonderland is a web UI for RabbitMQ implemented as a pure Ajax application which calls Alice. # 17th July 2009, 9:12 am

Rabbits and warrens. Handy tutorial introduction to using RabbitMQ and AMQP with Python. # 7th April 2009, 9:13 am

New Gearman Server & Library in C, MySQL UDFs. Gearman, the job queue written for LiveJournal and now used by Digg and Yahoo!, has been rewritten in C. Looks like a good candidate for an easily configured lightweight message queue. Also includes hooks for writing MySQL functions that can interact with queues. # 13th January 2009, 4:41 pm

Flickr Engineers Do It Offline. Flickr wrote their own queuing mechanism (in PHP), and currently run ten queue servers on dedicated hardware for tasks like pushing new photos in to indexes, denormalisation and “backfills” which move data between clusters and run bulk scripts against large numbers of existing rows. # 28th September 2008, 1:24 am

Queue everything and delight everyone. Les Orchard explains why I’ve been getting interested in queues recently: “One of the problems it seems most modern web apps face is the tendency to want to do everything all at once, and all in the same code that responds directly to a user.” # 4th July 2008, 10:38 pm

Standing in Line. Simon Wistow coins “CLAMP” for LAMP + Cache, and expresses the need for a dirt-simple, high performance open source queue system. # 19th March 2008, 9:41 am