Simon Willison’s Weblog

Items tagged messagequeues

Filters: messagequeues ×


cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity. # 21st September 2009, 11:09 pm

Announcing Alice and Wonderland. Continuing the RabbitMQ “stuff to do with rabbits” naming convention, Alice is a RESTful interface to RabbitMQ which exposes information about vhosts/queues/users/exchanges/etc as JSON. Wonderland is a web UI for RabbitMQ implemented as a pure Ajax application which calls Alice. # 17th July 2009, 9:12 am

PubSub-over-Webhooks with RabbitHub. RabbitMQ, the Erlang-powered AMQP message queue, is growing an HTTP interface based on webhooks and PubSubHubBub. # 1st July 2009, 8:22 pm

Working with Python and RabbitMQ. Nathan Borror eliminates the boilerplate needed to talk to RabbitMQ (or any other AMQP queue server) from Python. # 21st May 2009, 11:10 pm

Offline Processing on App Engine: a Look Ahead. A session at IO next week: “App Engine was designed to run request-driven web applications, although this will change in the coming year with the release of a number of offline computing components. In this session, we’ll explore the task queue/executor model of computation and some of the more interesting applications.” # 20th May 2009, 12:40 pm

disturbyte’s zenqueue. Simple, tiny and fast Python message queue server built on top of coroutines and Eventlet, using JSON over TCP as the message format. I’m impressed with how potentially useful this looks considering the small amount of code. The author benchmarks it at 28 thousand messages/second. # 11th May 2009, 1:27 pm

Rabbits and warrens. Handy tutorial introduction to using RabbitMQ and AMQP with Python. # 7th April 2009, 9:13 am

Kestrel. Twitter’s Robey Pointer rewrote their Starling message queue in 1500 lines of Scala, adding reliable fetch (where consumers can confirm their receipt of an item) and blocking fetches, which reduce the need for consumers to poll for updates (and hence solve my only beef with the original Starling). I haven’t tried running this on a low spec VPS yet but it looks very promising. # 26th February 2009, 10:20 am

Building and Scaling a Startup on Rails: 12 Things We Learned the Hard Way. Lessons learned from Posterous. Some good advice in here, in particular “Memcache later: If you memcache first, you will never feel the pain and never learn how bad your database indexes and Rails queries are”. Also recommends using job queues for offline processing of anything that takes more than 200ms. # 23rd February 2009, 8:28 am

Apache Qpid. A new open source AMQP message queue with implementations in C++ and Java, developed by engineers from Red Hat, IONA and JP Morgan Chase. Anyone tried this yet? Looks pretty good on paper. # 5th February 2009, 11:01 pm

A couple of years ago when I was working on a now defunct search engine, we were using ActiveMQ to pass messages between the frontend and the backend. The system was unreliable, flaky, and hard to debug. It delivered exactly none of the reliability queues promised. [...] More likely there’s something wrong with the whole design of network systems based on message queues, and we need to start developing alternatives.

Elliotte Rusty Harold # 5th February 2009, 10:53 pm

New Gearman Server & Library in C, MySQL UDFs. Gearman, the job queue written for LiveJournal and now used by Digg and Yahoo!, has been rewritten in C. Looks like a good candidate for an easily configured lightweight message queue. Also includes hooks for writing MySQL functions that can interact with queues. # 13th January 2009, 4:41 pm

Flickr Engineers Do It Offline. Flickr wrote their own queuing mechanism (in PHP), and currently run ten queue servers on dedicated hardware for tasks like pushing new photos in to indexes, denormalisation and “backfills” which move data between clusters and run bulk scripts against large numbers of existing rows. # 28th September 2008, 1:24 am

Using Python and Stompserver to Get Started With Message Queues. An eminently practical guide to this year’s Hot New Thing (for web developers at least) from Gareth Rushgrove. # 14th September 2008, 11:39 pm

stompserver. I think this is the lightweight message queue I’ve been looking for: written in Ruby and EventMachine, easy to set up (thanks to gems), interoperates perfectly with stomp.py. # 14th December 2007, 4:40 pm