Simon Willison’s Weblog

Subscribe
Atom feed for message-queues

21 items tagged “message-queues”

2024

Build your own SQS or Kafka with Postgres (via) Anthony Accomazzo works on Sequin, an open source "message stream" (similar to Kafka) written in Elixir and Go on top of PostgreSQL.

This detailed article describes how you can implement message queue patterns on PostgreSQL from scratch, including this neat example using a CTE, returning and for update skip locked to retrieve $1 messages from the messages table and simultaneously mark them with not_visible_until set to $2 in order to "lock" them for processing by a client:

with available_messages as (
  select seq
  from messages
  where not_visible_until is null
    or (not_visible_until <= now())
  order by inserted_at
  limit $1
  for update skip locked
)
update messages m
set
  not_visible_until = $2,
  deliver_count = deliver_count + 1,
  last_delivered_at = now(),
  updated_at = now()
from available_messages am
where m.seq = am.seq
returning m.seq, m.data;

# 31st July 2024, 5:34 pm / postgresql, sqs, message-queues, kafka, sql

2021

RabbitMQ Streams Overview. New in RabbitMQ 3.9: streams are a persisted, replicated append-only log with non-destructive consuming semantics. Sounds like it fits the same hole as Kafka and Redis Streams, an extremely useful pattern.

# 13th July 2021, 11:29 pm / rabbitmq, redis, message-queues, kafka

2009

cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity.

# 21st September 2009, 11:09 pm / cloudcrowd, webhooks, json, http, message-queues, workers, sinatra, thin, ruby, canvas

Announcing Alice and Wonderland. Continuing the RabbitMQ “stuff to do with rabbits” naming convention, Alice is a RESTful interface to RabbitMQ which exposes information about vhosts/queues/users/exchanges/etc as JSON. Wonderland is a web UI for RabbitMQ implemented as a pure Ajax application which calls Alice.

# 17th July 2009, 9:12 am / aliceinwonderland, rabbitmq, alice, wonderland, rest, json, ajax, javascript, message-queues, queues

PubSub-over-Webhooks with RabbitHub. RabbitMQ, the Erlang-powered AMQP message queue, is growing an HTTP interface based on webhooks and PubSubHubBub.

# 1st July 2009, 8:22 pm / rabbitmq, erlang, amqp, message-queues, http, webhooks, pubsubhubbub

Working with Python and RabbitMQ. Nathan Borror eliminates the boilerplate needed to talk to RabbitMQ (or any other AMQP queue server) from Python.

# 21st May 2009, 11:10 pm / nathan-borror, rabbitmq, amqp, message-queues, python

Offline Processing on App Engine: a Look Ahead. A session at IO next week: “App Engine was designed to run request-driven web applications, although this will change in the coming year with the release of a number of offline computing components. In this session, we’ll explore the task queue/executor model of computation and some of the more interesting applications.”

# 20th May 2009, 12:40 pm / appengine, io, google, message-queues, offlineprocessing, workers

disturbyte’s zenqueue. Simple, tiny and fast Python message queue server built on top of coroutines and Eventlet, using JSON over TCP as the message format. I’m impressed with how potentially useful this looks considering the small amount of code. The author benchmarks it at 28 thousand messages/second.

# 11th May 2009, 1:27 pm / zenqueue, message-queues, python, eventlet, coroutines, json, zachary-voase, github

Rabbits and warrens. Handy tutorial introduction to using RabbitMQ and AMQP with Python.

# 7th April 2009, 9:13 am / message-queues, queues, rabbitmq, amqp, python, tutorial

Kestrel. Twitter’s Robey Pointer rewrote their Starling message queue in 1500 lines of Scala, adding reliable fetch (where consumers can confirm their receipt of an item) and blocking fetches, which reduce the need for consumers to poll for updates (and hence solve my only beef with the original Starling). I haven’t tried running this on a low spec VPS yet but it looks very promising.

# 26th February 2009, 10:20 am / robey-pointer, message-queues, kestrel, scala, starling, twitter

Building and Scaling a Startup on Rails: 12 Things We Learned the Hard Way. Lessons learned from Posterous. Some good advice in here, in particular “Memcache later: If you memcache first, you will never feel the pain and never learn how bad your database indexes and Rails queries are”. Also recommends using job queues for offline processing of anything that takes more than 200ms.

# 23rd February 2009, 8:28 am / message-queues, rails, scaling, posterous, memcache

Apache Qpid. A new open source AMQP message queue with implementations in C++ and Java, developed by engineers from Red Hat, IONA and JP Morgan Chase. Anyone tried this yet? Looks pretty good on paper.

# 5th February 2009, 11:01 pm / apache, qpid, java, c-plus-plus, amqp, message-queues, red-hat, iona, jpmorganchase, open-source

A couple of years ago when I was working on a now defunct search engine, we were using ActiveMQ to pass messages between the frontend and the backend. The system was unreliable, flaky, and hard to debug. It delivered exactly none of the reliability queues promised. [...] More likely there's something wrong with the whole design of network systems based on message queues, and we need to start developing alternatives.

Elliotte Rusty Harold

# 5th February 2009, 10:53 pm / activemq, elliotte-rusty-harold, message-queues

New Gearman Server & Library in C, MySQL UDFs. Gearman, the job queue written for LiveJournal and now used by Digg and Yahoo!, has been rewritten in C. Looks like a good candidate for an easily configured lightweight message queue. Also includes hooks for writing MySQL functions that can interact with queues.

# 13th January 2009, 4:41 pm / mysql, message-queues, digg, gearman, livejournal, queues, scaling, yahoo, eric-day

2008

Beanstalkd / Python Basic Tutorial. How to get up and running quickly with my favourite light-weight queue server. If only it had persistence...

# 20th October 2008, 11:40 pm / beanstalkd, python, parand-tony-darugar, message-queues

Flickr Engineers Do It Offline. Flickr wrote their own queuing mechanism (in PHP), and currently run ten queue servers on dedicated hardware for tasks like pushing new photos in to indexes, denormalisation and “backfills” which move data between clusters and run bulk scripts against large numbers of existing rows.

# 28th September 2008, 1:24 am / flickr, message-queues, denormalisation, backfills, queues

Using Python and Stompserver to Get Started With Message Queues. An eminently practical guide to this year’s Hot New Thing (for web developers at least) from Gareth Rushgrove.

# 14th September 2008, 11:39 pm / message-queues, stompserver, stomppy, python, gareth-rushgrove

simple-thrift-queue (via) Phillip Pearson’s surprisingly concise in-memory message queue written in Python using Facebook’s Thrift library (which is similar to Protocol Buffers, but was open sourced much earlier on). Handles 4,000 requests per second on a laptop.

# 4th August 2008, 12:27 pm / messaging, phillip-pearson, python, facebook, thrift, protocolbuffers, message-queues

ZeroMQ. Open source message queue optimised for performance: claims 25μsec latency and 2.6 million messages per second.

# 27th July 2008, 4:57 pm / messaging, zeromq, performance, message-queues

RubyForge: Starling. “Starling is a light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter’s backend, and is in production across Twitter’s cluster.”

# 11th January 2008, 9:47 pm / memcached, twitter, ruby, messaging, queue, starling, blaine-cook, rubyforge, message-queues

2007

stompserver. I think this is the lightweight message queue I’ve been looking for: written in Ruby and EventMachine, easy to set up (thanks to gems), interoperates perfectly with stomp.py.

# 14th December 2007, 4:40 pm / python, ruby, stomp, messaging, message-queues, eventmachine, lightweight, gems