Simon Willison’s Weblog

Build your own SQS or Kafka with Postgres (via) Anthony Accomazzo works on Sequin, an open source "message stream" (similar to Kafka) written in Elixir and Go on top of PostgreSQL.

This detailed article describes how you can implement message queue patterns on PostgreSQL from scratch, including this neat example using a CTE, returning and for update skip locked to retrieve $1 messages from the messages table and simultaneously mark them with not_visible_until set to $2 in order to "lock" them for processing by a client:

with available_messages as (
  select seq
  from messages
  where not_visible_until is null
    or (not_visible_until <= now())
  order by inserted_at
  limit $1
  for update skip locked
)
update messages m
set
  not_visible_until = $2,
  deliver_count = deliver_count + 1,
  last_delivered_at = now(),
  updated_at = now()
from available_messages am
where m.seq = am.seq
returning m.seq, m.data;

# 31st July 2024, 5:34 pm / message-queues, postgresql, sql, sqs, kafka

Mass Video Conversion Using AWS. How to use S3, SQS, EC2, ffmpeg and some Python to bulk convert videos with Amazon Web Services.

# 3rd April 2007, 11:44 pm / amazon, aws, ec2, ffmpeg, python, s3, sqs

boto. Python library for accessing Amazon’s S3, SQS and EC2 Web Services, with excellent documentation.

# 11th February 2007, 12:17 am / amazon, aws, boto, ec2, python, s3, sqs