Simon Willison’s Weblog

HotQueue. A super-simple Python work queue using Redis. The API is neat, and makes clever use of generators for blocking consumption of queue items. # 22nd December 2010, 11:51 am

How we deploy new features. GitHub are experimenting with using Redis for configuration management. I’ve been thinking about this recently too—managing feature flags feels like an ideal use-case for Redis, since it lets you read multiple values on every page access without adding a bunch of extra read traffic on your regular database. # 8th July 2010, 10:04 am

Zero-downtime Redis upgrade discussion. GitHub have a short window of scheduled downtime in order to upgrade their Redis server. I asked in their comments if they’d considered trying to run the upgrade with no downtime at all using Redis replication, and Ryan Tomayko has posted some interesting replies. # 28th May 2010, 2:50 pm

A fast, fuzzy, full-text index using Redis. Interesting twist on building a reverse-index using Redis sets: this one indexes only the metaphones of the words, resulting in a phonetic fuzzy search. # 5th May 2010, 5:51 pm

tempalias.com development diary (via) tempalias.com is a e-mail forwarding service that lets you create an address that will only work for a few days (or a limited number of messages) and will forward messages on to your real account. It’s implemented using Node.js and Redis and the code is released under an MIT license. Philip Hofstetter, the developer, maintained a detailed development diary throughout which is worth reading if you’re interested in Node.js. # 23rd April 2010, 7:36 pm

Redis weekly update #3—Pub/Sub and more. Redis is now a publish/subscribe server—and it ended up only taking 150 lines of C code since Redis internals were already based on that paradigm. # 30th March 2010, 3:15 pm

VMware: the new Redis home. Redis creator Salvatore Sanfilippo is joining VMWare to work on Redis full time. Sounds like a good match. # 16th March 2010, 11:26 am

Redis weekly update #1—Hashes and... many more! Hashes were the big missing data type in Redis—support is only partial at the moment (no ability to list all keys in a hash or delete a specific key) but at the rate Redis is developed I expect that to be fixed within a week or two. # 13th March 2010, 12:06 am

Cache Machine: Automatic caching for your Django models. This is the third new ORM caching layer for Django I’ve seen in the past month! Cache Machine was developed for zamboni, the port of addons.mozilla.org to Django. Caching is enabled using a model mixin class (to hook up some post_delete hooks) and a custom caching manager. Invalidation works by maintaining a “flush list” of dependent cache entries for each object—this is currently stored in memcached and hence has potential race conditions, but a comment in the source code suggests that this could be solved by moving to redis. # 11th March 2010, 7:35 pm

Node.js, redis, and resque (via) Paul Gross has been experimenting with Node.js proxies for allowing web applications to be upgraded without missing any requests. Here he places all incoming HTTP requests in a redis queue, then has his backend Rails servers consume requests from the queue and push the responses back on to a queue for Node to deliver. When the backend application is upgraded, requests remain in the queue and users see a few seconds of delay before their request is handled. It’s not production ready yet (POST requests aren’t handled, for example) but it’s a very interesting approach. # 28th February 2010, 11:02 pm

A Collection Of Redis Use Cases. Lots of interesting case studies here, collated by Mathias Meyer. Redis clearly shines for anything involving statistics or high volumes of small writes. # 16th February 2010, 3:04 pm

Redis in Practice: Who’s Online? Using Redis to implement a “which of your friends are online now” feature, by maintaining a set of active user IDs for every minute, then intersecting the past five minutes of user IDs with a set containing the IDs of your friends. # 14th February 2010, 5:17 pm

Redis Virtual Memory: the story and the code. Fascinating overview of the virtual memory feature coming to Redis 2.0, which will remove the requirement that all Redis data fit in RAM. Keys still stay in RAM, but rarely accessed values will be swapped to disk. 16 GB of RAM will be enough to hold 100 million keys, each with a value as large as you like. # 9th February 2010, 3:59 pm

Distributed lock on top of memcached. A simple Python context manager (taking advantage of the with statement) that implements a distributed lock using memcached to store lock state: “memcached_lock can be used to ensure that some global data is only updated by one server”. Redis would work well for this kind of thing as well. # 1st February 2010, 10:15 am

Help pick the best photos, but watch out, it’s addictive! My favourite WildlifeNearYou feature yet—our new tool asks you to pick the best from two photos, then uses the results to rank all of the photos for each species. It’s surprisingly addictive—we had over 5,000 votes in the first two hours, peaking at 4 or 5 votes a second. The feature seems to be staying nice and speedy thanks to Redis under the hood. Photos in the top three for any given species display a medal on their photo page. # 25th January 2010, 12:36 am

BLPOP and BRPOP in Redis. Added over Christmas—Redis now has blocking list pop operations. This means you can use Redis to drive a queue server without the need for polling—simply BLPOP against a key and, if it’s empty, your client will block until another client pushes an item on to the list. Multiple clients can block against the same key and only the first client will return when an item becomes available. # 7th January 2010, 10:50 pm