Simon Willison’s Weblog

39 items tagged “redis”

For Redis 4.2 I’m moving Disque as a Redis module. To do this, Redis modules are getting a fully featured Cluster API. This means that it will be possible, for instance, to write a Redis module that orchestrates N Redis masters, using Raft or any other consensus algorithm, as a single distributed system. This will allow to also model strong guarantees easily.

Salvatore Sanfilippo # 8th November 2017, 10:51 pm

Redis Streams and the Unified Log. In which Brandur Leach explores the new Kafka-style streams functionality coming to Redis 4.0, and shows an example of a robust at-least once processing architecture built on a combination of Redis streams and PostgreSQL transactions. I really like the pattern of writing log records to a staging table in PostgreSQL first in order to bundle them up in the same transaction as the originating state change, then have a separate process read them from that table and publish them to Redis. # 8th November 2017, 4:37 pm

Redis streams aren’t exciting for their innovativeness, but rather than they bring building a unified log architecture within reach of a small and/or inexpensive app. Kafka is infamously difficult to configure and get running, and is expensive to operate once you do. [...] Redis on the other hand is probably already in your stack.

Brandur Leach # 8th November 2017, 4:23 pm

Secondary indexing with Redis. I haven’t seen this section of the official Redis documentation before, and it’s absolutely fantastic—well worth reading the whole thing. It talks through various ways in which you can set up indexes in Redis, mainly by leaning on sorted sets—which it turns out will binary lexicographically sort items with the same score. This makes it easy to implement autocomplete with Redis—but if you use them creatively you can implement subject/predicate/object graph searches or even N-dimensional range queries as well. # 7th November 2017, 2 am

walrus. Fascinating collection of Python utilities for working with Redis, by Charles Leifer. There are a ton of interesting ideas in here. It starts with Python object wrappers for Redis so you can interact with lists, sets, sorted sets and Redis hashes using Python-like objects. Then it gets really interesting: walrus ships with implementations of autocomplete, rate limiting, a graph engine (using a sorted set hexastore) and an ORM-style models mechanism which manages secondary indexes and even implements basic full-text search. # 6th November 2017, 1:14 am

Scaling the GitLab database. Lots of interesting details on how GitLab have worked to scale their PostgreSQL setup. They’ve avoided sharding so far, instead opting for database pooling with pgbouncer and read-only replicas using hot standbys. I like the way they deal with replica lag—they store the current WAL position in a redis key for the user every time there’s a write, then use pg_last_xlog_replay_location() on the various replicas to check and see if they have caught up next time the user makes a request that needs to read some data. # 30th October 2017, 8:53 pm

Streams: a new general purpose data structure in Redis. Exciting new Redis feature inspired by Kafka: redis streams, which allow you to construct an efficient, in-memory list of messages (similar to a Kafka log) which clients can read sections of or block against and await real-time delivery of new messages. As expected from Salvatore the API design is clean, obvious and covers a wide range of exciting use-cases. Planned for release with Redis 4 by the end of the year! # 3rd October 2017, 3:25 pm

Can you recommend a few video lectures on web application design with Redis?

We have a collection of 27 videos of conference talks that cover Redis on Lanyrd, gathered by our community: Redis Videos | Lanyrd

[... 74 words]

How we use Redis at Bump. A couple of neat tricks I hadn’t seen before: using Redis to aggregate log files from multiple servers (they all push in to a Redis queue, then one process pulls from the queue and writes to disk), and using Redis blocking queues for RPC by specifying a different temporary queue to return the result. # 16th July 2011, 4:37 pm

We can deploy new versions of our software, make database schema changes, or even rotate our primary database server, all without failing to respond to a single request. We can accomplish this because we gave ourselves the ability suspend our traffic, which gives us a window of a few seconds to make some changes before letting the requests through. To make this happen, we built a custom HTTP server and application dispatching infrastructure around Python’s Tornado and Redis.

Dan Manges, Braintree # 30th June 2011, 9:27 pm

HotQueue. A super-simple Python work queue using Redis. The API is neat, and makes clever use of generators for blocking consumption of queue items. # 22nd December 2010, 11:51 am

What is the largest production deployment of Redis?

I’d guess Twitter or Craigslist.

[... 19 words]

What are the advantages and disadvantages of using MongoDB vs CouchDB vs Cassandra vs Redis?

I see Redis as a different category from the other three—kind of like you wouldn’t say “what are the advantages of MySQL v.s. Memcached”. Redis makes an excellent complement to pretty much any other persistent storage mechanism. I expanded on this here: http://simonwillison.net/2009/Oc...

[... 67 words]

Will Redis support per-database persistence configuration?

I don’t know if that’s on the roadmap (you’d need to ask antirez on the mailing list or Twitter), but it should be easy enough to run multiple Redis instances with different settings—especially on a multi core machine.

[... 52 words]

How we deploy new features. GitHub are experimenting with using Redis for configuration management. I’ve been thinking about this recently too—managing feature flags feels like an ideal use-case for Redis, since it lets you read multiple values on every page access without adding a bunch of extra read traffic on your regular database. # 8th July 2010, 10:04 am

Zero-downtime Redis upgrade discussion. GitHub have a short window of scheduled downtime in order to upgrade their Redis server. I asked in their comments if they’d considered trying to run the upgrade with no downtime at all using Redis replication, and Ryan Tomayko has posted some interesting replies. # 28th May 2010, 2:50 pm

A fast, fuzzy, full-text index using Redis. Interesting twist on building a reverse-index using Redis sets: this one indexes only the metaphones of the words, resulting in a phonetic fuzzy search. # 5th May 2010, 5:51 pm

Comprehensive notes from my three hour Redis tutorial

Last week I presented two talks at the inaugural NoSQL Europe conference in London. The first was presented with Matthew Wall and covered the ways in which we have been exploring NoSQL at the Guardian. The second was a three hour workshop on Redis, my favourite piece of software to have the NoSQL label applied to it.

[... 263 words]

tempalias.com development diary (via) tempalias.com is a e-mail forwarding service that lets you create an address that will only work for a few days (or a limited number of messages) and will forward messages on to your real account. It’s implemented using Node.js and Redis and the code is released under an MIT license. Philip Hofstetter, the developer, maintained a detailed development diary throughout which is worth reading if you’re interested in Node.js. # 23rd April 2010, 7:36 pm

Redis weekly update #3—Pub/Sub and more. Redis is now a publish/subscribe server—and it ended up only taking 150 lines of C code since Redis internals were already based on that paradigm. # 30th March 2010, 3:15 pm

VMware: the new Redis home. Redis creator Salvatore Sanfilippo is joining VMWare to work on Redis full time. Sounds like a good match. # 16th March 2010, 11:26 am

Redis weekly update #1—Hashes and... many more! Hashes were the big missing data type in Redis—support is only partial at the moment (no ability to list all keys in a hash or delete a specific key) but at the rate Redis is developed I expect that to be fixed within a week or two. # 13th March 2010, 12:06 am

Cache Machine: Automatic caching for your Django models. This is the third new ORM caching layer for Django I’ve seen in the past month! Cache Machine was developed for zamboni, the port of addons.mozilla.org to Django. Caching is enabled using a model mixin class (to hook up some post_delete hooks) and a custom caching manager. Invalidation works by maintaining a “flush list” of dependent cache entries for each object—this is currently stored in memcached and hence has potential race conditions, but a comment in the source code suggests that this could be solved by moving to redis. # 11th March 2010, 7:35 pm

Node.js, redis, and resque (via) Paul Gross has been experimenting with Node.js proxies for allowing web applications to be upgraded without missing any requests. Here he places all incoming HTTP requests in a redis queue, then has his backend Rails servers consume requests from the queue and push the responses back on to a queue for Node to deliver. When the backend application is upgraded, requests remain in the queue and users see a few seconds of delay before their request is handled. It’s not production ready yet (POST requests aren’t handled, for example) but it’s a very interesting approach. # 28th February 2010, 11:02 pm

A Collection Of Redis Use Cases. Lots of interesting case studies here, collated by Mathias Meyer. Redis clearly shines for anything involving statistics or high volumes of small writes. # 16th February 2010, 3:04 pm

Redis in Practice: Who’s Online? Using Redis to implement a “which of your friends are online now” feature, by maintaining a set of active user IDs for every minute, then intersecting the past five minutes of user IDs with a set containing the IDs of your friends. # 14th February 2010, 5:17 pm

Redis Virtual Memory: the story and the code. Fascinating overview of the virtual memory feature coming to Redis 2.0, which will remove the requirement that all Redis data fit in RAM. Keys still stay in RAM, but rarely accessed values will be swapped to disk. 16 GB of RAM will be enough to hold 100 million keys, each with a value as large as you like. # 9th February 2010, 3:59 pm

Distributed lock on top of memcached. A simple Python context manager (taking advantage of the with statement) that implements a distributed lock using memcached to store lock state: “memcached_lock can be used to ensure that some global data is only updated by one server”. Redis would work well for this kind of thing as well. # 1st February 2010, 10:15 am

Help pick the best photos, but watch out, it’s addictive! My favourite WildlifeNearYou feature yet—our new tool asks you to pick the best from two photos, then uses the results to rank all of the photos for each species. It’s surprisingly addictive—we had over 5,000 votes in the first two hours, peaking at 4 or 5 votes a second. The feature seems to be staying nice and speedy thanks to Redis under the hood. Photos in the top three for any given species display a medal on their photo page. # 25th January 2010, 12:36 am

BLPOP and BRPOP in Redis. Added over Christmas—Redis now has blocking list pop operations. This means you can use Redis to drive a queue server without the need for polling—simply BLPOP against a key and, if it’s empty, your client will block until another client pushes an item on to the list. Multiple clients can block against the same key and only the first client will return when an item becomes available. # 7th January 2010, 10:50 pm