Simon Willison’s Weblog

46 items tagged “redis”

2010

Node.js, redis, and resque (via) Paul Gross has been experimenting with Node.js proxies for allowing web applications to be upgraded without missing any requests. Here he places all incoming HTTP requests in a redis queue, then has his backend Rails servers consume requests from the queue and push the responses back on to a queue for Node to deliver. When the backend application is upgraded, requests remain in the queue and users see a few seconds of delay before their request is handled. It’s not production ready yet (POST requests aren’t handled, for example) but it’s a very interesting approach. # 28th February 2010, 11:02 pm

A Collection Of Redis Use Cases. Lots of interesting case studies here, collated by Mathias Meyer. Redis clearly shines for anything involving statistics or high volumes of small writes. # 16th February 2010, 3:04 pm

Redis in Practice: Who’s Online? Using Redis to implement a “which of your friends are online now” feature, by maintaining a set of active user IDs for every minute, then intersecting the past five minutes of user IDs with a set containing the IDs of your friends. # 14th February 2010, 5:17 pm

Redis Virtual Memory: the story and the code. Fascinating overview of the virtual memory feature coming to Redis 2.0, which will remove the requirement that all Redis data fit in RAM. Keys still stay in RAM, but rarely accessed values will be swapped to disk. 16 GB of RAM will be enough to hold 100 million keys, each with a value as large as you like. # 9th February 2010, 3:59 pm

Distributed lock on top of memcached. A simple Python context manager (taking advantage of the with statement) that implements a distributed lock using memcached to store lock state: “memcached_lock can be used to ensure that some global data is only updated by one server”. Redis would work well for this kind of thing as well. # 1st February 2010, 10:15 am

Help pick the best photos, but watch out, it’s addictive! My favourite WildlifeNearYou feature yet—our new tool asks you to pick the best from two photos, then uses the results to rank all of the photos for each species. It’s surprisingly addictive—we had over 5,000 votes in the first two hours, peaking at 4 or 5 votes a second. The feature seems to be staying nice and speedy thanks to Redis under the hood. Photos in the top three for any given species display a medal on their photo page. # 25th January 2010, 12:36 am

BLPOP and BRPOP in Redis. Added over Christmas—Redis now has blocking list pop operations. This means you can use Redis to drive a queue server without the need for polling—simply BLPOP against a key and, if it’s empty, your client will block until another client pushes an item on to the list. Multiple clients can block against the same key and only the first client will return when an item becomes available. # 7th January 2010, 10:50 pm

2009

New Redis ZINCRBY command (via) Just added to Redis, a command which increments the “score” for an item in a sorted set and reorders the set to reflect the new scores. Looks ideally suited to real time stats, and I’m sure there are plenty of other exciting uses for it. # 22nd December 2009, 8:38 pm

Crowdsourced document analysis and MP expenses

As you may have heard, the UK government released a fresh batch of MP expenses documents a week ago on Thursday. I spent that week working with a small team at Guardian HQ to prepare for the release. Here’s what we built:

[... 2081 words]

I think that what’s particularly hard with C is not the details about pointers, automatic memory management, and so forth, but the fact that C is at the same time so low level and so flexible. So basically if you want to create a large project in C you have to build a number of intermediate layers (otherwise the code will be a complete mess full of bugs and 10 times bigger than required). This continue design exercise of creating additional layers is the hard part about C. You have to get very good at understanding when to write a function or not, when to create a layer of abstraction, and when it’s worth to generalize or when it is an overkill.

Salvatore Sanfilippo # 18th December 2009, 3:50 pm

Node.js is genuinely exciting

I gave a talk on Friday at Full Frontal, a new one day JavaScript conference in my home town of Brighton. I ended up throwing away my intended topic (JSONP, APIs and cross-domain security) three days before the event in favour of a technology which first crossed my radar less than two weeks ago.

[... 2009 words]

Introducing Resque. A new background worker management queue developed at GitHub, using Redis for the persistence layer. The blog post explains both the design and the shortcomings of previous solutions at length. Within 24 hours of the release code an external developer, Adam Cooke, has completely reskinned the UI. # 4th November 2009, 8:20 pm

Why I like Redis

I’ve been getting a lot of useful work done with Redis recently.

[... 900 words]

How We Made GitHub Fast. Detailed overview of the new GitHub architecture. It’s a lot more complicated than I would have expected—lots of moving parts are involved in ensuring they can scale horizontally when they need to. Interesting components include nginx, Unicorn, Rails, DRBD, HAProxy, Redis, Erlang, memcached, SSH, git and a bunch of interesting new open source projects produced by the GitHub team such as BERT/Ernie and ProxyMachine. # 21st October 2009, 9:14 pm

TwitterAlikeExample—redis. Excellent example of how you design a moderately complex system against a scalable key-value store (in this case redis). Most “how to build Twitter” code examples fail to address the hard problem of scaling user inboxes, but this one tackles it head on. # 21st May 2009, 11:14 pm

redis (via) An in-memory scalable key/value store but with an important difference: this one lets you perform list and set operations against keys, opening up a whole new set of possibilities for application development. It’s very young but already supports persistence to disk and master-slave replication. # 15th March 2009, 1:32 pm