Simon Willison’s Weblog

Keyspace. Yet Another Key-Value Store—this one focuses on high availability, with one server in the cluster serving as master (and handling all writes), and the paxos algorithm handling replication and ensuring a new master can be elected should the existing master become unavailable. Clients can chose to make dirty reads against replicated servers or clean reads by talking directly to the master. Underlying storage is BerkeleyDB, and the authors claim 100,000 writes/second. Released under the AGPL.

# 16th July 2009, 10:30 am / keyvaluepairs, keyspace, databases, agpl, berkeleydb, scaling, replication, paxos

TwitterAlikeExample—redis. Excellent example of how you design a moderately complex system against a scalable key-value store (in this case redis). Most “how to build Twitter” code examples fail to address the hard problem of scaling user inboxes, but this one tackles it head on.

# 21st May 2009, 11:14 pm / redis, keyvaluepairs, twitter, scaling

Some Notes on Distributed Key Stores. Another ringing endorsement for Tokyo Cabinet, this time from Leonard Lin.

# 21st April 2009, 9:15 am / tokyocabinet, leonardlin, keyvaluepairs

redis (via) An in-memory scalable key/value store but with an important difference: this one lets you perform list and set operations against keys, opening up a whole new set of possibilities for application development. It’s very young but already supports persistence to disk and master-slave replication.

# 15th March 2009, 1:32 pm / redis, keyvaluepairs, scaling, replication, masterslave, salvatore-sanfilippo

Tokyo Tyrant Tutorial. Buried at the bottom of the Tokyo Tyrant protocol documentation, this is the best resource I’ve seen yet for getting up and running with the database server (including setting up replication).

# 14th February 2009, 11:29 am / replication, tokyotyrant, tokyocabinet, databases, keyvaluepairs

Tokyo Cabinet: Beyond Key-Value Store. Useful overview of Yet Another Scalable Key Value Store. Interesting points: multiple backends (hash table, B-Tree, in memory, on disk), a “table” engine which enables more advanced queries, a network server that supports HTTP, memcached or its own binary protocol and the ability to extend the engine with Lua scripts.

# 14th February 2009, 11:17 am / keyvaluepairs, tokyocabinet, http, memcached, lua, hash, databases

Project Voldemort. Yet Another “big, distributed, persistent, fault-tolerant hash table”—this time from LinkedIn, released under the Apache 2.0 license. The approach to consistency is interesting—instead of using distributed transactions, they use versioning and “resolve inconsistencies at read time”. It also uses consistent hashing (as seen in libketama) to select servers. The design document has lots more information.

# 17th January 2009, 7:45 pm / consistency, hashtable, keyvaluepairs, linkedin, open-source, scaling, versioning, voldemort, libketama

MemcacheDB. A server that speaks the memcache protocol but uses Berkeley DB for reliable persistent storage. Speedy: 20,000 writes/second and 60,000+ reads/second. Includes a full replication mechanism (with custom memcache protocol commands) based on Berkeley DB’s.

# 5th January 2009, 12:37 pm / replication, memcachedb, berkeleydb, memcache, scaling, keyvaluepairs

The Universal Design Pattern. Steve Yegge presents a small book on key/value pairs and prototypal inheritance. “I call it the Universal design pattern because it is (by far) the best known solution to the problem of designing open-ended systems, which in turn translates to long-lived systems.”

# 20th October 2008, 11:13 pm / steve-yegge, prototypes, keyvaluepairs, programming