Simon Willison’s Weblog

Items tagged scaling in Dec

Filters: Month: Dec × scaling ×

Scaling Datastores at Slack with Vitess (via) Slack spent three years migrating 99% of their MySQL query load to run against Vitess, the open source MySQL sharding system originally built by YouTube. “Today, we serve 2.3 million QPS at peak. 2M of those queries are reads and 300K are writes. Our median query latency is 2 ms, and our p99 query latency is 11 ms.” # 1st December 2020, 9:30 pm

Scaling React Server-Side Rendering (via) Outstanding, detailed essay from 2017 on challenges and solutions for scaling React server-side rendering at Kijiji, Canada’s largest classified site (owned by eBay). There’s a lot of great stuff in here, including a detailed discussion of different approaches to load balancing, load shedding, component caching, client-side rendering fallbacks and more. # 30th December 2019, 10:26 pm

What are good ways to develop software architectures using multiple languages?

There are a bunch of options for communicating between different languages, but these days the simplest is definitely JSON—it maps directly to common data structures in PHP, Python, Ruby and so on. Treat it as your common interchange format and you can’t go far wrong. It’s very easy to build simple internal web services on top of JSON.

[... 109 words]

What are you some good blogs, videos, papers, etc. on scaling Django?

We’re building up a pretty sizable collection of video (and slides) from talks about Django on—including plenty that talk about scaling issues. Try this:—we have 16 videos and 16 slide decks from talks at events all over the world.

[... 102 words]

What is the largest production deployment of Redis?

I’d guess Twitter or Craigslist.

[... 19 words]

PostgreSQL 8.5alpha3 now available. “Hot Standby, allowing read-only connections during recovery, provides a built-in master-slave replication solution.” Woohoo! # 23rd December 2009, 9:57 am

Django | Multiple Databases. Russell just checked in the final patch developed from Alex Gaynor’s Summer of Code project to add multiple database support to Django. I’d link to the 21,000 line changeset but it crashed our Trac, so here’s the documentation instead. # 22nd December 2009, 5:22 pm

Scaling memcached at Facebook. Fascinating techie details on how Facebook forked memcache to use UDP and increase performance from 50,000 requests a second to 200,000. Now running on 800 servers with 28 TB of memory, and their code is on GitHub. (They may scale like crazy, but they can’t put their blog entry title in the title element?) # 13th December 2008, 10:08 am

Spock Proxy. A MySQL Proxy fork (no Lua) that concentrates solely on sharding, by parsing incoming SQL statements and redirecting them across multiple databases. There are some limitations on the SQL that can be handled (no nested queries, joins across a maximum of two tables) but generally it looks pretty impressive. # 11th December 2008, 9:49 am

Eventually Consistent. Werner Vogels explains the trade-offs involved in building scalable, highly-available data stores such as Amazon’s SimpleDB. # 20th December 2007, 5:59 pm

Techniques for safely consuming external HTTP on demand? I asked this question on yesterday and got some really insightful answers, including Joe Stump from Digg describing how Digg Images uses Danga’s Gearman worker queue. # 15th December 2007, 12:29 pm

NginxMemcachedModule. nginx can be set up to directly serve a URL from memcache if the corresponding cache key is set, and fall back to a backend application server otherwise. Application servers can then write directly to memcache when content needs to be cached or goes stale. # 15th December 2007, 1:59 am

What You Need To Know About Amazon SimpleDB. Amazon have finally launched the database component of their web service suite. It fits a bunch of current trends: key/value pairs, schemaless, built on top of Erlang. “Eventual consistency” is an interesting characteristic. # 14th December 2007, 11:21 am

Curse launches with Django platform. Handles 500k visits/hour! # 14th December 2006, 3:02 am

The Architecture of Mailinator. 3 million e-mails a day on a 2GHz server with 1GB of RAM. # 7th December 2006, 3:11 pm