Items tagged twitter, scaling
Filters: twitter × scaling × Sorted by date
Reducing search indexing latency to one second. Really detailed dive into the nuts and bolts of Twitter’s latest iteration of search indexing technology, including a great explanation of skip lists. # 26th June 2020, 5:06 pm
How does Twitter select trending topics?
They use stream processing algorithms—they mention trending topics calculation in their technical blog entry about Storm, their open source stream processing software: http://engineering.twitter.com/2...
[... 38 words]twitter’s gizzard (via) Intriguing new open source project from Twitter. Gizzard is a sharding framework which provides a network service for partitioning data across arbitrary backend datastores, managing its own forwarding table to map key ranges to partitions and adding support for tree-based replication. # 11th April 2010, 9:39 pm
Up and running with Cassandra. Twitter are beginning to use Cassandra, the open source branch of Facebook’s BigTable-like non-relational database. Evan Weaver explains how to get started with it, but warns that it’s not yet a good idea to trust data to it without having a full backup in an unrelated storage engine. # 7th July 2009, 11:18 am
TwitterAlikeExample—redis. Excellent example of how you design a moderately complex system against a scalable key-value store (in this case redis). Most “how to build Twitter” code examples fail to address the hard problem of scaling user inboxes, but this one tackles it head on. # 21st May 2009, 11:14 pm
peeping into memcached. “Peep uses ptrace to freeze a running memcached server, dump the internal key metadata, and return the server to a running state”—you can then load the resulting data in to MySQL using LOAD LOCAL INFILE and analyse it using standard SQL queries. # 20th April 2009, 6:35 pm
Streams, affordances, Facebook, and rounding errors. I asked Kellan about scaling activity streams the other day. Here he suggests the best technique is not to promise a perfect stream (like Twitter does)—Facebook used to get away with 80% loss of update messages, but their new redesign has changed the contract with their users. # 19th March 2009, 2:02 pm
Scoble writes something—6,800 writes are kicked off, 1 for each follower. Michael Arrington replies—another 6,600 writes. Jason Calacanis jumps in—another 6,500 writes. Beyond the 19,900 writes, there’s a lot of additional overhead too. You have to hit a DB to figure out who the 19,900 followers are. [...] And here’s the kicker: that giant processing and delivery effort—possibly a combined 100K disk IOs—was caused by 3 users, each just sending one, tiny, 140 char message. How innocent it all seemed.
— Isreal L'Heureux # 23rd May 2008, 7:28 pm
Internet Asshattery, Armchair Scaling Experts Edition (via) Leonard says what needs to be said about the most recent case of Twitter scaling flame-bait. # 25th April 2008, 11:19 pm
SlideShare: Webapps scalability. Lots of great presentations on scaling, from Twitter, Digg, Vox, LiveJournal, Last.fm and more. # 4th July 2007, 12:53 am
SELECT * FROM everything, or why databases are awesome. I’m beginning to think that for scalable applications the thinner your ORM is the better—if you even use one at all. # 22nd June 2007, 12:40 am
The top 10 presentations on scaling websites: twitter, Flickr, Bloglines, Vox and more. I normally avoid linking to “top 10” lists on principle, but this one pulls together some great resources and adds extra context to each one. # 1st May 2007, 1:51 pm
Scaling Twitter (via) Slides from Blaine’s recent talk. # 23rd April 2007, 11:02 am
In the big picture, Twitter did exactly the right thing. They had a good idea and they buckled down and focused on delivering something as cool as possible as fast as possible, and it’s really hard, in early 2007, to beat Rails for that. When all of a sudden there were a few tens of thousands of people using it, then they went to work on the scaling.
— Tim Bray # 14th April 2007, 9:13 am
None of these scaling approaches are as fun and easy as developing for Rails. All the convenience methods and syntactical sugar that makes Rails such a pleasure for coders ends up being absolutely punishing, performance-wise.
— Alex Payne, Twitter # 12th April 2007, 2:51 pm