Simon Willison's Weblog: friendfeed

How was FriendFeed's schema less db faster than pure MySQL?

2013-10-30T16:27:00+00:00

My answer to How was FriendFeed's schema less db faster than pure MySQL? on Quora

The principle reason they switched to a schemaless DB was to work around the challenges of having to make schemes changes in MySQL, which can lock the table and take hours if bit days to complete in large tables.

The performance improvement shown in the graph is almost certainly because they almost entirely eliminated joins and complex queries when they switched to the new mechanism. This meant that all it their database traffic was now simple queries, which have much more predictable performance characteristics. MySQL (in fact all databases) are extremely fast at primary key lookups and index scans.

Tags: databases, friendfeed, mysql, nosql, quora, rdbms

How does FriendFeed work, and what programming languages are used?

2011-01-22T13:19:00+00:00

My answer to How does FriendFeed work, and what programming languages are used? on Quora

It was written in Python, using the Tornado asynchronous web server (which the FriendFeed team developed themselves). They used their own schemaless NoSQL-style store based on keeping serialized objects in MySQL.

http://www.tornadoweb.org/
http://bret.appspot.com/entry/ho...

Tags: friendfeed, web-development, quora

Quoting Bret Taylor

2009-09-11T17:31:57+00:00

We experimented with different async DB approaches, but settled on synchronous at FriendFeed because generally if our DB queries were backlogging our requests, our backends couldn't scale to the load anyway. Things that were slow enough were abstracted to separate backend services which we fetched asynchronously via the async HTTP module.

— Bret Taylor

Tags: async, bret-taylor, friendfeed, http, tornado

Tornado Web Server

2009-09-10T21:32:57+00:00

Tornado Web Server

An extremely exciting addition to the Python web landscape, Tornado is the open sourced version of FriendFeed’s custom web stack. It’s a non-blocking (epoll) Python web server designed for handling thousands of simultaneous connections, perfect for building Comet applications. The web framework is cosmetically similar to web.py or App Engine’s webapp but has decorators for writing asynchronous request handlers. The template language uses Django-style syntax but allows you to use full Python expressions. FriendFeed have benchmarked it handling 8,000 requests a second running as four load-balanced processes on a 4 core server.

Via Bret Taylor

Tags: bret-taylor, comet, django, epoll, friendfeed, google-app-engine, python, tornado, webapp, webpy

Paul Buchheit: Make your site faster and cheaper to operate in one easy step

2009-04-17T17:19:44+00:00

Paul Buchheit: Make your site faster and cheaper to operate in one easy step

Paul promotes gzip encoding using nginx as a proxy, and mentions that FriendFeed use a “custom, epoll-based python server” as their application server. Does that mean that they’re serving their real-time comet feeds directly from Python?

Tags: comet, epoll, friendfeed, gzip, nginx, paul-buchheit, python

How FriendFeed uses MySQL to store schema-less data

2009-02-27T14:33:35+00:00

How FriendFeed uses MySQL to store schema-less data

The pain of altering/ adding indexes to tables with 250 million rows was killing their ability to try out new features, so they’ve moved to storing pickled Python objects and manually creating the indexes they need as denormalised two column tables. These can be created and dropped much more easily, and are continually populated by an off-line index building process.

Tags: bret-taylor, databases, friendfeed, mysql, python, scaling, sharding

Simple Update Protocol: Update

2008-12-18T23:33:46+00:00

Simple Update Protocol: Update

Already implemented by more than five services, each of which now have near-real-time updates in to the FriendFeed syndication engine.

Tags: friendfeed, realtime, sup, syndication

FriendFeed launch a real-time API

2008-10-22T14:18:56+00:00

FriendFeed launch a real-time API

This is huge: JSONP plus long polling Comet, with “everything since X” tokens to ensure you don’t miss anything. This is the first open Comet API I’ve seen anywhere. Combine this with FriendFeed’s regular API (which allows arbitrary message posting) and you’ve got a really powerful tool for hackers who want to experiment with Comet without rigging up their own infrastructure.

Tags: apis, comet, friendfeed, javascript, json, jsonp, realtime

View your FriendFeed in real-time

2008-10-16T14:06:16+00:00

View your FriendFeed in real-time

FriendFeed become the latest site to enable real-time updates using the long-polling variant of Comet. The real-time Web was something of a theme at this year’s FOWA, with talks on message queues, XMPP and scaling Comet at Meebo.

Tags: comet, fowa2008, friendfeed, meebo, realtimeweb, xmpp

FriendFeed Blog: Simple Update Protocol

2008-08-28T12:16:38+00:00

FriendFeed Blog: Simple Update Protocol

FriendFeed infamously poll RSS feeds on the 43 services they support millions of times an hour in an effort to keep their content as real-time as possible. SUP is a new proposal by FriendFeed for a sort of “master feed” of changes to a site—instead of hitting the Flickr feed for each of their users they would just poll Flickr’s SUP feed every minute or so to find out who had uploaded a new photo, and only retrieve the RSS feed for those users.

Tags: atom, feeds, flickr, friendfeed, polling, rss, sup

Quoting Merlin Mann

2008-08-26T22:28:25+00:00

As duplicitous and sad as "fake following" sounds - and let's be honest: the whole idea's pathetic on a number of levels - for a certain kind of user, I can see why there’s a desire for this functionality. Especially on a site like FriendFeed, which has quickly become the platform of choice for the web's least interesting narcissists - and the slow-witted woodland creatures who enjoy grooming their fur - this is a major breakthrough in the makebelieve friendship space. Yes, primate culture may be primitive, but it is not without its evolving needs.

— Merlin Mann

Tags: fakefollowing, friendfeed, funny, merlinmann, slowwittedwoodlandcreatures