Simon Willison’s Weblog

Curating conversations. Chris Thorpe has open-sourced the Guardian’s moderated Twitter backchannel app, for displaying back channels at high profile (and hence high potential for abuse) events. It’s a Python application that runs on App Engine. # 16th July 2009, 7:34 pm

You should follow me on Twitter. Dustin Curtis did a simple A/B testing experiment on his blog and found that the text “you should follow me on Twitter” had the highest click-through rate—173% more effective than “I’m on Twitter”. # 15th July 2009, 10:43 am

YQL: INSERT INTO internet. insert into twitter.status (status,username,password) values (“Playing with INSERT, UPDATE and DELETE in YQL”, “twitterusername”,“twitterpassword”) # 8th July 2009, 8:19 pm

Up and running with Cassandra. Twitter are beginning to use Cassandra, the open source branch of Facebook’s BigTable-like non-relational database. Evan Weaver explains how to get started with it, but warns that it’s not yet a good idea to trust data to it without having a full backup in an unrelated storage engine. # 7th July 2009, 11:18 am

cache-money. A “write-through caching library for ActiveRecord”, maintained by Nick Kallen from Twitter. Queries hit memcached first, and caches are automatically kept up-to-date when objects are created, updated and deleted. Only some queries are supported—joins and comparisons won’t hit the cache, for example. # 28th June 2009, 3:17 pm

Twitter, an Evolving Architecture. The most detailed write-up of Twitter’s current architecture I’ve seen, explaining the four layers of cache (all memcached) used by the Twitter API. # 28th June 2009, 3:09 pm

C64 Twitter client. Awesome. # 17th June 2009, 9:14 am

The Twitpocalypse is Near: Will Your Twitter Client Survive? Twitter tweet IDs will shortly tick over past the maximum signed 32 bit integer, potentially breaking applications. I learnt this lesson when the same thing happened to Flickr photo IDs: never store numeric IDs from external systems as integers, always use strings. # 9th June 2009, 10:52 am

Muck Rack: Links posted by Guardian Journalists on Twitter. I’m rather impressed by the Sawhorse Media collection of Twitter aggregation sites (Muck Rack aggregates journalists)—a simple idea very well executed. Here’s a nice example—this page shows links posted to Twitter by known Guardian journalists, but goes a step further and scrapes in the favicon, the real title of the page and resolves the domain from any shortened links. # 22nd May 2009, 10:02 pm

TwitterAlikeExample—redis. Excellent example of how you design a moderately complex system against a scalable key-value store (in this case redis). Most “how to build Twitter” code examples fail to address the hard problem of scaling user inboxes, but this one tackles it head on. # 21st May 2009, 11:14 pm

peeping into memcached. “Peep uses ptrace to freeze a running memcached server, dump the internal key metadata, and return the server to a running state”—you can then load the resulting data in to MySQL using LOAD LOCAL INFILE and analyse it using standard SQL queries. # 20th April 2009, 6:35 pm

Sign in with Twitter. Intriguing: Twitter are now an OpenID-style identity provider... using OAuth. # 20th April 2009, 4:10 am

17-year-old claims responsibility for Twitter worm. It was a text book XSS attack—the URL on the user profile wasn’t properly escaped, allowing an attacker to insert a script element linking out to externally hosted JavaScript which then used Ajax to steal any logged-in user’s anti-CSRF token and use it to self-replicate in to their profile. # 12th April 2009, 7:22 pm

Twitter: blaming Ruby for their mistakes? The comments on the entry include replies from Twitter employees and the RabbitMQ consultant they brought in, and provide a full rebuttal to the various accusations of NIH that were thrown around recently. # 6th April 2009, 11:06 am

Mending The Bitter Absence of Reasoned Technical Discussion. Not at all surprised to see Alex Payne write this considering the low quality of discussion around anything technical to do with Twitter. # 5th April 2009, 7:59 pm

Streams, affordances, Facebook, and rounding errors. I asked Kellan about scaling activity streams the other day. Here he suggests the best technique is not to promise a perfect stream (like Twitter does)—Facebook used to get away with 80% loss of update messages, but their new redesign has changed the contract with their users. # 19th March 2009, 2:02 pm

Parallel merge sort in Erlang. Thoughts on an Erlang-y way of implementing a combined activity stream (e.g. Facebook and Twitter). Activity streams are a Really Hard Problem—as far as I know there’s no best practise for implementing them yet. # 15th March 2009, 1:36 pm

How search.twitter.com uses Varnish. Includes examples of the configuration options they use. # 2nd March 2009, 5:08 pm

Kestrel. Twitter’s Robey Pointer rewrote their Starling message queue in 1500 lines of Scala, adding reliable fetch (where consumers can confirm their receipt of an item) and blocking fetches, which reduce the need for consumers to poll for updates (and hence solve my only beef with the original Starling). I haven’t tried running this on a low spec VPS yet but it looks very promising. # 26th February 2009, 10:20 am

Oscars 2009: the interactive results | guardian.co.uk. My latest project for the Guardian, put together on very short notice. Updates live as the results are announced, and allows Twitter users to vote on their favourite for each category by sending a specially formatted message to @guardianfilm—jQuery and Ajax polling against S3 under the hood. # 23rd February 2009, 2:19 am

Twitter Don’t Click Exploit. Someone ran a successful ClickJacking exploit against Twitter users, using a transparent iframe holding the Twitter homepage with a status message fed in by a query string parameter. Thiss will definitely help raise awareness of ClickJacking! Twitter has now added framebusting JavaScript to prevent the exploit. # 12th February 2009, 7:56 pm

Four reasons why public Facebook status updates won’t kill Twitter. Mike Butcher highlights the importance of “follow” rather than “friend” in social software. # 9th February 2009, 7:04 pm

FluidDB domain names available early (and free) for Twitter users. It’s interesting how Twitter has revitalised the concept of usernames as first class identifiers. FluidDB hasn’t even launched yet, but it’s allowing people to reserve their Twitter username within the FluidDB system just by following @fluidDB. # 24th January 2009, 11:44 pm

Weak Password Brings “Happiness” to Twitter Hacker. The full story on the Twitter admin account hack. I bet there are a LOT of web applications out there that don’t track and rate-limit failed password attempts. # 7th January 2009, 12:04 pm

The Twitter administrator hack was a dictionary attack. I quoted Blaine earlier suggesting that the recent Twitter mass-hack was due to a Twitter admin password being scooped up by a rogue third party application—this was not the case, as Alex Payne explains in a comment. # 6th January 2009, 11:56 pm

Update on the “antipatterns for sale” Twply auction (via) The collected username and password database is NOT included in the auction. # 6th January 2009, 9:41 am

Antipatterns for sale. Twply collected over 800 Twitter usernames and passwords (OAuth can’t arrive soon enough) and was promptly auctioned off on SitePoint to the highest bidder. # 2nd January 2009, 10:48 am

Now You Can Sign Into Friend Connect Sites With Your Twitter ID. Great. Now even Google is asking me for my Twitter password. Slow clap. How’s that Twitter OAuth beta coming along? # 15th December 2008, 5:20 pm

Interview @MarsPhoenix (via) “For over a year, Veronica McGregor has been Twittering from Mars.”—an interview with the Twitter voice of the Mars Phoenix lander. # 11th November 2008, 12:17 pm

Tweetersation. Nat and my latest side project: a JSONP API powered tool to more easily follow conversations between people on Twitter, by combining their tweets in to a single timeline. # 2nd October 2008, 5:08 pm