Simon Willison’s Weblog

GitHub: Announcing SVN Support. The best kind of April Fool’s joke: one that works. It’s read-only, but that’s good enough to support referencing GitHub repositories from SVN externals. # 1st April 2010, 11:33 am

Ryan Tomayko on Github’s development process. In the comments—a fascinating insight in to how GitHub’s “developers work on whatever is most interesting to them” process manages to achieve really good results. # 22nd February 2010, 9:18 am

Algorithmic recruitment with GitHub. Matt Biddulph crawls GitHub’s social graph using JUNG (the Java Universal Network/Graph Framework), JRuby and Yahoo! BOSS to find good leads on interesting developers in specific geographic locations. # 12th February 2010, 1:17 pm

jQuery.require() implementation. John Resig has added a new jQuery.require() function to a jQuery development branch, for release as part of jQuery 1.4. The commit on GitHub has an extensive discussion attached to it (scroll to the bottom). # 17th December 2009, 11:24 am

Introducing the YUI 3 Gallery. Write a plugin for YUI3, BSD license it and sign a CLA and Yahoo! will push your module out to their CDN and make it loadable using the YUI().use() statement. They’re coordinating the submissions using GitHub. # 4th November 2009, 11:14 pm

Introducing Resque. A new background worker management queue developed at GitHub, using Redis for the persistence layer. The blog post explains both the design and the shortcomings of previous solutions at length. Within 24 hours of the release code an external developer, Adam Cooke, has completely reskinned the UI. # 4th November 2009, 8:20 pm

Introducing BERT and BERT-RPC. Justification for inventing a brand new serialisation protocol: Thrift and Protocol Buffers both use IDLs and code generation, XML “is not convertible to a simple unambiguous data structure in any language I’ve ever used” and JSON lacks support for unencoded binary data. The result is BERT—Binary ERlang Term—which extracts a format from Erlang in much the same way that JSON extracted one from JavaScript. # 21st October 2009, 10:11 pm

How We Made GitHub Fast. Detailed overview of the new GitHub architecture. It’s a lot more complicated than I would have expected—lots of moving parts are involved in ensuring they can scale horizontally when they need to. Interesting components include nginx, Unicorn, Rails, DRBD, HAProxy, Redis, Erlang, memcached, SSH, git and a bunch of interesting new open source projects produced by the GitHub team such as BERT/Ernie and ProxyMachine. # 21st October 2009, 9:14 pm

Scriptlets—Quick web scripts (via) From the prolific Jeff Lindsay, a pastebin-style tool for short server-side scripts written in Python, JavaScript or PHP that executes them within a Google App Engine powered sandbox. The Java code that implements the service is available on GitHub. # 13th August 2009, 1:51 pm

optfunc. Command line parsing libraries in Python such as optparse frustrate me because I can never remember how to use them without consulting the manual. optfunc is a new experimental interface to optparse which works by introspecting a function definition (including its arguments and their default values) and using that to construct a command line argument parser. Feedback and suggestions welcome! # 28th May 2009, 7:38 pm

geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services. # 27th May 2009, 10:02 am

disturbyte’s zenqueue. Simple, tiny and fast Python message queue server built on top of coroutines and Eventlet, using JSON over TCP as the message format. I’m impressed with how potentially useful this looks considering the small amount of code. The author benchmarks it at 28 thousand messages/second. # 11th May 2009, 1:27 pm

djangopeople.net on GitHub. I’ve released the source code for Django People, the geographical community site developed last year by myself and Natalie Downe (it hasn’t otherwise been touched since April last year, so it needs porting to Django 1.1). If you want a new feature on the site, implement it and I’ll see about merging it in. # 4th May 2009, 6:12 pm

Web Hooks and the Programmable World of Tomorrow. Tour de force presentation on Web Hooks by Jeff Lindsay. Tons of really good ideas—provided your application isn’t Flickr sized, there’s a good chance you could implement web hooks pretty cheaply and unleash a huge flurry of creativity from your users. GitHub makes a great case study here. # 16th February 2009, 9 pm

fireeagle_api.py. Steve Marshall’s Fire Eagle python binding on GitHub. # 22nd March 2008, 11:57 pm