Simon Willison’s Weblog

Product design at GitHub. At GitHub, every employee is a product designer.

# 2nd April 2011, 7:51 am / github, recovered

URL Design. Thoughtful tips on modern URL design, from GitHub designer Kyle Neath. GitHub has the best designed URLs of any application I can think of.

# 31st December 2010, 10:03 am / github, urls, recovered

10K Apart Contest: Cheating by Compressing Your JavaScript and CSS to PNG Images. Fascinating hack: transform your JS and CSS in to coloured pixels, save the result as a PNG to benefit from PNG’s built in compression algorithms, then read the data back out of the PNG and convert it back to text using JavaScript and canvas—all to reduce the on-disk filesize when entering the 10K app competition. Alex’s GithubFinder entry is worth checking out too.

# 23rd August 2010, 9:45 am / 10kapart, alex-le, compression, github, hacks, javascript, png, recovered

How we deploy new features. GitHub are experimenting with using Redis for configuration management. I’ve been thinking about this recently too—managing feature flags feels like an ideal use-case for Redis, since it lets you read multiple values on every page access without adding a bunch of extra read traffic on your regular database.

# 8th July 2010, 10:04 am / github, redis, recovered, feature-flags

Zero-downtime Redis upgrade discussion. GitHub have a short window of scheduled downtime in order to upgrade their Redis server. I asked in their comments if they’d considered trying to run the upgrade with no downtime at all using Redis replication, and Ryan Tomayko has posted some interesting replies.

# 28th May 2010, 2:50 pm / github, ops, redis, ryan-tomayko, upgrades, zero-downtime, recovered

GitHub: Announcing SVN Support. The best kind of April Fool’s joke: one that works. It’s read-only, but that’s good enough to support referencing GitHub repositories from SVN externals.

# 1st April 2010, 11:33 am / aprilfools, subversion, git, github

Ryan Tomayko on Github’s development process. In the comments—a fascinating insight in to how GitHub’s “developers work on whatever is most interesting to them” process manages to achieve really good results.

# 22nd February 2010, 9:18 am / github, ryan-tomayko, process, agile

Algorithmic recruitment with GitHub. Matt Biddulph crawls GitHub’s social graph using JUNG (the Java Universal Network/Graph Framework), JRuby and Yahoo! BOSS to find good leads on interesting developers in specific geographic locations.

# 12th February 2010, 1:17 pm / matt-biddulph, socialgraph, recruiting, github

jQuery.require() implementation. John Resig has added a new jQuery.require() function to a jQuery development branch, for release as part of jQuery 1.4. The commit on GitHub has an extensive discussion attached to it (scroll to the bottom).

# 17th December 2009, 11:24 am / github, jquery, john-resig, javascript

Introducing the YUI 3 Gallery. Write a plugin for YUI3, BSD license it and sign a CLA and Yahoo! will push your module out to their CDN and make it loadable using the YUI().use() statement. They’re coordinating the submissions using GitHub.

# 4th November 2009, 11:14 pm / cla, bsd, github, javascript, git, open-source, yahoo, yui, yui3

Introducing Resque. A new background worker management queue developed at GitHub, using Redis for the persistence layer. The blog post explains both the design and the shortcomings of previous solutions at length. Within 24 hours of the release code an external developer, Adam Cooke, has completely reskinned the UI.

# 4th November 2009, 8:20 pm / resque, open-source, redis, github, queue, workers, ruby, sinatra

Introducing BERT and BERT-RPC. Justification for inventing a brand new serialisation protocol: Thrift and Protocol Buffers both use IDLs and code generation, XML “is not convertible to a simple unambiguous data structure in any language I’ve ever used” and JSON lacks support for unencoded binary data. The result is BERT—Binary ERlang Term—which extracts a format from Erlang in much the same way that JSON extracted one from JavaScript.

# 21st October 2009, 10:11 pm / protocolbuffers, json, erlang, javascript, serialisation, thrift, xml, github

How We Made GitHub Fast. Detailed overview of the new GitHub architecture. It’s a lot more complicated than I would have expected—lots of moving parts are involved in ensuring they can scale horizontally when they need to. Interesting components include nginx, Unicorn, Rails, DRBD, HAProxy, Redis, Erlang, memcached, SSH, git and a bunch of interesting new open source projects produced by the GitHub team such as BERT/Ernie and ProxyMachine.

# 21st October 2009, 9:14 pm / github, scaling, nginx, unicorn, rails, drbd, haproxy, replication, redis, erlang, memcached, ssh, git, proxymachine, ruby, ernie

Scriptlets—Quick web scripts (via) From the prolific Jeff Lindsay, a pastebin-style tool for short server-side scripts written in Python, JavaScript or PHP that executes them within a Google App Engine powered sandbox. The Java code that implements the service is available on GitHub.

# 13th August 2009, 1:51 pm / github, jeff-lindsay, webhooks, scriptlets, python, javascript, php, googleappengine, appengine, open-source, java

optfunc. Command line parsing libraries in Python such as optparse frustrate me because I can never remember how to use them without consulting the manual. optfunc is a new experimental interface to optparse which works by introspecting a function definition (including its arguments and their default values) and using that to construct a command line argument parser. Feedback and suggestions welcome!

# 28th May 2009, 7:38 pm / optfunc, github, introspection, commandlines, optparse, projects, python

geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services.

# 27th May 2009, 10:02 am / geocoders, github, projects, geocoding, placemaker, google, yahoo, geonames, jacob-kaplan-moss, python, web-services

disturbyte’s zenqueue. Simple, tiny and fast Python message queue server built on top of coroutines and Eventlet, using JSON over TCP as the message format. I’m impressed with how potentially useful this looks considering the small amount of code. The author benchmarks it at 28 thousand messages/second.

# 11th May 2009, 1:27 pm / zenqueue, message-queues, python, eventlet, coroutines, json, zachary-voase, github

djangopeople.net on GitHub. I’ve released the source code for Django People, the geographical community site developed last year by myself and Natalie Downe (it hasn’t otherwise been touched since April last year, so it needs porting to Django 1.1). If you want a new feature on the site, implement it and I’ll see about merging it in.

# 4th May 2009, 6:12 pm / github, git, django, django-people, open-source, projects, python, natalie-downe

Web Hooks and the Programmable World of Tomorrow. Tour de force presentation on Web Hooks by Jeff Lindsay. Tons of really good ideas—provided your application isn’t Flickr sized, there’s a good chance you could implement web hooks pretty cheaply and unleash a huge flurry of creativity from your users. GitHub makes a great case study here.

# 16th February 2009, 9 pm / webhooks, github, jeff-lindsay, apis

fireeagle_api.py. Steve Marshall’s Fire Eagle python binding on GitHub.

# 22nd March 2008, 11:57 pm / github, steve-marshall, fireeagle, python