Simon Willison’s Weblog

Subscribe
Atom feed for github

132 items tagged “github”

2009

Why I like Redis

I’ve been getting a lot of useful work done with Redis recently.

[... 900 words]

Introducing BERT and BERT-RPC. Justification for inventing a brand new serialisation protocol: Thrift and Protocol Buffers both use IDLs and code generation, XML “is not convertible to a simple unambiguous data structure in any language I’ve ever used” and JSON lacks support for unencoded binary data. The result is BERT—Binary ERlang Term—which extracts a format from Erlang in much the same way that JSON extracted one from JavaScript.

# 21st October 2009, 10:11 pm / protocolbuffers, json, erlang, javascript, bert, serialisation, thrift, xml, github

How We Made GitHub Fast. Detailed overview of the new GitHub architecture. It’s a lot more complicated than I would have expected—lots of moving parts are involved in ensuring they can scale horizontally when they need to. Interesting components include nginx, Unicorn, Rails, DRBD, HAProxy, Redis, Erlang, memcached, SSH, git and a bunch of interesting new open source projects produced by the GitHub team such as BERT/Ernie and ProxyMachine.

# 21st October 2009, 9:14 pm / github, scaling, nginx, unicorn, rails, drbd, haproxy, replication, redis, erlang, memcached, ssh, git, proxymachine, ruby, bert, ernie

Scriptlets—Quick web scripts (via) From the prolific Jeff Lindsay, a pastebin-style tool for short server-side scripts written in Python, JavaScript or PHP that executes them within a Google App Engine powered sandbox. The Java code that implements the service is available on GitHub.

# 13th August 2009, 1:51 pm / github, jeff-lindsay, webhooks, scriptlets, python, javascript, php, googleappengine, appengine, open-source, java

optfunc. Command line parsing libraries in Python such as optparse frustrate me because I can never remember how to use them without consulting the manual. optfunc is a new experimental interface to optparse which works by introspecting a function definition (including its arguments and their default values) and using that to construct a command line argument parser. Feedback and suggestions welcome!

# 28th May 2009, 7:38 pm / optfunc, github, introspection, commandlines, optparse, projects, python

geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services.

# 27th May 2009, 10:02 am / geocoders, github, projects, geocoding, placemaker, google, yahoo, geonames, jacob-kaplan-moss, python, web-services

djng—a Django powered microframework

djng is nearly two weeks old now, so it’s about time I wrote a bit about the project.

[... 1,501 words]

disturbyte’s zenqueue. Simple, tiny and fast Python message queue server built on top of coroutines and Eventlet, using JSON over TCP as the message format. I’m impressed with how potentially useful this looks considering the small amount of code. The author benchmarks it at 28 thousand messages/second.

# 11th May 2009, 1:27 pm / zenqueue, message-queues, python, eventlet, coroutines, json, zachary-voase, github

djangopeople.net on GitHub. I’ve released the source code for Django People, the geographical community site developed last year by myself and Natalie Downe (it hasn’t otherwise been touched since April last year, so it needs porting to Django 1.1). If you want a new feature on the site, implement it and I’ll see about merging it in.

# 4th May 2009, 6:12 pm / github, git, django, django-people, open-source, projects, python

Web Hooks and the Programmable World of Tomorrow. Tour de force presentation on Web Hooks by Jeff Lindsay. Tons of really good ideas—provided your application isn’t Flickr sized, there’s a good chance you could implement web hooks pretty cheaply and unleash a huge flurry of creativity from your users. GitHub makes a great case study here.

# 16th February 2009, 9 pm / webhooks, github, jeff-lindsay, apis

Rate limiting with memcached

On Monday, several high profile “celebrity” Twitter accounts started spouting nonsense, the victims of stolen passwords. Wired has the full story—someone ran a dictionary attack against a Twitter staff member, discovered their password and used Twitter’s admin tools to reset the passwords on the accounts they wanted to steal.

[... 910 words]

2008

fireeagle_api.py. Steve Marshall’s Fire Eagle python binding on GitHub.

# 22nd March 2008, 11:57 pm / github, steve-marshall, fireeagle, python