Simon Willison’s Weblog

On javascript, python, django, node, openstreetmap, ...

Recent entries

Getting married and going travelling one month ago

It’s been a busy month. On Saturday the 5th of June I married the wonderful Natalie Downe in a beautiful ceremony at Roedean School in Brighton. The reception had owls, cheese, a ferret, a golden eagle, amazing Turkish food, Jewish chair dancing and lovely guests. It was the happiest day of my life.

The official wedding photos were taken by Drew McLellan, and there’s a Flickr group pool as well. The day after the wedding Natalie’s sister Louise took some fun photos of us running around Brighton in our wedding clothes.

Thanks to everyone who helped out with the preparations, and also to everyone who came along to share the special day with us. And a big thanks to Tom Coates, my best man.

Yesterday afternoon, we set out on our honeymoon. I’m writing this from the beach in Nice, on the south coast of France. Tomorrow we take the ferry to Corsica for a week in relative luxury. After that, we’re backpacking around Europe, then Africa, then the rest of the world. We’ve given up our flat and put our stuff in to storage, and the plan is to keep on travelling until we get fed up or run out of money. We expect to be gone for at least 18 months.

Since we’re both web developers, we’re lucky to be able to take some of our work with us. I’ll still be doing some work for the Guardian and Natalie is available for freelance work. If you have something you think we can help you with, drop us a line.

Naturally we’ll be blogging, tweeting and Flickring our adventures. You can follow our updates at http://sparkabout.net/.

11:01 pm / 21st June 2010 /

/ nataliedowne, natbat, personal, sparkabout, tomcoates, travel, wedding

Comprehensive notes from my three hour Redis tutorial three months ago

Last week I presented two talks at the inaugural NoSQL Europe conference in London. The first was presented with Matthew Wall and covered the ways in which we have been exploring NoSQL at the Guardian. The second was a three hour workshop on Redis, my favourite piece of software to have the NoSQL label applied to it.

I’ve written about Redis here before, and it has since earned a place next to MySQL/PostgreSQL and memcached as part of my default web application stack. Redis makes write-heavy features such as real-time statistics feasible for small applications, while effortlessly scaling up to handle larger projects as well. If you haven’t tried it out yet, you’re sorely missing out.

For the workshop, I tried to give an overview of each individual Redis feature along with detailed examples of real-world problems that the feature can help solve. I spent the past day annotating each slide with detailed notes, and I think the result makes a pretty good stand-alone tutorial. Here’s the end result:

Redis tutorial slides and notes

In unrelated news, Nat and I both completed the first ever Brighton Marathon last weekend, in my case taking 4 hours, 55 minutes and 17 seconds. Sincere thanks to everyone who came out to support us—until the race I had never appreciated how important the support of the spectators is to keep going to the end. We raised £757 for the Have a Heart children’s charity. Thanks in particular to Clearleft who kindly offered to match every donation.

10:36 pm / 25th April 2010 /

/ brightonmarathon, guardian, marathon, nosql, redis, running

WildlifeNearYou talk at £5 app, and being Wired (not Tired) three months ago

Two quick updates about WildlifeNearYou. First up, I gave a talk about the site at £5 app, my favourite Brighton evening event which celebrates side projects and the joy of Making Stuff. I talked about the site’s genesis on a fort, crowdsourcing photo ratings, how we use Freebase and DBpedia and how integrating with Flickr’s machine tags gave us a powerful location API for free. Here’s the video of the talk, courtesy of Ian Oszvald:

£5 App #22 WildLifeNearYou by Simon Willison and Natalie Downe from IanProCastsCoUk on Vimeo.

Secondly, I’m excited to note that WildlifeNearYou spin-off OwlsNearYou.com is featured in UK Wired magazine’s Wired / Tired / Expired column... and we’re Wired!

8:42 pm / 11th April 2010 /

/ api, crowdsourcing, fivepoundapp, flickr, freebase, owlsnearyou, talks, wildlifenearyou, wired

Elsewhere

Yesterday

Hookbox (via) For most web projects, I believe implementing any real-time comet features on a separate stack from the rest of the application makes sense—keep using Rails, Django or PHP for the bulk of the application logic, and offload any WebSocket or Comet requests to a separate stack built on top of something like Node.js, Twisted, EventMachine or Jetty. Hookbox is the best example of that philosophy I’ve yet seen—it’s a Comet server that makes WebHook requests back to your regular application stack to check if a user has permission to publish or subscribe to a given channel. “The key insight is that all application development with hookbox happens either in JavaScript or in the native language of the web application itself”.
canto.js: An Improved HTML5 Canvas API (via) Improved is an understatement: canto adds jQuery-style method chaining, the ability to multiple coordinates to e.g. lineTo at once, relative coordinate methods (regular Canvas does everything in terms of absolute coordinates), the ability to use degrees instead of radians, a rounded corner shortcut, a more convenient .revert() method and a simple parser that can understand SVG path expressions! The only catch: it uses getters and setters so won’t work in IE.

28th July 2010

nodejitsu's node-http-proxy (via) Exactly what I’ve been waiting for—a robust HTTP proxy library for Node that makes it trivial to proxy requests to a backend with custom proxy behaviour added in JavaScript. The example app adds an artificial delay to every request to simulate a slow connection, but other exciting potential use cases could include rate limiting, API key restriction, logging, load balancing, lint testing and more besides.

22nd July 2010

Jeremiah Grossman: I know who your name, where you work, and live. Appalling unfixed vulnerability in Safari 4 and 5 —if you have the “AutoFill web forms using info from my Address Book card” feature enabled (it’s on by default) malicious JavaScript on any site can steal your name, company, state and e-mail address—and would be able to get your phone number too if there wasn’t a bug involving strings that start with a number. The temporary fix is to disable that preference.

21st July 2010

What to do when PyPI goes down. My deployment scripts tend to rely on PyPI these days (they install dependencies in to a virtualenv) which makes me distinctly uncomfortable. Jacob explains how to use the PyPI mirrors that are starting to come online, but that won’t help if the PyPI listing links to an externally hosted file which starts to 404, as happened with the python-openid package quite recently (now fixed). The comments on the post discuss workarounds, including hosting your own PyPI mirror or bundling tar.gz files of your dependencies with your project.

20th July 2010

[UPDATE] Spatial Search in Apache Lucene and Solr. Spacial search is finally coming (back) to Solr—trunk now supports sorting and boosting by distance.
Easier custom Model Manager Chaining. A neat solution to the problem of wanting to write a custom QuerySet method (.published() for example) which is also available on that model’s objects manager, without having to write much boilerplate.
Three new features for reddit gold. Reddit’s experiments with a subscriber program are interesting to watch. 9,000 people signed up as subscribers without there being any benefit at all, and they’re now being rewarded with the ability to opt out of ads and access to computationally expensive features (like different ways of sorting their own user page) that wouldn’t scale for the entire user base.