Simon Willison’s Weblog

Datasettes · simonw/datasette. I’m collecting examples of datasette-powered APIs on the project wiki. # 14th November 2017, 7:39 am

Datasette for Polar Bears. I found a fun dataset of Polar Bear ear tag tracking data put out by the USGS Alaska Science Center and deployed it using datasette in just a couple of minutes—here’s how I did it. # 14th November 2017, 5:41 am

simonw/csvs-to-sqlite. I built a simple tool for bulk converting multiple CSV files into a SQLite database. # 13th November 2017, 6:49 am

Find conferences to speak at with Lanyrd. We just launched calls for participation on Lanyrd. You can list calls for any conference, browse them by topic, and subscribe to an Atom feed of calls for your area of interest. # 24th November 2010, 2:38 am

Welcome to Lanyrd | The Lanyrd Blog. We’ve started a blog for Lanyrd, our social conference directory project. We’re off to a great start: “Lanyrd is now listing 1,508 conferences and 5,167 individual speaker profiles. 5,637 people have signed in to the site and made 13,293 edits to our data.” # 11th September 2010, 9:32 pm

Lanyrd—the social conference directory. Nat and my new project, launched today and doing pretty well despite some early server hiccups. Sign in with Twitter to see conferences that your friends are speaking at, attending or tracking, then add your own events. We’re particularly keen on helping people build up a detailed profile of their previous talks, so adding older conferences is encouraged. # 31st August 2010, 7:41 pm

getlatlon.com commit dae961a... I’ve finally added an OpenStreetMap tab to getlatlon.com—here’s the diff, it turns out adding a custom OpenStreetMap layer to an existing Google Maps application only takes a few lines of boilerplate code. # 10th July 2010, 12:22 pm

webhook-relay. Another of my experiments with Node.js: webhook-relay is a self-contained queue and webhook request sending agent. Your application can POST to it specifying a webhook alert to be sent off, and webhook-relay will place that request in an in-memory queue and send it on its own time, avoiding the need for your main application server to block until the outgoing request has been processed. # 19th March 2010, 10:17 am

dogproxy. Another of my experiments with Node.js—this is a very simple HTTP proxy which addresses the dog pile effect (also known as the thundering herd) by watching out for multiple requests for a URL that is currently “in flight” and bundling them together. # 3rd February 2010, 1:05 pm

They Write For You. I helped put together this visualisation of stories written by MPs for various newspapers at last Friday’s ’Hackers and Hacks" hack day. # 2nd February 2010, 9:27 am

World Government Data. Launched last week, this is the Guardian’s meta-search engine for searching and browsing through data from four different government data sites (with more sites planned). Under the hood it’s Django, Solr, Haystack and the Scrapy crawling library. The application was built by Ben Firshman during an internship over Christmas. # 27th January 2010, 12:27 pm

Applications: the real stars of the data.gov.uk launch. A write-up of the data.gov.uk launch event at the Guardian. I demonstrated the Guardian’s World Government Data search engine and a small data.gov.uk inspired feature on WildlifeNearYou. # 27th January 2010, 12:23 pm

Help pick the best photos, but watch out, it’s addictive! My favourite WildlifeNearYou feature yet—our new tool asks you to pick the best from two photos, then uses the results to rank all of the photos for each species. It’s surprisingly addictive—we had over 5,000 votes in the first two hours, peaking at 4 or 5 votes a second. The feature seems to be staying nice and speedy thanks to Redis under the hood. Photos in the top three for any given species display a medal on their photo page. # 25th January 2010, 12:36 am

Owls, Otters, Monkeys and Lions Near You.com. It’s not just Owls—we also registered ottersnearyou.com, monkeysnearyou.com and lionsnearyou.com. We’ll probably stop there though, or this could turn in to a very expensive marketing gimmick. # 19th January 2010, 2:54 pm

owlsnearyou.com. Nat and I built this over the weekend. It asks for your location, then tells you where your nearest Owl is (using sightings data people have entered on WildlifeNearYou.com). If you’re using Firefox 3.6 or an iPhone it grabs your location using the W3C geolocation API so you don’t have to type anything at all. # 19th January 2010, 2:45 pm

WildlifeNearYou: Help identify animals in other people’s photos. The first of a number of crowdsourcing-style features we have planned for WildlifeNearYou—users can now help identify the animals in each other’s photos, and photo owners get a simple queue interface to approve or reject the suggestions. # 15th January 2010, 1:35 am

countdown_to_newyear.py. A quick Python / OS X script I knocked up last night to count in the new year (using the OS X “say” command). # 1st January 2010, 4:24 pm

Investigate your MP’s expenses. Launched today, this is the project that has been keeping me ultra-busy for the past week—we’re crowdsourcing the analysis of the 700,000+ scanned MP expenses documents released this morning. It’s the Guardian’s first live Django-powered application, and also the first time we’ve hosted something on EC2. # 18th June 2009, 11:16 pm

optfunc. Command line parsing libraries in Python such as optparse frustrate me because I can never remember how to use them without consulting the manual. optfunc is a new experimental interface to optparse which works by introspecting a function definition (including its arguments and their default values) and using that to construct a command line argument parser. Feedback and suggestions welcome! # 28th May 2009, 7:38 pm

geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services. # 27th May 2009, 10:02 am

djangopeople.net on GitHub. I’ve released the source code for Django People, the geographical community site developed last year by myself and Natalie Downe (it hasn’t otherwise been touched since April last year, so it needs porting to Django 1.1). If you want a new feature on the site, implement it and I’ll see about merging it in. # 4th May 2009, 6:12 pm

Oscars 2009: the interactive results | guardian.co.uk. My latest project for the Guardian, put together on very short notice. Updates live as the results are announced, and allows Twitter users to vote on their favourite for each category by sending a specially formatted message to @guardianfilm—jQuery and Ajax polling against S3 under the hood. # 23rd February 2009, 2:19 am

lightningtimer.net. I’m fed up of having to dig out or knock up a timer script every time I manage lightning talks, so I’ve given one a domain name. You can use lightningtimer.net/#90 to set a different start time for the counter. # 12th November 2008, 4:43 pm

Tweetersation. Nat and my latest side project: a JSONP API powered tool to more easily follow conversations between people on Twitter, by combining their tweets in to a single timeline. # 2nd October 2008, 5:08 pm

csrf_protect.php. A PHP class for applying CSRF protection to existing PHP applications, using output buffering to rewrite any POST forms on a page. Heavily inspired by Django’s CSRF middleware. Tell me if you spot any bugs! # 24th September 2008, 2:52 pm

backup_to_s3.py. I wrote Yet Another S3 backup script today. It’s a thin wrapper about boto that doesn’t do anything particularly impressive, but it fits my brain. # 21st September 2008, 6:51 pm

django-html. A small project I’m working on to make Django behave better with regards to HTML v.s. XHTML. # 9th September 2008, 11:59 pm

json-tinyurl. Because sometimes you want to be able to create a shorter version of a URL directly from JavaScript without hosting your own server-side proxy. # 27th August 2008, 10:58 am

Get Lat Lon now has a “Get my location (by IP)” button. It took all of five minutes to add using the new google.loader.ClientLocation API. The button is only visible if your location can be resolved. # 22nd August 2008, 10:16 am

json-head. I’ve deployed another App Engine mini-app, which provides a JSON-P API for running HEAD requests against an arbitrary URL (useful for checking things like Content-Length and Content-Type headers and whether a URL returns 200). App Engine’s urlfetch limitations mean it can only deal with port 80 and 443 requests. # 29th July 2008, 3:41 pm