Simon Willison’s Weblog

Items in May, 2009

Filters: Year: 2009 × Month: May ×

Knockbrex Castle. I’m off to a Scottish castle with 11 fellow geeks for /dev/fort—offline for six days, back next Saturday. # 30th May 2009, 10:51 am

Perl 6: The MAIN sub (via) “Calling subs and running a typical Unix program from the command line is visually very similar: you can have positional, optional and named arguments.”—that’s exactly what I was thinking when I came up with optfunc. # 28th May 2009, 9:32 pm

PostgreSQL Development Priorities. The top two for 8.4 are “Simple built-in replication” and “Upgrade-in-place”, Josh Berkus is seeking feedback on priorities for future work on 8.5. # 28th May 2009, 8:08 pm

optfunc. Command line parsing libraries in Python such as optparse frustrate me because I can never remember how to use them without consulting the manual. optfunc is a new experimental interface to optparse which works by introspecting a function definition (including its arguments and their default values) and using that to construct a command line argument parser. Feedback and suggestions welcome! # 28th May 2009, 7:38 pm

TiddlyPocketBook. Paul Downey took Nat’s dinky pocketbooks CSS and combined it with TiddlyWiki to create a single page pocketbook editor. # 28th May 2009, 1:24 am

Announcing Google Maps API v3. Sounds like a complete rewrite, with performance as the key goal. Only a developer preview at the moment, but my favourite feature is that API keys are no longer required. # 28th May 2009, 1:22 am

Changes in Opera’s user agent string format (via) How depressing... Opera 10 will ship with 9.80 in the User-Agent string because badly written browser sniffing scripts can’t cope with double digits. # 28th May 2009, 1:16 am

Dice-O-Matic hopper and elevator. An outstanding piece of applied geekery, now generating dice rolls for “It is a 7 foot tall, 104 pound, dice-eating monster, capable of generating 1.3 million rolls a day.” # 27th May 2009, 7:32 pm

You ask, they answer: Neal’s Yard Remedies. After reading the comments, something tells me Neal’s Yard Remedies may be regretting their decision to answer questions from Guardian readers. # 27th May 2009, 10:35 am

geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services. # 27th May 2009, 10:02 am

Testing Django Views for Concurrency Issues. Neat decorator for executing a Django view under high concurrency in your unit tests, to help spot errors caused by database race conditions that should be executed inside a transaction. # 27th May 2009, 10:01 am

The Web vs. the Fallacies. Tim Bray on how the architecture of the Web helps developers handle the Fallacies of Distributed Computing. # 25th May 2009, 11:49 pm Neat implementation of an ID server from Mike Malone—it serves up incrementing integers over a socket (using Python’s asyncore for fast IO) and records state to a file only after every 10,000 IDs served, so most of the time it’s not reading or writing to disk at all. If the server crashes it doesn’t matter because it can start up again at an integer it’s sure hasn’t been used before. # 25th May 2009, 9:34 pm

On the Anonymity of Home/Work Location Pairs. Most people can be uniquely identified by the rough location of their home combined with the rough location of their work. US Census data shows that 5% of people can be uniquely identified by this combination even at just census tract level (1,500 people). # 24th May 2009, 1:14 pm

iPlayer usage, for streaming, peaks about 10pm—just a little later from TV. But interestingly, iPlayer on the iPhone peaks at about midnight. So people are clearly going to bed with their iPhone and watching in bed. And we also see on the weekends, there’s a peak of Saturday and Sunday morning usage at about 8 to 10am in the morning on iPhone.

Anthony Rose # 23rd May 2009, 12:42 am

JS-Placemaker—geolocate texts in JavaScript. Chris Heilmann exposed Placemaker to JavaScript (JSONP) using a YQL execute table. Try his examples—I’m impressed that “My name is Jack London, I live in Ontario” returns just Ontario, demonstrating that Placemaker’s NLP is pretty well tuned. # 23rd May 2009, 12:36 am

Introducing Yardbird. I absolutely love it—an IRC bot built on top of Twisted that passes incoming messages off to Django code running in a separate thread. Requests and Response objects are used to represent incoming and outgoing messages, and Django’s regex-based URL routing is used to dispatch messages to different handling functions based on their content. # 22nd May 2009, 11:13 pm

Muck Rack: Links posted by Guardian Journalists on Twitter. I’m rather impressed by the Sawhorse Media collection of Twitter aggregation sites (Muck Rack aggregates journalists)—a simple idea very well executed. Here’s a nice example—this page shows links posted to Twitter by known Guardian journalists, but goes a step further and scrapes in the favicon, the real title of the page and resolves the domain from any shortened links. # 22nd May 2009, 10:02 pm

Flickr Shapefiles Public Dataset 1.0. Another awesome Geo dataset from the Yahoo! stable—this time it’s Flickr releasing shapefiles (geometrical shapes) for hundreds of thousands of places around the world, under the CC0 license which makes them essentially public domain. The shapes themselves have been crowdsourced from geocoded photos uploaded to Flickr, where users can “correct” the textual location assigned to each photo. Combine this with the GeoPlanet WOE data and you get a huge, free dataset describing the human geography of the world. # 22nd May 2009, 6:12 pm

Fake Reviews. Now now kids, play nice... Not at all surprised to hear this—nefarious iPhone app developers (in this case the team behind “London Tube”, an inferior version of Malcolm Barclay’s marvellous “Tube Deluxe”) have been caught leaving fake negative reviews on rival applications in the App Store. This is an excellent argument for adding friends/followers or importing an existing social graph—I’d much rather see reviews from people in my social network than strangers who may turn out to be sock puppets. # 22nd May 2009, 12:49 am

Dinky pocketbooks with WebKit transforms. Nat used 90 degree CSS transform rotations in print stylesheets for WebKit and Safari to create printable cut-out-and-fold pocketbooks from A4 pages. Very neat. # 22nd May 2009, 12:33 am

TwitterAlikeExample—redis. Excellent example of how you design a moderately complex system against a scalable key-value store (in this case redis). Most “how to build Twitter” code examples fail to address the hard problem of scaling user inboxes, but this one tackles it head on. # 21st May 2009, 11:14 pm

Working with Python and RabbitMQ. Nathan Borror eliminates the boilerplate needed to talk to RabbitMQ (or any other AMQP queue server) from Python. # 21st May 2009, 11:10 pm

If you review your first site version and don’t feel embarrassment, you spent too much time on it.

Reid Hoffman # 21st May 2009, 9:56 pm

AWS Import/Export: Ship Us That Disk! Andrew Tanenbaum said “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway”, and now you can ship your storage device direct to Amazon and have them load the data in to an S3 bucket for you. # 21st May 2009, 11:22 am

Yahoo! Placemaker. Really exciting new API from Yahoo!—Placemaker accepts a block of text (or a URL to HTML or RSS) and extracts and returns geographical locations mentioned in the text. I just ran my djng blog entry through it and it pulled out “Prague” as the only location mentioned. This should be really useful for adding geodata to existing textual content. # 20th May 2009, 9:34 pm

Yahoo! Geo: Announcing GeoPlanet Data. The Yahoo! WhereOnEarth geographic data set is fantastic, but I’ve always felt slightly uncomfortable about building applications against it in case the API went away. That’s not an issue any more—the entire dataset is now available to download and use under a Creative Commons Attribution license. It’s not entirely clear what the attribution requirements are—do you have to put “data from GeoPlanet” on every page or can you get away with just tucking the attribution away in an “about this site” page? UPDATE: The data doesn’t include latitude/longitude or bounding boxes, which severely reduces its utility. # 20th May 2009, 9:12 pm

Google Maps Data API (via) I’m disappointed by this one—it’s really just a CRUD store for the KML files used in Google MyMaps. It would be a lot more useful if it let you perform geospatial calculations against your stored map data using some kind of query API—a cloud service alternative to tools like PostGIS. # 20th May 2009, 9:07 pm

Offline Processing on App Engine: a Look Ahead. A session at IO next week: “App Engine was designed to run request-driven web applications, although this will change in the coming year with the release of a number of offline computing components. In this session, we’ll explore the task queue/executor model of computation and some of the more interesting applications.” # 20th May 2009, 12:40 pm

EC2: Creating an Image. Here’s the easier way of creating your own AMI: start with a running instance in EC2, then customise it to fit your purposes and create a new bundle (and then AMI) using the ec2-bundle-vol command. # 19th May 2009, 7:50 pm