Simon Willison’s Weblog

geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services. # 27th May 2009, 10:02 am

Offline Processing on App Engine: a Look Ahead. A session at IO next week: “App Engine was designed to run request-driven web applications, although this will change in the coming year with the release of a number of offline computing components. In this session, we’ll explore the task queue/executor model of computation and some of the more interesting applications.” # 20th May 2009, 12:40 pm

Google container data center tour (on YouTube). 45,000 servers in 45 shipping containers, along with some serious looking plumbing. # 26th April 2009, 10:14 pm

And Now For Something Entire... Oooh! Shiny! Alex Russell on O3D, the new 3D browser plugin from Google that makes OpenGL accessible to JavaScript (and embeds V8 so performance won’t suck even on slower browsers). # 22nd April 2009, 12:19 pm

London’s abandoned Underground Stations on Google Street View. “The network is littered with buildings that belonged to stations that closed their doors to the public because routes were changed and diverted, or because there was just too little traffic to make them viable. Here are some of the remnants of disused Underground stations that you can see on Google’s Street View of London.” # 14th April 2009, 2:51 pm

Reducing XSS by way of Automatic Context-Aware Escaping in Template Systems (via) The Google Online Security Blog reminds us that simply HTML-escaping everything isn’t enough—the type of escaping needed depends on the current markup context, for example variables inside JavaScript blocks should be escaped differently. Google’s open source Ctemplate library uses an HTML parser to keep track of the current context and apply the correct escaping function automatically. # 14th April 2009, 9:26 am

Running Rhino and Helma NG on Google App Engine. Helma NG is a JavaScript web app framework, which now works on App Engine out of the box. # 12th April 2009, 12:52 pm

Using Scala with Google App Engine. Scala works, but I haven’t seen confirmation on actors yet (which are likely to break due to their dependency on threads). # 11th April 2009, 3:28 pm

Dynamic languages on Google App Engine—an overview. Ola Bini’s notes on exploring the new Java support for App Engine with the aim of getting JVM dynamic languages such as JRuby running. Restrictions include a complete lack of threads (which will make it hard to get Scala up and running), but JRuby trunk now works without modification. # 8th April 2009, 2:08 pm

App Engine: Scheduled Tasks With Cron. Cron tasks simply hit a URL on your application, and can be run as frequently as once a minute. They made up their own syntax, which much nicer than traditional unix cron. # 8th April 2009, 2:04 pm

Google uncloaks once-secret server. Instead of a data centre wide UPS and redundant power supplies, each Google server has its own 12V battery. They live in standard shipping containers, each holding 1,160 servers. # 2nd April 2009, 10:47 am

ProjectPlan—unladen-swallow. A branch of Python 2.6 aiming to radically improve performance (the target is a 5x improvement), by compiling Python to machine code using LLVM’s JIT engine. I think this is a Google 20% time project (or maybe not, see the comments). An early version without LLVM is already available for download. # 30th March 2009, 10:09 am

django-gae2django. An implementation of the Google App Engine API (datastore, memcache, urlfetch, users and mail) that runs on Django, allowing you to take an existing application written for App Engine and deploy it on your own server on top of Django. # 9th March 2009, 3:37 pm

Map Maker for Developers. Tiles from Google’s Map Maker crowdsourcing effort are now available in the JS and static maps APIs on an opt-in basis. Maybe I’m misunderstanding something here, but Google Map Maker seems like a big step backwards for open geographic data. People donate their mapping efforts to Google, who keep them—unlike OpenStreetMap, where the donated efforts are made available under a Creative Commons license. # 21st February 2009, 9:05 am

Write to a Google Spreadsheet from a Python script. I didn’t know Google Spreadsheets could directly serve dynamic images that automatically update when the underlying data changes. # 16th February 2009, 9:02 pm

Google App Engine 1.1.9 boosts capacity and compatibility. Niall summarises the recent changes to App Engine. urllib and urllib2 support plus massively increased upload limits and request duration quotas will make it a whole lot easier to deploy serious projects on the platform. # 16th February 2009, 8:35 pm

Specify your canonical. You can now use a link rel=“canonical” to tell Google that a page has a canonical URL elsewhere. I’ve run in to this problem a bunch of times—in some sites it really does make sense to have the same content shown in two different places—and this seems like a neat solution that could apply to much more than just metadata for external search engines. # 14th February 2009, 11:28 am

Plaxo sees 92% success rate with OpenID/OAuth hybrid method. Really wish I could have been at the OpenID UX Summit hosted by Facebook yesterday—sounds like an awful lot of important problems are being solved. # 11th February 2009, 5:20 pm

Yahoo! Query Language thoughts. An engineer on Google’s App Engine provides an expert review of Yahoo!’s YQL. I found this more useful than the official documentation. # 9th February 2009, 10:29 pm

Google App Engine: A roadmap update! Receiving e-mail, background tasks and XMPP. I predict a bunch of sites will start building small parts of their overall functionality on App Engine when some of these features land (much easier than hosting your own custom XMPP server). # 9th February 2009, 7 pm

Recreating the button. Fascinating article from Doug Bowman on the work that went in to creating custom CSS buttons for use across Google’s different applications, avoiding images to improve performance ensure they could be easily styled using just CSS. I’d love to see the Google Code team turn this in to a full open source release—the more sites using these buttons the more familiar they will become to users at large. # 5th February 2009, 9:50 pm

Post-Commit Web Hooks for Google Code Project Hosting (via) I really, really like web hooks (which I’ve been calling “callback APIs”, but it looks like “web hooks” is the term that’s sticking). I’m interested in their scaling challenges—I’ve heard XMPP advocates argue that a web hook style model simply won’t scale for really large sites. # 4th February 2009, 10:22 am

Sharding Counters on Google App Engine. “While the datastore for App Engine scales to support a huge number of entities it is important to note that you can only expect to update any single entity, or entity-group, about five times a second”. This article explains a technique for sharding writes across multiple counters in detail, including a way to keep a memcache counter updated at the same time for faster reads. # 27th January 2009, 8:27 pm

google-mobwrite. Neal Fraser’s terrifyingly clever differential synchronization algorithm (for SubEthaEdit-style collaboration over the web) is now available as an open source Python and JavaScript library. # 24th January 2009, 11:55 pm

AJAX APIs Playground. Ferociously useful collection of executable and editable example code for all(?) of Google’s JavaScript APIs, including Google Maps and the increasingly interesting Visualization API. # 22nd January 2009, 6:38 pm

Leo Hickman on the carbon cost of Googling. Alex Wissner-Gross (who published the 7g/search figures) appears to be including Google’s extra capacity, so total CO2 output divided by number of searches. Google’s 0.2g/search estimate includes just the energy used by the servers processing your query. # 12th January 2009, 3:31 pm

Powering a Google search. I thought the recent estimate of each Google search producing 7g of CO2 was a little high—Google have responded with a claim that the amount is 0.2g instead. # 12th January 2009, 11:15 am

Amazon SimpleDB—Now With Select. So now all three of Yahoo!, Amazon and Google have invented their own SQL-like languages (YQL, SimpleDB and GQL)—though it looks like Yahoo!’s is the only one that attempts to provide joins. # 18th December 2008, 8:59 am