Simon Willison’s Weblog

Polymaps. Absurdly classy: “a JavaScript library for image- and vector-tiled maps using SVG”. It can pull in image tiles from sources such as OpenStreetMap, then overlay SVG paths specified using GeoJSON. The demos make use of GeoJSON tiles for US states and counties hosted on AppEngine. The library is developed by Stamen and SimpleGeo, and released under a BSD license. SVG support in the browser is required. # 20th August 2010, 6:46 pm

App Engine at Google I/O 2010. OpenID and OAuth are now baked in to the AppEngine users API. They’re also demoing two very exciting new features—a mapper API for doing map/reduce style queries against the data store, and a Channel API for building comet applications. # 20th May 2010, 3:30 pm

Why toppcloud will not be agnostic. Ian Bicking’s toppcloud aims to offer deployment with the ease of use of AppEngine against a standard, open source Ubuntu + Python 2.6 + mod_wsgi + Varnish stack. Here he explains why he’s not going to vary the required components: keeping everything completely standardised means everyone gets the same bugs (and the same fixes). # 12th February 2010, 9:21 am

What’s hot? Introducing Zeitgeist. Dan Catt’s first project at the Guardian. “When something appears on the Zeitgeist page, it’s because it performed better (got more attention) than the norm for that content type/section/day”. The application itself is written in Python and runs on Google App Engine. # 5th February 2010, 12:17 pm

Introducing docent. Paul Mison’s clever little Flickr app for viewing galleries that have been added by your contacts. It runs in Python on App Engine and makes extensive use of the Task Queue API. # 28th January 2010, 8:35 pm

Djangopeople JSON parser. Awesome—Andy McKay has compensated for the lack of an official DjangoPeople API by creating a JSONP screen scraped API and hosting it on App Engine. As far as I’m concerned this is an officially supported feature—I’ll make sure future site changes don’t break it, and when I do add an API I’ll try to keep it compatible and help Andy set up redirects. # 28th November 2009, 11:29 am

Tornado Web Server (via) An extremely exciting addition to the Python web landscape, Tornado is the open sourced version of FriendFeed’s custom web stack. It’s a non-blocking (epoll) Python web server designed for handling thousands of simultaneous connections, perfect for building Comet applications. The web framework is cosmetically similar to web.py or App Engine’s webapp but has decorators for writing asynchronous request handlers. The template language uses Django-style syntax but allows you to use full Python expressions. FriendFeed have benchmarked it handling 8,000 requests a second running as four load-balanced processes on a 4 core server. # 10th September 2009, 9:32 pm

Scriptlets—Quick web scripts (via) From the prolific Jeff Lindsay, a pastebin-style tool for short server-side scripts written in Python, JavaScript or PHP that executes them within a Google App Engine powered sandbox. The Java code that implements the service is available on GitHub. # 13th August 2009, 1:51 pm

Curating conversations. Chris Thorpe has open-sourced the Guardian’s moderated Twitter backchannel app, for displaying back channels at high profile (and hence high potential for abuse) events. It’s a Python application that runs on App Engine. # 16th July 2009, 7:34 pm

App Engine outage postmortem. Interesting peek behind the scenes. The primary cause of the error was a bug in a GFS (Google File System) Master server caused by a MapReduce process sending a malformed filehandle, reminiscent of the error which took down S3 last year. # 9th July 2009, 12:49 pm

Address Extractor. Running on App Engine, an address extractor web service using code from the EveryBlock open source release. # 1st July 2009, 8:03 pm

Offline Processing on App Engine: a Look Ahead. A session at IO next week: “App Engine was designed to run request-driven web applications, although this will change in the coming year with the release of a number of offline computing components. In this session, we’ll explore the task queue/executor model of computation and some of the more interesting applications.” # 20th May 2009, 12:40 pm

python-sqlparse (via) Python library for re-identing SQL statements. This could make debugging Django’s generated SQL a whole lot easier. You can try the library out using an App Engine hosted application (complete with an API). # 28th April 2009, 8:25 pm

pubsubhubbub. From Brad Fitzpatrick, a simple but clever way of using web hooks (HTTP callbacks) to inform subscribers that an Atom feed has updated in almost real-time—solving the constant polling problem and making it easier for small sites to offer publish-subscribe APIs. Any Atom feed can delegate subscriber updates to a “hub” server. An example hub server implementation is provided running on App Engine. # 20th April 2009, 6:49 pm

Running Rhino and Helma NG on Google App Engine. Helma NG is a JavaScript web app framework, which now works on App Engine out of the box. # 12th April 2009, 12:52 pm

Using Scala with Google App Engine. Scala works, but I haven’t seen confirmation on actors yet (which are likely to break due to their dependency on threads). # 11th April 2009, 3:28 pm

Dynamic languages on Google App Engine—an overview. Ola Bini’s notes on exploring the new Java support for App Engine with the aim of getting JVM dynamic languages such as JRuby running. Restrictions include a complete lack of threads (which will make it hard to get Scala up and running), but JRuby trunk now works without modification. # 8th April 2009, 2:08 pm

App Engine: Scheduled Tasks With Cron. Cron tasks simply hit a URL on your application, and can be run as frequently as once a minute. They made up their own syntax, which much nicer than traditional unix cron. # 8th April 2009, 2:04 pm

django-gae2django. An implementation of the Google App Engine API (datastore, memcache, urlfetch, users and mail) that runs on Django, allowing you to take an existing application written for App Engine and deploy it on your own server on top of Django. # 9th March 2009, 3:37 pm

Google App Engine 1.1.9 boosts capacity and compatibility. Niall summarises the recent changes to App Engine. urllib and urllib2 support plus massively increased upload limits and request duration quotas will make it a whole lot easier to deploy serious projects on the platform. # 16th February 2009, 8:35 pm

Yahoo! Query Language thoughts. An engineer on Google’s App Engine provides an expert review of Yahoo!’s YQL. I found this more useful than the official documentation. # 9th February 2009, 10:29 pm

Google App Engine: A roadmap update! Receiving e-mail, background tasks and XMPP. I predict a bunch of sites will start building small parts of their overall functionality on App Engine when some of these features land (much easier than hosting your own custom XMPP server). # 9th February 2009, 7 pm

Sharding Counters on Google App Engine. “While the datastore for App Engine scales to support a huge number of entities it is important to note that you can only expect to update any single entity, or entity-group, about five times a second”. This article explains a technique for sharding writes across multiple counters in detail, including a way to keep a memcache counter updated at the same time for faster reads. # 27th January 2009, 8:27 pm

Why Google App Engine is broken and what Google must do to fix it. Aral Balkan describes a number of critical issues with App Engine. If you’re considering building something serious on it you need to read this article; I’ve run in to several of these problems myself just running toy projects on the platform. Here’s hoping they get addressed in the near future. # 3rd October 2008, 10 pm

html-whitelist (via) DeWitt Clinton’s web service wrapper aroud the html5lib HTML sanitiser, hosted on AppEngine. # 24th September 2008, 11:54 pm

We haven’t changed the name of the conference to “Over Quota”. Aral is having intermittent App Engine quota problems, which are proving impossible to debug. I had a similar problem with an App Engine app a while ago—the quota / debugging story really needs fixing. # 3rd September 2008, 1:37 pm

json-tinyurl. Because sometimes you want to be able to create a shorter version of a URL directly from JavaScript without hosting your own server-side proxy. # 27th August 2008, 10:58 am

IMG-2-JSON (via) I’m not the only person deploying simple JSON-P APIs on App Engine: Adam Burmister’s tool extracts dimension, mimetype and EXIF metadata when provided the URL to an image file. # 12th August 2008, 9:43 am

json-head. I’ve deployed another App Engine mini-app, which provides a JSON-P API for running HEAD requests against an arbitrary URL (useful for checking things like Content-Length and Content-Type headers and whether a URL returns 200). App Engine’s urlfetch limitations mean it can only deal with port 80 and 443 requests. # 29th July 2008, 3:41 pm

How-to: Full-text search in Google App Engine. Use search.SearchableModel instead of db.Model—it’s pretty rough at the moment which is probably why it’s still undocumented. # 27th June 2008, 8:25 am