Simon Willison’s Weblog

cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity. # 21st September 2009, 11:09 pm

“MongoDB is fantastic for logging”. Sounds tempting... high performance inserts, JSON structured records and capped collections if you only want to keep the past X entries. If you care about older historic data but still want to preserve space you could run periodic jobs to roll up log entries in to summarised records. It shouldn’t be too hard to write a command-line script that hooks in to Apache’s logging directive and writes records to MongoDB. # 26th August 2009, 7:09 pm

Tile Drawer (via) The most inspired use of EC2 I’ve seen yet: center a map on an area, pick a Cascadenik stylesheet URL (or write and link to your own) and Tile Drawer gives you an Amazon EC2 AMI and a short JSON snippet. Launch the AMI with the JSON as the “user data” parameter and you get your own OpenStreetMap tile rendering server, which self-configures on startup and starts rendering and serving tiles using your custom design. # 26th August 2009, 9:32 am

Announcing Alice and Wonderland. Continuing the RabbitMQ “stuff to do with rabbits” naming convention, Alice is a RESTful interface to RabbitMQ which exposes information about vhosts/queues/users/exchanges/etc as JSON. Wonderland is a web UI for RabbitMQ implemented as a pure Ajax application which calls Alice. # 17th July 2009, 9:12 am

MongoDB. Lots of discussions about this at EuroPython today—it’s a document database, very similar to CouchDB but significantly faster and suggested for production use. Best of all, trying it out on OS X is as easy as extracting the tarball and running “bin/mongod --dbpath /tmp/test-mongo-db run”. # 30th June 2009, 7:13 pm

Firefox 3.5 for developers. It’s out today, and the feature list is huge. Highlights include HTML 5 drag ’n’ drop, audio and video elements, offline resources, downloadable fonts, text-shadow, CSS transforms with -moz-transform, localStorage, geolocation, web workers, trackpad swipe events, native JSON, cross-site HTTP requests, text API for canvas, defer attribute for the script element and TraceMonkey for better JS performance! # 30th June 2009, 6:08 pm

disturbyte’s zenqueue. Simple, tiny and fast Python message queue server built on top of coroutines and Eventlet, using JSON over TCP as the message format. I’m impressed with how potentially useful this looks considering the small amount of code. The author benchmarks it at 28 thousand messages/second. # 11th May 2009, 1:27 pm

With YQL Execute, the Internet becomes your database. This is nuts (in a good way). Yahoo!’s intriguing universal SQL-style XML/JSONP web service interface now supports JavaScript as a kind of stored procedure language, meaning you can use JavaScript and E4X to screen-scrape web pages, then query the results with YQL. # 29th April 2009, 10:50 pm

Building Fast Client-side Searches. Flickr now lazily loads your entire contact list in to memory for auto-completion. Extensive benchmarking found that a control character delimited string was the fastest option for shipping thousands of contacts around as quickly as possible. # 19th March 2009, 3:35 pm

Combine JSONP and jQuery to quickly build powerful mashups. jQuery’s JSONP support is one of my favourite little-known features of the library. # 3rd March 2009, 3:17 pm

JsonML (JSON Markup Language). An almost non-lossy serialization format for sending XML as JSON (plain text in between elements is ignored). Uses the (element-name, attribute-dictionary, list-of-children) tuple format, which sadly means many common cases end up taking more bytes than the original XML. Still an improvement on serializations that behave differently when a list of children has only one item in it. # 10th February 2009, 3:03 pm

Open in Browser Firefox Add-on (via) Solves the “application/json wants to download” problem, among others. # 9th February 2009, 10:24 pm

husk.org. a flickr machine tag browser (via) Flickr recently added API methods for exploring the machine tags used by the community. Paul Mison has built a neat OS X Finder style interface for exploring them, using JSONP and jQuery. # 15th December 2008, 11:24 pm

YQL—converting the web to JSON with mock SQL. YQL just got a whole lot more interesting to me—I had no idea they were exposing an HTML and RSS scraping tool over a JSONP API in addition to all of the Yahoo! web service methods. # 13th December 2008, 9:39 am

Code your own election mashup with Google’s JSON data. The data that powered Google’s US election results map is available to download as a bunch of JSON files. # 6th November 2008, 8:24 pm

typeface.js. Outstanding hack—renders custom fonts using VML in IE and canvas in everything else, using fonts that are defined as a set of vector paths stored using JSON. # 27th October 2008, 11:45 pm

CSSHttpRequest (via) Devious cross-domain Ajax hack that uses CSS for transport (@import rules with data URIs, but it still works in IE). Similar to JSONP but safer, since JSONP can cause arbitrary JavaScript to execute. # 23rd October 2008, 6:25 pm

FriendFeed launch a real-time API. This is huge: JSONP plus long polling Comet, with “everything since X” tokens to ensure you don’t miss anything. This is the first open Comet API I’ve seen anywhere. Combine this with FriendFeed’s regular API (which allows arbitrary message posting) and you’ve got a really powerful tool for hackers who want to experiment with Comet without rigging up their own infrastructure. # 22nd October 2008, 2:18 pm

Page Inlink Analyzer (via) Here’s why I’m so keen on JSONP APIs—Eric Miraglia’s tool fires off dozens of cross-domain JSON requests to pull together information about inbound links to your site from Yahoo! Site Explorer and del.icio.us. I imagine it would have been uneconomic for him to provide the tool if it had to proxy every request through his own server. # 15th October 2008, 5:23 pm

What’s New in Python 2.6 (via) Python 2.6 final has been released (the last 2.x version before 3.0). multiprocessing and simplejson (as json) are now in the standard library, any backwards compatible 3.0 features have been added and the official docs are now powered by Sphinx (used by Django 1.0 as well). There’s plenty more. # 2nd October 2008, 11:47 am

simplejson 2.0.1. Python’s simplejson JSON library got a whole lot faster while I wasn’t looking. # 1st October 2008, 10:55 pm

addSizes.js: Snazzy automatic link file-size generation. Posted to Nat’s snazzy new blog: a script that uses my json-head API to grab the file size of linked documents on a page and insert those sizes in to the document. # 30th August 2008, 10:39 am

json-tinyurl. Because sometimes you want to be able to create a shorter version of a URL directly from JavaScript without hosting your own server-side proxy. # 27th August 2008, 10:58 am

Dare left something out (and it’s important). Dave Winer: “You should at least learn the lessons and add to REST what it needs to catch up with XML-RPC. Seriously. What’s missing in REST, btw, is a standard method of serializing structs, lists and scalar types.” That would be JSON. # 18th August 2008, 9:39 am

IMG-2-JSON (via) I’m not the only person deploying simple JSON-P APIs on App Engine: Adam Burmister’s tool extracts dimension, mimetype and EXIF metadata when provided the URL to an image file. # 12th August 2008, 9:43 am

json-head. I’ve deployed another App Engine mini-app, which provides a JSON-P API for running HEAD requests against an arbitrary URL (useful for checking things like Content-Length and Content-Type headers and whether a URL returns 200). App Engine’s urlfetch limitations mean it can only deal with port 80 and 443 requests. # 29th July 2008, 3:41 pm

Browser Uploads to S3 using HTML POST Forms. I didn’t know you could do this: create a regular HTML form that gives people permission to upload direct to your own S3 bucket, using a signed JSON policy statement in a hidden form field to prevent third parties from abusing your S3 account. # 27th June 2008, 12:11 pm

jsontime. Nat and I threw this together this morning—it runs on Google App Engine and exposes Python’s pytz timezone library over JSONP. # 21st June 2008, 7:07 pm