Simon Willison’s Weblog

Indexing JSON in Solr 3.1. The next release of Solr will support indexing documents provided as JSON—Solr currently requires incoming documents to be formatted as XML. 10th December 2010, 9:46 am

ijson. A SAX-style streaming JSON parser for Python, using ctypes to talk to the yajl C library. 22nd September 2010, 9:59 pm

DNode: Asynchronous Remote Method Invocation for Node.js and the Browser. Mind-bendingly clever. DNode lets you expose a JavaScript function so that it can be called from another machine using a simple JSON-based network protocol. That’s relatively straight-forward... but DNode is designed for asynchronous environments, and so also lets you pass callback functions which will be translated in to references and used to make remote method invocations back to your original client. And to top it off, there’s a browser client library so you can perform the same trick over a WebSocket between a browser and a server. 11th July 2010, 2:27 pm

The Guardian’s Open Platform is open for business. The Guardian’s Content API is now out of beta. Of particular interest: you can access basic article metadata (headline, URL and tags) without using an API key at all, and the API supports JSONP—just request format=json and include a callback=foo argument. 20th May 2010, 5:40 pm

Google Charts release notes, February 2010. More new Google Charts Image API features I hadn’t noticed before: charts of large data sets can now be generated using a POST request, but the killer feature is the ability to add ?chof=validate to see useful error messages. ?chof=json is interesting too—it gives you back a JSON object detailing the coordinates of various interesting shapes on the associated chart, which you can then use to create your own image maps or JavaScript tooltips. It’s a shame it doesn’t support JSON-P. 27th April 2010, 12:07 pm

The MessagePack Project. A cross-language efficient binary-based serialization library—“It’s like JSON, but very fast and small”. Claims to outperform protocol buffers for at least some benchmarks. 21st April 2010, 10:55 pm

ElasticSearch: Your Data, Your Search. A neat example of how ElasticSearch’s schemaless indexes and native JSON support make it ridiculously easy to index different types of data and run queries across them. 12th February 2010, 3:22 pm

Elastic Search (via) Solr has competition! Like Solr, Elastic Search provides a RESTful JSON HTTP interface to Lucene. The focus here is on distribution, auto-sharding and high availability. It’s even easier to get started with than Solr, partly due to the focus on providing a schema-less document store, but it’s currently missing out on a bunch of useful Solr features (a web interface and faceting are the two that stand out). The high availability features look particularly interesting. UPDATE: I was incorrect, basic faceted queries are already supported. 11th February 2010, 6:33 pm

jsondns. A JSONP API for making DNS queries, with a nice URL structure. 30th December 2009, 5:37 pm

Orderly JSON. Essentially the JSON equivalent of RelaxNG’s compact syntax—a pleasant mini-language for describing JSON objects which compiles to the more verbose JSONSchema format. 23rd December 2009, 2:44 pm

Djangopeople JSON parser. Awesome—Andy McKay has compensated for the lack of an official DjangoPeople API by creating a JSONP screen scraped API and hosting it on App Engine. As far as I’m concerned this is an officially supported feature—I’ll make sure future site changes don’t break it, and when I do add an API I’ll try to keep it compatible and help Andy set up redirects. 28th November 2009, 11:29 am

Introducing BERT and BERT-RPC. Justification for inventing a brand new serialisation protocol: Thrift and Protocol Buffers both use IDLs and code generation, XML “is not convertible to a simple unambiguous data structure in any language I’ve ever used” and JSON lacks support for unencoded binary data. The result is BERT—Binary ERlang Term—which extracts a format from Erlang in much the same way that JSON extracted one from JavaScript. 21st October 2009, 10:11 pm

cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity. 21st September 2009, 11:09 pm

“MongoDB is fantastic for logging”. Sounds tempting... high performance inserts, JSON structured records and capped collections if you only want to keep the past X entries. If you care about older historic data but still want to preserve space you could run periodic jobs to roll up log entries in to summarised records. It shouldn’t be too hard to write a command-line script that hooks in to Apache’s logging directive and writes records to MongoDB. 26th August 2009, 7:09 pm

Tile Drawer (via) The most inspired use of EC2 I’ve seen yet: center a map on an area, pick a Cascadenik stylesheet URL (or write and link to your own) and Tile Drawer gives you an Amazon EC2 AMI and a short JSON snippet. Launch the AMI with the JSON as the “user data” parameter and you get your own OpenStreetMap tile rendering server, which self-configures on startup and starts rendering and serving tiles using your custom design. 26th August 2009, 9:32 am

Announcing Alice and Wonderland. Continuing the RabbitMQ “stuff to do with rabbits” naming convention, Alice is a RESTful interface to RabbitMQ which exposes information about vhosts/queues/users/exchanges/etc as JSON. Wonderland is a web UI for RabbitMQ implemented as a pure Ajax application which calls Alice. 17th July 2009, 9:12 am

MongoDB. Lots of discussions about this at EuroPython today—it’s a document database, very similar to CouchDB but significantly faster and suggested for production use. Best of all, trying it out on OS X is as easy as extracting the tarball and running “bin/mongod --dbpath /tmp/test-mongo-db run”. 30th June 2009, 7:13 pm

Firefox 3.5 for developers. It’s out today, and the feature list is huge. Highlights include HTML 5 drag ’n’ drop, audio and video elements, offline resources, downloadable fonts, text-shadow, CSS transforms with -moz-transform, localStorage, geolocation, web workers, trackpad swipe events, native JSON, cross-site HTTP requests, text API for canvas, defer attribute for the script element and TraceMonkey for better JS performance! 30th June 2009, 6:08 pm

disturbyte’s zenqueue. Simple, tiny and fast Python message queue server built on top of coroutines and Eventlet, using JSON over TCP as the message format. I’m impressed with how potentially useful this looks considering the small amount of code. The author benchmarks it at 28 thousand messages/second. 11th May 2009, 1:27 pm

With YQL Execute, the Internet becomes your database. This is nuts (in a good way). Yahoo!’s intriguing universal SQL-style XML/JSONP web service interface now supports JavaScript as a kind of stored procedure language, meaning you can use JavaScript and E4X to screen-scrape web pages, then query the results with YQL. 29th April 2009, 10:50 pm

Building Fast Client-side Searches. Flickr now lazily loads your entire contact list in to memory for auto-completion. Extensive benchmarking found that a control character delimited string was the fastest option for shipping thousands of contacts around as quickly as possible. 19th March 2009, 3:35 pm

Combine JSONP and jQuery to quickly build powerful mashups. jQuery’s JSONP support is one of my favourite little-known features of the library. 3rd March 2009, 3:17 pm

JsonML (JSON Markup Language). An almost non-lossy serialization format for sending XML as JSON (plain text in between elements is ignored). Uses the (element-name, attribute-dictionary, list-of-children) tuple format, which sadly means many common cases end up taking more bytes than the original XML. Still an improvement on serializations that behave differently when a list of children has only one item in it. 10th February 2009, 3:03 pm

Open in Browser Firefox Add-on (via) Solves the “application/json wants to download” problem, among others. 9th February 2009, 10:24 pm

husk.org. a flickr machine tag browser (via) Flickr recently added API methods for exploring the machine tags used by the community. Paul Mison has built a neat OS X Finder style interface for exploring them, using JSONP and jQuery. 15th December 2008, 11:24 pm

YQL—converting the web to JSON with mock SQL. YQL just got a whole lot more interesting to me—I had no idea they were exposing an HTML and RSS scraping tool over a JSONP API in addition to all of the Yahoo! web service methods. 13th December 2008, 9:39 am

Code your own election mashup with Google’s JSON data. The data that powered Google’s US election results map is available to download as a bunch of JSON files. 6th November 2008, 8:24 pm

typeface.js. Outstanding hack—renders custom fonts using VML in IE and canvas in everything else, using fonts that are defined as a set of vector paths stored using JSON. 27th October 2008, 11:45 pm

CSSHttpRequest (via) Devious cross-domain Ajax hack that uses CSS for transport (@import rules with data URIs, but it still works in IE). Similar to JSONP but safer, since JSONP can cause arbitrary JavaScript to execute. 23rd October 2008, 6:25 pm

FriendFeed launch a real-time API. This is huge: JSONP plus long polling Comet, with “everything since X” tokens to ensure you don’t miss anything. This is the first open Comet API I’ve seen anywhere. Combine this with FriendFeed’s regular API (which allows arbitrary message posting) and you’ve got a really powerful tool for hackers who want to experiment with Comet without rigging up their own infrastructure. 22nd October 2008, 2:18 pm

Page Inlink Analyzer (via) Here’s why I’m so keen on JSONP APIs—Eric Miraglia’s tool fires off dozens of cross-domain JSON requests to pull together information about inbound links to your site from Yahoo! Site Explorer and del.icio.us. I imagine it would have been uneconomic for him to provide the tool if it had to proxy every request through his own server. 15th October 2008, 5:23 pm

What’s New in Python 2.6 (via) Python 2.6 final has been released (the last 2.x version before 3.0). multiprocessing and simplejson (as json) are now in the standard library, any backwards compatible 3.0 features have been added and the official docs are now powered by Sphinx (used by Django 1.0 as well). There’s plenty more. 2nd October 2008, 11:47 am

simplejson 2.0.1. Python’s simplejson JSON library got a whole lot faster while I wasn’t looking. 1st October 2008, 10:55 pm

addSizes.js: Snazzy automatic link file-size generation. Posted to Nat’s snazzy new blog: a script that uses my json-head API to grab the file size of linked documents on a page and insert those sizes in to the document. 30th August 2008, 10:39 am

json-tinyurl. Because sometimes you want to be able to create a shorter version of a URL directly from JavaScript without hosting your own server-side proxy. 27th August 2008, 10:58 am

Dare left something out (and it’s important). Dave Winer: “You should at least learn the lessons and add to REST what it needs to catch up with XML-RPC. Seriously. What’s missing in REST, btw, is a standard method of serializing structs, lists and scalar types.” That would be JSON. 18th August 2008, 9:39 am

IMG-2-JSON (via) I’m not the only person deploying simple JSON-P APIs on App Engine: Adam Burmister’s tool extracts dimension, mimetype and EXIF metadata when provided the URL to an image file. 12th August 2008, 9:43 am

json-head. I’ve deployed another App Engine mini-app, which provides a JSON-P API for running HEAD requests against an arbitrary URL (useful for checking things like Content-Length and Content-Type headers and whether a URL returns 200). App Engine’s urlfetch limitations mean it can only deal with port 80 and 443 requests. 29th July 2008, 3:41 pm

Browser Uploads to S3 using HTML POST Forms. I didn’t know you could do this: create a regular HTML form that gives people permission to upload direct to your own S3 bucket, using a signed JSON policy statement in a hidden form field to prevent third parties from abusing your S3 account. 27th June 2008, 12:11 pm

jsontime. Nat and I threw this together this morning—it runs on Google App Engine and exposes Python’s pytz timezone library over JSONP. 21st June 2008, 7:07 pm

Dopplr place googlemaps, with and without Yahoo Geo API bounding box adjustment. Dopplr uses Geonames for most geo information, but is now mixing in bounding box data from the Yahoo! Geo web service to improve the default zoom level for their maps. The JSON callback API means no server-side code is required on Dopplr’s end. 17th May 2008, 11:35 pm

Session variables without cookies. Brilliant but terrifying hack—you can store up to 2 MB of data in window.name and it persists between multiple pages, even across domains. Doesn’t work with new tabs though, and storing JSON in it and eval()ing it is a bad idea—a malicious site could populate it before sending the user to you. 13th May 2008, 9:59 pm

Persevere adds Comet Support. Persevere sounds neat: a RESTful HTTP/JSON data store (the interface reminds me of CouchDB) which recently gained the ability to “subscribe” to a resource and receive notifications of updates via comet. 13th May 2008, 8:09 am

Google AJAX Search API: Flash and Server Side Access. Over a year after Google shot down their SOAP Search API, they’ve quietly released a JSON based one under the guise of supporting “Flash and other non JavaScript environments”. Comes with the strange requirement that an HTTP referer be sent with every request; the API key is optional. 22nd April 2008, 7:16 pm

CouchDB, XML, and E4X. Brilliant—CouchDB now enables SpiderMonkey’s E4X support, meaning CouchDB views can easily query XML documents stored inside JSON objects using E4X syntax. 5th March 2008, 12:31 am

Social Graph API. This is freaking awesome. Input one or more URLs to your profile pages and it returns a huge dump of crawled relationship data, based on XFN, FOAF and OpenID links. No API key required and it supports JSON callbacks so you can incorporate it in to a site without even needing to write any extra server-side code. 3rd February 2008, 10:34 pm

The Art & Science of JavaScript. The JavaScript book I contributed to is now shipping! My chapter describes how to build a Flickr / Google Maps mashup entirely using client-side code (via JSON-P). 12th January 2008, 7:05 pm

[Release] CouchDB 0.7.0. This is a huge milestone for the project—it’s the first official release to include the JSON REST API instead of XML, and it’s also the first release that is “intended for widespread use”. 17th November 2007, 12:25 am

CouchDB first impressions. Jacob’s been poking at CouchDB. Inserting data is slow, but everything else looks pretty slick considering how recently the JSON / JavaScript views functionality was added. 19th October 2007, 11:43 am

hasAccount. Stuart proposes a light-weight API for letting any site know if a user has an account (and is signed in) on another service. I wouldn’t want to deploy this without being confident that my CSRF protection was in order. 28th September 2007, 9:10 am

How should JSON strings be represented in Erlang? Erlang’s poor support for strings makes this a surprisingly tricky question. 14th September 2007, 8:17 am

CouchDB: Thinking beyond the RDBMS. CouchDB is a fascinating project—an Erlang powered non-relational database with a JSON API that lets you define “views” (really computed tables) based on JavaScript functions that execute using map/reduce. Damien Katz, the main developer currently works for MySQL and used to work on Lotus Notes. 3rd September 2007, 9:48 am

Freebase developer documentation. The JSON API and particularly the query language are fascinating. 3rd September 2007, 2:38 am

Cabochon event server. Written in Python (on top of SQLObject and Paste), uses JSON for messages, allows event consumers to subscribe with a callback URL. 2nd August 2007, 8:36 am

JSON and Browser Security. Douglas Crockford suggests using secret tokens to protect JSON content, and avoiding wrapper hacks to protect unauthorised JSON delivery as they may fall foul of undiscovered browser bugs in the future. 11th April 2007, 12:52 am

XML and JSON. James Clark on JSON’s strengths and weaknesses compared to XML. 9th April 2007, 8:57 pm

Fortify JavaScript Hijacking FUD. Bob Ippolito points out the flaws in the recent widely disseminated JavaScript Hijacking paper. While the paper does miss some important details, it’s good that more people are now aware of the security implications involved in serving JSON up wrapped in an array. 5th April 2007, 10:51 pm

Triplr. Ultra simple GET-based web service for converting RSS / Atom / RDF / Microformats+GRDDL to HTML / ntriples / RDF / RSS / JSON / Turtle. Small pieces, loosely joined. 30th March 2007, 3:30 pm

JSON and JSON-RPC for Erlang. Nice example of using lists:reverse and an accumulator to efficiently build a string in reverse order. 25th March 2007, 4:29 pm

Security; AJAX; JSON; Satisfaction. The JSON attack I linked to earlier only works against raw arrays, which technically aren’t valid JSON anyway. 6th March 2007, 8:06 am

JSON is not as safe as people think it is. Joe Walker reminds us that even authenticated JSON served without a callback or variable assignment is vulnerable to CSRF in Firefox, thanks to that browser letting you redefine the Array constructor. 5th March 2007, 10:51 pm

json-taglib. Because JSON just doesn’t have enough angle brackets. 4th March 2007, 8:52 pm

Safe JSON (via) Subtle but important point about JSON APIs: you shouldn’t use a callback or variable assignment for JSON incorporating private user data, especially if it’s at a predictable URL. 2nd March 2007, 1:11 pm

Badge Any RSS Feed With Yahoo! Pipes. Smart hack from Kent Brewster. Uses Yahoo! Pipes’ JSON output plus a few lines of JavaScript to create a badge from any RSS feed. 16th February 2007, 8:21 am

Apache Solr 1.1. Solr is the search Web Service built on top of Lucene. The latest release introduces JSON, Python and Ruby response formats in addition to XML. 13th January 2007, 1:16 am

Why doesn’t Python have more data format readers in the stdlib? I for one would love to see simplejson included in the standard library, with or without a C implementation. 8th January 2007, 1:03 am

Javascript character set screw-ups (via) Some browsers treat JavaScript files as having the same content-type as the page from which they are linked. This could cause problems with UTF-8 encoded JSON; the workaround is serving up ASCII with unicode escape sequences. 21st December 2006, 3:20 pm

PHP: JSON Functions. Now bundled in PHP 5. A great way to move data from PHP to some other language. 3rd November 2006, 12:25 pm

Proposed RFC for application/json (via) Douglas Crockford is putting JSON through the IETF. 1st August 2006, 9:29 pm

Flickr API + JSON + MochiKit + Django. Short and sweet. 26th July 2006, 6:12 pm

Don’t serve JSON as text/html. Another sneaky XSS trick. 5th July 2006, 11:46 pm

Django and JSON. Neat trick to serialize Djange QuerySets as JSON. 20th June 2006, 8:16 am

Using JSON with Yahoo! Web Services (via) No more cross-domain script access problems. 15th December 2005, 11:53 pm

Introducing JSON (via) A data-interchange format based on JavaScript objects and arrays. 7th February 2005, 11:42 pm