OpenIDPragmatism, purity and JSON content types
I started a conversation about this on Twitter the other day, but Twitter is a horrible place to have an archived discussion so I’m going to try again here.
If you’re producing a JSON API for other people to use (as opposed to an API that’s only really meant for your own local Ajax responses), you need to decide which Content-Type to use. The best option is not entirely obvious.
RFC 4672 defines JSON and reserves application/json as the preferred media type. The problem is that most browsers will prompt you to download the file rather than displaying it inline (as they would for text/plain or application/javascript). One of my favourite qualities of REST-style APIs is that they enable exploration and debugging using just a browser—using application/json throws a big, frustrating road block in the way. There are ways of telling your browser to treat application/json in the same way as text/plain but that doesn’t really help you if your aim is to create an API that’s easy for other developers to use.
It’s also worth mentioning that if you are returning JSONP (with an extra callback function wrapped around the JSON response to enable the dynamic script tag hack) you HAVE to serve as application/javascript—otherwise the script you are providing won’t be executed by the browser. Don’t forget to include charset=UTF8 as well (for both types of response).
So, it’s pragmatism v.s. purity. The correct thing to do is to return application/json, but doing so makes your API harder for developers to use.
In a brief, non-comprehensive review of some existing JSON APIs (FriendFeed, Flickr, Google Social Graph etc) I couldn’t find any that were using application/json, presumably for this exact reason.
Using the Accept: header
The Accept: header is one of my least favourite parts of HTTP. I like to be confident that if I send a URL to someone, they’ll get back exactly the same bytes as I did when I retrieved it myself (I distrust language negotiation for the same reason). However, a number of people suggested it on Twitter and it looks like it could be a useful solution to this problem.
I’m currently considering the following: ONLY use the application/json Content-Type in reply to requests that include application/json in their Accept header—essentially allowing clients that care about the correct content type to opt-in to receiving it. Everyone else (browsers included) gets application/javascript, which is less correct (though not an all-out lie, since JSON is a subset of JavaScript) but solves the usability problem.
A couple of things worry me about this. Firstly, is this a reasonable thing to use Accept for? Secondly, is there a chance that browsers might add application/json to their Accept header at some point in the future? Safari currently sends text/xml,application/xml,application/xhtml+xml,text/html; q=0.9,text/plain; q=0.8,image/png,*/*; q=0.5 while Firefox sends text/html,application/xhtml+xml,application/xml; q=0.9,*/*; q=0.8. Would it be smarter to look for */* and serve the incorrect Content-Type to those requests and the correct one to everything else?
An alternative is to simply allow people to specify “JSON with a browsable Content-Type” as an alternative format option, or to enable a “pretty=1” query string argument which returns the response as text/plain and potentially pretty prints it as well. I haven’t yet decided if this is better than messing around with the Accept header.
Just so you know, you can't serve JSONP as utf8 and expect it to work.
http://jerakeen.org/test/unicode/
IE6/7 doesn't respect the content-type on the script, it assume the script has the content-type of the requesting page. So you have to send jsonp as ascii. Which is possible, but you need to rememebr to do it..
Browsers, eh?
It's solving it from the wrong end, but how about bothering Firefox devs to make application/json displayed like application/javascript? They would most likely do it, and all devs use Firefox only, so that's an easy win.
Tomasz Wegrzanowski - 6th February 2009 10:37 - #
Some thoughts:
The thing is, at the end of the day I'm just not convinced by your argument that it's a useful design goal to enable people to just shove stuff into a web browser to get this done. A user agent, sure; and there are plenty of these that will do this better than a standard web browser. GET(1) and wget(1) spring to mind, but there must be some good GUI tools for people who are scared of the command line. Something that better organises the GET parameters, and provides access to other methods, authentication, and so on all laid out in a nice fashion.
If you really need an in-browser way of doing this, you can build an "API" browser in a webpage in not very much code. For people who aren't going to find a better tool, this will be more powerful than using a web browser anyway.
(Note my sarcastic quoting of API. I'm not convinced that the 'A' is remotely apposite in the case of network protocols over HTTP. That's a different issue though ;-) .)
By the way, you seem to be styling the useful bits out of ordered lists :-)
I tried looking for an existing mime-type display bug for this, but instead found https://bugzilla.mozilla.org/show_bug.cgi?id=37525 0 which argues (unsuccessfully, but it's a good argument) that JSON served as application/javascript is dangerous and shouldn't be returned to the script.
Oh, and personally, I use Safari for development. When it's not randomly eating all my cookies. Stupid browser.
The issue about not being able to use the browser as a debugging tool is just an issue of appropriate tool choice. A browser isn't the only debugging tool that is available for web APIs. I often use Todd Ditchendorf's HttpClient:
http://ditchnet.org/httpclient/
This lets you see everything that is going on in a request, gives you the raw content back, and allows you to easily stimulate non-GET requests.
HttpClient is OS X specific, but I'm sure there are analogous non-OS X alternatives (and if there aren't, there should be).
Russell Keith-Magee - 6th February 2009 11:11 - #
My preference here is to do the Right Thing and return application/json. If you want to help your users debug the "API" with a browser, process the Accept header and offer a HTML/XHTML representation (which could show the JSON response inline, along with other useful debug information, if you like) for clients that request it (i.e. browsers).
Applications that specifically want the JSON representation should explicitly specify "Accept: application/json", but you can still default to returning JSON for those clients that forget to include the Accept header.
The query string argument seems much stronger to me as the "browse the API" use case is the less common one. Those building applications using your API will find it much simpler not to have to bother with content negotiation. Also, I would venture to say that most developers will not take issue with adding query string arguments while developing.
I also like Chris Miles's suggestion of offering an HTML representation as it's likely to carry more useful information to a human.
I'm going to go with: writing proper APIs, against the specs, because browsers get it wrong is optimising for the wrong people. It only really makes your API harder for developers who are just using a browser for the wrong purpose anyway--it's not an API explorer, it's a web page browser.
Browsers ignore the Content-Type when loading through <script> (though they do adhere to the charset parameter, apart from perhaps Internet Explorer as the first comment indicates) so whether you use application/json, application/javascript, application/x-javascript, text/plain, text/html, or nothing at all does not really matter.
The HTTP gods would like you to do the correct thing though and would also like browsers to be changed to force you do the right thing and break half of the interwebs.
My pattern is to send application/json when it is in fact what I'm sending, and to only send that when the user agent requests it with an Accept header. mimeparse is nice for dealing with those:
http://code.google.com/p/mimeparse/
To allow human browsers to browse representations other than that asked for by the browser (i.e. not HTML) I usually parse an extension or query string on the URL that simulates content-negotiation.
I like the idea of letting a human browser declare "I'd like to see this representation inined in some HTML in my browser".
Basically, I like content negotiation, at least in a basic form, and think HTTP-based APIs should try to support it. JSON should be one of several representations available. This has worked wonderfully in the Socialtext and TiddlyWeb APIs.
James: finally fixed <ol> styling in comments, thanks for the nudge.
Maybe I'm being selfish here, but I don't want to settle with just using application/json because I personally always use my browser to try out new APIs, I use enough different browsers that I don't want to have to configure each one to work around this issue and I get really frustrated when a browser downloads an API response instead of showing it to me.
Here's how we do it with the Last.fm API:
When you're asking for JSON, you get
application/json, when you're asking for a Javascript callback, you gettext/javascript, which is displayed as text in browsers and satisfies your discoverability requirement.text/javascriptis preferred overapplication/javascriptto reduce confusion around browser compatibility issues.In short, you cannot use
application/javascriptin thetypeattribute of the<script>tag or IE will not load it. You can serve it with that MIME type from the server (browsers ignore the server's MIME type for script includes) but doing so and documenting it as such will be a source of confusion for developers working with our API.Digg's API use the Accept header but also allow you to choose the format with a parameter (they also use
text/javascript)James Wheare - 6th February 2009 13:13 - #
If you are building an API on top of a specific protocol, use the protocol the way it is intended. As many others have pointed out in this discussion, you are catering your API to those who browse the API rather than the actual applications that use the API if you don't return the correct content type.
If you really want a human-readable explanation of what to expect from your API, consider actually documenting the expected format. This does seem to be rare in a lot of today's web APIs, I know, but building a client to an API specification makes for cleaner applications all around.
Ken Hoxworth - 6th February 2009 13:39 - #
I agree with the "purity" crowd. The current browser behavior will never be changed to the correct way unless we as web developers force them by following the spec in the first place. Secondly, as far as browsing the API with a web browser goes, I do that too. However, I prefer my browser download application/json so that it opens in my default application/json text editor. This allows me to pretty-print the response, get begin/end brace highlighting, type colorizing (string, null, int) and even folding. I can't get that when the browser displays it all in one long string.
Your notion of using the Accept header is a fine one here, and is what we do. However, I do not recommend doing so in the ad-hoc way you describe but instead by using a Web server that does proper HTTP content negotiation.
I just wrote a short note on this: http://blog.therestfulway.com/2009/02/content-nego tiation-for-humans.html
I'm with the purity-over-pragmatism crowd, especially since the usability issue is already solved in Firefox by the "Open in browser" extension. And for everyone else, your "pretty=1" argument should be workable.
Brandt - 6th February 2009 16:15 - #
reedunderwood - 6th February 2009 16:25 - #
Simon-good discussion! It's definitely annoying that application/json prompts for download in the browser, but in practice, you need pretty=1 support anyway, since normally your JSON will be very compact and not properly indented. At Plaxo, we have pretty=1 and plaintext=1 for our JSON APIs, and the two together give you nicely formatted output in the browser as text/html but with a plaintext element at the top. I think only sending application/json based on an application/json Accept: header sounds reasonable, but as always I'd look at consumer libraries in the wild and make sure that doesn't break them. :)
Why do you always use a browser to try out these APIs? They're not actually meant for browsers, per se, unless HTML is one of the output choices.
I mean, you can use a fingernail clipper as a screwdriver in some cases, but it's not a surprise when it doesn't work.
Personally, I've always used cURL from the command line because it gives me repeatability, scriptability, and more control over all the various aspects of an HTTP request (ie. cookies, HTTP method, request body).
And, just to throw one more wrench in, when I worked at Delicious we actually started trying to make the API as inaccessible as possible in browsers - all because the choice of HTTP basic auth and the caching of credentials in browsers made the API quite open for exploits via inclusion in img tags on random 3rd party pages.
l.m. orchard - 6th February 2009 16:38 - #
James - Digg uses application/json for JSON responses, and text/javascript for javascript callbacks. The Accept: header seemed like a worthy idea at the time (two years ago) but I'm willing to bet that no one has ever actually used it in the wild.
It'd be nice if text/json were the blessed content type for JSON responses - why isn't it?
Michal Migurski - 6th February 2009 16:40 - #
If readable JSON response is what you're looking for, you have to return it prettified as it's a bit hard to read compact JSON responses that are more than 3 levels deep. With that in mind I like the pretty=1 parameter of the Google Social Graph API - it returns a text/plain response with added spaces to make the whole thing readable. In the absence of pretty it will return text/javascript which I'd change into application/json.
This also makes the links to the API more obvious as you're always returning the same content for the same URL, no hidden stuff, no weird "unexpected" behaviour.
Oh, and of course the fact that the whole del API runs on HTTP GET doesn't help the issue. But, you live and learn.
Actually, only some browsers assume that application/json is a "download" file. WebKit/Safari displays JSON inline. Mozilla/Firefox does prompt you to download the file, but it is not universal, and I think the solution is to server the proper content type, and go bug Mozilla to fix their browser.
Woody Gilk - 6th February 2009 17:20 - #
Also, for checking data structures, it can be helpful to pipe json from curl through a python or php script and generate pretty xml to make it more readable.
reedunderwood - 6th February 2009 17:49 - #
The problem that you cannot debug JSON output that is server as application/json in the browser can - at least in Firefox - be solved by prefixing the URL with "view-source:".
Example URL: view-source:http://twitter.com/statuses/user_timel ine/simonw.json
application/javascriptis definitely wrong for JSON, since not all JSON is legal Javascript. JSON in which the{"root node is": "an object"}is a Javascript expression, but not a Javascript program. To be an actual piece of executable Javascript, it would have to be wrapped in parentheses,({"like":"this"}), which is not valid JSON. (This is actually an accidental protection against<script>-based CSRF, that JSON arrays do not enjoy.)Probably the best solution to the debugging problem is for browsers to change. In the meantime, I'd either send
text/plainor ignore the problem until browsers catch up.One thing worth noting is that according to the specification of application/json it should always be UTF-8 and never have a charset defined in the header. The specification of application/json can be compared to the one of javascript - this came up in the discussions of an issue at drupal.org.
Also - regarding what to send - the Accept-headers can define what content types are preferred by the client. By parsing the header with for example the above mentioned mimeparser you could send text/javascript to every application not specifically telling that they would prefer application/json.
Pelle - 6th February 2009 18:42 - #
Purity-schurity. What you're really looking for is the browser to render the JSON is a sane, human-readable manner.
I was thinking about this myself earlier today, and figured that somebody must have ran into this before, so I did a quick search and turns up this plugin for Firefox: http://lackoftalent.org/michael/blog/json-in-firef ox/
Don't do the 'pragmatic' thing, just use better tools.
I didn't really run into this problem because it's pretty easy to tell Opera to display application/json as plain text in the browser window.
In my implementation, the JSON version of the API uses application/json while the JSON-P mode uses text/javascript. My advice to those using browsers that aren't so amenable is to just use the JSON-P output, which is identical to the JSON output but for the callback function wrapped around it.
I'm sure it won't be long before browsers add application/json to their default list of things that render as plain text in the browser window.
(The JSONovich Firefox Extension is another way to deal with this, if you don't mind installing some extra gunk in your browser.)
James Aylett wrote:
Quite true. The "more powerful" part especially involves being able to follow links as specified in whatever microformat/subprotocol of JSON you're following.
Simon said:
I'd sooner recommend another path segment, like
http:/my.host.com/api/resource/1. In general, I've found the URI-space design constraints are different enough between JSON representations and, say, HTML, to warrant a separate URI hierarchy.But both approaches are probably better than Content-Negotiation; the latter is harder to mashup, at least in the short term.
Another way to get around this problem which may work for some people is to return a different header depending on whether the request is a GET or POST request. For example, you could return content with a plain-text header (and with pretty print while you are at it) when the a GET request is sent, and return using the json header when a POST request is sent.
I'd worry that the further you abstract yourself from the data with querystrings and separate URLs for testing, you lose a connection with the data and your faith in being able to transfer your testing to working code will diminish.
James Wheare - 7th February 2009 02:04 - #
Should you change the behaviour of the server to accommodate the poor visual output of one client? No!
I both produce and consume RESTful JSON APIs and totally agree that Firefox does with application/json is quite annoying, but there are solutions.
Some client-side ways to deal with JSON:
1. Use another client
2. Use Poster (https://addons.mozilla.org/en-US/firefox/addon/269 1 ) which will just show you the JSON, and is also very handy when it comes to those other HTTP methods used in RESTful API development.
3. Use JSONovich (http://lackoftalent.org/michael/blog/json-in-firef ox/ ) as somebody else already posted... let the client take care of pretty printing.
Helen
Helen - 7th February 2009 13:26 - #
there is no issue whatsoever with the Accept header; it's used to negotiate representation - not the resource. Why should it matter to you what representation you and your friend are seeing as long as it's the same resource?
Mike - 19th February 2009 10:20 - #
New JSONView FF extension:
http://brh.numbera.com/blog/index.php/2009/02/24/j sonview-view-json-documents-in-firefox/
Here's how I deal with this: I offer a special text/html representation for all my resources, alongside the application/json representation.
You can generate a decent HTML rendering of a JSON syntax tree quite easily.
So if the client's Accept header assigns a higher quality value to text/html than to application/json (as a browser will), we return the HTML-ified representation of the resource.
There are other advantages of this approach - you can serialize links to other resources as actual anchor tags in HTML, meaning a human can click around the API and explore in the browser. This is what REST is all about, the whole linking / discoverablility / HATEOS thing. Making it work for HTML forces you to think about how to make the API discoverable in JSON too.
Also if you can implement a way to add docs into the HTML representations, you have yourself a self-documenting, human-browsable API, which is pretty awesome.
Not necessarily worth doing for everyone, but an interesting technique, worth considering...
Matthew - 11th January 2010 21:06 - #