Simon Willison’s Weblog

Subscribe

130 items tagged “json”

2018

jq recipes. Remy Sharp’s handy collection of jq recipes, each one linking to an interactive demo on jqterm.com. I thought jq was just for extracting values from a JSON document—I hadn’t realized how powerful it was for modifying and extending those documents as well. # 22nd August 2018, 3:23 pm

JSON Escape Text. I built a tiny tool for turning text into an escaped JSON string—I needed it to help create descriptions and canned SQL queries for adding to Datasette’s metadata.json files. # 25th April 2018, 4:13 am

react-jsonschema-form. Exciting library from the Mozilla Services team: given a JSON Schema definition, react-jsonschema-form can produce a cleanly designed React-powered form for adding and editing data that matches that schema. Includes support for adding multiple items in a nested array, re-ordering them, custom form widgets and more. # 23rd April 2018, 9:38 pm

gron. Ingenious tool for working with JSON on the command line: run “gron URL/filepath” to transform a JSON document into a multi-line assignment structure designed to be easy to run grep against. Grep it, then pipe it back into “gron --ungron” to convert the filtered data back to JSON again. It solves a similar problem to jq—which is addressed in the README: “gron’s primary purpose is to make it easy to find the path to a value in a deeply nested JSON blob when you don’t already know the structure; much of jq’s power is unlocked only once you know that structure”. # 3rd April 2018, 9:16 pm

How I made a Who’s On First subset database. Inspired by Paul Ford on Twitter, I tried out a new trick with SQLite: connect to a database containing JSON, attach a brand new empty database file using “attach database”, then populate it using INSERT INTO ... SELECT plus the json_extract() function to extract out a subset of the JSON properties into a new table in the new database. # 3rd February 2018, 5:25 am

[On SQLite] The JSON interface is like, “we save the text and when you retrieve it we parse the JSON at several hundred MB/s and let you do path queries against it please stop overthinking it, this is filing cabinet.”

Paul Ford # 29th January 2018, 4:29 pm

How to turn a list of JSON objects into a Datasette. ramadis on GitHub cleaned up data on 184,879 crimes reported in Buenos Aires since 2016 and shared them on GitHub as a JSON file. Here are my notes on how to use Pandas to convert JSON into SQLite and publish it using Datasette. # 20th January 2018, 1:07 am

How to compile and run the SQLite JSON1 extension on OS X. Thanks, Stack Overflow! I’ve been battling this one for a while—it turns out you can download the SQLite source bundle, compile just the json1.c file using gcc and load that extension in Python’s sqlite3 module (or with Datasette’s --load-extension= option) to gain access to the full suite of SQLite JSON functions—json(), json_extract() etc. # 10th January 2018, 9:01 pm

2017

Datasette: instantly create and publish an API for your SQLite databases

I just shipped the first public version of datasette, a new tool for creating and publishing JSON APIs for SQLite databases.

[... 968 words]

Select Transform: JSON Template over JSON (via) A barrage of interesting ideas here. Having clients transmit up a JSON template which is then executed against data on the server and used to return exactly the data the client needs is just one of them (significant overlap with GraphQL there). # 18th October 2017, 5:12 pm

2013

How does a web page interact with a server to parse a dynamic JSON file?

If you’re only dealing with 60 records there’s no need to add a full database. I’ve actually hand coded a 50 record JSON file before and it was fine- use an editor with good JSON support (I like Sublime Text 2) and it’s pretty easy to hand write.

[... 103 words]

2012

Has JSON pretty much replaced XML for string processing for the web, or are there use cases where XML is still necessary?

It’s replaced XML as the default format for most APIs. XML is still necessary for Atom/RSS feeds and other existing standards built on top of XML. It’s also a better choice than JSON for markup-style data—stuff like XHTML where tags are applied to sequences of characters within larger chunks of text.

[... 81 words]

How can I parse unquoted JSON with JavaScript?

Unquoted JSON isn’t JSON—the JSON spec requires that strings are quoted (with double quotes, not single quotes).

[... 104 words]

2011

What are the JSON security concerns in web development?

Be very careful when implementing JSON-P for authenticated actions—evil third party sites could assemble URLs to your user’s private data and steal it. This attack has worked against Gmail in the past.

[... 203 words]

2010

Indexing JSON in Solr 3.1. The next release of Solr will support indexing documents provided as JSON—Solr currently requires incoming documents to be formatted as XML. # 10th December 2010, 9:46 am

I think the Web community has spoken, and it’s clear that what it wants is HTML5, JavaScript and JSON. XML isn’t going away but I see it being less and less a Web technology; it won’t be something that you send over the wire on the public Web, but just one of many technologies that are used on the server to manage and generate what you do send over the wire.

James Clark # 2nd December 2010, 6:48 pm

If I have data that loads using  json / JavaScript will it get indexed by Google?

No. Personally I dislike sites with content that is only accessible through JavaScript, but if you absolutely insist on doing this you should look in to implementing the Google Ajax Crawling mechanism: http://code.google.com/web/ajaxc...

[... 56 words]

JSON sucks. [...] Every time I need to (correctly) represent a large integer such as 4611686018427387900, I’m forced to do so in a string. It causes me to throw up in mouth a little.

Theo Schlossnagle # 11th October 2010, 11:06 am

ijson. A SAX-style streaming JSON parser for Python, using ctypes to talk to the yajl C library. # 22nd September 2010, 9:59 pm

DNode: Asynchronous Remote Method Invocation for Node.js and the Browser. Mind-bendingly clever. DNode lets you expose a JavaScript function so that it can be called from another machine using a simple JSON-based network protocol. That’s relatively straight-forward... but DNode is designed for asynchronous environments, and so also lets you pass callback functions which will be translated in to references and used to make remote method invocations back to your original client. And to top it off, there’s a browser client library so you can perform the same trick over a WebSocket between a browser and a server. # 11th July 2010, 2:27 pm

The Guardian’s Open Platform is open for business. The Guardian’s Content API is now out of beta. Of particular interest: you can access basic article metadata (headline, URL and tags) without using an API key at all, and the API supports JSONP—just request format=json and include a callback=foo argument. # 20th May 2010, 5:40 pm

Google Charts release notes, February 2010. More new Google Charts Image API features I hadn’t noticed before: charts of large data sets can now be generated using a POST request, but the killer feature is the ability to add ?chof=validate to see useful error messages. ?chof=json is interesting too—it gives you back a JSON object detailing the coordinates of various interesting shapes on the associated chart, which you can then use to create your own image maps or JavaScript tooltips. It’s a shame it doesn’t support JSON-P. # 27th April 2010, 12:07 pm

The MessagePack Project. A cross-language efficient binary-based serialization library—“It’s like JSON, but very fast and small”. Claims to outperform protocol buffers for at least some benchmarks. # 21st April 2010, 10:55 pm

ElasticSearch: Your Data, Your Search. A neat example of how ElasticSearch’s schemaless indexes and native JSON support make it ridiculously easy to index different types of data and run queries across them. # 12th February 2010, 3:22 pm

Elastic Search (via) Solr has competition! Like Solr, Elastic Search provides a RESTful JSON HTTP interface to Lucene. The focus here is on distribution, auto-sharding and high availability. It’s even easier to get started with than Solr, partly due to the focus on providing a schema-less document store, but it’s currently missing out on a bunch of useful Solr features (a web interface and faceting are the two that stand out). The high availability features look particularly interesting. UPDATE: I was incorrect, basic faceted queries are already supported. # 11th February 2010, 6:33 pm

2009

jsondns. A JSONP API for making DNS queries, with a nice URL structure. # 30th December 2009, 5:37 pm

Orderly JSON. Essentially the JSON equivalent of RelaxNG’s compact syntax—a pleasant mini-language for describing JSON objects which compiles to the more verbose JSONSchema format. # 23rd December 2009, 2:44 pm

Djangopeople JSON parser. Awesome—Andy McKay has compensated for the lack of an official DjangoPeople API by creating a JSONP screen scraped API and hosting it on App Engine. As far as I’m concerned this is an officially supported feature—I’ll make sure future site changes don’t break it, and when I do add an API I’ll try to keep it compatible and help Andy set up redirects. # 28th November 2009, 11:29 am

Introducing BERT and BERT-RPC. Justification for inventing a brand new serialisation protocol: Thrift and Protocol Buffers both use IDLs and code generation, XML “is not convertible to a simple unambiguous data structure in any language I’ve ever used” and JSON lacks support for unencoded binary data. The result is BERT—Binary ERlang Term—which extracts a format from Erlang in much the same way that JSON extracted one from JavaScript. # 21st October 2009, 10:11 pm

cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity. # 21st September 2009, 11:09 pm