Simon Willison’s Weblog

Datasette 0.23: CSV, SpatiaLite and more (via) The big new feature in 0.23 is CSV export: any Datasette table or query can now be exported as CSV, including the option to get all matching rows in one giant CSV file taking advantage of Python 3 async and Datasette’s efficient keyset pagination. Also in this release: improved support for SpatiaLite and various JSON API improvements including the ability to expand foreign key labels in JSON and CSV responses.

# 18th June 2018, 3:34 pm / projects, csv, datasette

Datasette 0.21: New _shape=, new _size=, search within columns. Nothing earth-shattering here but it’s accumulated enough small improvements that it warranted a new release. You can now send ?_shape=array to get back a plain JSON array of results, ?_size=XXX|max to get back a specific number of rows from a table view and ?_search_COLUMN=text to run full-text search against a specific column.

# 5th May 2018, 11:25 pm / datasette, projects

Make Near Me (via) The natural evolution of owlsnearme.com—Make Near Me uses the Zeit Now API to allow anyone to deploy their own version of Owls Near Me for any species! I announced this on stage at Zeit Day SF 2018 as part of my talk on Datasette and Datasette Publish.

# 28th April 2018, 9:28 pm / zeit-now, projects, owlsnearyou

JSON Escape Text. I built a tiny tool for turning text into an escaped JSON string—I needed it to help create descriptions and canned SQL queries for adding to Datasette’s metadata.json files.

# 25th April 2018, 4:13 am / projects, json, datasette

csvs-to-sqlite 0.8. I released a new version of my csvs-to-sqlite tool this morning with a bunch of handy new features. It can now rename columns and define their types, add the CSV filenames as an additional column, add create indexes on columns and parse dates and datetimes into SQLite-friendly ISO formatted values.

# 24th April 2018, 4:11 pm / projects, csv, sqlite

Datasette 0.15: sort by column (via) I’ve released the latest version of Datasette to PyPI. The key new feature is the ability to sort tables by column, using clickable column headers or directly via the new _sort= and _sort_desc= querystring parameters.

# 9th April 2018, 5:25 pm / projects, datasette

owlsnearme source code on GitHub. Here’s the source code for our new owlsnearme.com project. It’s a single-page React application that pulls all of its data from the iNaturalist API. We built it this weekend with the SuperbOwl kick-off as a hard deadline so it’s not the most beautiful React code, but it’s a nice demonstration of how React (and create-react-app in particular) can be used for rapid development.

# 4th February 2018, 10:33 pm / react, natalie-downe, javascript, projects, inaturalist, github

Owls Near Me. Back in 2010 Natalie and I shipped owlsnearyou.com—a website for finding your nearest owls, using data from the sadly deceased WildlifeNearYou (RIP). To celebrate #SuperbOwl Sunday we rebuilt the same concept on top of the excellent iNaturalist API. Search for a place to see which owls have been spotted there, or click the magic button to geolocate your device and see which owls have been spotted in your nearby area!

# 4th February 2018, 10:26 pm / wildlifenearyou, inaturalist, natalie-downe, projects

ftfy—fix unicode that’s broken in various ways (via) I shipped a small web UI wrapper around the excellent Python FTFY library, which can take broken unicode strings and suggest a sequence of operations that can be applied to get back sensible text.

# 9th January 2018, 3:22 am / unicode, projects, zeit-now

Datasette 0.14: customization edition. I just released the latest version of Datasette with a strong theme of customization: Datasette now supports custom templates and CSS styling hooks, and the metadata format has been expanded to allow per-database and per-table source/license/description information. You can also now define named canned queries which will be packaged up with your data.

# 10th December 2017, 1:55 am / projects, datasette

gzthermal-web (via) I built a quick web application wrapping the gzthermal gzip visualization tool and deployed it to Zeit Now wrapped up in a Docker container. Give it a URL and it shows you a PNG visualization of how gzip encodes that page.

# 21st November 2017, 6:24 pm / zeit-now, projects, docker, sanic

csvs-to-sqlite: Refactoring columns into separate lookup tables. I just shipped a new version of csvs-to-sqlite with the ability to extract specified columns into a separate SQLite lookup table by passing additional command-line arguments.

# 17th November 2017, 6:41 am / projects, csv

Datasettes · simonw/datasette. I’m collecting examples of datasette-powered APIs on the project wiki.

# 14th November 2017, 7:39 am / datasette, projects, github

Datasette for Polar Bears. I found a fun dataset of Polar Bear ear tag tracking data put out by the USGS Alaska Science Center and deployed it using datasette in just a couple of minutes—here’s how I did it.

# 14th November 2017, 5:41 am / datasette, projects

simonw/csvs-to-sqlite. I built a simple tool for bulk converting multiple CSV files into a SQLite database.

# 13th November 2017, 6:49 am / csv, sqlite, github, datasette, projects

Find conferences to speak at with Lanyrd. We just launched calls for participation on Lanyrd. You can list calls for any conference, browse them by topic, and subscribe to an Atom feed of calls for your area of interest.

# 24th November 2010, 2:38 am / conferences, lanyrd, projects, recovered

Welcome to Lanyrd | The Lanyrd Blog. We’ve started a blog for Lanyrd, our social conference directory project. We’re off to a great start: “Lanyrd is now listing 1,508 conferences and 5,167 individual speaker profiles. 5,637 people have signed in to the site and made 13,293 edits to our data.”

# 11th September 2010, 9:32 pm / blogging, conferences, lanyrd, projects, recovered

Lanyrd—the social conference directory. Nat and my new project, launched today and doing pretty well despite some early server hiccups. Sign in with Twitter to see conferences that your friends are speaking at, attending or tracking, then add your own events. We’re particularly keen on helping people build up a detailed profile of their previous talks, so adding older conferences is encouraged.

# 31st August 2010, 7:41 pm / conferences, lanyrd, natimon, projects, twitter, recovered

getlatlon.com commit dae961a... I’ve finally added an OpenStreetMap tab to getlatlon.com—here’s the diff, it turns out adding a custom OpenStreetMap layer to an existing Google Maps application only takes a few lines of boilerplate code.

# 10th July 2010, 12:22 pm / getlatlon, google-maps, javascript, openstreetmap, projects, recovered

webhook-relay. Another of my experiments with Node.js: webhook-relay is a self-contained queue and webhook request sending agent. Your application can POST to it specifying a webhook alert to be sent off, and webhook-relay will place that request in an in-memory queue and send it on its own time, avoiding the need for your main application server to block until the outgoing request has been processed.

# 19th March 2010, 10:17 am / nodejs, node, javascript, webhooks, experiments, projects, webhookrelay

dogproxy. Another of my experiments with Node.js—this is a very simple HTTP proxy which addresses the dog pile effect (also known as the thundering herd) by watching out for multiple requests for a URL that is currently “in flight” and bundling them together.

# 3rd February 2010, 1:05 pm / scaling, node, nodejs, projects, javascript, dogproxy, dogpile, thunderingherd

They Write For You. I helped put together this visualisation of stories written by MPs for various newspapers at last Friday’s ’Hackers and Hacks" hack day.

# 2nd February 2010, 9:27 am / hackday, mps, politics, newspapers, visualisations, projects