Simon Willison’s Weblog

niche-museums.com, powered by Datasette

25th November 2019

I just released a major upgrade to my www.niche-museums.com website (launched last month).

• The site is now rendered server-side. The previous version used lit-html to render content using JavaScript.
• Each museum now has its own page. Here’s today’s new museum listing for the Conservatory of Flowers in San Francisco. These pages have a map on them.
• The site has an about page.
• You can now link to the page for a specific latitude and longitude, e.g. this location in Golden Gate Park.
• The source code for the site is now available on GitHub.

Notably, the site is entirely powered by Datasette. It’s a heavily customized Datasette instance, making extensive use of custom templates and plugins.

It’s a really fun experiment. I’m essentially using Datasette as a weird twist on a static site generator—no moving parts since the database is immutable but there’s still stuff happening server-side to render the pages.

Continuous deployment

The site is entirely stateless and is published using Circle CI to a serverless hosting provider (currently Zeit Now v1, but I’ll probably move it to Google Cloud Run in the near future.)

The site content—46 museums and counting—lives in the museums.yaml file. I’ve been adding a new museum listing every day by editing the YAML file using Working Copy on my iPhone.

The build script runs automatically on every commit. It converts the YAML file into a SQLite database using my yaml-to-sqlite tool, then runs datasette publish now... to deploy the resulting database.

The full deployment command is as follows:

datasette publish now browse.db about.db \
    --token=$NOW_TOKEN \
    --alias=www.niche-museums.com \
    --name=niche-museums \
    --install=datasette-haversine \
    --install=datasette-pretty-json \
    --install=datasette-template-sql \
    --install=datasette-json-html \
    --install=datasette-cluster-map~=0.8 \
    --metadata=metadata.json \
    --template-dir=templates \
    --plugins-dir=plugins \
    --branch=master

There’s a lot going on here.

browse.db is the SQLite database file that was built by running yaml-to-sqlite.

about.db is an empty database built using sqlite3 about.db ''—more on this later.

The --alias= option tells Zeit Now to alias that URL to the resulting deployment. This is the single biggest feature that I’m missing from Google Cloud Run at the moment. It’s possible to point domains at deployments there but it’s not nearly as easy to script.

The --install= options tell datasette publish which plugins should be installed on the resulting instance.

--metadata=, --template-dir= and --plugins-dir= are the options that customize the instance.

--branch=master means we always deploy the latest master of Datasette directly from GitHub, ignoring the most recent release to PyPI. This isn’t strictly necessary here.

Customization

The site itself is built almost entirely using Datasette custom templates. I have four of them:

• index.html is the template used for the homepage, and for the page you see when you search for museums near a specific latitude and longitude.
• row-browse-museums.html is the template used for the individual museum pages. It includes the JavaScript used for the map (which is powered by Leaflet and uses Wikimedia’s OpenStreetMap tiles, which I discovered thanks to this Observable notebook by Tom MacWright).
• _museum_card.html is an included template rendering a card for a museum, shared by the index and museum pages.
• database-about.html is the template for the about page.

The about page uses a particularly devious hack.

Datasette doesn’t have an easy way to create additional custom pages with URLs at the moment (without abusing the asgi_wrapper() hook, which is pretty low-level).

But... every attached database gets its own URL at /database-name.

So, to create the /about page I create an empty database called about.db using the sqlite3 about.db "" command. I serve that using Datasette, then create a custom template for that specific database using Datasette’s template naming conventions.

I’ll probably come up with a less grotesque way of doing this and bake it into Datasette in the future. For the moment this seems to work pretty well.

Plugins

The two key plugins here are datasette-haversine and datasette-template-sql.

datasette-haversine adds a custom SQL function to Datasette called haversine(), which calculates the haversine distance between two latitude/longitude points.

It’s used by the SQL query which finds the nearest museums to the user.

This is very inefficient—it’s essentially a brute-force approach which calculates that distance for every museum in the database and sorts them accordingly—but it will be years before I have enough museums listed for that to cause any kind of performance issue.

datasette-template-sql is the new plugin I described last week, made possible by Datasette dropping Python 3.5 support. It allows SQL queries to be executed directly from templates. I’m using it here to run the queries that power homepage.

I tried to get the site working just using code in the templates, but it got pretty messy. Instead, I took advantage of Datasette’s --plugins-dir option, which causes Datasette to treat all Python modules in a specific directory as plugins and attempt to load them.

index_vars.py is a single custom plugin that I’m bundling with the site. It uses the extra_template_vars() plugin took to detect requests to the index page and inject some additional custom template variables based on values read from the querystring.

This ends up acting a little bit like a custom Django view function. It’s a slightly weird pattern but again it does the job—and helps me further explore the potential of Datasette as a tool for powering websites in addition to just providing an API.

Weeknotes

This post is standing in for my regular weeknotes, because it represents most of what I achieved this last week. A few other bits and pieces:

• I’ve been exploring ways to enable CSV upload directly into a Datasette instance. I’m building a prototype of this on top of Starlette, because it has solid ASGI file upload support. This is currently a standalone web application but I’ll probably make it work as a Datasette ASGI plugin once I have something I like.
• Shortcuts in iOS 13 got some very interesting new features, most importantly the ability to trigger shortcuts automatically on specific actions—including every time you open a specific app. I’ve been experimenting with using this to automatically copy data from my iPhone up to a custom web application—maybe this could help ingest notes and photos into Dogsheep.
• Posted seven new museums to niche-museums.com:
  • Cable Car Museum in San Francisco
  • Audium in San Francisco
  • House of Broel Dollhouse Museum in New Orleans
  • Neptune Society Columbarium in San Francisco
  • Recoleta Cemetery in Buenos Aires
  • NASA Glenn Visitor Center in Cleveland
  • Conservatory of Flowers in San Francisco
• I composed devious SQL query for generating the markdown for the seven most recently added museums.