niche-museums.com, powered by Datasette
25th November 2019
- Each museum now has its own page. Here’s today’s new museum listing for the Conservatory of Flowers in San Francisco. These pages have a map on them.
- The site has an about page.
- You can now link to the page for a specific latitude and longitude, e.g. this location in Golden Gate Park.
- The source code for the site is now available on GitHub.
It’s a really fun experiment. I’m essentially using Datasette as a weird twist on a static site generator—no moving parts since the database is immutable but there’s still stuff happening server-side to render the pages.
The site is entirely stateless and is published using Circle CI to a serverless hosting provider (currently Zeit Now v1, but I’ll probably move it to Google Cloud Run in the near future.)
The build script runs automatically on every commit. It converts the YAML file into a SQLite database using my yaml-to-sqlite tool, then runs
datasette publish now... to deploy the resulting database.
The full deployment command is as follows:
datasette publish now browse.db about.db \ --token=$NOW_TOKEN \ --alias=www.niche-museums.com \ --name=niche-museums \ --install=datasette-haversine \ --install=datasette-pretty-json \ --install=datasette-template-sql \ --install=datasette-json-html \ --install=datasette-cluster-map~=0.8 \ --metadata=metadata.json \ --template-dir=templates \ --plugins-dir=plugins \ --branch=master
There’s a lot going on here.
browse.db is the SQLite database file that was built by running
about.db is an empty database built using
sqlite3 about.db ''—more on this later.
--alias= option tells Zeit Now to alias that URL to the resulting deployment. This is the single biggest feature that I’m missing from Google Cloud Run at the moment. It’s possible to point domains at deployments there but it’s not nearly as easy to script.
--install= options tell
datasette publish which plugins should be installed on the resulting instance.
--plugins-dir= are the options that customize the instance.
--branch=master means we always deploy the latest master of Datasette directly from GitHub, ignoring the most recent release to PyPI. This isn’t strictly necessary here.
The site itself is built almost entirely using Datasette custom templates. I have four of them:
- index.html is the template used for the homepage, and for the page you see when you search for museums near a specific latitude and longitude.
- _museum_card.html is an included template rendering a card for a museum, shared by the index and museum pages.
- database-about.html is the template for the about page.
The about page uses a particularly devious hack.
Datasette doesn’t have an easy way to create additional custom pages with URLs at the moment (without abusing the asgi_wrapper() hook, which is pretty low-level).
But... every attached database gets its own URL at
So, to create the
/about page I create an empty database called
about.db using the
sqlite3 about.db "" command. I serve that using Datasette, then create a custom template for that specific database using Datasette’s template naming conventions.
I’ll probably come up with a less grotesque way of doing this and bake it into Datasette in the future. For the moment this seems to work pretty well.
The two key plugins here are
datasette-haversine adds a custom SQL function to Datasette called
haversine(), which calculates the haversine distance between two latitude/longitude points.
It’s used by the SQL query which finds the nearest museums to the user.
This is very inefficient—it’s essentially a brute-force approach which calculates that distance for every museum in the database and sorts them accordingly—but it will be years before I have enough museums listed for that to cause any kind of performance issue.
datasette-template-sql is the new plugin I described last week, made possible by Datasette dropping Python 3.5 support. It allows SQL queries to be executed directly from templates. I’m using it here to run the queries that power homepage.
I tried to get the site working just using code in the templates, but it got pretty messy. Instead, I took advantage of Datasette’s
--plugins-dir option, which causes Datasette to treat all Python modules in a specific directory as plugins and attempt to load them.
index_vars.py is a single custom plugin that I’m bundling with the site. It uses the extra_template_vars() plugin took to detect requests to the
index page and inject some additional custom template variables based on values read from the querystring.
This ends up acting a little bit like a custom Django view function. It’s a slightly weird pattern but again it does the job—and helps me further explore the potential of Datasette as a tool for powering websites in addition to just providing an API.
This post is standing in for my regular weeknotes, because it represents most of what I achieved this last week. A few other bits and pieces:
- I’ve been exploring ways to enable CSV upload directly into a Datasette instance. I’m building a prototype of this on top of Starlette, because it has solid ASGI file upload support. This is currently a standalone web application but I’ll probably make it work as a Datasette ASGI plugin once I have something I like.
- Shortcuts in iOS 13 got some very interesting new features, most importantly the ability to trigger shortcuts automatically on specific actions—including every time you open a specific app. I’ve been experimenting with using this to automatically copy data from my iPhone up to a custom web application—maybe this could help ingest notes and photos into Dogsheep.
- Posted seven new museums to niche-museums.com:
- I composed devious SQL query for generating the markdown for the seven most recently added museums.
More recent articles
- Weeknotes: Embeddings, more embeddings and Datasette Cloud - 17th September 2023
- Build an image search engine with llm-clip, chat with models with llm chat - 12th September 2023
- LLM now provides tools for working with embeddings - 4th September 2023
- Datasette 1.0a4 and 1.0a5, plus weeknotes - 30th August 2023
- Making Large Language Models work for you - 27th August 2023
- Datasette Cloud, Datasette 1.0a3, llm-mlc and more - 16th August 2023
- How I make annotated presentations - 6th August 2023
- Weeknotes: Plugins for LLM, sqlite-utils and Datasette - 5th August 2023
- Catching up on the weird world of LLMs - 3rd August 2023
- Run Llama 2 on your own Mac using LLM and Homebrew - 1st August 2023