Simon Willison’s Weblog

Plugin support for Datasette Lite

17th August 2022

I’ve added a new feature to Datasette Lite, my distribution of Datasette that runs entirely in the browser using Python and SQLite compiled to WebAssembly. You can now install additional Datasette plugins by passing them in the URL.

Datasette Lite background

Datasette Lite runs Datasette in the browser. I initially built it as a fun technical proof of concept, but I’m increasingly finding it to be a genuinely useful tool for quick ad-hoc data analysis and publication. Not having any server-side components at all makes it effectively free to use without fear of racking up cloud computing costs for a throwaway project.

You can read more about Datasette Lite in these posts:

• Datasette Lite: a server-side Python web application running in a browser
• Joining CSV files in your browser using Datasette Lite
• Scraping data into Datasette Lite shows an example project where I scraped PSF board resolutions, stored the results in a CSV file in a GitHub Gist and then constructed this URL to open the result in Datasette Lite and execute a SQL query.

Adding plugins to Datasette Lite

One of Datasette’s key features is support for plugins. There are over 90 listed in the plugin directory now, with more emerging all the time. They’re a fantastic way to explore new feature ideas and extend the software to handle non-default use cases.

Plugins are Python packages, published to PyPI. You can add them to Datasette Lite using the new ?install=name-of-plugin query string parameter.

Here’s an example URL that loads the datasette-jellyfish plugin, which adds new SQL functions for calculating distances between strings, then executes a SQL query that demonstrates that plugin:

https://lite.datasette.io/?install=datasette-jellyfish#/fixtures?sql=SELECT%0A++++levenshtein_distance%28%3As1%2C+%3As2%29%2C%0A++++damerau_levenshtein_distance%28%3As1%2C+%3As2%29%2C%0A++++hamming_distance%28%3As1%2C+%3As2%29%2C%0A++++jaro_similarity%28%3As1%2C+%3As2%29%2C%0A++++jaro_winkler_similarity%28%3As1%2C+%3As2%29%2C%0A++++match_rating_comparison%28%3As1%2C+%3As2%29%3B&s1=barrack+obama&s2=barrack+h+obama

That URL uses ?install=datasette-jellyfish to install the plugin, then executes the following SQL query:

SELECT
    levenshtein_distance(:s1, :s2),
    damerau_levenshtein_distance(:s1, :s2),
    hamming_distance(:s1, :s2),
    jaro_similarity(:s1, :s2),
    jaro_winkler_similarity(:s1, :s2),
    match_rating_comparison(:s1, :s2);

It sets s1 to "barack obama" and s2 to "barrack h obama".

Plugin compatibility

Unfortunately, many existing Datasette plugins aren’t yet compatible with Datasette Lite. Most importantly, visualization plugins such as datasette-cluster-map and datasette-vega don’t work.

This is because I haven’t yet solved the challenge of loading additional JavaScript and CSS into Datasette Lite—see issue #8.

Here’s the full list of plugins that I’ve confirmed work with Datasette Lite so far:

• datasette-packages—Show a list of currently installed Python packages—demo
• datasette-dateutil—dateutil functions for Datasette—demo
• datasette-schema-versions—Datasette plugin that shows the schema version of every attached database—demo
• datasette-debug-asgi—Datasette plugin for dumping out the ASGI scope.—demo
• datasette-query-links—Turn SELECT queries returned by a query into links to execute them—demo
• datasette-json-html—Datasette plugin for rendering HTML based on JSON values—demo
• datasette-haversine—Datasette plugin that adds a custom SQL function for haversine distances—demo
• datasette-jellyfish—Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the Jellyfish Python library—demo
• datasette-pretty-json—Datasette plugin that pretty-prints any column values that are valid JSON objects or arrays.—demo
• datasette-yaml—Export Datasette records as YAML—demo
• datasette-copyable—Datasette plugin for outputting tables in formats suitable for copy and paste—demo

How it works

The implementation is pretty simple—it can be seen in this commit. The short version is that ?install= options are passed through to the Python web worker that powers Datasette Lite, which then runs the following:

for install_url in install_urls:
    await micropip.install(install_url)

micropip is a component of Pyodide which knows how to install pure Python wheels directly from PyPI into the browser’s emulated Python environment. If you open up the browser devtools networking panel you can see that in action!

Since the ?install= parameter is being passed directly to micropip.install() you don’t even need to provide names of packages hosted on PyPI—you could instead provide the URL to a wheel file that you’re hosting elsewhere.

This means you can use ?install= as a code injection attack—you can install any Python code you want into the environent. I think that’s fine—the only person who will be affected by this is the user who is viewing the page, and the lite.datasette.io domain deliberately doesn’t have any cookies set that could cause problems if someone were to steal them in some way.