Weeknotes: Still pretty distracted

17th January 2021

Not a lot to report this week. U.S. politics continues to make it extremely difficult to focus on anything substantial.

Ongoing Datasette projects

I have a lot of balls in the air regarding Datasette core at the moment. I’m mainly thinking rather than coding, but I feel like I’ve made some good progress that should turn into a flurry of code relatively soon.

Some of the areas I’ve been thinking about this week:

Ability for plugins to collaborate when adding extra HTML to blocks in default templates.

Plugins like datasette-cluster-map add content to the top of the table page (example). I’m increasingly seeing a need for plugins to do this in a way that respects potential additions by other plugins.

I want to add hooks that let plugins cleanly add content at the top or bottom of the table, database, query, row and index pages in a way that won’t over-write additions from other plugins.
Take advantage of sqlite-utils cached table counts, if available. I added a cached table count mechanism in sqlite-utils 3.2, now I’d like to take advantage of it to speed up Datasette.
Datasette Enrichments. This is still purely in the idea phase but I’m really excited about it. I want to provide a plugin-driven mechanism for Datasette which allows users to select a column and apply an “enrichment” against that column which will expand it with additional data.

A classic example is geocoding: imagine being able to select an “address” column, hit “geocode” and have extra latitude/longitude columns added to the table and populated using a geocoder. If it takes ten hours that’s fine—come back ten hours later.

Now imagine if it was trivial to write additional enrichments using a simple plugin hook—OCR, entity extraction and so on.
Rewriting the Datasette TableView. The table page is by far the most complex component of Datasette, supporting faceting and filtering and pagination and various other features. The code is pretty messy. I’d like to clean it up.

Meanwhile for Datasette 1.0 I’m planning on introducing a mechanism for users of the .json API to request extras for things like suggested facets. And I’m looking to improve the plugin hook used for custom output formats like .yaml.

I’m beginning to suspect that the solution to all of these issues are related.

<script type="module"> for Datasette plugins

One change I did complete this week is adding support for <script type="module"> JavaScript scripts in Datasette’s plugin system.

This kind of script tag is required if you want to use JavaScript modules directly in modern browsers, without needing to run your code through a pre-processor.

Datasette has a mechanism for adding additional scripts to a page which can be configured both by plugins and in the metadata.json configuration file. I’ve updated both of those to support a "module": true key which adds the type="module" attribute to the script tag. See Custom CSS and JavaScript and extra_js_urls() in the documentation for details. Short version: add this to metadata.json:

{
    "extra_js_urls": [
        {
            "url": "https://example.datasette.io/module.js",
            "module": true
        }
    ]
}

To add the following line to every page of your Datasette instance:

<script type="module" src="https://example.datasette.io/module.js"></script>

I also modified the extra_body_script() plugin hook to allow scripts to use module syntax:

@hookimpl
def extra_body_script():
    return {
        "module": True,
        "script": "console.log('Your JavaScript goes here...')"
    }

Which will add the following to the bottom of the <body> element:

<script type="module">console.log('Your JavaScript goes here...')</script>

Releases this week

markdown-to-sqlite: 0.2—2021-01-14
CLI tool for loading markdown files into a SQLite database
sqlite-utils: 3.2.1—2021-01-12
Python CLI utility and library for manipulating SQLite databases
datasette-export-notebook: 0.2—2021-01-11
Datasette plugin providing instructions for exporting data to Jupyter or Observable

TIL this week

Posted 17th January 2021 at 5 am · Follow me on Mastodon or Twitter or subscribe to my newsletter

Simon Willison’s Weblog