Simon Willison's Weblog: datasette

datasette-ports 0.4

2026-10-12T04:57:07+00:00

My plugin for detecting and listing all Datasette instances on the current machine now works with the latest version, and displays the detected PID as well. Knowing the PID makes it easier to attach the new Python 3.15 profiler.

Tags: datasette

Using Parseable with Datasette for OpenTelemetry traces

2026-10-06T19:07:31+00:00

TIL: Using Parseable with Datasette for OpenTelemetry traces

I saw Parseable in a Show HN today - it's a new observability platform with both an open source (AGPL) Rust implementation (a single ~180MB binary), an "Enterprise" version with extra features and a cloud hosted option.

Since Datasette 1.0a41 added OpenTelemetry support (thanks, Alex Garcia), I decided to fire up Codex and have it figure out how to run Parseable and feed it traces from Datasette.

Here's my (human-written) TIL showing the patterns that worked, and here's a screenshot of a Datasette trace displayed within the Parseable localhost web application:

Tags: datasette, observability, alex-garcia, opentelemetry

datasette-atom 0.11a0

2026-10-06T17:35:39+00:00

Release: datasette-atom 0.11a0

A minor fix for compatibility with the latest Datasette alphas. This meant we could upgrade the datasette.io site to Datasette 1.0a41.

Tags: atom, datasette

datasette 1.0a41

2026-09-24T19:15:23+00:00

Release: datasette 1.0a41

Alec Garcia added support for OpenTelemetry to Datasette in this release.

I've also refactored all of Datasette's modal dialogs to a single Web Component, which is now documented for other plugins to use.

Tags: javascript, datasette, web-components, alex-garcia, opentelemetry

datasette-explain 0.2.2

2026-09-20T00:22:04+00:00

Release: datasette-explain 0.2.2

Explain plans now work on read-only stored-query pages.

I upgraded datasette.simonwillison.net to Datasette 1.0a40, which inspired me to ship a new version of this explain plugin.

Tags: sqlite, datasette

datasette-auth-github 1.0

2026-09-19T19:52:02+00:00

Release: datasette-auth-github 1.0

I run this GitHub login plugin on the agent.datasette.io demo site and I noticed that my authenticated sessions weren't lasting very long. It turned out that the plugin was setting cookies without a Max-Age parameter, so they were expiring at the end of a browser session (which in Mobile Safari seems to happen pretty often, independently of how you are using the app.)

I fixed that in #80 and, since this plugin has been around for quite a while and is tested against both Datasette 0.65.x and Datasette 1.0ax, I decided to bump it up to a 1.0 release. I'm trying to get better at promoting stable plugins to 1.0.

Tags: github, plugins, datasette

datasette 1.0a40

2026-09-16T23:51:43+00:00

Release: datasette 1.0a40

Same security fix as 0.65.5, plus some neat new features and bug fixes:

Plugins can now launch and manage background tasks using the new datasette.add_background_task() method. Thanks, Alex Garcia.
I've migrated Datasette to httpx2 for features like the internal datasette.client.get() method.
A whole lot of bug fixes, many of them stemming from a recent effort to triage issues for a 1.0 stable release.

Tags: security, datasette

datasette 0.65.5

2026-09-16T23:51:08+00:00

Release: datasette 0.65.5

Security fix for an issue where a trailing newline in a requested table name could bypass table permissions and expose private rows, reported by dpfkdlemtp in GHSA-h547-rmjf-5m2m.

Tags: security, datasette

Datasette 1.0a39 and 0.65.4 security releases

2026-09-11T03:27:16+00:00

Datasette 1.0a39 and 0.65.4 security releases

Today we're releasing two new security patch versions of Datasette: 1.0a39 and 0.65.4 - one for the current alpha series and one for the stable 0.65.x family.

These are security fixes which you should apply if you are running a Datasette instance on the public web - in particular if that instance mixes both public and private tables.

Following issues reported by Sevban Dönmez, Alex Garcia and I ran an extensive audit of Datasette using Claude Fable 5.1, GPT-5.6, and GPT-6 Astra. We then spent almost a week collaborating on and reviewing the fixes.

They helped find some very subtle bugs. We'll be incorporating security audits by frontier models into all of our development work going forward.

Alex came up with a way of splitting the work which I found extremely productive:

Alex Garcia and I worked together running and then responding to the audit, working in a shared private repository. For most of the issues we split the work: one of us would create the automated tests highlighting the issue, then the other would implement the fix. This ensured that two separate humans had eyes on each of the issues, in addition to our coding agents running different models.

Tags: releases, security, ai, datasette, generative-ai, llms, agentic-engineering, ai-security-research

datasette-publish-fly 1.4

2026-09-11T02:58:52+00:00

Release: datasette-publish-fly 1.4

Sets force_https=true in fly.toml. #31

Fix for Volume could not be found bug. #32

Compatible with app-scoped deploy tokens. #34

Tags: datasette, fly

datasette 0.65.4

2026-09-11T00:06:55+00:00

Release: datasette 0.65.4

See Datasette 1.0a39 and 0.65.4 security releases on the Datasette blog.

Tags: security, datasette

datasette 1.0a39

2026-09-11T00:05:54+00:00

Release: datasette 1.0a39

See Datasette 1.0a39 and 0.65.4 security releases on the Datasette blog.

Tags: security, datasette

Creepy crawlies

2026-09-07T23:08:58+00:00

Creepy crawlies

Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of git.kernel.org, the official Git repository for the Linux kernel:

TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.

I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages.

Via Hacker News

Tags: crawling, git, linux, datasette, ai-ethics

datasette-mcp 0.2

2026-09-01T15:30:12+00:00

Release: datasette-mcp 0.2

"rows" from execute_sql is now an array of objects. Previously it was an array of arrays. This should help weaker models avoid losing track of which positional array element maps to which column. #1

Now depends on mcp>=2.1.1.

This is the first non-alpha release of the plugin. I'm confident it's ready as I've been using it quite a bit myself.

Tags: datasette, model-context-protocol

datasette-upload-dbs 0.5a0

2026-08-11T20:35:13+00:00

Release: datasette-upload-dbs 0.5a0

This plugin has been around for a while - it lets users upload a brand new SQLite database to a hosted Datasette instance, at which point that database will start being served by that instance.

It can also be used to atomically swap a database with a more recent version. The uploaded database is saved to a file, verified, then swapped in so /name starts serving the new one.

The new release adds a formalized API, so you can replace an existing database (or add a new one) like this:

curl -X POST \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Accept: application/json" \
  -F "db=@content.db" \
  -F "db_name=content" \
  https://your-instance.example.com/-/upload-dbs

This means you can build fresh databases in an environment such as GitHub Actions and swap them in production as soon as that build has completed.

Tags: datasette

datasette-auth-tokens 0.4a13

2026-08-06T23:07:47+00:00

Release: datasette-auth-tokens 0.4a13

Upgraded for compatibility with `sqlite-utils 4.

Tags: datasette

datasette 1.0a38

2026-08-06T18:24:34+00:00

Release: datasette 1.0a38

This release fixes a SQL injection security issue that affects Datasette instances that serve a mixture of public and private tables in the same database, with access configured using the Datasette permissions system.

Site administrators who serve private tables in this way are advised to disable the execute-sql permission on that database to prevent users from accessing private tables using raw SQL queries. The bug that has been fixed would have allowed users with access to any public table to execute SQL injection attacks despite that restriction, giving them read-only access to data in private tables in the same database.

This fix is also available in Datasette 0.65.3.

Thankfully this particular configuration - private tables and public tables exposed for the same database within the same instance - is likely to be rare. I've not encountered an instance like that myself.

Tags: security, sql-injection, datasette

datasette 0.65.3

2026-08-06T18:22:07+00:00

Release: datasette 0.65.3

Back-ported the SQL Injection security fix from 1.0a38.

Tags: datasette

datasette-apps 0.2a0

2026-08-01T21:23:56+00:00

Release: datasette-apps 0.2a0

Changes that improve Datasette Apps when created and edited using Datasette Agent:

New app_debug() tool allowing agent to open an app (invisibly) and test it using JavaScript. #33

New app_list() tool for listing apps the user has permission to edit, so the agent can edit them. #36

The app_debug() tool is pretty neat: it works by displaying the app in a opacity: 0 iframe with pointer-events: none (so it can't be seen or interacted with) and then executing agent-provided JavaScript inside that sandboxed iframe. This means the agent can smoke test that the app is working and even do things like measure the dimensions of different elements.

This uses the new context.browser_task() mechanism added in datasette-agent 0.4a0.

Tags: iframes, datasette, datasette-apps

Stateless MCP has recaptured my interest (and inspired mcp-explorer and datasette-mcp)

2026-07-31T23:13:22+00:00

Tuesday was Stateless MCP day - the rollout of MCP 2.0, or the 2026-07-28 Model Context Protocol specification to use the more formal but less memorable name. This is the most significant change to the MCP spec since it first launched, and has also served to reignite my personal interest in the protocol.

For background: MCP is the Model Context Protocol, which describes a standard way to expose new tools to LLM-powered agent frameworks. It was introduced by Anthropic back in November 2024, had a huge spike of interest through much of 2025, and then became somewhat eclipsed by Skills (another Anthropic invention) when it became apparent that an agent harness with access to a terminal and curl could do most of what MCP did in a more flexible way. I wrote about that in my review of 2025.

I'm coming back around to MCP now. Giving an agent a shell environment with the ability to access the internet is fraught with risk, and requires a strong model that is capable of effectively driving such an environment. MCP tools are easier to audit and control, and simple enough that smaller models that run on a laptop can still drive them reasonably well.

The new stateless MCP specification also greatly decreases the complexity of implementing both clients and servers for the protocol. I built three of those this week!

What's easier with stateless MCP

The best demonstration of the difference between stateful and stateless MCP is in this May 21st blog post that introduced the RC for the new specification. It included a clear before-and-after example.

The older stateful MCP (I'm going to call it "legacy MCP") required two HTTP requests - the first to initialize a session and obtain a Mcp-Session-Id, and the second to actually call the tool:

POST /mcp HTTP/1.1
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "initialize",
  "params": {
    "protocolVersion": "2025-11-25",
    "capabilities": {
    },
    "clientInfo": {
      "name": "my-app",
      "version": "1.0"
    }
  }
}

POST /mcp HTTP/1.1
Mcp-Session-Id: 1868a90c-3a3f-4f5b
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "search",
    "arguments": {
      "q": "otters"
    }
  }
}

The new stateless way uses a single HTTP request which looks like this:

POST /mcp HTTP/1.1
MCP-Protocol-Version: 2026-07-28
Mcp-Method: tools/call
Mcp-Name: search
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "search",
    "arguments": {
      "q": "otters"
    },
    "_meta": {
      "io.modelcontextprotocol/clientInfo": {
        "name": "my-app",
        "version": "1.0"
      }
    }
  }
}

This is so much cleaner from both a client- and server-side implementation perspective. It's also a better fit for building scalable web applications, since now you don't need to maintain server-side state to keep track of those session IDs, or worry about routing the same session to the same backend machine.

mcp-explorer

I couldn't find a great CLI tool for interactively probing an MCP server, so I had Codex help build my own.

mcp-explorer is the result. It's a stateless Python CLI tool, so you don't even need to install it to try it out - it works with uvx like this:

uvx mcp-explorer list https://agentic-mermaid.dev/mcp

This queries Ade Oshineye's agentic-mermaid.dev demo MCP. The above command returns the following list of tools:

execute(code: string, timeoutMs?: integer) - Execute Mermaid SDK code
  Run JavaScript in an isolated sandbox; return a value.

describe_sdk(family: string, detail?: string) - Describe Mermaid SDK operations
  Return version-matched mutation operations for one diagram family.

render_svg(source: string, options?: object) - Render Mermaid as SVG
  Render a Mermaid source string to themeable SVG. Returns { ok, svg }.

render_ascii(source: string, useAscii?: boolean, targetWidth?: integer, options?: object) - Render Mermaid as text
  Render a Mermaid source string to text. Returns { ok, text }.

render_png(source: string, scale?: number, background?: string, fitTo?: object, options?: object) - Render Mermaid as PNG
  Rasterize a Mermaid source string to PNG. Returns { ok, png_base64 }.
...

Then to inspect a tool:

uvx mcp-explorer inspect render_svg

This outputs a whole bunch of information, including the JSON schema of the inputs and outputs.

To call that tool and pass arguments to it:

uvx mcp-explorer call \
  https://agentic-mermaid.dev/mcp \
  render_svg \
  -a source 'graph TD; A-->B' \
  -a options '{"padding":24}'

Which returns:

{"ok":true,"svg":"<svg xmlns=\"http://www.w3.org/2000/svg\" width=...

To get just the raw SVG try adding | jq .svg -r to that command. I got back this image:

There are a few more commands in the README, but you get the general idea. I find building CLI tools like this to be a really productive way to get familiar with a specification, even if an agent writes most of the actual code.

datasette-mcp

The second project is datasette-mcp, a Datasette plugin which adds a /-/mcp endpoint to any Datasette instance.

This is probably the fourth time I've tried building this plugin, but thanks to the new stateless MCP specification I finally have a version that feels good to release.

It provides just three tools: list_databases(), get_database_schema(database_name), and execute_sql(database_name, sql). They do exactly what you would expect them to do - though execute_sql() is read-only for the moment.

Wire these into an agent, or a chat tool like ChatGPT or Claude, and they'll gain the ability to run SQL queries against your hosted Datasette instance.

So far I'm running it on the Datasette mirror of my blog, at datasette.simonwillison.net/-/mcp. It took a bit of fiddling to figure out how to attach that to ChatGPT and Claude, but I got there in the end. Here's a new TIL showing exactly how to do that.

Here's a shared Claude session where I asked it:

list tables in simonwillison.net

And then:

what has Simon said recently about MCP?

It ran 7 separate SQL queries to figure out the answer.

llm-mcp-client

My LLM tool is long overdue for an official MCP integration. The new alpha llm-mcp-client plugin is my attempt at exactly that:

llm install llm-mcp-client
llm -T 'MCP("https://datasette.simonwillison.net/-/mcp")' 'count the notes'

Here's the output (including reasoning trace, I'm using LLM 0.32rc2):

Considering note count

I see the question "count the notes" is probably asking me to tally up blog notes. It could also mean published notes or drafts, so there's some ambiguity there. I'll need to figure out the total number of notes, likely by querying the count for both published notes and drafts to get a clear answer. Let's execute that count!

There are 151 notes.

And the output of llm logs for that prompt.

Once this is fully baked, I'm considering bringing it directly into LLM core. I'm excited to experiment with MCP in Datasette Agent and llm-coding-agent as well.

MCP is a safer way to build with agents

A few months after MCP was first released, I wrote Model Context Protocol has prompt injection security problems, where I noted that the pattern of having end users mix and match tools pushed responsibility for avoiding data exfiltration attacks out to the users themselves. I hadn't coined the Lethal Trifecta yet, but that was absolutely what I had in mind.

Then general agents with arbitrary shell and curl access came along, and that's so much harder to keep secure!

Something I've come to appreciate about MCP is that it's much easier to reason about agent capabilities and what might go wrong than with arbitrary command execution in an open network environment - the default for most of today's general and coding agent tools.

I plan to lean into MCP a whole lot more when I'm building sensitive applications on top of LLMs.

Tags: projects, ai, datasette, mermaid, generative-ai, llms, llm, anthropic, model-context-protocol

datasette-agent 0.4a0

2026-07-31T14:14:23+00:00

Release: datasette-agent 0.4a0

New await context.browser_task() mechanism allowing agent tools to run code directly in the user's browser. #33

This is an exciting new capability: it makes it easy for Datasette Agent plugins to provide tools that execute custom JavaScript in the user's browser.

I used this to add a debug loop to Datasette Apps in datasette-apps 0.2a0.

Tags: datasette, llm-tool-use, datasette-agent

datasette-agent-modeler 0.1a0

2026-07-29T02:47:05+00:00

Release: datasette-agent-modeler 0.1a0

Bit of a wild vibe coding experiment with Claude Fable 5. I told it to "Build a plugin for datasette agent which adds a tool for creating a 3D model and then making edits to that model" (full prompt here) and... it did.

Tags: datasette, vibe-coding, datasette-agent, claude-mythos-fable

datasette-apps 0.1a4

2026-07-14T23:44:56+00:00

Release: datasette-apps 0.1a4

Tags: datasette, datasette-apps

datasette 1.0a37

2026-07-14T16:31:41+00:00

Release: datasette 1.0a37

A minor release. Performance and documentation improvements to the permissions system, plus I reverted a cosmetic API change which caused almost every existing plugin test suite to break.

Tags: datasette

DOOMQL

2026-07-13T22:34:41+00:00

DOOMQL

Peter Gostev built this using GPT-5.6 Sol. This is a lot of fun:

DOOMQL started with a deliberately unreasonable question: what if SQLite were the game engine, not merely the place where a game stores data?

The result is a small, original Doom-like game in which SQL owns movement, collision, enemies, combat, progression and every RGB pixel on screen.

It's implemented as a Python terminal script - I tried it out like this:

cd /tmp
git clone https://github.com/petergpt/doomql
cd doomql
uv run host/doomql.py

Here's the huge SQL query that implements a full ray tracer in SQLite using a recursive CTE.

Running the above script creates a /tmp/doomql/.doomql/doomql.sqlite SQLite database, which you can explore using Datasette like this:

uvx --prerelease=allow  --with datasette-apps datasette \
  /tmp/doomql/.doomql/doomql.sqlite \
  -p 4444 --root --secret 1 --internal internal.db

The --with datasette-apps option installs the new Datasette Apps plugin, which supports creating custom HTML+JavaScript apps that can run SQL queries directly within the Datasette interface.

I created a new app, pasted the copy-paste prompt into Claude chat (Fable 5) and told it:

Build an app that displays the current state of the screen using the frame_pixels view with its x, y, r, g, b columns. have it refresh once a second.

This got me a working HTML+JavaScript app inside Datasette that could reflect the current state while I played the game in my terminal. Then I added:

add a minimap

And now my Datasette App looks like this:

Here's the HTML app code - paste that into your own Datasette instance (using the uvx --with datasette-apps recipe from above) to try it yourself.

Via @petergostev

Tags: games, sql, sqlite, ai, datasette, generative-ai, llms, ai-assisted-programming, gpt, datasette-apps

datasette code-frequency chart on GitHub

2026-07-13T21:45:27+00:00

datasette code-frequency chart on GitHub

Out of curiosity I decided to see if I could find a useful illustration of the impact of coding agents and Opus 4.5 class models on my own output. The best I've found so far is this GitHub chart of frequency of code changes to my Datasette open source project:

The big spike in activity at the end aligns with Opus 4.8, GPT-5.5, Fable 5 and GPT-5.6 Sol.

Tags: github, ai, datasette, generative-ai, llms, ai-assisted-programming, coding-agents

datasette 1.0a36

2026-07-07T21:43:27+00:00

Release: datasette 1.0a36

The signature features of this alpha are new UIs for inserting multiple rows at once (from TSV, CSV or JSON) and for creating a table from rows, plus a large number of small JSON API consistency fixes in preparation for a 1.0 stable release.

Tags: datasette

Using DSPy to evaluate and improve Datasette Agent's SQL system prompts

2026-07-02T18:25:00+00:00

Research: Using DSPy to evaluate and improve Datasette Agent's SQL system prompts

One of this morning's AIE keynotes covered dspy, which reminded me I've been meaning to see if it could help me improve the system prompt used by Datasette Agent - so I fired off an asynchronous research task in Claude Code for web using Claude Fable 5:

Pip install the latest Datasette alpha and datasette-agent and dspy - then figure out how to use dspy to evaluate and improve the main system prompts used by Datasette Agent for the feature where it can execute read only SQL queries to answer user questions about data.

Fable chose to test using GPT 4.1 mini and nano, and identified several promising looking directions for improvements. I particularly like this one:

The schema listing gives only table names; the "don't call describe_table if you already have the information" advice caused column-name guessing (page_count, o.order_id, first_name) and error-retry loops in baseline traces. Either include column names in the prompt's schema listing or soften that advice.

Tags: ai, datasette, generative-ai, llms, evals, dspy, datasette-agent, claude-mythos-fable

Have your agent record video demos of its work with shot-scraper video

2026-06-30T16:54:26+00:00

shot-scraper video is a new command introduced in today's shot-scraper 1.10 release which accepts a storyboard.yml file defining a routine to run against a web application and uses Playwright to record a video of that routine. I've written before about the importance of having coding agents produce demos of their work; this is my latest attempt at enabling them to do that.

Here's an example video created using shot-scraper video, exercising a still in development feature adding the ability to create new tables in Datasette from pasted CSV, TSV or JSON data:

That video was created by running this command:

shot-scraper video datasette-bulk-insert-storyboard.yml \
  --auth datasette-demo-auth.json --mp4

(That --auth JSON file contains a cookie, as described here in the documentation.)

Here's the datasette-bulk-insert-storyboard.yml file:

output: /tmp/datasette-bulk-insert-demo.webm
server:
  - uv
  - --directory
  - /Users/simon/Dropbox/dev/datasette
  - run
  - datasette
  - -p
  - 6419
  - --root
  - --secret
  - "1"
  - /tmp/demo.db
url: http://127.0.0.1:6419/demo/tasks
viewport:
  width: 1280
  height: 720
cursor: true
wait_for: 'button[data-table-action="insert-row"]'
javascript: |
  (() => {
    let clipboardText = "";
    Object.defineProperty(navigator, "clipboard", {
      configurable: true,
      get: () => ({
        writeText: async (text) => {
          clipboardText = String(text);
        },
        readText: async () => clipboardText,
      }),
    });
  })();
scenes:
  - name: Bulk insert existing table rows
    do:
      - pause: 0.8
      - click: 'button[data-table-action="insert-row"]'
      - wait_for: "#row-edit-dialog[open]"
      - pause: 0.5
      - click: ".row-edit-bulk-insert"
      - wait_for: ".row-edit-bulk-textarea"
      - pause: 0.5
      - click: ".row-edit-copy-template"
      - wait_for: "text=Copied"
      - pause: 0.8
      - fill:
          into: ".row-edit-bulk-textarea"
          text: |
            title,owner,status,priority,notes
            Prepare release video,Ana,doing,1,Recorded with shot-scraper
            Check pasted CSV import,Ben,review,3,Previewed before inserting
            Share the branch demo,Chen,queued,2,Bulk insert creates three rows
      - pause: 0.8
      - click: ".row-edit-save"
      - wait_for: "text=Previewing 3 rows."
      - pause: 1.2
      - click: ".row-edit-save"
      - wait_for: "text=3 rows inserted."
      - pause: 1.0
      - click: ".row-edit-cancel"
      - wait_for: "text=Prepare release video"
      - pause: 1.0
  - name: Create a table from pasted CSV
    open: http://127.0.0.1:6419/demo
    wait_for: 'details.actions-menu-links summary'
    do:
      - pause: 0.8
      - click: 'details.actions-menu-links summary'
      - click: 'button[data-database-action="create-table"]'
      - wait_for: "#table-create-dialog[open]"
      - pause: 0.5
      - fill:
          into: ".table-create-table-name"
          text: "launch_metrics"
      - click: ".table-create-from-data"
      - wait_for: ".table-create-data-textarea"
      - pause: 0.5
      - fill:
          into: ".table-create-data-textarea"
          text: |
            metric_id,name,score,recorded_on
            m001,Activation rate,87.5,2026-06-29
            m002,Retention check,72.25,2026-06-30
            m003,CSV import health,95,2026-07-01
      - pause: 0.8
      - click: ".table-create-save"
      - wait_for: "text=Previewing 3 rows."
      - pause: 1.2
      - click: ".table-create-save"
      - wait_for_url: "**/demo/launch_metrics"
      - wait_for: "text=Activation rate"
      - pause: 1.2

The video command documentation includes simpler examples, but for the purpose of this post I thought I'd go with something more comprehensive.

That demo YAML storyboard was constructed entirely by GPT-5.5 xhigh running in Codex Desktop, using the following prompt run inside my ~/dev/datasette checkout of this branch:

Review the changes on this branch.

cd to ~/dev/shot-scraper and run the command "uv run shot-scraper video --help"

Now use that new video command to record a video demo of the new features from this branch, including running a "uv run datasette -p 6419 --root --secret 1 /tmp/demo.db" development server so you can record the video against a demo DB that you first create.

Now that I've released the feature the prompt could say "run uvx shot-scraper video --help" instead and it should achieve the same result.

I really like this pattern where the --help output for a command provides enough detail that a coding agent can use it - it works kind of like bundling a SKILL.md file directly inside the tool. I used the same pattern for showboat and rodney.

How I built this

shot-scraper video started as an experimental prototype. shot-scraper is built on top of Playwright, and the key feature it needed was for Playwright to be able to record video of browser sessions with enough control to create the desired demo.

I first tried this a few years ago and found that the Playwright-produced videos included additional chrome that was useful for debugging a test failure but unwanted for a product demo.

They fixed that a while ago, but there were still some minor blockers. In particular I was getting a few white frames at the start of the videos, since the recording mechanism kicked in before the first URL was loaded by the browser.

Playwright 1.59 added a new screencast mechanism providing much more finely grained control over video recording. This was very nearly what I needed, but the resulting videos were fixed at 800px wide.

I found a landed PR fixing that but it wasn't yet in a release. Then yesterday they shipped it in playwright-python 1.61.0 and I was finally unblocked to finish implementing the feature!

The code itself was all written by GPT-5.5 xhigh in Codex Desktop. I had it write the documentation as well which gave me a very useful frame for reviewing the design - much of the iteration on the feature came from reviewing that documentation, spotting things that were redundant, inconsistent or confusing, and requesting (or dictating) a better design.

The YAML format itself was mostly defined by the coding agent. I had it use Pydantic to both define and validate the format, partly to make the design easier to review.

This is a great example of the kind of feature that I almost certainly wouldn't have taken on without coding agent support. I filed the original issue in February 2024, and had difficulty finding the necessary time to solve this in amongst all of my other projects.

Tags: projects, python, yaml, ai, datasette, playwright, shot-scraper, generative-ai, llms, pydantic, coding-agents, agentic-engineering

datasette-export-database 0.3a2

2026-06-25T17:21:09+00:00

Release: datasette-export-database 0.3a2

An embarrassingly tiny release. The pyproject.toml had pinned to datasette==1.0a27, inadvertently making this plugin incompatible with all other Datasette versions. It's now datasette>=1.0a27 instead.

Tags: datasette