Simon Willison’s Weblog

Newsletter
Atom feed for datasette Random

1,557 posts tagged “datasette”

Datasette is an open source tool for exploring and publishing data.

2026

Hide your slop PR descriptions behind <details><summary>

Visit Hide your slop PR descriptions behind <details><summary>

Most of the changes I make to my projects these days come in the form of PRs opened for me by Claude Code or Codex. The PR is my unit of work—I use them to bundle commits together, iterating on them until a feature is complete, document decisions and then merge the finished result. Today’s coding agents are great at writing PR descriptions.

[... 987 words]

datasette #3046: Don’t serve link: rel=next HTTP headers to Firefox for HTML pages. I was looking at the Firefox network panel today when I noticed that requests to a Datasette table page were triggering an HTTP prefetch to the next page of rows in that table.

This was happening because Datasette pages serve an HTTP link header like this:

link: <https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv>; rel="next"

This is bad. A tiny fraction of the visitors to one of those pages will click the "next" link, which means for everyone else their browser is fetching unnecessary data and the Datasette server is having to render an entire page that then gets thrown away.

I dug in and learned that Firefox has an issue that has been open for 24 years about this behavior...

And one of the early comments on that issue links to my blog post from 2002 complaining about this feature. I hadn't just forgotten that I had written about it, I had forgotten the feature entirely!

# 7th November 2026, 11:57 pm / firefox, html, mozilla, datasette

Fixes a bug that affected my agent.datasette.io demo site, where the set of available LLM models was not correctly restricted to just those in the configuration.

Datasette Apps adds support for hosting and editing custom HTML applications inside of Datasette.

These can be created by hand, but it's easier to create them using LLMs via Datasette Agent. This release improves that integration:

  • Apps can now be tagged in Datasette Agent chat messages by typing #, which autocompletes from the apps the user has permission to view.
  • New app_add_stored_queries() and app_remove_stored_queries() tools let the agent manage which stored queries an app is allowed to call. Granting a write query requires approval from the user first.
  • app_create() now checks that each requested stored query exists and that the user can view it, and asks the user to approve any write queries before the app is created.
  • The "View app" buttons shown by the agent tools, and the links on # app references, now open the app full screen in the Datasette Agent panel.

This is a significant release of Datasette Agent. The most notable new feature is called the panel - it lets the agent chat open and then sit next to pages from Datasette, for example a table or query page.

When combined with datasette-apps this means you can prompt a database-backed app into existence and then modify it from the chat that sits next to that app.

Screenshot of a split-screen chat interface titled "evaluate llm prices" (model tag "gpt-6-luna"): the conversation runs in a narrow column on the left, and a wide side panel on the right displays the app the assistant built, with "EXPORT" and "Close panel" buttons above it. In the chat, blue user bubbles read "Build an app to help explore this data" and "give it a more hacker aesthetic design", each followed by collapsed tool calls (app_create, app_render, app_view, app_str_replace), "LLM Price Explorer v1." and "v2." entries with "View app" and "Edit" buttons, and short assistant replies describing the app and its new dark terminal-inspired theme. The right panel shows the resulting "LLM Price Explorer" in neon green monospace on black: stat cards for 185 models from 12 providers with workload costs from $0.042 (Jev 1.13) to $300.00 (o1 Pro), filter and workload cost estimate controls, and a table of matching models with input, output and cached input prices per million tokens.

You can try it by signing in with your GitHub account to agent.datasette.io.

I have some big upgrades to Datasette Agent in the pipeline, but in the meantime here's a tiny user experience improvement to the underlying datasette-llm plugin.

It now picks up the default model used by LLM, which means you can start using Datasette Agent without needing to explicitly configure a default model:

uvx --with datasette-agent datasette content.db --internal internal.db

Previously you would also need to specify a model with -s plugins.datasette-agent.default_model gpt-6-astra.

When I switched LLM over to httpx2 in LLM 0.33 I forget that this would break any plugins which used httpx without directly depending on it. llm-tools-datasette was one of those plugins, now fixed.

My plugin for detecting and listing all Datasette instances on the current machine now works with the latest version, and displays the detected PID as well. Knowing the PID makes it easier to attach the new Python 3.15 profiler.

None

I saw Parseable in a Show HN today - it's a new observability platform with both an open source (AGPL) Rust implementation (a single ~180MB binary), an "Enterprise" version with extra features and a cloud hosted option.

Since Datasette 1.0a41 added OpenTelemetry support (thanks, Alex Garcia), I decided to fire up Codex and have it figure out how to run Parseable and feed it traces from Datasette.

Here's my (human-written) TIL showing the patterns that worked, and here's a screenshot of a Datasette trace displayed within the Parseable localhost web application:

Screenshot of a trace detail view in an observability web app, with a span waterfall overlaid on a dimmed navigation sidebar and filter column. Dimmed sidebar: breadcrumb "Community > Traces > datas" (cut off), search box "Search... ⌘K", nav items "Home", "Ingest telemetry", section "ANALYZE": "Keystone", "Dashboards", "SQL Editor", section "OBSERVE": "Logs", "Metrics", "Traces" (selected), "APM", "Agents", section "MONITOR": "Alerts", "Errors", section "DATA": "Datasets", and at the bottom "Settings", "Book a call", "Support". Dimmed filter column, cut off at the right edge: "Search fi", "Core", "Log format", "User agent", "Source IPs", "Error", "Service", "service.ins", "service.na", "datasett" (checked), "Span", "Database", "HTTP", "http.reque", "NULL" (unchecked), "GET" (checked), "http.respo", "http.route", "Server", "server.add", "Telemetry", "URL", "All fields". Trace panel header: "Trace detail > 6f819a2170bcd1e91c6ea3ae236ec60b" with a copy icon, a "Related logs" button and a close X. Summary: "Start time 6:56 PM, Oct 6, 2026 UTC", "Duration 40.9 ms", "Spans 247". A minimap with axis "0ns 10.2ms 20.5ms 30.7ms 40.9ms" shows many short span bars cascading diagonally from top left toward the lower middle, with a few longer bars. Below is a span table with a "Span name" header, a "Search spans..." box, collapse and expand buttons, and a timeline axis "0ns 10.2ms 20.5ms 30.7ms 40.9ms". Rows (name, service, duration): root span with collapse toggle "123", "GET /..." "datasette..." 40.9ms spanning the full timeline; then alternating rows where each "db.query" has a collapse toggle "1": db.query datasette-local 679µs, db.query.execute datasette-local 278µs, db.query datasette-local 341µs, db.query.execute datasette-local 55µs, db.query datasette-local 357µs, db.query.execute datasette-local 197µs, db.query datasette-local 330µs, db.query.execute datasette-local 73µs, db.query datasette-local 2.26ms, db.query.execute datasette-local 2.02ms, db.query datasette-local 232µs, db.query.execute datasette-local 64µs, db.query datasette-local 207µs, db.query.execute datasette-local 71µs, db.query datasette-local 6.11ms. The child span bars start progressively later across the early part of the timeline.

A minor fix for compatibility with the latest Datasette alphas. This meant we could upgrade the datasette.io site to Datasette 1.0a41.

Alec Garcia added support for OpenTelemetry to Datasette in this release.

I've also refactored all of Datasette's modal dialogs to a single Web Component, which is now documented for other plugins to use.

  • Explain plans now work on read-only stored-query pages.

I upgraded datasette.simonwillison.net to Datasette 1.0a40, which inspired me to ship a new version of this explain plugin.

I run this GitHub login plugin on the agent.datasette.io demo site and I noticed that my authenticated sessions weren't lasting very long. It turned out that the plugin was setting cookies without a Max-Age parameter, so they were expiring at the end of a browser session (which in Mobile Safari seems to happen pretty often, independently of how you are using the app.)

I fixed that in #80 and, since this plugin has been around for quite a while and is tested against both Datasette 0.65.x and Datasette 1.0ax, I decided to bump it up to a 1.0 release. I'm trying to get better at promoting stable plugins to 1.0.

Same security fix as 0.65.5, plus some neat new features and bug fixes:

  • Plugins can now launch and manage background tasks using the new datasette.add_background_task() method. Thanks, Alex Garcia.
  • I've migrated Datasette to httpx2 for features like the internal datasette.client.get() method.
  • A whole lot of bug fixes, many of them stemming from a recent effort to triage issues for a 1.0 stable release.

Security fix for an issue where a trailing newline in a requested table name could bypass table permissions and expose private rows, reported by dpfkdlemtp in GHSA-h547-rmjf-5m2m.

Datasette 1.0a39 and 0.65.4 security releases. Today we're releasing two new security patch versions of Datasette: 1.0a39 and 0.65.4 - one for the current alpha series and one for the stable 0.65.x family.

These are security fixes which you should apply if you are running a Datasette instance on the public web - in particular if that instance mixes both public and private tables.

Following issues reported by Sevban Dönmez, Alex Garcia and I ran an extensive audit of Datasette using Claude Fable 5.1, GPT-5.6, and GPT-6 Astra. We then spent almost a week collaborating on and reviewing the fixes.

They helped find some very subtle bugs. We'll be incorporating security audits by frontier models into all of our development work going forward.

Alex came up with a way of splitting the work which I found extremely productive:

Alex Garcia and I worked together running and then responding to the audit, working in a shared private repository. For most of the issues we split the work: one of us would create the automated tests highlighting the issue, then the other would implement the fix. This ensured that two separate humans had eyes on each of the issues, in addition to our coding agents running different models.

# 11th September 2026, 3:27 am / releases, security, ai, datasette, generative-ai, llms, agentic-engineering, ai-security-research

  • Sets force_https=true in fly.toml. #31
  • Fix for Volume could not be found bug. #32
  • Compatible with app-scoped deploy tokens. #34

Creepy crawlies (via) Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of git.kernel.org, the official Git repository for the Linux kernel:

TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.

I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages.

# 7th September 2026, 11:08 pm / crawling, git, linux, datasette, ai-ethics

  • "rows" from execute_sql is now an array of objects. Previously it was an array of arrays. This should help weaker models avoid losing track of which positional array element maps to which column. #1
  • Now depends on mcp>=2.1.1.

This is the first non-alpha release of the plugin. I'm confident it's ready as I've been using it quite a bit myself.

This plugin has been around for a while - it lets users upload a brand new SQLite database to a hosted Datasette instance, at which point that database will start being served by that instance.

It can also be used to atomically swap a database with a more recent version. The uploaded database is saved to a file, verified, then swapped in so /name starts serving the new one.

The new release adds a formalized API, so you can replace an existing database (or add a new one) like this:

curl -X POST \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Accept: application/json" \
  -F "db=@content.db" \
  -F "db_name=content" \
  https://your-instance.example.com/-/upload-dbs

This means you can build fresh databases in an environment such as GitHub Actions and swap them in production as soon as that build has completed.

Upgraded for compatibility with `sqlite-utils 4.

This release fixes a SQL injection security issue that affects Datasette instances that serve a mixture of public and private tables in the same database, with access configured using the Datasette permissions system.

Site administrators who serve private tables in this way are advised to disable the execute-sql permission on that database to prevent users from accessing private tables using raw SQL queries. The bug that has been fixed would have allowed users with access to any public table to execute SQL injection attacks despite that restriction, giving them read-only access to data in private tables in the same database.

This fix is also available in Datasette 0.65.3.

Thankfully this particular configuration - private tables and public tables exposed for the same database within the same instance - is likely to be rare. I've not encountered an instance like that myself.

Back-ported the SQL Injection security fix from 1.0a38.

Changes that improve Datasette Apps when created and edited using Datasette Agent:

  • New app_debug() tool allowing agent to open an app (invisibly) and test it using JavaScript. #33
  • New app_list() tool for listing apps the user has permission to edit, so the agent can edit them. #36

The app_debug() tool is pretty neat: it works by displaying the app in a opacity: 0 iframe with pointer-events: none (so it can't be seen or interacted with) and then executing agent-provided JavaScript inside that sandboxed iframe. This means the agent can smoke test that the app is working and even do things like measure the dimensions of different elements.

This uses the new context.browser_task() mechanism added in datasette-agent 0.4a0.

Stateless MCP has recaptured my interest (and inspired mcp-explorer and datasette-mcp)

Visit Stateless MCP has recaptured my interest (and inspired mcp-explorer and datasette-mcp)

Tuesday was Stateless MCP day—the rollout of MCP 2.0, or the 2026-07-28 Model Context Protocol specification to use the more formal but less memorable name. This is the most significant change to the MCP spec since it first launched, and has also served to reignite my personal interest in the protocol.

[... 1,316 words]

  • New await context.browser_task() mechanism allowing agent tools to run code directly in the user's browser. #33

This is an exciting new capability: it makes it easy for Datasette Agent plugins to provide tools that execute custom JavaScript in the user's browser.

I used this to add a debug loop to Datasette Apps in datasette-apps 0.2a0.

Bit of a wild vibe coding experiment with Claude Fable 5. I told it to "Build a plugin for datasette agent which adds a tool for creating a 3D model and then making edits to that model" (full prompt here) and... it did.

Screenshot of a Datasette chat interface showing a 3D rendered cartoon orange cat with black whiskers, closed eyes, pink nose and curled tail sitting in a teal decorative pot with gold trim and handles on a grid floor. Prompt: put the cat in the pot. Text: Thinking: …will be cone shapes, and I'll add torus-shaped eyes along with a nose and smile., Tool: edit_model, Happy Cat Sitting in Decorative Pot · revision 3 · 50 objects · drag to orbit, scroll to zoom, right-drag to pan, Result: edit_model