Simon Willison’s Weblog

datasette-scraper walkthrough on YouTube (via) datasette-scraper is Colin Dellow’s new plugin that turns Datasette into a powerful web scraping tool, with a web UI based on plugin-driven customizations to the Datasette interface. It’s really impressive, and this ten minute demo shows quite how much it is capable of: it can crawl sitemaps and fetch pages, caching them (using zstandard with optional custom dictionaries for extra compression) to speed up subsequent crawls... and you can add your own plugins to extract structured data from crawled pages and save it to a separate SQLite table! # 29th January 2023, 5:23 am

Examples of sites built using Datasette (via) I gave the examples page on the Datasette website a significant upgrade today: it now includes screenshots (taken using shot-scraper) of six projects chosen to illustrate the variety of problems Datasette can be used to tackle. # 29th January 2023, 3:40 am

Cyber (via) “Cyber is a new language for fast, efficient, and concurrent scripting.” Lots of interesting ideas in here, but the one that really caught my eye is that its designed to be easily embedded into other languages and “will allow the host to insert gas mileage checks in user scripts. This allows the host to control how long a script can run”—my dream feature for implementing a safe, sandboxed extension mechanism! Cyber is implemented using Zig and LLVM. # 28th January 2023, 4:25 am

sqlite-jsonschema. “A SQLite extension for validating JSON objects with JSON Schema”, building on the jsonschema Rust crate. SQLite and JSON are already a great combination—Alex suggests using this extension to implement check constraints to validate JSON columns before inserting into a table, or just to run queries finding existing data that doesn’t match a given schema. # 28th January 2023, 3:50 am

sqlite-ulid. Alex Garcia’s sqlite-ulid adds lightning-fast SQL functions for generating ULIDs—Universally Unique Lexicographically Sortable Identifiers. These work like UUIDs but are smaller and faster to generate, and can be canonically encoded as a URL-safe 26 character string (UUIDs are 36 characters). Again, this builds on a Rust crate—ulid-rs—and can generate 1 million byte-represented ULIDs with the ulid_bytes() function in just 88.4ms. # 28th January 2023, 3:45 am

sqlite-fastrand. Alex Garcia just dropped three new SQLite extensions, and I’m going to link to all of them. The first is sqlite-fastrand, which adds new functions for generating random numbers (and alphanumeric characters too). Impressively, these out-perform the default SQLite random() and randomblob() functions by about 1.6-2.6x, thanks to being built on the Rust fastrand crate which builds on wyhash, an extremely fast (though not cryptographically secure) hashing function. # 28th January 2023, 3:41 am

graphql-voyager. Neat tool for producing an interactive graph visualization of any GraphQL API. Click “Change schema” and then “Introspection” and it will give you a GraphQL query you can run against your own API—copy and paste back the JSON results and the visualizer will show you how your API fits together. I tested this against a datasette-graphql instance and it worked exactly as described. # 27th January 2023, 11:58 pm

babelmark3 (via) I found this tool today while investigating an bug in Datasette’s datasette-render-markdown plugin: it lets you run a fragment of Markdown through dozens of different Markdown libraries across multiple different languages and compare the results. Under the hood it works with a registry of API URL endpoints for different implementations, most of which are encrypted in the configuration file on GitHub because they are only intended to be used by this comparison tool. # 27th January 2023, 11:34 pm

Guppe Groups. This is a really neat mechanism for helping build topic-oriented communities on Mastodon: follow @any-group-name@a.gup.pe to join (or create) a group, then that account will re-broadcast any messages from people in that group who mention the group in their message.

I found it via the histodons group. I was pondering how something like this might work this recently, so it’s great to see someone has built it already. # 26th January 2023, 7:45 pm

Python Sandbox in Web Assembly (via) Jim Kring responded to my questions on Mastodon about running Python in a WASM sandbox by building this repo, which demonstrates using wasmer-python to run a build of Python 3.6 compiled to WebAssembly, complete with protected access to a sandbox directory. # 25th January 2023, 9:10 pm

Jortage Communal Cloud. An interesting pattern that’s emerging in the Mastodon / Fediverse community: Jortage is “a communal project providing object storage and hosting”. Each Mastodon server needs to host copies of files—not just for their users, but files that have been imported into the instance because they were posted by other people followed by that instance’s users. Jortage lets multiple instances share the same objects, reducing costs and making things more efficient. I like the idea that multiple projects like this can co-exist, improving the efficiency of the overall network without introducing single centralized services. # 24th January 2023, 11:23 pm

Wildebeest (via) New project from Cloudflare, first quietly unveiled three weeks ago: “Wildebeest is an ActivityPub and Mastodon-compatible server”. It’s built using a flurry of Cloudflare-specific technology, including Workers, Pages and their SQLite-based D1 database. # 23rd January 2023, 12:03 am

Inside the Globus INK: a mechanical navigation computer for Soviet spaceflight (via) Absolutely beautiful piece of Soviet spacecraft engineering, explained in detail by Ken Shirriff. # 21st January 2023, 7:53 pm

The Page With No Code (via) A fun demo by Dan Q, who created a web page with no HTML at all—but in Firefox it still renders content, thanks to a data URI base64 encoded stylesheet served in a link: header that uses html::before, html::after, body::before and body::after with content: properties to serve the content. It even has a background image, encoded as a base64 SVG nested inside another data URI. # 21st January 2023, 6:59 pm

OpenAI Cookbook: Techniques to improve reliability (via) “Let’s think step by step” is a notoriously successful way of getting large language models to solve problems, but it turns out that’s just the tip of the iceberg: this article includes a wealth of additional examples and techniques that can be used to trick GPT-3 into being a whole lot more effective. # 21st January 2023, 5:15 am

datasette-granian (via) Granian is a new Python web server—similar to Gunicorn—written in Rust. I built a small plugin that adds a “datasette granian” command starting a Granian server that serves Datasette’s ASGI application, using the same pattern as my existing datasette-gunicorn plugin. # 20th January 2023, 2:12 am

Hctree Design Documentation. More detailed information on the design of the new Hctree SQLite branch. # 20th January 2023, 12:50 am

Hctree: an experimental high-concurrency database backend for SQLite (via) Really interesting new research branch from the core SQLite team. “Hctree uses optimistic row-level locking and is designed to support dozens of concurrent writers running at full-speed”—with very impressive benchmarks supporting that claim. Also two bonuses: it has a replication mechanism based on the existing SQLite sessions extension, and it bumps up the maximum size of a SQLite database from 16TiB to 1EiB (roughly one million TiB). # 20th January 2023, 12:47 am

Datasette is my data hammer (via) Jeremia Kimelman—a data journalist at CalMatters in Sacramento—enthuses about how he uses Datasette as his default hammer for all kinds of data projects—in particular how much he appreciates Datasette’s focus on URLs. So nice to see this! # 17th January 2023, 5:23 pm

Igalia: the Open Source Powerhouse You’ve Never Heard of (via) An in-depth article about Igalia from July 2022. I had no idea how much stuff they had worked on: arrow functions, generators, async/await, MathML, CSS Grid and a whole bunch more. # 16th January 2023, 8:28 pm

Servo to Advance in 2023 (via) This is excellent news: Serve, the browser-in-Rust project started by Mozilla in 2012 that produced the Rust programming language, is getting re-activated with four new full-time developers provided by Igalia.

Igalia are a fascinating organization—I hadn’t realized quite how influential they’ve been until I read their Wikipedia page just now

They’ve been around since 2001, and “in 2019 they were the #2 committers to both the WebKit and Chromium codebases and in the top 10 contributors to Gecko/Servo”—including implementing and maintaining CSS Grid Layout! # 16th January 2023, 5:08 pm

How to simulate a broken database connection for testing in Django (via) Neil Kakkar explores the options using unittest.patch() and then settles on a neater pattern using “with connection.execute_wrapper(QueryTimeoutWrapper()):” to simulate the exact exception he needs to test against. # 15th January 2023, 8:31 pm

Introducing sqlite-xsv: The Fastest CSV Parser for SQLite. Alex Garcia continues to push the boundaries of SQLite extensions. This new extension in Rust wraps the lightning fast Rust csv crate and provides a new csv_reader() virtual table that can handle regular, gzipped and zstd compressed files. # 14th January 2023, 9:54 pm

Examples of floating point problems (via) I learned so much practical stuff from this post by Julia Evans. There are no 32-bit floating point numbers between 262144.0 and 262144.03125, which breaks code that attempts to keep incrementing by 0.01. I knew about the JavaScript tweet ID problem (JavaScript can’t handle numbers like 1612850010110005250) but I didn’t realize it affected jq as well. Lots more great examples in here. # 13th January 2023, 3:41 pm

Mapping Python to LLVM (via) Codon is a fascinating new entry in the “compile Python code to something else” world—this time targeting LLVM. Ariya Shajii describes in great detail how it pulls this off, including tricks such as transforming Python generators to LLVM coroutines. Codon doesn’t promise that all Python code will work—it’s best thought of as a Python-like language which can be used to create compiled modules which can then be imported back into regular Python projects. # 10th January 2023, 2:08 am

Retiring Pinafore (via) Nolan Lawson built Pinafore, which became my default Mastodon client on both desktop and mobile over the past month. He thoughtfully explains why he’s ending his involvement in the project—and why, for trust reasons, he’s not planning on handing over the reigns to someone else. Pinafore is everything I want a good SPA to be—it loads fast, works offline and packs a whole lot of functionality into a tiny package. I’m sad to see Nolan’s involvement come to end—it’s a superb piece of software. # 10th January 2023, 2:05 am

Petals (via) The challenge with large language models in the same scale ballpark as GPT-3 is that they’re large—really large. Far too big to run on a single machine at home. Petals is a fascinating attempt to address that problem: it works a little bit like BitTorrent, in that each user of Petal runs a subset of the overall language model on their machine and participates in a larger network to run inference across potentially hundreds of distributed GPUs. I tried it just now in Google Colab and it worked exactly as advertised, after downloading an 8GB subset of the 352GB BLOOM-176B model. # 2nd January 2023, 11:29 pm

nanoGPT. “The simplest, fastest repository for training/finetuning medium-sized GPTs”—by Andrej Karpathy, in about 600 lines of Python. # 2nd January 2023, 11:27 pm

In 2022, web3 went just great. Molly White’s essential roundup of 2022 in cryptocurrency. “$4.27 billion was stolen in various hacks and scams this year alone”. # 1st January 2023, 5:13 am