| An MVCC-like columnar table on S3 with constant-time deletes |
https://www.shayon.dev/post/2025/277/an-mvcc-like-columnar-table-on-s3-with-constant-time-deletes/ |
s3's support for conditional writes ([previously](https://simonwillison.net/2024/Nov/26/s3-conditional-writes/)) makes it an interesting, scalable and often inexpensive platform for all kinds of database patterns.
Shayon Mukherjee presents an ingenious design for a Parquet-backed database in S3 which accepts concurrent writes, presents a single atomic view for readers and even supports reliable row deletion despite Parquet requiring a complete file rewrite in order to remove data.
The key to the design is a `_latest_manifest` JSON file at the top of the bucket, containing an integer version number. Clients use compare-and-swap to increment that version - only one client can succeed at this, so the incremented version they get back is guaranteed unique to them.
Having reserved a version number the client can write a unique manifest file for that version - `manifest/v00000123.json` - with a more complex data structure referencing the current versions of every persisted file, including the one they just uploaded.
Deleted rows are written to tombstone files as either a list of primary keys or a list of of ranges. Clients consult these when executing reads, filtering out deleted rows as part of resolving a query.
The pricing estimates are especially noteworthy:
> For a workload ingesting 6 TB/day with 2 TB of deletes and 50K queries/day:
>
> - PUT requests: ~380K/day \(≈4 req/s\) = $1.88/day
> - GET requests: highly variable, depends on partitioning effectiveness
> - Best case \(good time-based partitioning\): ~100K-200K/day = $0.04-$0.08/day
> - Worst case \(poor partitioning, scanning many files\): ~2M/day = $0.80/day
~$3/day for ingesting 6TB of data is pretty fantastic!
Watch out for storage costs though - each new TB of data at $0.023/GB/month adds $23.55 to the ongoing monthly bill. |
2025-10-11 03:47:16+00:00 |
| simonw/claude-skills |
https://github.com/simonw/claude-skills |
One of the tips I picked up from [Jesse Vincent's Claude Code Superpowers post]() ([previously](https://simonwillison.net/2025/Oct/10/superpowers/)) was this:
> Skills are what give your agents Superpowers.
>
> The first time they really popped up on my radar was a few weeks ago when Anthropic rolled out improved Office document creation. When the feature rolled out, I went poking around a bit -- I asked Claude to tell me all about its new skills. And it was [only too happy to dish](https://claude.ai/share/0fe5a9c0-4e5a-42a1-9df7-c5b7636dad92).
In that transcript Jesse asks how the skill system works and Claude replies:
> Looking at the skills system, here's how it works:
>
> **Current Skills** (in `/mnt/skills/public/`):
>
> - **docx** - Word document handling
> - **pdf** - PDF processing and form filling
> - **pptx** - PowerPoint presentations
> - **xlsx** - Excel spreadsheets
This looks like it's part of Claude's brand new Code Interpreter feature! I [wrote about that extensively](https://simonwillison.net/2025/Sep/9/claude-code-interpreter/) last month, but I missed that there was a `/mnt/skills/public/` folder full of fascinating implementation details.
So I fired up a fresh Claude instance (fun fact: Code Interpreter also works in the Claude iOS app now, which it didn't when they first launched) and prompted:
> `Create a zip file of everything in your /mnt/skills folder`
This worked, and gave me a `.zip` to download. You can [run the prompt yourself here](https://claude.ai/new?q=Create%20a%20zip%20file%20of%20everything%20in%20your%20%2Fmnt%2Fskills%20folder), though you'll need to [enable the new feature first](https://simonwillison.net/2025/Sep/9/claude-code-interpreter/#switching-it-on-in-settings-features).
I've pushed the contents of that zip to my [new simonw/claude-skills GitHub repo](https://github.com/simonw/claude-skills).
So now you can see the prompts Anthropic wrote to enable the creation and manipulation of the following files in their Claude consumer applications:
- [pdf](https://github.com/simonw/claude-skills/blob/initial/mnt/skills/public/pdf/SKILL.md) - PDF files
- [docx](https://github.com/simonw/claude-skills/blob/initial/mnt/skills/public/docx/SKILL.md) - Microsoft Word
- [pptx](https://github.com/simonw/claude-skills/blob/initial/mnt/skills/public/pptx/SKILL.md) - Microsoft PowerPoint decks
- [xlsx](https://github.com/simonw/claude-skills/blob/initial/mnt/skills/public/xlsx/SKILL.md) - Microsoft Excel
In each case the prompts spell out detailed instructions for manipulating those file types using Python, using libraries that come pre-installed on Claude's containers.
Skills are more than just prompts though: the repository also includes dozens of pre-written Python scripts for performing common operations.
[pdf/scripts/fill_fillable_fields.py](https://github.com/simonw/claude-skills/blob/initial/mnt/skills/public/pdf/scripts/fill_fillable_fields.py) for example is a custom CLI tool that uses [pypdf](https://pypi.org/project/pypdf/) to find and then fill in a bunch of PDF form fields, specified as JSON, then render out the resulting combined PDF.
This is a really sophisticated set of tools for document manipulation, and I love that Anthropic have made those visible - presumably deliberately - to users of Claude who know how to ask for them. |
2025-10-10 23:57:19+00:00 |
| Superpowers: How I'm using coding agents in October 2025 |
https://blog.fsck.com/2025/10/09/superpowers/ |
A follow-up to Jesse Vincent's post [about September](https://blog.fsck.com/2025/10/05/how-im-using-coding-agents-in-september-2025/), but this is a really significant piece in its own right.
Jesse is one of the most creative users of coding agents (Claude Code in particular) that I know. He's put a great amount of work into evolving an effective process for working with them, encourage red/green TDD (watch the test fail first), planning steps, self-updating memory notes and even implementing a [feelings journal](https://blog.fsck.com/2025/05/28/dear-diary-the-user-asked-me-if-im-alive/) ("I feel engaged and curious about this project" - Claude).
Claude Code [just launched plugins](https://www.anthropic.com/news/claude-code-plugins), and Jesse is celebrating by wrapping up a whole host of his accumulated tricks as a new plugin called [Superpowers](https://github.com/obra/superpowers). You can add it to your Claude Code like this:
/plugin marketplace add obra/superpowers-marketplace
/plugin install superpowers@superpowers-marketplace
There's a lot in here! It's worth spending some time [browsing the repository](https://github.com/obra/superpowers) - here's just one fun example, in [skills/debugging/root-cause-tracing/SKILL.md](https://github.com/obra/superpowers/blob/main/skills/debugging/root-cause-tracing/SKILL.md):
> ---
> name: Root Cause Tracing
> description: Systematically trace bugs backward through call stack to find original trigger
> when_to_use: Bug appears deep in call stack but you need to find where it originates
> version: 1.0.0
> languages: all
> ---
>
> **Overview**
>
> Bugs often manifest deep in the call stack (git init in wrong directory, file created in wrong location, database opened with wrong path). Your instinct is to fix where the error appears, but that's treating a symptom.
>
> **Core principle:** Trace backward through the call chain until you find the original trigger, then fix at the source.
>
> **When to Use**
>
> digraph when_to_use {
> "Bug appears deep in stack?" [shape=diamond];
> "Can trace backwards?" [shape=diamond];
> "Fix at symptom point" [shape=box];
> "Trace to original trigger" [shape=box];
> "BETTER: Also add defense-in-depth" [shape=box];
>
> "Bug appears deep in stack?" -> "Can trace backwards?" [label="yes"];
> "Can trace backwards?" -> "Trace to original trigger" [label="yes"];
> "Can trace backwards?" -> "Fix at symptom point" [label="no - dead end"];
> "Trace to original trigger" -> "BETTER: Also add defense-in-depth";
> }
>
> [...]
This one is particularly fun because it then includes a [Graphviz DOT graph](https://en.wikipedia.org/wiki/DOT_(graph_description_language)) illustrating the process - it turns out Claude can interpret those as workflow instructions just fine, and Jesse has been [wildly experimenting with them](https://blog.fsck.com/2025/09/29/using-graphviz-for-claudemd/).
I [vibe-coded up](https://claude.ai/share/2b78a93e-cdc3-4b1d-9b02-457eb62140a5) a quick URL-based DOT visualizer, [here's that one rendered](https://tools.simonwillison.net/dot#digraph%20when_to_use%20%7B%0A%20%20%20%20%22Bug%20appears%20deep%20in%20stack%3F%22%20%5Bshape%3Ddiamond%5D%3B%0A%20%20%20%20%22Can%20trace%20backwards%3F%22%20%5Bshape%3Ddiamond%5D%3B%0A%20%20%20%20%22Fix%20at%20symptom%20point%22%20%5Bshape%3Dbox%5D%3B%0A%20%20%20%20%22Trace%20to%20original%20trigger%22%20%5Bshape%3Dbox%5D%3B%0A%20%20%20%20%22BETTER%3A%20Also%20add%20defense-in-depth%22%20%5Bshape%3Dbox%5D%3B%0A%0A%20%20%20%20%22Bug%20appears%20deep%20in%20stack%3F%22%20-%3E%20%22Can%20trace%20backwards%3F%22%20%5Blabel%3D%22yes%22%5D%3B%0A%20%20%20%20%22Can%20trace%20backwards%3F%22%20-%3E%20%22Trace%20to%20original%20trigger%22%20%5Blabel%3D%22yes%22%5D%3B%0A%20%20%20%20%22Can%20trace%20backwards%3F%22%20-%3E%20%22Fix%20at%20symptom%20point%22%20%5Blabel%3D%22no%20-%20dead%20end%22%5D%3B%0A%20%20%20%20%22Trace%20to%20original%20trigger%22%20-%3E%20%22BETTER%3A%20Also%20add%20defense-in-depth%22%3B%0A%7D):
There is *so much* to learn about putting these tools to work in the most effective way possible. Jesse is way ahead of the curve, so it's absolutely worth spending some time exploring what he's shared so far.
And if you're worried about filling up your context with a bunch of extra stuff, here's [a reassuring note from Jesse](https://bsky.app/profile/s.ly/post/3m2srmkergc2p):
> The core of it is VERY token light. It pulls in one doc of fewer than 2k tokens. As it needs bits of the process, it runs a shell script to search for them. The long end to end chat for the planning and implementation process for that todo list app was 100k tokens.
>
> It uses subagents to manage token-heavy stuff, including all the actual implementation.
(Jesse's post also tipped me off about Claude's `/mnt/skills/public` folder, see [my notes here](https://simonwillison.net/2025/Oct/10/claude-skills/).) |
2025-10-10 23:30:14+00:00 |
| A Retrospective Survey of 2024/2025 Open Source Supply Chain Compromises |
https://words.filippo.io/compromise-survey/ |
Filippo Valsorda surveyed 18 incidents from the past year of open source supply chain attacks, where package updates were infected with malware thanks to a compromise of the project itself.
These are important lessons:
> I have the growing impression that software supply chain compromises have a few predominant causes which we might have a responsibility as a professional open source maintainers to robustly mitigate.
>
> To test this impression and figure out any such mitigations, I collected all 2024/2025 open source supply chain compromises I could find, and categorized their root cause.
This is a fascinating piece of research. 5 were the result of phishing (maintainers should use passkeys/WebAuthn!), ~5 were stolen long-lived credentials, 3 were "control handoff" where a maintainer gave project access to someone who later turned out to be untrustworthy, 4 were caused by GitHub Actions workflows that triggered on pull requests or issue comments in a way that could leak credentials, and one ([MavenGate](https://blog.oversecured.com/Introducing-MavenGate-a-supply-chain-attack-method-for-Java-and-Android-applications/)) was caused by [an expired domain](https://blog.oversecured.com/Introducing-MavenGate-a-supply-chain-attack-method-for-Java-and-Android-applications/#method-of-attacks) being resurrected. |
2025-10-10 23:00:52+00:00 |
| Video of GPT-OSS 20B running on a phone |
https://twitter.com/nexa_ai/status/1975232300985291008 |
GPT-OSS 20B is a [very good model](https://simonwillison.net/2025/Aug/5/gpt-oss/). At launch OpenAI claimed:
> The gpt-oss-20b model delivers similar results to OpenAI o3‑mini on common benchmarks and can run on edge devices with just 16 GB of memory
[Nexa AI](https://nexa.ai/) just posted a video on Twitter demonstrating exactly that: the full GPT-OSS 20B running on a Snapdragon Gen 5 phone in their [Nexa Studio](https://play.google.com/store/apps/details?id=com.nexa.studio) Android app. It requires at least 16GB of RAM, and benefits from Snapdragon using a similar trick to Apple Silicon where the system RAM is available to both the CPU and the GPU.
The latest iPhone 17 Pro Max is still stuck at 12GB of RAM, presumably not enough to run this same model. |
2025-10-10 22:37:21+00:00 |
| TIL: Testing different Python versions with uv with-editable and uv-test |
https://til.simonwillison.net/python/uv-tests |
While tinkering with upgrading various projects to handle Python 3.14 I finally figured out a universal `uv` recipe for running the tests for the current project in any specified version of Python:
uv run --python 3.14 --isolated --with-editable '.[test]' pytest
This should work in any directory with a `pyproject.toml` (or even a `setup.py`) that defines a `test` set of extra dependencies and uses `pytest`.
The `--with-editable '.[test]'` bit ensures that changes you make to that directory will be picked up by future test runs. The `--isolated` flag ensures no other environments will affect your test run.
I like this pattern so much I built a little shell script that uses it, [shown here](https://til.simonwillison.net/python/uv-tests#user-content-uv-test). Now I can change to any Python project directory and run:
uv-test
Or for a different Python version:
uv-test -p 3.11
I can pass additional `pytest` options too:
uv-test -p 3.11 -k permissions |
2025-10-09 03:37:06+00:00 |
| Python 3.14 Is Here. How Fast Is It? |
https://blog.miguelgrinberg.com/post/python-3-14-is-here-how-fast-is-it |
Miguel Grinberg uses some basic benchmarks (like `fib(40)`) to test the new Python 3.14 on Linux and macOS and finds some substantial speedups over Python 3.13 - around 27% faster.
The optional JIT didn't make a meaningful difference to his benchmarks. On a threaded benchmark he got 3.09x speedup with 4 threads using the free threading build - for Python 3.13 the free threading build only provided a 2.2x improvement. |
2025-10-08 18:36:33+00:00 |
| Why NetNewsWire Is Not a Web App |
https://inessential.com/2025/10/04/why-netnewswire-is-not-web-app.html |
In the wake of Apple [removing ICEBlock from the App Store](https://daringfireball.net/2025/10/iceblock_removed_from_app_store), Brent Simmons talks about why he still thinks his veteran (and actively maintained) [NetNewsWire](https://netnewswire.com/) feed reader app should remain a native application.
Part of the reason is cost - NetNewsWire is free these days ([MIT licensed in fact]()) and the cost to Brent is an annual Apple developer subscription:
> If it were a web app instead, I could drop the developer membership, but I’d have to pay way more money for web and database hosting. [...] I could charge for NetNewsWire, but that would go against my political goal of making sure there’s a good and *free* RSS reader available to everyone.
A bigger reason is around privacy and protecting users:
> Second issue. Right now, if law enforcement comes to me and demands I turn over a given user’s subscriptions list, I can’t. Literally can’t. I don’t have an encrypted version, even — I have nothing at all. The list lives on their machine (iOS or macOS).
And finally it's about the principle of what a personal computing device should mean:
> My computer is *not* a terminal. It’s a world I get to control, and I can use — and, especially, *make* — whatever I want. I’m not stuck using just what’s provided to me on some other machines elsewhere: I’m not dialing into a mainframe or doing the modern equivalent of using only websites that other people control. |
2025-10-08 16:12:14+00:00 |
| Python 3.14 |
https://www.python.org/downloads/release/python-3140/ |
This year's major Python version, Python 3.14, just made its first stable release!
As usual the [what's new in Python 3.14](https://docs.python.org/3.14/whatsnew/3.14.html) document is the best place to get familiar with the new release:
> The biggest changes include [template string literals](https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-template-string-literals), [deferred evaluation of annotations](https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-deferred-annotations), and support for [subinterpreters](https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-multiple-interpreters) in the standard library.
>
> The library changes include significantly improved capabilities for [introspection in asyncio](https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-asyncio-introspection), [support for Zstandard](https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-zstandard) via a new [compression.zstd](https://docs.python.org/3.14/library/compression.zstd.html#module-compression.zstd) module, syntax highlighting in the REPL, as well as the usual deprecations and removals, and improvements in user-friendliness and correctness.
Subinterpreters look particularly interesting as a way to use multiple CPU cores to run Python code despite the continued existence of the GIL. If you're feeling brave and [your dependencies cooperate](https://hugovk.github.io/free-threaded-wheels/) you can also use the free-threaded build of Python 3.14 - [now officially supported](https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-free-threaded-now-supported) - to skip the GIL entirely.
A new major Python release means an older release hits the [end of its support lifecycle](https://devguide.python.org/versions/) - in this case that's Python 3.9. If you maintain open source libraries that target every supported Python versions (as I do) this means features introduced in Python 3.10 can now be depended on! [What's new in Python 3.10](https://docs.python.org/3.14/whatsnew/3.10.html) lists those - I'm most excited by [structured pattern matching](https://docs.python.org/3.14/whatsnew/3.10.html#pep-634-structural-pattern-matching) (the `match/case` statement) and the [union type operator](https://docs.python.org/3.14/whatsnew/3.10.html#pep-604-new-type-union-operator), allowing `int | float | None` as a type annotation in place of `Optional[Union[int, float]]`.
If you use `uv` you can grab a copy of 3.14 using:
uv self update
uv python upgrade 3.14
uvx python@3.14
Or for free-threaded Python 3.1;:
uvx python@3.14t
The `uv` team wrote [about their Python 3.14 highlights](https://astral.sh/blog/python-3.14) in their announcement of Python 3.14's availability via `uv`.
The GitHub Actions [setup-python action](https://github.com/actions/setup-python) includes Python 3.14 now too, so the following YAML snippet in will run tests on all currently supported versions:
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
steps:
- uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
[Full example here](https://github.com/simonw/datasette-pretty-traces/blob/3edddecab850d6ac47ed128a400b6a0ff8b0c012/.github/workflows/test.yml) for one of my many Datasette plugin repos. |
2025-10-08 04:10:06+00:00 |
| Deloitte to pay money back to Albanese government after using AI in $440,000 report |
https://www.theguardian.com/australia-news/2025/oct/06/deloitte-to-pay-money-back-to-albanese-government-after-using-ai-in-440000-report |
Ouch:
> Deloitte will provide a partial refund to the federal government over a $440,000 report that contained several errors, after admitting it used generative artificial intelligence to help produce it.
(I was initially confused by the "Albanese government" reference in the headline since this is a story about the Australian federal government. That's because the current Australia Prime Minister is Anthony Albanese.)
Here's [the page for the report](https://www.dewr.gov.au/assuring-integrity-targeted-compliance-framework/resources/targeted-compliance-framework-assurance-review-final-report). The PDF now includes this note:
> This Report was updated on 26 September 2025 and replaces the Report dated 4 July 2025. The Report has been updated to correct those citations and reference list entries which contained errors in the previously issued version, to amend the summary of the Amato proceeding which contained errors, and to make revisions to improve clarity and readability. The updates made in no way impact or affect the substantive content, findings and recommendations in the Report. |
2025-10-06 23:35:53+00:00 |