Simon Willison's Weblog: linux

Any Nix package, live in your browser

2026-09-10T23:44:15+00:00

Any Nix package, live in your browser

Farid Zakaria calls this his "magnum opus of Nix work", and I can see why.

trynix.dev provides a qemu-wasm powered x86_64 Linux virtual machine running entirely in your browser through WebAssembly. That VM can then be booted with any Nix package from the past 13 years. They are URL addressable, so you can navigate to this page:

https://trynix.dev/?pkg=python3%403.6.2

Then click "Load" and get an interactive shell against a virtual machine running Python 3.6.2 from 2017.

Farid is building all sorts of neat things on top of this. One recent example: Review a pull request by booting it introduces trynix-preview, described like this:

GitHub action that comments a link on a pull request which lets you boot the PR’s build in the browser using https://trynix.dev. No servers, just browsers.

Via Lobste.rs

Tags: code-review, linux, webassembly, github-actions

Creepy crawlies

2026-09-07T23:08:58+00:00

Creepy crawlies

Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of git.kernel.org, the official Git repository for the Linux kernel:

TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.

I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages.

Via Hacker News

Tags: crawling, git, linux, datasette, ai-ethics

Quoting Rick Brewster

2026-09-02T05:50:57+00:00

Direct2D has always been the biggest hurdle for Paint.NET on WINE, and it's clear that it will never be completed enough for Paint.NET's use. And I can't just "disable" the use of Direct2D. So, instead, Paint.NET now has an internal, from-scratch, clean-room reverse-engineered rewrite of Direct2D that it uses on WINE (triggered by using /wine). It lives in PaintDotNet.Windows.Direct2D1.Managed.dll. This was written by our good friend Claude, without whom this would NOT have been possible and would NEVER have happened. [...]

Most of this code is, as they say, "vibe coded." By that I mean that it has not been thoroughly reviewed, it's more "trust me bro" style. I cannot possibly review 180,000 lines of code, it's just way way way too much. For reference, the rest of Paint.NET is about 700,000 lines of code and I've been working on it for over 20 years. [...]

At times, Claude was working with the fury of 10 freshly unshackled Einstein genius-level 10x coders. And other times ... well, not so much. I had to babysit Claude quite a bit to make sure it did resource management correctly (for awhile it just wasn't doing the COM equivalent of AddRef() for reference counted objects, oops). I had to slap it a few times when I found some really bad design or architecture decisions. And I was also impressed at some rather clever and tireless reverse engineering work it did to figure out all the formulas needed for implementing Direct2D's built-in effects library.

— Rick Brewster, author of Paint.NET

Tags: dotnet, linux, reverse-engineering, ai, generative-ai, llms, claude, vibe-coding, coding-agents

Your executable is a SQLite database

2026-08-24T11:38:15+00:00

Your executable is a SQLite database

Farid Zakaria describes a neat Linux pattern for creating a SQLite database file that can be directly used as an executable binary.

The trick sets the SQLite file format's 4-byte application ID (68 bytes into the file) to SELF, standing for Structured Executable & Linkable Format. The various components of the ELF executable format are then arranged into a number of different SQLite tables, using this schema.

Their self-exec interpreter (C code here) can then extract and execute the necessary pieces.

You can additionally use a Linux mechanism called binfmt_misc to teach the kernel to execute that any time it encounters an executable matching that binary pattern. Farid uses NixOS here, but without NixOS I think registration looks something like this:

printf '%s\n' ':self:M:68:SELF::/usr/local/bin/self-exec:' \
  > /proc/sys/fs/binfmt_misc/register

Via Hacker News

Tags: c, linux, sqlite

Quoting Linus Torvalds

2026-08-22T21:04:26+00:00

And this was a debug session from hell, enormously helped by an AI doing much of the grunt-work.

I'd like to call it my tireless helper, but the AI several times stated flat out that this was impossible and unsolvable and that we should just write a report about it.

I suspect those things have been trained by people who may not be quite as stubborn as I am.

But while the AI was ready to give up several times, it did keep adding debug code and analyzing it faithfully when I pushed. So credit where credit is due and I let the AI write the commit message above.

— Linus Torvalds, drm/xe: Don't hand out the flat CCS storage as usable VRAM

Tags: linus-torvalds, linux, ai, generative-ai, llms, ai-assisted-programming

Quoting Linus Torvalds

2026-07-16T13:26:10+00:00

I realize that some people really dislike AI, but this is an area where I'm willing to absolutely put my foot down as the top-level maintainer.

Linux is not one of those anti-AI projects, and if somebody has issues with that, they can do the open-source thing and fork it.

Or just walk away.

AI is a tool, just like other tools we use. And it's clearly a useful one.

It may not have been that "clearly" even just a year ago, but it's no longer in question today.

There are other questions around AI (like what the economy of it will actually look like in the end), but "is it useful" is no longer one of those questions. Anybody who doubts that clearly hasn't actually used it.

— Linus Torvalds, Linux Media Mailing List

Tags: linus-torvalds, linux, open-source, ai, generative-ai, llms

Quoting Willy Tarreau

2026-04-03T21:48:22+00:00

On the kernel security list we've seen a huge bump of reports. We were between 2 and 3 per week maybe two years ago, then reached probably 10 a week over the last year with the only difference being only AI slop, and now since the beginning of the year we're around 5-10 per day depending on the days (fridays and tuesdays seem the worst). Now most of these reports are correct, to the point that we had to bring in more maintainers to help us.

And we're now seeing on a daily basis something that never happened before: duplicate reports, or the same bug found by two different people using (possibly slightly) different tools.

— Willy Tarreau, Lead Software Developer. HAPROXY

Tags: linux, security, ai, generative-ai, llms, ai-security-research

Quoting Greg Kroah-Hartman

2026-04-03T21:44:41+00:00

Months ago, we were getting what we called 'AI slop,' AI-generated security reports that were obviously wrong or low quality. It was kind of funny. It didn't really worry us.

Something happened a month ago, and the world switched. Now we have real reports. All open source projects have real reports that are made with AI, but they're good, and they're real.

— Greg Kroah-Hartman, Linux kernel maintainer (bio), in conversation with Steven J. Vaughan-Nichols

Tags: linux, security, ai, generative-ai, llms, ai-security-research

Quoting Julian Andres Klode

2025-11-01T17:34:34+00:00

I plan to introduce hard Rust dependencies and Rust code into APT, no earlier than May 2026. This extends at first to the Rust compiler and standard library, and the Sequoia ecosystem.

In particular, our code to parse .deb, .ar, .tar, and the HTTP signature verification code would strongly benefit from memory safe languages and a stronger approach to unit testing.

If you maintain a port without a working Rust toolchain, please ensure it has one within the next 6 months, or sunset the port.

— Julian Andres Klode, debian-devel mailing list

Tags: debian, linux, open-source, rust

No More Blue Fridays

2024-07-22T18:33:00+00:00

No More Blue Fridays

Brendan Gregg: "In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code."

New-to-me things I picked up from this:

eBPF - a technology I had thought was unique to the a Linux kernel - is coming Windows!
A useful mental model to have for eBPF is that it provides a WebAssembly-style sandbox for kernel code.
eBPF doesn't stand for "extended Berkeley Packet Filter" any more - that name greatly understates its capabilities and has been retired. More on that in the eBPF FAQ.
From this Hacker News thread eBPF programs can be analyzed before running despite the halting problem because eBPF only allows verifiably-halting programs to run.

Via Hacker News

Tags: linux, security, windows, webassembly, brendan-gregg

Hello World

2024-04-09T01:06:46+00:00

Hello World

Lennon McLean dives deep down the rabbit hole of what happens when you execute the binary compiled from “Hello world” in C on a Linux system, digging into the details of ELF executables, objdump disassembly, the C standard library, stack frames, null-terminated strings and taking a detour through musl because it’s easier to read than Glibc.

Via Hacker News

Tags: c, linux

iSH: The Linux shell for iOS

2023-12-31T04:20:00+00:00

iSH: The Linux shell for iOS

Installing this iOS app gives you a full Linux shell environment running on your phone, using a “usermode x86 emulator”. You can even install packages: “apk add python3” gave me a working Python 3.9 interpreter, installed from the apk.ish.app repository.

I didn’t think this kind of thing was allowed by the App Store, but that’s not been the case for a few years now: Section 4.5.2 of the App Store guidelines clarifies that “Educational apps designed to teach, develop, or allow students to test executable code may, in limited circumstances, download code provided that such code is not used for other purposes.”

Via Hacker News

Tags: appstore, emulator, linux, python, ios

Dynamic linker tricks: Using LD_PRELOAD to cheat, inject features and investigate programs

2023-09-08T22:05:10+00:00

Dynamic linker tricks: Using LD_PRELOAD to cheat, inject features and investigate programs

This tutorial by Rafał Cieślak from 2013 filled in a bunch of gaps in my knowledge about how C works on Linux.

Via Hacker News

Tags: c, linux

Lima VM - Linux Virtual Machines On macOS

2023-07-10T19:01:21+00:00

Lima VM - Linux Virtual Machines On macOS

This looks really useful: “brew install lima” to install, then “limactl start default” to start an Ubuntu VM running and “lima” to get a shell. Julia Evans wrote about the tool this morning, and here Adam Gordon Bell includes details on adding a writable directory (by default lima mounts your macOS home directory in read-only mode).

Via Lima: a nice way to run Linux VMs on Mac

Tags: linux, macos, virtualization, julia-evans

Weeknotes: Getting my personal Dogsheep up and running again

2021-08-22T04:46:13+00:00

I gave a talk about Dogsheep at Noisebridge's Five Minutes of Fame on Thursday. Just one problem: my regular Dogsheep demo was broken, so I ended up building it from scratch again. In doing so I fixed a few bugs in some Dogsheep tools.

Dogsheep on a Digital Ocean droplet

The latest iteration of my personal Dogsheep runs on a $20/month 4GB/2CPU Digital Ocean Droplet running Ubuntu 20.04 LTS.

It runs a private Datasette instance and a bunch of cron jobs to fetch data from Twitter, GitHub, Foursquare Swarm, Pocket and Hacker News.

It also has copies of my Apple Photos and Apple HealthKit data which I upload manually - plus a copy of my genome for good measure.

Some abbreviated notes on how I set it up, copied from a private GitHub Issues thread:

Create a new Ubuntu droplet, and configure its IP address as the A record for dogsheep.simonwillison.net
Install Python 3 and NGINX and SQLite: apt-get install python3 python3-venv nginx sqlite -y
Use letsencrypt to get an HTTPS certificate for it: apt-get update and then apt install certbot python3-certbot-nginx -y, then certbot --nginx -d dogsheep.simonwillison.net
I had to remove the ipv6only=on; bit from the NGINX configuration due to this bug
Created a dogsheep user, useradd -s /bin/bash -d /home/dogsheep/ -m -G
As that user, created a virtual environment: python3 -mvenv datasette-venv and then datasette-venv/bin/pip install wheel and datasette-venv/bin/pip install datasette datasette-auth-passwords
Created a /etc/systemd/system/datasette.service file with this contents

Created a set of blank SQLite database files in WAL mode in /home/dogsheep using the following:

for f in beta.db twitter.db healthkit.db github.db \
  swarm.db photos.db genome.db simonwillisonblog.db \
  pocket.db hacker-news.db memories.db
do
  sqlite3 $f vacuum
  # And enable WAL mode:
  sqlite3 $f 'PRAGMA journal_mode=WAL;'
done

Started the Datasette service: service datasette start
Configured NGINX to proxy to localhost port 8001, using this configuration

It's a few more steps than I'd like, but the end result was a password-protected Datasette instance running against a bunch of SQLite database files on my new server.

With Datasette up and running, the next step was to start loading in data.

Importing my tweets

I started with Twitter. I dropped my Twitter API access credentials into an auth.json file (as described here) and ran the following:

source /home/dogsheep/datasette-venv/bin/activate
pip install twitter-to-sqlite
twitter-to-sqlite user-timeline /home/dogsheep/twitter.db \
  -a /home/dogsheep/auth.json
@simonw [###############################-----] 26299/29684 00:02:06

That pulled in all 29,684 of my personal tweets.

(Actually, first it broke with an error, exposing a bug that had already been reported. I shipped a fix for that and tried again and it worked.)

Favourited tweets were a little harder - I have 39,904 favourited tweets, but the Twitter API only returns the most recent 3,200. I grabbed those more recent ones with:

twitter-to-sqlite favorites /home/dogsheep/twitter.db \
  -a /home/dogsheep/auth.json

Then I requested my Twitter archive, waited 24 hours and uploaded the resulting like.js file to the server, then ran:

twitter-to-sqlite import twitter.db /tmp/like.js

This gave me an archive_like table with the data from that file - but it wasn't the full tweet representation, just the subset that Twitter expose in the archive export.

The README shows how to inflate those into full tweets:

twitter-to-sqlite statuses-lookup twitter.db \
  --sql='select tweetId from archive_like' \
  --skip-existing
Importing 33,382 tweets [------------------------------------] 0% 00:18:28

Once that was done I wrote additional records into the favorited_by table like so:

sqlite3 twitter.db '
  INSERT OR IGNORE INTO favorited_by (tweet, user)
  SELECT tweetId, 12497 FROM archive_like
'

(12497 is my Twitter user ID.)

I also came up with a SQL view that lets me see just media attached to tweets:

sqlite-utils create-view twitter.db media_details "
select
  json_object('img_src', media_url_https, 'width', 400) as img,
  tweets.full_text,
  tweets.created_at,
  tweets.id as tweet_id,
  users.screen_name,
  'https://twitter.com/' || users.screen_name || '/status/' || tweets.id as tweet_url
from
  media
  join media_tweets on media.id = media_tweets.media_id
  join tweets on media_tweets.tweets_id = tweets.id
  join users on tweets.user = users.id
order by
  tweets.id desc
"

Now I can visit /twitter/media_details?_where=tweet_id+in+(select+tweet+from+favorited_by+where+user+=+12497) to see the most recent media tweets that I've favourited!

Swarm checkins

Swarm checkins were a lot easier. I needed my previously-created Foursquare API token, and swarm-to-sqlite:

pip install swarm-to-sqlite
swarm-to-sqlite /home/dogsheep/swarm.db --token=...

This gave me a full table of my Swarm checkins, which I can visualize using datasette-cluster-map:

Apple HealthKit

I don't yet have full automation for my Apple HealthKit data (collected by my Apple Watch) or my Apple Photos - both require me to run scripts on my laptop to create the SQLite database file and then copy the result to the server via scp.

healthkit-to-sqlite runs against the export.zip that is produced by the Apple Health app on the iPhone's export data button - for me that was a 158MB zip file which I AirDropped to my laptop and converted (after fixing a new bug) like so:

healthkit-to-sqlite ~/Downloads/export.zip healthkit.db
Importing from HealthKit  [-----------------------------]    2%  00:02:25

I uploaded the resulting 1.5GB healthkit.db file and now I can do things like visualize my 2017 San Francisco Half Marathon run on a map:

Apple Photos

For my photos I use dogsheep-photos, which I described last year in Using SQL to find my best photo of a pelican according to Apple Photos. The short version: I run this script on my laptop:

# Upload original photos to my S3 bucket
dogsheep-photos upload photos.db \
  ~/Pictures/Photos\ Library.photoslibrary/originals
dogsheep-photos apple-photos photos.db \
  --image-url-prefix "https://photos.simonwillison.net/i/" \
  --image-url-suffix "?w=600"
scp photos.db dogsheep:/home/dogsheep/photos.db

photos.db is only 171MB - it contains the metadata, including the machine learning labels, but not the photos themselves.

And now I can run queries for things like photos of food I've taken in 2021:

Automation via cron

I'm still working through the last step, which involves setting up cron tasks to refresh my data periodically from various sources. My crontab currently looks like this:

# Twitter
1,11,21,31,41,51 * * * * /home/dogsheep/datasette-venv/bin/twitter-to-sqlite user-timeline /home/dogsheep/twitter.db -a /home/dogsheep/auth.json --since
4,14,24,34,44,54 * * * * run-one /home/dogsheep/datasette-venv/bin/twitter-to-sqlite mentions-timeline /home/dogsheep/twitter.db -a /home/dogsheep/auth.json --since
11 * * * * run-one /home/dogsheep/datasette-venv/bin/twitter-to-sqlite user-timeline /home/dogsheep/twitter.db cleopaws -a /home/dogsheep/auth.json --since
6,16,26,36,46,56 * * * * run-one /home/dogsheep/datasette-venv/bin/twitter-to-sqlite favorites /home/dogsheep/twitter.db -a /home/dogsheep/auth.json --stop_after=50

# Swarm
25 */2 * * * /home/dogsheep/datasette-venv/bin/swarm-to-sqlite /home/dogsheep/swarm.db --token=... --since=2w

# Hacker News data every six hours
35 0,6,12,18 * * * /home/dogsheep/datasette-venv/bin/hacker-news-to-sqlite user /home/dogsheep/hacker-news.db simonw

# Re-build dogsheep-beta search index once an hour
32 * * * * /home/dogsheep/datasette-venv/bin/dogsheep-beta index /home/dogsheep/beta.db /home/dogsheep/dogsheep-beta.yml

I'll be expanding this out as I configure more of the Dogsheep tools for my personal instance.

TIL this week

Releases this week

healthkit-to-sqlite: 1.0.1 - (9 releases total) - 2021-08-20
Convert an Apple Healthkit export zip to a SQLite database
twitter-to-sqlite: 0.21.4 - (27 releases total) - 2021-08-20
Save data from Twitter to a SQLite database
datasette-block-robots: 1.0 - (5 releases total) - 2021-08-19
Datasette plugin that blocks robots and crawlers using robots.txt
sqlite-utils: 3.16 - (85 releases total) - 2021-08-18
Python CLI utility and library for manipulating SQLite databases
datasette-debug-asgi: 1.1 - (3 releases total) - 2021-08-17
Datasette plugin for dumping out the ASGI scope

Tags: linux, nginx, datasette, dogsheep, weeknotes

TLDR pages

2017-11-24T05:38:04+00:00

TLDR pages

This is an absurdly good idea: a community maintained set of alternative man pages for common commands with a focus on usage examples, plus a “tldr netstat” command to see them. The man pages themselves are maintained on GitHub.

Tags: documentation, github, linux

Docker Containers on the Desktop

2017-11-05T04:16:38+00:00

Docker Containers on the Desktop

Jessie Frazelle’s classic explanation from 2015 of how she runs every desktop application on her Linux machine in its own Docker container.

Via Jeff Triplett

Tags: linux, docker

What is the difference between Windows and Linux for web hosting, in other words, what are the pros and cons of each, each’s limitations, performance development environment and deployment between Windows and Linux?

2013-08-05T12:24:00+00:00

My answer to What is the difference between Windows and Linux for web hosting, in other words, what are the pros and cons of each, each’s limitations, performance development environment and deployment between Windows and Linux? on Quora

Any and every operation you perform on a Linux server can be trivially automated by copying the commands you ran in to a text file. I haven't managed a Windows server in years and I hear PowerShell is pretty great these days but an OS based around a GUI is always going to be harder to automate than one based around a command line.

Linux has a much stronger complement of high quality open source software - and new open source server software (stuff like node.js) usually becomes available for Linux first.

If you don't know anything about Linux but are comfortable with Windows, you'll find it easier to manage a Windows server in the short term.

Tags: hosting, linux, web-development, windows, quora, ecommerce

How can I download a web server's directory and all subdirectories with one command?

2012-01-15T18:55:00+00:00

My answer to How can I download a web server's directory and all subdirectories with one command? on Quora

Use wget (you can install it with apt-get install wget)

$ wget --recursive http://example.com

That will create a directory called example.com and put the mirrored downloaded files in the right sub-directories inside it.

If you just want to download a subdirectory, do this:

$ wget --recursive http://example.com/subdirectory --no-parent

The --no-parent option ensures wget won't follow links up to parent directories of the one you want to download.

Tags: http, linux, ubuntu, quora

What is a painless way for a non-Unix programmer to get started learning Unix or GNU/Linux?

2010-10-04T17:01:00+00:00

My answer to What is a painless way for a non-Unix programmer to get started learning Unix or GNU/Linux? on Quora

I'd suggest getting yourself an Ubuntu virtual machine running on your own laptop - VirtualBox is free, so that's a good starting point. That way you can play with Linux all you like without fear of breaking anything, since you can always delete the image or roll back to a snapshot.

Tags: linux, programming, software-engineering, quora

Undelete!

2010-08-21T10:56:00+00:00

Undelete!

How to undelete a file accidentally removed using rm on Linux, by grepping through the raw bytes on the hard drive searching for a unique string that was contained in the file. “grep -a -B 25 -A 100 ’some string in the file’ /dev/sda1 > results.txt”

Tags: linux, sysadmin, recovered

Running Processes

2010-03-02T09:55:18+00:00

Running Processes

I’ve been searching for a good solution to this problem (“run this program, and restart it if it falls over”) for years. I’m currently using god which works pretty well, but according to this article I should be learning upstart instead. It never ceases to amaze me how difficult this is, and how obtuse the tools are.

Tags: god, linux, processes, ubuntu, unix, upstart

Linux performance basics

2010-01-24T13:50:39+00:00

Linux performance basics

This kind of Linux knowledge is rapidly becoming a key skill for server-side web development.

Tags: jonathan-ellis, linux, ops, performance, sysadmin

The Maximal Usage Doctrine for Open Source

2010-01-06T17:23:31+00:00

The Maximal Usage Doctrine for Open Source

Yehuda Katz shares my own philosophy on Open Source licensing—stick BSD or MIT on it to maximise the number of people who can use it. The projects I work on are small enough that I don’t care if someone makes big private improvements and refuses to share them. I can see how much larger projects like Linux would disagree though.

Tags: bsd, licensing, linux, mit, open-source, yehuda-katz

Using Graphics Card Memory as Swap

2009-11-03T11:01:18+00:00

Using Graphics Card Memory as Swap

Interesting idea: “Graphic cards contain a lot of very fast RAM, typically between 64 and 512 MB. With Linux, it’s possible to use it as swap space, or even as RAM disk.”

Via Anchor Hosting

Tags: graphicscards, linux, memory, ops, performance, ram, sysadmin

Learning to compile things from source (on Unix/Linux/OSX)

2009-07-27T16:21:47+00:00

Learning to compile things from source (on Unix/Linux/OSX)

I asked on serverfault.com for tips on learning how to solve configure/make/install problems on my own, and got some extremely useful replies.

Tags: compiling, linux, macos, questions, serverfault

Ubuntu brings advanced Screen features to the masses

2009-04-28T21:52:50+00:00

Ubuntu brings advanced Screen features to the masses

Ubuntu 9.04’s screen-profiles package adds a taskbar to screen and emulates the gnome panel. You can even add a widget showing the cost of your current EC2 session.

Tags: ec2, linux, screen, ubuntu

Introducing the Karmic Koala, our mascot for Ubuntu 9.10

2009-02-21T17:19:18+00:00

Introducing the Karmic Koala, our mascot for Ubuntu 9.10

Ubuntu 9.10 will have a strong focus on cloud computing, including tools for easily creating EC2 AMIs and Eucalyptus, an open-source system for running an EC2-compatible cloud in your own data centre.

Via Simon Wardley

Tags: cloud-computing, ec2, eucalyptus, karmickoala, linux, mark-shuttleworth, ubuntu

Quoting Jeremy Allison

2008-12-09T08:03:09+00:00

[In Mali...] The outcome of this rampant illegal software copying is that Windows is seen as "the first world standard" and any attempt to push a cheaper alternative is strongly resisted. They consider it trying to cheat local people out of getting the same quality of software that is used in the developed world, even though it's a legal way of getting quality software for free.

— Jeremy Allison

Tags: africa, jeremy-allison, linux, mali, open-source, piracy, windows

Ubuntu and Debian AMIs for Amazon EC2

2008-12-08T18:04:09+00:00

Ubuntu and Debian AMIs for Amazon EC2

Exactly what it says on the tin.

Tags: amis, debian, ec2, linux, ubuntu