Simon Willison's Weblog: http

datasette-referrer-policy 0.1

2026-05-05T23:44:27+00:00

Release: datasette-referrer-policy 0.1

The OpenStreetMap tiles on the Datasette global-power-plants demo weren't displaying correctly. This turned out to be caused by two bugs.

The first is that the CAPTCHA I added to that site a few weeks ago was triggering for the .json fetch requests used by the map plugin, and since those weren't HTML the user was not being asked to solve them. Here's the fix.

The second was that OpenStreetMap quite reasonably block tile requests from sites that use a Referrer-Policy: no-referrer header.

Datasette does this by default, and I didn't want to change that default on people without warning - so I had Codex + GPT-5.5 build me a new plugin to help set that header to another value.

Tags: http, openstreetmap, datasette

Unicode Explorer using binary search over fetch() HTTP range requests

2026-02-27T17:50:54+00:00

Unicode Explorer using binary search over fetch() HTTP range requests

Here's a little prototype I built this morning from my phone as an experiment in HTTP range requests, and a general example of using LLMs to satisfy curiosity.

I've been collecting HTTP range tricks for a while now, and I decided it would be fun to build something with them myself that used binary search against a large file to do something useful.

So I brainstormed with Claude. The challenge was coming up with a use case for binary search where the data could be naturally sorted in a way that would benefit from binary search.

One of Claude's suggestions was looking up information about unicode codepoints, which means searching through many MBs of metadata.

I had Claude write me a spec to feed to Claude Code - visible here - then kicked off an asynchronous research project with Claude Code for web against my simonw/research repo to turn that into working code.

Here's the resulting report and code. One interesting thing I learned is that Range request tricks aren't compatible with HTTP compression because they mess with the byte offset calculations. I added 'Accept-Encoding': 'identity' to the fetch() calls but this isn't actually necessary because Cloudflare and other CDNs automatically skip compression if a content-range header is present.

I deployed the result to my tools.simonwillison.net site, after first tweaking it to query the data via range requests against a CORS-enabled 76.6MB file in an S3 bucket fronted by Cloudflare.

The demo is fun to play with - type in a single character like ø or a hexadecimal codepoint indicator like 1F99C and it will binary search its way through the large file and show you the steps it takes along the way:

Tags: algorithms, http, research, tools, unicode, ai, generative-ai, llms, ai-assisted-programming, vibe-coding, http-range-requests

Unicode Explorer — Binary Search Over HTTP

2026-02-27T16:01:33+00:00

Tool: Unicode Explorer — Binary Search Over HTTP

Tags: http, unicode, http-range-requests

Unicode Explorer — Binary Search Over HTTP

2026-02-27T16:00:00+00:00

Research: Unicode Explorer — Binary Search Over HTTP

Tags: http, unicode, http-range-requests

Introducing gisthost.github.io

2026-01-01T22:12:20+00:00

I am a huge fan of gistpreview.github.io, the site by Leon Huang that lets you append ?GIST_id to see a browser-rendered version of an HTML page that you have saved to a Gist. The last commit was ten years ago and I needed a couple of small changes so I've forked it and deployed an updated version at gisthost.github.io.

Some background on gistpreview

The genius thing about gistpreview.github.io is that it's a core piece of GitHub infrastructure, hosted and cost-covered entirely by GitHub, that wasn't built with any involvement from GitHub at all.

To understand how it works we need to first talk about Gists.

Any file hosted in a GitHub Gist can be accessed via a direct URL that looks like this:

https://gist.githubusercontent.com/simonw/d168778e8e62f65886000f3f314d63e3/raw/79e58f90821aeb8b538116066311e7ca30c870c9/index.html

That URL is served with a few key HTTP headers:

Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff

These ensure that every file is treated by browsers as plain text, so HTML file will not be rendered even by older browsers that attempt to guess the content type based on the content.

Via: 1.1 varnish
Cache-Control: max-age=300
X-Served-By: cache-sjc1000085-SJC

These confirm that the file is sever via GitHub's caching CDN, which means I don't feel guilty about linking to them for potentially high traffic scenarios.

Access-Control-Allow-Origin: *

This is my favorite HTTP header! It means I can hit these files with a fetch() call from any domain on the internet, which is fantastic for building HTML tools that do useful things with content hosted in a Gist.

The one big catch is that Content-Type header. It means you can't use a Gist to serve HTML files that people can view.

That's where gistpreview comes in. The gistpreview.github.io site belongs to the dedicated gistpreview GitHub organization, and is served out of the github.com/gistpreview/gistpreview.github.io repository by GitHub Pages.

It's not much code. The key functionality is this snippet of JavaScript from main.js:

fetch('https://api.github.com/gists/' + gistId)
.then(function (res) {
  return res.json().then(function (body) {
    if (res.status === 200) {
      return body;
    }
    console.log(res, body); // debug
    throw new Error('Gist <strong>' + gistId + '</strong>, ' + body.message.replace(/\(.*\)/, ''));
  });
})
.then(function (info) {
  if (fileName === '') {
    for (var file in info.files) {
      // index.html or the first file
      if (fileName === '' || file === 'index.html') {
        fileName = file;
      }
    }
  }
  if (info.files.hasOwnProperty(fileName) === false) {
    throw new Error('File <strong>' + fileName + '</strong> is not exist');
  }
  var content = info.files[fileName].content;
  document.write(content);
})

This chain of promises fetches the Gist content from the GitHub API, finds the section of that JSON corresponding to the requested file name and then outputs it to the page like this:

document.write(content);

This is smart. Injecting the content using document.body.innerHTML = content would fail to execute inline scripts. Using document.write() causes the browser to treat the HTML as if it was directly part of the parent page.

That's pretty much the whole trick! Read the Gist ID from the query string, fetch the content via the JSON API and document.write() it into the page.

Here's a demo:

https://gistpreview.github.io/?d168778e8e62f65886000f3f314d63e3

Fixes for gisthost.github.io

I forked gistpreview to add two new features:

A workaround for Substack mangling the URLs
The ability to serve larger files that get truncated in the JSON API

I also removed some dependencies (jQuery and Bootstrap and an old fetch() polyfill) and inlined the JavaScript into a single index.html file.

The Substack issue was small but frustrating. If you email out a link to a gistpreview page via Substack it modifies the URL to look like this:

https://gistpreview.github.io/?f40971b693024fbe984a68b73cc283d2=&utm_source=substack&utm_medium=email

This breaks gistpreview because it treats f40971b693024fbe984a68b73cc283d2=&utm_source... as the Gist ID.

The fix is to read everything up to that equals sign. I submitted a PR for that back in November.

The second issue around truncated files was reported against my claude-code-transcripts project a few days ago. Here's the fix.

That project provides a CLI tool for exporting HTML rendered versions of Claude Code sessions. It includes a --gist option which uses the gh CLI tool to publish the resulting HTML to a Gist and returns a gistpreview URL that the user can share.

These exports can get pretty big, and some of the resulting HTML was past the size limit of what comes back from the Gist API.

As of claude-code-transcripts 0.5 the --gist option now publishes to gisthost.github.io instead, fixing both bugs.

Here's the Claude Code transcript that refactored Gist Host to remove those dependencies, which I published to Gist Host using the following command:

uvx claude-code-transcripts web --gist

Tags: github, http, javascript, projects, ai-assisted-programming, cors

HTTP Range Requests for Wheel Metadata

2025-12-26T23:29:00+00:00

Research: HTTP Range Requests for Wheel Metadata

More details here.

Tags: http, http-range-requests

YouTube embeds fail with a 153 error

2025-12-01T05:26:23+00:00

YouTube embeds fail with a 153 error

I just fixed this bug on my blog. I was getting an annoying "Error 153: Video player configuration error" on some of the YouTube video embeds (like this one) on this site. After some digging it turns out the culprit was this HTTP header, which Django's SecurityMiddleware was sending by default:

Referrer-Policy: same-origin

YouTube's embedded player terms documentation explains why this broke:

API Clients that use the YouTube embedded player (including the YouTube IFrame Player API) must provide identification through the HTTP Referer request header. In some environments, the browser will automatically set HTTP Referer, and API Clients need only ensure they are not setting the Referrer-Policy in a way that suppresses the Referer value. YouTube recommends using strict-origin-when-cross-origin Referrer-Policy, which is already the default in many browsers.

The fix, which I outsourced to GitHub Copilot agent since I was on my phone, was to add this to my settings.py:

SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"

This explainer on the Chrome blog describes what the header means:

strict-origin-when-cross-origin offers more privacy. With this policy, only the origin is sent in the Referer header of cross-origin requests.

This prevents leaks of private data that may be accessible from other parts of the full URL such as the path and query string.

Effectively it means that any time you follow a link from my site to somewhere else they'll see this in the incoming HTTP headers even if you followed the link from a page other than my homepage:

Referer: https://simonwillison.net/

The previous header, same-origin, is explained by MDN here:

Send the origin, path, and query string for same-origin requests. Don't send the Referer header for cross-origin requests.

This meant that previously traffic from my site wasn't sending any HTTP referer at all!

Tags: django, http, privacy, youtube

httpjail

2025-09-19T21:57:29+00:00

httpjail

Here's a promising new (experimental) project in the sandboxing space from Ammar Bandukwala at Coder. httpjail provides a Rust CLI tool for running an individual process against a custom configured HTTP proxy.

The initial goal is to help run coding agents like Claude Code and Codex CLI with extra rules governing how they interact with outside services. From Ammar's blog post that introduces the new tool, Fine-grained HTTP filtering for Claude Code:

httpjail implements an HTTP(S) interceptor alongside process-level network isolation. Under default configuration, all DNS (udp:53) is permitted and all other non-HTTP(S) traffic is blocked.

httpjail rules are either JavaScript expressions or custom programs. This approach makes them far more flexible than traditional rule-oriented firewalls and avoids the learning curve of a DSL.

Block all HTTP requests other than the LLM API traffic itself:
$ httpjail --js "r.host === 'api.anthropic.com'" -- claude "build something great"

I tried it out using OpenAI's Codex CLI instead and found this recipe worked:

brew upgrade rust
cargo install httpjail # Drops it in `~/.cargo/bin`
httpjail --js "r.host === 'chatgpt.com'" -- codex

Within that Codex instance the model ran fine but any attempts to access other URLs (e.g. telling it "Use curl to fetch simonwillison.net)" failed at the proxy layer.

This is still at a really early stage but there's a lot I like about this project. Being able to use JavaScript to filter requests via the --js option is neat (it's using V8 under the hood), and there's also a --sh shellscript option which instead runs a shell program passing environment variables that can be used to determine if the request should be allowed.

At a basic level it works by running a proxy server and setting HTTP_PROXY and HTTPS_PROXY environment variables so well-behaving software knows how to route requests.

It can also add a bunch of other layers. On Linux it sets up nftables rules to explicitly deny additional network access. There's also a --docker-run option which can launch a Docker container with the specified image but first locks that container down to only have network access to the httpjail proxy server.

It can intercept, filter and log HTTPS requests too by generating its own certificate and making that available to the underlying process.

I'm always interested in new approaches to sandboxing, and fine-grained network access is a particularly tricky problem to solve. This looks like a very promising step in that direction - I'm looking forward to seeing how this project continues to evolve.

Via Fine-grained HTTP filtering for Claude Code

Tags: http, javascript, proxies, sandboxing, security, v8, rust, claude-code, codex

tidwall/pogocache

2025-07-21T23:58:53+00:00

tidwall/pogocache

New project from Josh Baker, author of the excellent tg C geospatial libarry (covered previously) and various other interesting projects:

Pogocache is fast caching software built from scratch with a focus on low latency and cpu efficency.

Faster: Pogocache is faster than Memcache, Valkey, Redis, Dragonfly, and Garnet. It has the lowest latency per request, providing the quickest response times. It's optimized to scale from one to many cores, giving you the best single-threaded and multithreaded performance.

Faster than Memcache and Redis is a big claim! The README includes a design details section that explains how the system achieves that performance, using a sharded hashmap inspired by Josh's shardmap project and clever application of threads.

Performance aside, the most interesting thing about Pogocache is the server interface it provides: it emulates the APIs for Redis and Memcached, provides a simple HTTP API and lets you talk to it over the PostgreSQL wire protocol as well!

psql -h localhost -p 9401
=> SET first Tom;
=> SET last Anderson;
=> SET age 37;

$ curl http://localhost:9401/last
Anderson

Via Show HN

Tags: c, caching, http, memcached, postgresql, redis

Some Go web dev notes

2024-09-27T23:43:31+00:00

Some Go web dev notes

Julia Evans on writing small, self-contained web applications in Go:

In general everything about it feels like it makes projects easy to work on for 5 days, abandon for 2 years, and then get back into writing code without a lot of problems.

Go 1.22 introduced HTTP routing in February of this year, making it even more practical to build a web application using just the Go standard library.

Tags: go, http, web-development, julia-evans

How streaming LLM APIs work

2024-09-22T03:48:12+00:00

How streaming LLM APIs work

New TIL. I used curl to explore the streaming APIs provided by OpenAI, Anthropic and Google Gemini and wrote up detailed notes on what I learned.

Also includes example code for receiving streaming events in Python with HTTPX and receiving streaming events in client-side JavaScript using fetch().

Tags: apis, http, json, llms

SQL Injection Isn't Dead: Smuggling Queries at the Protocol Level

2024-08-12T15:36:47+00:00

SQL Injection Isn't Dead: Smuggling Queries at the Protocol Level

PDF slides from a presentation by Paul Gerste at DEF CON 32. It turns out some databases have vulnerabilities in their binary protocols that can be exploited by carefully crafted SQL queries.

Paul demonstrates an attack against PostgreSQL (which works in some but not all of the PostgreSQL client libraries) which uses a message size overflow, by embedding a string longer than 4GB (2**32 bytes) which overflows the maximum length of a string in the underlying protocol and writes data to the subsequent value. He then shows a similar attack against MongoDB.

The current way to protect against these attacks is to ensure a size limit on incoming requests. This can be more difficult than you may expect - Paul points out that alternative paths such as WebSockets might bypass limits that are in place for regular HTTP requests, plus some servers may apply limits before decompression, allowing an attacker to send a compressed payload that is larger than the configured limit.

Via lobste.rs

Tags: http, mongodb, postgresql, security, sql-injection, websockets

django-http-debug, a new Django app mostly written by Claude

2024-08-08T15:26:27+00:00

Yesterday I finally developed something I’ve been casually thinking about building for a long time: django-http-debug. It’s a reusable Django app - something you can pip install into any Django project - which provides tools for quickly setting up a URL that returns a canned HTTP response and logs the full details of any incoming request to a database table.

This is ideal for any time you want to start developing against some external API that sends traffic to your own site - a webhooks provider like Stripe, or an OAuth or OpenID connect integration (my task yesterday morning).

You can install it right now in your own Django app: add django-http-debug to your requirements (or just pip install django-http-debug), then add the following to your settings.py:

INSTALLED_APPS = [
    # ...
    'django_http_debug',
    # ...
]

MIDDLEWARE = [
    # ...
    "django_http_debug.middleware.DebugMiddleware",
    # ...
]

You'll need to have the Django Admin app configured as well. The result will be two new models managed by the admin - one for endpoints:

And a read-only model for viewing logged requests:

It’s possible to disable logging for an endpoint, which means django-http-debug doubles as a tool for adding things like a robots.txt to your site without needing to deploy any additional code.

How it works

The key to how this works is this piece of middleware:

class DebugMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        if response.status_code == 404:
            path = request.path.lstrip("/")
            debug_response = debug_view(request, path)
            if debug_response:
                return debug_response
        return response

This dispatches to the default get_response() function, then intercepts the result and checks if it's a 404. If so, it gives the debug_view() function an opportunity to respond instead - which might return None, in which case that original 404 is returned to the client.

That debug_view() function looks like this:

@csrf_exempt
def debug_view(request, path):
    try:
        endpoint = DebugEndpoint.objects.get(path=path)
    except DebugEndpoint.DoesNotExist:
        return None  # Allow normal 404 handling to continue

    if endpoint.logging_enabled:
        log_entry = RequestLog(
            endpoint=endpoint,
            method=request.method,
            query_string=request.META.get("QUERY_STRING", ""),
            headers=dict(request.headers),
        )
        log_entry.set_body(request.body)
        log_entry.save()

    content = endpoint.content
    if endpoint.is_base64:
        content = base64.b64decode(content)

    response = HttpResponse(
        content=content,
        status=endpoint.status_code,
        content_type=endpoint.content_type,
    )
    for key, value in endpoint.headers.items():
        response[key] = value

    return response

It checks the database for an endpoint matching the incoming path, then logs the response (if the endpoint has logging_enabled set) and returns a canned response based on the endpoint configuration.

Here are the models:

from django.db import models
import base64


class DebugEndpoint(models.Model):
    path = models.CharField(max_length=255, unique=True)
    status_code = models.IntegerField(default=200)
    content_type = models.CharField(max_length=64, default="text/plain; charset=utf-8")
    headers = models.JSONField(default=dict, blank=True)
    content = models.TextField(blank=True)
    is_base64 = models.BooleanField(default=False)
    logging_enabled = models.BooleanField(default=True)

    def __str__(self):
        return self.path

    def get_absolute_url(self):
        return f"/{self.path}"


class RequestLog(models.Model):
    endpoint = models.ForeignKey(DebugEndpoint, on_delete=models.CASCADE)
    method = models.CharField(max_length=10)
    query_string = models.CharField(max_length=255, blank=True)
    headers = models.JSONField()
    body = models.TextField(blank=True)
    is_base64 = models.BooleanField(default=False)
    timestamp = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"{self.method} {self.endpoint.path} at {self.timestamp}"

    def set_body(self, body):
        try:
            # Try to decode as UTF-8
            self.body = body.decode("utf-8")
            self.is_base64 = False
        except UnicodeDecodeError:
            # If that fails, store as base64
            self.body = base64.b64encode(body).decode("ascii")
            self.is_base64 = True

    def get_body(self):
        if self.is_base64:
            return base64.b64decode(self.body.encode("ascii"))
        return self.body

The admin screens are defined in admin.py.

Claude built the first version of this for me

This is a classic example of a project that I couldn’t quite justify building without assistance from an LLM. I wanted it to exist, but I didn't want to spend a whole day building it.

Claude 3.5 Sonnet got me 90% of the way to a working first version. I had to make a few tweaks to how the middleware worked, but having done that I had a working initial prototype within a few minutes of starting the project.

Here’s the full sequence of prompts I used, each linking to the code that was produced for me (as a Claude artifact):

I want a Django app I can use to help create HTTP debugging endpoints. It should let me configure a new path e.g. /webhooks/receive/ that the Django 404 handler then hooks into - if one is configured it can be told which HTTP status code, headers and content to return.

ALL traffic to those endpoints is logged to a Django table - full details of incoming request headers, method and body. Those can be browsed read-only in the Django admin (and deleted)

Produced Claude v1

make it so I don't have to put it in the urlpatterns because it hooks ito Django's 404 handling mechanism instead

Produced Claude v2

Suggestions for how this could handle request bodies that don't cleanly decode to utf-8

Produced Claude v3

don't use a binary field, use a text field but still store base64 data in it if necessary and have a is_base64 boolean column that gets set to true if that happens

Produced Claude v4

I took that code and ran with it - I fired up a new skeleton library using my python-lib cookiecutter template, copied the code into it, made some tiny changes to get it to work and shipped it as an initial alpha release - mainly so I could start exercising it on a couple of sites I manage.

Using it in the wild for a few minutes quickly identified changes I needed to make. I filed those as issues:

Then I worked though fixing each of those one at a time. I did most of this work myself, though GitHub Copilot helped me out be typing some of the code for me.

Adding the base64 preview

There was one slightly tricky feature I wanted to add that didn’t justify spending much time on but was absolutely a nice-to-have.

The logging mechanism supports binary data: if incoming request data doesn’t cleanly encode as UTF-8 it gets stored as Base 64 text instead, with the is_base64 flag set to True (see the set_body() method in the RequestLog model above).

I asked Claude for a curl one-liner to test this and it suggested:

curl -X POST http://localhost:8000/foo/ \
  -H "Content-Type: multipart/form-data" \
  -F "image=@pixel.gif"

I do this a lot - knocking out quick curl commands is an easy prompt, and you can tell it the URL and headers you want to use, saving you from having to edit the command yourself later on.

I decided to have the Django Admin view display a decoded version of that Base 64 data. But how to render that, when things like binary file uploads may not be cleanly renderable as text?

This is what I came up with:

The trick here I'm using here is to display the decoded data as a mix between renderable characters and hex byte pairs, with those pairs rendered using a different font to make it clear that they are part of the binary data.

This is achieved using a body_display() method on the RequestLogAdmin admin class, which is then listed in readonly_fields. The full code is here, this is that method:

    def body_display(self, obj):
        body = obj.get_body()
        if not isinstance(body, bytes):
            return format_html("<pre>{}</pre>", body)

        # Attempt to guess filetype
        suggestion = None
        match = filetype.guess(body[:1000])
        if match:
            suggestion = "{} ({})".format(match.extension, match.mime)

        encoded = repr(body)
        # Ditch the b' and trailing '
        if encoded.startswith("b'") and encoded.endswith("'"):
            encoded = encoded[2:-1]

        # Split it into sequences of octets and characters
        chunks = sequence_re.split(encoded)
        html = []
        if suggestion:
            html.append(
                '<p style="margin-top: 0; font-family: monospace; font-size: 0.8em;">Suggestion: {}</p>'.format(
                    suggestion
                )
            )
        for chunk in chunks:
            if sequence_re.match(chunk):
                octets = octet_re.findall(chunk)
                octets = [o[2:] for o in octets]
                html.append(
                    '<code style="color: #999; font-family: monospace">{}</code>'.format(
                        " ".join(octets).upper()
                    )
                )
            else:
                html.append(chunk.replace("\\\\", "\\"))

        return mark_safe(" ".join(html).strip().replace("\\r\\n", "<br>"))

I got Claude to write that using one of my favourite prompting tricks. I'd solved this problem once before in the past, in my datasette-render-binary project. So I pasted that code into Claude, told it:

With that code as inspiration, modify the following Django Admin code to use that to display decoded base64 data:

And then pasted in my existing Django admin class. You can see my full prompt here.

Claude replied with this code, which almost worked exactly as intended - I had to make one change, swapping out the last line for this:

        return mark_safe(" ".join(html).strip().replace("\\r\\n", "<br>"))

I love this pattern: "here's my existing code, here's some other code I wrote, combine them together to solve this problem". I wrote about this previously when I described how I built my PDF OCR JavaScript tool a few months ago.

Adding automated tests

The final challenge was the hardest: writing automated tests. This was difficult because Django tests need a full Django project configured for them, and I wasn’t confident about the best pattern for doing that in my standalone django-http-debug repository since it wasn’t already part of an existing Django project.

I decided to see if Claude could help me with that too, this time using my files-to-prompt and LLM command-line tools:

files-to-prompt . --ignore LICENSE | \
  llm -m claude-3.5-sonnet -s \
  'step by step advice on how to implement automated tests for this, which is hard because the tests need to work within a temporary Django project that lives in the tests/ directory somehow. Provide all code at the end.'

Here's Claude's full response. It almost worked! It gave me a minimal test project in tests/test_project and an initial set of quite sensible tests.

Sadly it didn’t quite solve the most fiddly problem for me: configuring it so running pytest would correctly set the Python path and DJANGO_SETTINGS_MODULE in order run the tests. I saw this error instead:

django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

I spent some time with the relevant pytest-django documentation and figure out a pattern that worked. Short version: I added this to my pyproject.toml file:

[tool.pytest.ini_options]
DJANGO_SETTINGS_MODULE = "tests.test_project.settings"
pythonpath = ["."]

For the longer version, take a look at my full TIL: Using pytest-django with a reusable Django application.

Test-supported cleanup

The great thing about having comprehensive tests in place is it makes iterating on the project much faster. Claude had used some patterns that weren’t necessary. I spent a few minutes seeing if the tests still passed if I deleted various pieces of code, and cleaned things up quite a bit.

Was Claude worth it?

This entire project took about two hours - just within a tolerable amount of time for what was effectively a useful sidequest from my intended activity for the day.

Claude didn't implement the whole project for me. The code it produced didn't quite work - I had to tweak just a few lines of code, but knowing which code to tweak took a development environment and manual testing and benefited greatly from my 20+ years of Django experience!

This is yet another example of how LLMs don't replace human developers: they augment us.

The end result is a tool that I'm already using to solve real-world problems, and a code repository that I'm proud to put my name to. Without LLM assistance this project would have stayed on my ever-growing list of "things I'd love to build one day".

I'm also really happy to have my own documented solution to the challenge of adding automated tests to a standalone reusable Django application. I was tempted to skip this step entirely, but thanks to Claude's assistance I was able to break that problem open and come up with a solution that I'm really happy with.

Last year I wrote about how AI-enhanced development makes me more ambitious with my projects. It's also helping me be more diligent in not taking shortcuts like skipping setting up automated tests.

Tags: django, django-admin, http, projects, python, webhooks, ai, generative-ai, llms, ai-assisted-programming, anthropic, claude, claude-3-5-sonnet

django-http-debug 0.2

2024-08-07T20:17:55+00:00

Release: django-http-debug 0.2

Tags: http

django-http-debug 0.1a0

2024-08-07T18:15:27+00:00

Release: django-http-debug 0.1a0

Tags: http

How I write HTTP services in Go after 13 years

2024-02-09T20:40:23+00:00

How I write HTTP services in Go after 13 years

Useful set of current best practices for deploying HTTP servers written in Go. I guess Go counts as boring technology these days, which is high praise in my book.

Via Hacker News

Tags: go, http

Cloudflare does not consider vary values in caching decisions

2023-11-20T05:08:52+00:00

Cloudflare does not consider vary values in caching decisions

Here’s the spot in Cloudflare’s documentation where they hide a crucially important detail:

“Cloudflare does not consider vary values in caching decisions. Nevertheless, vary values are respected when Vary for images is configured and when the vary header is vary: accept-encoding.”

This means you can’t deploy an application that uses content negotiation via the Accept header behind the Cloudflare CDN—for example serving JSON or HTML for the same URL depending on the incoming Accept header. If you do, Cloudflare may serve cached JSON to an HTML client or vice-versa.

There’s an exception for image files, which Cloudflare added support for in September 2021 (for Pro accounts only) in order to support formats such as WebP which may not have full support across all browsers.

Tags: caching, http, cloudflare

See this page fetch itself, byte by byte, over TLS

2023-05-10T13:58:36+00:00

See this page fetch itself, byte by byte, over TLS

George MacKerron built a TLS 1.3 library in TypeScript and used it to construct this amazing educational demo, which performs a full HTTPS request for its own source code over a WebSocket and displays an annotated byte-by-byte representation of the entire exchange. This is the most useful illustration of how HTTPS actually works that I’ve ever seen.

Via Julia Evans

Tags: encryption, http, https, tls, websockets, explorables

urllib3 v2.0.0 is now generally available

2023-04-26T22:00:16+00:00

urllib3 v2.0.0 is now generally available

urllib3 is 12 years old now, and is a common low-level dependency for packages like requests and httpx. The biggest new feature in v2 is a higher-level API: resp = urllib3.request(“GET”, “https://example.com”)—a very welcome addition to the library.

Tags: http, python

RFC 7807: Problem Details for HTTP APIs

2022-11-01T03:15:05+00:00

RFC 7807: Problem Details for HTTP APIs

This RFC has been brewing for quite a while, and is currently in last call (ends 2022-11-03). I’m designing the JSON error messages for Datasette at the moment so this could not be more relevant for me.

Via Nicolas Fränkel

Tags: errors, http, json, mark-nottingham, rfc, standards

Introducing sqlite-http: A SQLite extension for making HTTP requests

2022-08-10T22:22:42+00:00

Introducing sqlite-http: A SQLite extension for making HTTP requests

Characteristically thoughtful SQLite extension from Alex, following his sqlite-html extension from a few days ago. sqlite-http lets you make HTTP requests from SQLite—both as a SQL function that returns a string, and as a table-valued SQL function that lets you independently access the body, headers and even the timing data for the request.

This write-up is excellent: it provides interactive demos but also shows how additional SQLite extensions such as the new-to-me “define” extension can be combined with sqlite-http to create custom functions for parsing and processing HTML.

Via @agarcia_me

Tags: http, sqlite, alex-garcia

Making HTTP calls using IPv6

2022-06-08T22:58:28+00:00

TIL: Making HTTP calls using IPv6

Tags: http

curlconverter.com

2022-03-10T20:12:44+00:00

curlconverter.com

This is pretty magic: paste in a “curl” command (including the ones you get from browser devtools using copy-as-curl) and this will convert that into code for making the same HTTP request... using Python, JavaScript, PHP, R, Go, Rust, Elixir, Java, MATLAB, Ansible URI, Strest, Dart or JSON.

Via Julia Evans

Tags: curl, http

Hurl

2021-11-22T03:32:33+00:00

Hurl

Hurl is “a command line tool that runs HTTP requests defined in a simple plain text format”—written in Rust on top of curl, it lets you run HTTP requests and then execute assertions against the response, defined using JSONPath or XPath for HTML. It can even assert that responses were returned within a specified duration.

Via @humphd

Tags: curl, http, rust

New HTTP standards for caching on the modern web

2021-10-21T22:40:50+00:00

New HTTP standards for caching on the modern web

Cache-Status is a new HTTP header (RFC from August 2021) designed to provide better debugging information about which caches were involved in serving a request—“Cache-Status: Nginx; hit, Cloudflare; fwd=stale; fwd-status=304; collapsed; ttl=300” for example indicates that Nginx served a cache hit, then Cloudflare had a stale cached version so it revalidated from Nginx, got a 304 not modified, collapsed multiple requests (dogpile prevention) and plans to serve the new cached value for the next five minutes. Also described is $Target-Cache-Control: which allows different CDNs to respond to different headers and is already supported by Cloudflare and Akamai (Cloudflare-CDN-Cache-Control: and Akamai-Cache-Control:).

Via Hacker News

Tags: caching, dogpile, http, cloudflare

Notes on streaming large API responses

2021-06-25T16:26:49+00:00

I started a Twitter conversation last week about API endpoints that stream large amounts of data as an alternative to APIs that return 100 results at a time and require clients to paginate through all of the pages in order to retrieve all of the data:

Any unexpected downsides to offering streaming HTTP API endpoints that serve up eg 100,000 JSON objects in a go rather than asking users to paginate 100 at a time over 1,000 requests, assuming efficient implementation of that streaming endpoint?
— Simon Willison (@simonw) June 17, 2021

I got a ton of great replies. I tried to tie them together in a thread attached to the tweet, but I'm also going to synthesize them into some thoughts here.

Bulk exporting data

The more time I spend with APIs, especially with regard to my Datasette and Dogsheep projects, the more I realize that my favourite APIs are the ones that let you extract all of your data as quickly and easily as possible.

There are generally three ways an API might provide this:

Click an "export everything" button, then wait for a while for an email to show up with a link to a downloadable zip file. This isn't really an API, in particular since it's usually hard if not impossible to automate that initial "click", but it's still better than nothing. Google's Takeout is one notable implementation of this pattern.
Provide a JSON API which allows users to paginate through their data. This is a very common pattern, although it can run into difficulties: what happens if new data is added while you are paginating through the original data, for example? Some systems only allow access to the first N pages too, for performance reasons.
Providing a single HTTP endpoint you can hit that will return ALL of your data - potentially dozens or hundreds of MBs of it - in one go.

It's that last option that I'm interested in talking about today.

Efficiently streaming data

It used to be that most web engineers would quickly discount the idea of an API endpoint that streams out an unlimited number of rows. HTTP requests should be served as quickly as possible! Anything more than a couple of seconds spent processing a request is a red flag that something should be reconsidered.

Almost everything in the web stack is optimized for quickly serving small requests. But over the past decade the tide has turned somewhat: Node.js made async web servers commonplace, WebSockets taught us to handle long-running connections and in the Python world asyncio and ASGI provided a firm foundation for handling long-running requests using smaller amounts of RAM and CPU.

I've been experimenting in this area for a few years now.

Datasette has the ability to use ASGI trickery to stream all rows from a table (or filtered table) as CSV, potentially returning hundreds of MBs of data.

Django SQL Dashboard can export the full results of a SQL query as CSV or TSV, this time using Django's StreamingHttpResponse (which does tie up a full worker process, but that's OK if you restrict it to a controlled number of authenticated users).

VIAL implements streaming responses to offer an "export from the admin" feature. It also has an API-key-protected search API which can stream out all matching rows in JSON or GeoJSON.

Implementation notes

The key thing to watch out for when implementing this pattern is memory usage: if your server buffers 100MB+ of data any time it needs to serve an export request you're going to run into trouble.

Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON.

Regular JSON requires a bit more thought: you can output a [ character, then output each row in a stream with a comma suffix, then skip the comma for the last row and output a ]. Doing that requires peeking ahead (looping two at a time) to verify that you haven't yet reached the end.

Or... Martin De Wulf pointed out that you can output the first row, then output every other row with a preceeding comma - which avoids the whole "iterate two at a time" problem entirely.

The next challenge is efficiently looping through every database result without first pulling them all into memory.

PostgreSQL (and the psycopg2 Python module) offers server-side cursors, which means you can stream results through your code without loading them all at once. I use these in Django SQL Dashboard.

Server-side cursors make me nervous though, because they seem like they likely tie up resources in the database itself. So the other technique I would consider here is keyset pagination.

Keyset pagination works against any data that is ordered by a unique column - it works especially well against a primary key (or other indexed column). Each page of data is retrieved using a query something like this:

select * from items order by id limit 21

Note the limit 21 - if we are retrieving pages of 20 items we ask for 21, since then we can use the last returned item to tell if there is a next page or not.

Then for subsequent pages take the 20th id value and ask for things greater than that:

select * from items where id > 20 limit 21

Each of these queries is fast to respond (since it's against an ordered index) and uses a predictable, fixed amount of memory. Using keyset pagination we can loop through an abitrarily large table of data, streaming each page out one at a time, without exhausting any resources.

And since each query is small and fast, we don't need to worry about huge queries tying up database resources either.

What can go wrong?

I really like these patterns. They haven't bitten me yet, though I've not deployed them for anything truly huge scale. So I asked Twitter what kind of problems I should look for.

Based on the Twitter conversation, here are some of the challenges that this approach faces.

Challenge: restarting servers

If the stream takes a significantly long time to finish then rolling out updates becomes a problem. You don't want to interrupt a download but also don't want to wait forever for it to finish to spin down the server.
— Adam Lowry (@robotadam) June 17, 2021

This came up a few times, and is something I hadn't considered. If your deployment process involves restarting your servers (and it's hard to imagine one that doesn't) you need to take long-running connections into account when you do that. If there's a user half way through a 500MB stream you can either truncate their connection or wait for them to finish.

Challenge: how to return errors

If you're streaming a response, you start with an HTTP 200 code... but then what happens if an error occurs half-way through, potentially while paginating through the database?

You've already started sending the request, so you can't change the status code to a 500. Instead, you need to write some kind of error to the stream that's being produced.

If you're serving up a huge JSON document, you can at least make that JSON become invalid, which should indicate to your client that something went wrong.

Formats like CSV are harder. How do you let your user know that their CSV data is incomplete?

And what if someone's connection drops - are they definitely going to notice that they are missing something, or will they assume that the truncated file is all of the data?

Challenge: resumable downloads

If a user is paginating through your API, they get resumability for free: if something goes wrong they can start again at the last page that they fetched.

Resuming a single stream is a lot harder.

The HTTP range mechanism can be used to provide resumable downloads against large files, but it only works if you generate the entire file in advance.

There is a way to design APIs to support this, provided the data in the stream is in a predictable order (which it has to be if you're using keyset pagination, described above).

Have the endpoint that triggers the download take an optional ?since= parameter, like this:

GET /stream-everything?since=b24ou34
[
    {"id": "m442ecc", "name": "..."},
    {"id": "c663qo2", "name": "..."},
    {"id": "z434hh3", "name": "..."},
]

Here the b24ou34 is an identifier - it can be a deliberately opaque token, but it needs to be served up as part of the response.

If the user is disconnected for any reason, they can start back where they left off by passing in the last ID that they successfully retrieved:

GET /stream-everything?since=z434hh3

This still requires some level of intelligence from the client application, but it's a reasonably simple pattern both to implement on the server and as a client.

Easiest solution: generate and return from cloud storage

It seems the most robust way to implement this kind of API is the least technically exciting: spin off a background task that generates the large response and pushes it to cloud storage (S3 or GCS), then redirect the user to a signed URL to download the resulting file.

This is easy to scale, gives users complete files with content-length headers that they know they can download (and even resume-downloading, since range headers are supported by S3 and GCS). It also avoids any issues with server restarts caused by long connections.

This is how Mixpanel handle their export feature, and it's the solution Sean Coates came to when trying to find a workaround for the AWS Lambda/API Gate response size limit.

If your goal is to provide your users a robust, reliable bulk-export mechanism for their data, export to cloud storage is probably the way to go.

But streaming dynamic responses are a really neat trick, and I plan to keep exploring them!

Tags: apis, http, scaling, streaming, asgi, http-range-requests

Weeknotes: Archiving coronavirus.data.gov.uk, custom pages and directory configuration in Datasette, photos-to-sqlite

2020-04-29T19:41:11+00:00

I mainly made progress on three projects this week: Datasette, photos-to-sqlite and a cleaner way of archiving data to a git repository.

Archiving coronavirus.data.gov.uk

The UK goverment have a new portal website sharing detailed Coronavirus data for regions around the country, at coronavirus.data.gov.uk.

As with everything else built in 2020, it's a big single-page JavaScript app. Matthew Somerville investigated what it would take to build a much lighter (and faster loading) site displaying the same information by moving much of the rendering to the server.

One of the best things about the SPA craze is that it strongly encourages structured data to be published as JSON files. Matthew's article inspired me to take a look, and sure enough the government figures are available in an extremely comprehensive (and 3.3MB in size) JSON file, available from https://c19downloads.azureedge.net/downloads/data/data_latest.json.

Any time I see a file like this my first questions are how often does it change - and what kind of changes are being made to it?

I've written about scraping to a git repository (see my new gitscraping tag) a bunch in the past:

Scraping hurricane Irma - September 2017
Changelogs to help understand the fires in the North Bay - October 2017
Generating a commit log for San Francisco’s official list of trees - March 2019
Tracking PG&E outages by scraping to a git repo - October 2019
Deploying a data API using GitHub Actions and Cloud Run - January 2020

Now that I've figured out a really clean way to Commit a file if it changed in a GitHub Action knocking out new versions of this pattern is really quick.

simonw/coronavirus-data-gov-archive is my new repo that does exactly that: it periodically fetches the latest versions of the JSON data files powering that site and commits them if they have changed. The aim is to build a commit history of changes made to the underlying data.

The first implementation was extremely simple - here's the entire action:

name: Fetch latest data

on:
push:
repository_dispatch:
schedule:
    - cron:  '25 * * * *'

jobs:
scheduled:
    runs-on: ubuntu-latest
    steps:
    - name: Check out this repo
    uses: actions/checkout@v2
    - name: Fetch latest data
    run: |-
        curl https://c19downloads.azureedge.net/downloads/data/data_latest.json | jq . > data_latest.json
        curl https://c19pub.azureedge.net/utlas.geojson | gunzip | jq . > utlas.geojson
        curl https://c19pub.azureedge.net/countries.geojson | gunzip | jq . > countries.geojson
        curl https://c19pub.azureedge.net/regions.geojson | gunzip | jq . > regions.geojson
    - name: Commit and push if it changed
    run: |-
        git config user.name "Automated"
        git config user.email "actions@users.noreply.github.com"
        git add -A
        timestamp=$(date -u)
        git commit -m "Latest data: ${timestamp}" || exit 0
        git push

It uses a combination of curl and jq (both available in the default worker environment) to pull down the data and pretty-print it (better for readable diffs), then commits the result.

Matthew Somerville pointed out that inefficient polling sets a bad precedent. Here I'm hitting azureedge.net, the Azure CDN, so that didn't particularly worry me - but since I want this pattern to be used widely it's good to provide a best-practice example.

Figuring out the best way to make conditional get requests in a GitHub Action lead me down something of a rabbit hole. I wanted to use curl's new ETag support but I ran into a curl bug, so I ended up rolling a simple Python CLI tool called conditional-get to solve my problem. In the time it took me to release that tool (just a few hours) a new curl release came out with a fix for that bug!

Here's the workflow using my conditional-get tool. See the issue thread for all of the other potential solutions, including a really neat Action shell-script solution by Alf Eaton.

To my absolute delight, the project has already been forked once by Daniel Langer to capture Canadian Covid-19 cases!

New Datasette features

I pushed two new features to Datasette master, ready for release in 0.41.

Configuration directory mode

This is an idea I had while building datasette-publish-now. Datasette instances can be run with custom metadata, custom plugins and custom templates. I'm increasingly finding myself working on projects that run using something like this:

$ datasette data1.db data2.db data3.db \
    --metadata=metadata.json
    --template-dir=templates \
    --plugins-dir=plugins

Directory configuration mode introduces the idea that Datasette can configure itself based on a directory layout. The above example can instead by handled by creating the following layout:

my-project/data1.db
my-project/data2.db
my-project/data3.db
my-project/metadatata.json
my-project/templates/index.html
my-project/plugins/custom_plugin.py

Then run Datasette directly targetting that directory:

$ datasette my-project/

See issue #731 for more details. Directory configuration mode is documented here.

Define custom pages using templates/pages

In niche-museums.com, powered by Datasette I described how I built the www.niche-museums.com website as a heavily customized Datasette instance.

That site has /about and /map pages which are served by custom templates - but I had to do some gnarly hacks with empty about.db and map.db files to get them to work.

Issue #648 introduces a new mechanism for creating this kind of page: create a templates/pages/map.html template file and custom 404 handling code will ensure that any hits to /map serve the rendered contents of that template.

This could work really well with the datasette-template-sql plugin, which allows templates to execute abritrary SQL queries (ala PHP or ColdFusion).

Here's the new documentation on custom pages, including details of how to use the new custom_status(), custom_header() and custom_redirect() template functions to go beyond just returning HTML.

photos-to-sqlite

My Dogsheep personal analytics project brings my tweets, GitHub activity, Swarm checkins and more together in one place. But the big missing feature is my photos.

As-of yesterday, I have 39,000 photos from Apple Photos uploaded to an S3 bucket using my new photos-to-sqlite tool. I can run the following SQL query and get back ten random photos!

select
  json_object(
    'img_src',
    'https://photos.simonwillison.net/i/' || 
    sha256 || '.' || ext || '?w=400'
  ),
  filepath,
  ext
from
  photos
where
  ext in ('jpeg', 'jpg', 'heic')
order by
  random()
limit
  10

photos.simonwillison.net is running a modified version of my heic-to-jpeg image converting and resizing proxy, which I'll release at some point soon.

There's still plenty of work to do - I still need to import EXIF data (including locations) into SQLite, and I plan to use osxphotos to export additional metadata from my Apple Photos library. But this week it went from a pure research project to something I can actually start using, which is exciting.

TIL this week

Generated using this query.

Tags: git, http, matthew-somerville, photos, projects, datasette, weeknotes, covid19, git-scraping

Async Support - HTTPX

2020-01-10T04:49:59+00:00

Async Support - HTTPX

HTTPX is the new async-friendly HTTP library for Python spearheaded by Tom Christie. It works in both async and non-async mode with an API very similar to requests. The async support is particularly interesting - it's a really clean API, and now that Jupyter supports top-level await you can run (await httpx.AsyncClient().get(url)).text directly in a cell and get back the response. Most excitingly the library lets you pass an ASGI app directly to the client and then perform requests against it - ideal for unit tests.

Via @_tomchristie

Tags: async, http, python, asgi, kim-christie, httpx

Usage of ARIA attributes via HTTP Archive

2018-07-12T03:16:26+00:00

Usage of ARIA attributes via HTTP Archive

A neat example of a Google BigQuery query you can run against the HTTP Archive public dataset (a crawl of the “top” websites run periodically by the Internet Archive, which captures the full details of every resource fetched) to see which ARIA attributes are used the most often. Linking to this because I used it successfully today as the basis for my own custom query—I love that it’s possible to analyze a huge representative sample of the modern web in this way.

Tags: aria, http, internet-archive, big-data

Cookies-over-HTTP Bad

2018-04-07T14:39:06+00:00

Cookies-over-HTTP Bad

Mike West from the Chrome security team proposes a way for browsers to start discouraging the use of tracking cookies sent over HTTP—which represent a significant threat to user privacy from network attackers. It’s a clever piece of thinking: browsers would slowly ramp up the forced expiry deadline for non-HTTPS cookies, further encouraging sites to switch to HTTPS cookies while giving them ample time to adapt.

Via @mikewest

Tags: cookies, http, https, privacy