Simon Willison’s Weblog

Replaying logs to exercise the new API

3rd March 2021

Originally posted to my internal blog at VaccinateCA

22 days ago n1mmy pushed a change to help.vaccinate which logged full details of inoming Netlify function API traffic to an Airtable database.

What an asset that is! The Airtable table over here currently contains over 9,000 logged API calls, including the full JSON POST body, when the call was receieved and which authenticated user made the call.

This morning I exported that data as CSV from Airtable, and wrote a Python script to replay those requests against my new imitation implementation of the API.

Here’s what that script looks like running against my localhost development server:

You can track the work in this issue—the replay script helped me get to a place where every single report from the past 22 days can be safely ingested by the new API, with the exception of a tiny number of reports against locations which have since been deleted (which isn’t supposed to happen—we try to soft-delete rather than full-delete things—but apparently a few deletes had slipped through).

API logging

Since the Airtable API logs have so clearly proved their value, Jesse proposed using the same trick for the Django app. I implemented that today: the full incoming request body and outgoing response are now recorded in an ApiLog model in Django. You can see those in the Django admin here: https://vaccinateca-preview.herokuapp.com/admin/api/apilog/

Here’s the ORM model and the view decorator that logs requests.

Unit tests for the new API

I added tests for the submitReport API. The tests are driven by example JSON fixtures—so far I’ve created two of those, but I hope that having them in this format will make it really easy to add more as we find edge-cases in the API and expand it with new features.

Those API test fixtures live in vaccinate/api/test-data/submitReport. Here’s the test code that executes them.

Documentation for the new API

I wrote API documentation! You can find that here.

Now that the API is documented I intend to update the documentation in lock-step with changes made to the API itself—using the pattern where every commit includes the change, the tests for the change AND the documentation for the change in a single unit.

Dual-writing to Django

The combination of the replay script and the unit tests has left me feeling pretty confident that the replacement API is ready to start accepting traffic.

The plan is to run the new system in parallel with Airtable for a few days to thoroughly test it and make sure it covers everything we need. Our Netlify functions offer a great place to do this, so this afternoon I submitted a pull request to help.vaccinate to silently dual-write incoming API requests to the Django API, catching and logging any exceptions without intefering with the rest of the application flow.

Testing this locally helped me identify some bugs in the way the Django app verified JWT tokens that originated with the help.vaccinate application.

Everything else

Here are my other commits from today.