Replaying logs to exercise the new API
3rd March 2021
Originally posted to my internal blog at VaccinateCA
22 days ago n1mmy pushed a change to
help.vaccinate which logged full details of inoming Netlify function API traffic to an Airtable database.
What an asset that is! The Airtable table over here currently contains over 9,000 logged API calls, including the full JSON POST body, when the call was receieved and which authenticated user made the call.
This morning I exported that data as CSV from Airtable, and wrote a Python script to replay those requests against my new imitation implementation of the API.
Here’s what that script looks like running against my localhost development server:
You can track the work in this issue—the replay script helped me get to a place where every single report from the past 22 days can be safely ingested by the new API, with the exception of a tiny number of reports against locations which have since been deleted (which isn’t supposed to happen—we try to soft-delete rather than full-delete things—but apparently a few deletes had slipped through).
Since the Airtable API logs have so clearly proved their value, Jesse proposed using the same trick for the Django app. I implemented that today: the full incoming request body and outgoing response are now recorded in an
ApiLog model in Django. You can see those in the Django admin here: https://vaccinateca-preview.herokuapp.com/admin/api/apilog/
Unit tests for the new API
I added tests for the
submitReport API. The tests are driven by example JSON fixtures—so far I’ve created two of those, but I hope that having them in this format will make it really easy to add more as we find edge-cases in the API and expand it with new features.
Documentation for the new API
I wrote API documentation! You can find that here.
Now that the API is documented I intend to update the documentation in lock-step with changes made to the API itself—using the pattern where every commit includes the change, the tests for the change AND the documentation for the change in a single unit.
Dual-writing to Django
The combination of the replay script and the unit tests has left me feeling pretty confident that the replacement API is ready to start accepting traffic.
The plan is to run the new system in parallel with Airtable for a few days to thoroughly test it and make sure it covers everything we need. Our Netlify functions offer a great place to do this, so this afternoon I submitted a pull request to
help.vaccinate to silently dual-write incoming API requests to the Django API, catching and logging any exceptions without intefering with the rest of the application flow.
Testing this locally helped me identify some bugs in the way the Django app verified JWT tokens that originated with the
Here are my other commits from today.
More recent articles
- Datasette Enrichments: a new plugin framework for augmenting your data - 1st December 2023
- llamafile is the new best way to run a LLM on your own computer - 29th November 2023
- Prompt injection explained, November 2023 edition - 27th November 2023
- I'm on the Newsroom Robots podcast, with thoughts on the OpenAI board - 25th November 2023
- Weeknotes: DevDay, GitHub Universe, OpenAI chaos - 22nd November 2023
- Deciphering clues in a news article to understand how it was reported - 22nd November 2023
- Exploring GPTs: ChatGPT in a trench coat? - 15th November 2023
- Financial sustainability for open source projects at GitHub Universe - 10th November 2023
- ospeak: a CLI tool for speaking text in the terminal via OpenAI - 7th November 2023
- DALL-E 3, GPT4All, PMTiles, sqlite-migrate, datasette-edit-schema - 30th October 2023