Replaying logs to exercise the new API
Originally posted to my internal blog at VaccinateCA
22 days ago n1mmy pushed a change to
help.vaccinate which logged full details of inoming Netlify function API traffic to an Airtable database.
What an asset that is! The Airtable table over here currently contains over 9,000 logged API calls, including the full JSON POST body, when the call was receieved and which authenticated user made the call.
This morning I exported that data as CSV from Airtable, and wrote a Python script to replay those requests against my new imitation implementation of the API.
Here’s what that script looks like running against my localhost development server:
You can track the work in this issue—the replay script helped me get to a place where every single report from the past 22 days can be safely ingested by the new API, with the exception of a tiny number of reports against locations which have since been deleted (which isn’t supposed to happen—we try to soft-delete rather than full-delete things—but apparently a few deletes had slipped through).
Since the Airtable API logs have so clearly proved their value, Jesse proposed using the same trick for the Django app. I implemented that today: the full incoming request body and outgoing response are now recorded in an
ApiLog model in Django. You can see those in the Django admin here: https://vaccinateca-preview.herokuapp.com/admin/api/apilog/
Here’s the ORM model and the view decorator that logs requests.
Unit tests for the new API
I added tests for the
submitReport API. The tests are driven by example JSON fixtures—so far I’ve created two of those, but I hope that having them in this format will make it really easy to add more as we find edge-cases in the API and expand it with new features.
Those API test fixtures live in vaccinate/api/test-data/submitReport. Here’s the test code that executes them.
Documentation for the new API
I wrote API documentation! You can find that here.
Now that the API is documented I intend to update the documentation in lock-step with changes made to the API itself—using the pattern where every commit includes the change, the tests for the change AND the documentation for the change in a single unit.
Dual-writing to Django
The combination of the replay script and the unit tests has left me feeling pretty confident that the replacement API is ready to start accepting traffic.
The plan is to run the new system in parallel with Airtable for a few days to thoroughly test it and make sure it covers everything we need. Our Netlify functions offer a great place to do this, so this afternoon I submitted a pull request to
help.vaccinate to silently dual-write incoming API requests to the Django API, catching and logging any exceptions without intefering with the rest of the application flow.
Testing this locally helped me identify some bugs in the way the Django app verified JWT tokens that originated with the
Here are my other commits from today.
More recent articles
- ChatGPT should include inline tips - 30th May 2023
- Lawyer cites fake cases invented by ChatGPT, judge is not amused - 27th May 2023
- llm, ttok and strip-tags - CLI tools for working with ChatGPT and other LLMs - 18th May 2023
- Delimiters won't save you from prompt injection - 11th May 2023
- Weeknotes: sqlite-utils 3.31, download-esm, Python in a sandbox - 10th May 2023
- Leaked Google document: "We Have No Moat, And Neither Does OpenAI" - 4th May 2023
- Midjourney 5.1 - 4th May 2023
- Prompt injection explained, with video, slides, and a transcript - 2nd May 2023
- download-esm: a tool for downloading ECMAScript modules - 2nd May 2023
- Let's be bear or bunny - 1st May 2023