Entries
Filters: Sorted by date
Weeknotes: sqlite-utils updates, Datasette and asgi-csrf, open-sourcing VIAL
Some work on sqlite-utils, asgi-csrf, a Datasette alpha and we open-sourced VIAL.
Notes on streaming large API responses
I started a Twitter conversation last week about API endpoints that stream large amounts of data as an alternative to APIs that return 100 results at a time and require clients to paginate through all of the pages in order to retrieve all of the data:
[... 1,692 words]Joining CSV and JSON data with an in-memory SQLite database
The new sqlite-utils memory command can import CSV and JSON data directly into an in-memory SQLite database, combine and query it using SQL and output the results as CSV, JSON or various other formats of plain text tables.
Weeknotes: New releases across nine different projects
A new release and security patch for Datasette, plus releases of sqlite-utils, datasette-auth-passwords, django-sql-dashboard, datasette-upload-csvs, xml-analyser, datasette-placekey, datasette-mask-columns and db-to-sqlite.
[... 861 words]Weeknotes: Docker architectures, sqlite-utils 3.7, nearly there with Datasette 0.57
This week I learned a whole bunch about using Docker to emulate different architectures, released sqlite-utils 3.7 and made a ton of progress towards the almost-ready-to-ship Datasette 0.57.
[... 1,081 words]Weeknotes: Spinning back up on Datasette
I’ve been somewhat distracted from Datasette for the past couple of months, thanks to my work on VIAL and the accompanying open source project django-sql-dashboard. This week I scraped back some time to work on Datasette.
[... 401 words]Weeknotes: Velma, more Django SQL Dashboard
Matching locations for Vaccinate The States, fun with GeoJSON and more improvements to Django SQL Dashboard.
[... 555 words]Django SQL Dashboard
I’ve released the first non-alpha version of Django SQL Dashboard, which provides an interface for running arbitrary read-only SQL queries directly against a PostgreSQL database, protected by the Django authentication scheme. It can also be used to create saved dashboards that can be published or shared internally.
[... 2,171 words]Adding GeoDjango to an existing Django project
Work on VIAL for Vaccinate The States continues.
[... 1,503 words]One year of TILs
Just over a year ago I started tracking TILs, inspired by Josh Branchaud’s collection. I’ve since published 148 TILs across 43 different topics. It’s a great format!
[... 224 words]A CSV export, JSON import workflow for bulk updating our data
I just added missing counties to around 1200 of our locations using a combination of tricks, and I thought they’d make a good blog post.
[... 1,429 words]Weeknotes: Vaccinate The States, and how I learned that returning dozens of MB of JSON works just fine these days
On Friday VaccinateCA grew in scope, a lot: we launched a new website called Vaccinate The States. Patrick McKenzie wrote more about the project here—the short version is that we’re building the most comprehensive possible dataset of vaccine availability in the USA, using a combination of data collation, online research and continuing to make a huge number of phone calls.
[... 1,109 words]Porting VaccinateCA to Django
As I mentioned back in February, I’ve been working with the VaccinateCA project to try to bring the pandemic to an end a little earlier by helping gather as accurate a model as possible of where the Covid vaccine is available in California and how people can get it.
[... 2,157 words]Animated choropleth of vaccinations by US county
Last week I mentioned that I’ve recently started scraping and storing the CDC’s per-county vaccination numbers in my cdc-vaccination-history GitHub repository. This week I used an Observable notebook and d3’s TopoJSON support to render those numbers on an animated choropleth map.
[... 1,138 words]VIAL: Preparing for some collaborative testing
With the Airtable limits fast approaching, I’m going to start leaning heavily on people to help verify that VIAL can do the jobs that it needs to do.
[... 465 words]Weeknotes: SpatiaLite 5, Datasette on Azure, more CDC vaccination history
This week I got SpatiaLite 5 working in the Datasette Docker image, improved the CDC vaccination history git scraper, figured out Datasette on Azure and we closed on a new home!
[... 986 words]The Airtable formulas at the heart of everything
While working on building a Counties.json API endpoint for VIAL I realized I wasn’t entirely sure how the “Total reports” and “Yeses” numbers in this piece of JSON were calculated:
[... 323 words]Weeknotes: django-sql-dashboard widgets
A few small releases this week, for django-sql-dashboard, datasette-auth-passwords and datasette-publish-vercel.
VIAL is now live, plus django-sql-dashboard
Our new Django backend has now officially graduated from preview mode! We’ve been running it to collect caller reports for Oregon for over a week now, and today we finally turned off the old Heroku app and promoted https://vial.calltheshots.us/ to be the place that our caller app writes to.
[... 672 words]Weeknotes: tableau-to-sqlite, django-sql-dashboard
This week I started a limited production run of my new backend for Vaccinate CA calling, built a tableau-to-sqlite import tool and started working on a subset of Datasette for PostgreSQL and Django called django-sql-dashboard.
APIs for importing locations
An important aspect of the new backend is the ability to import new locations.
[... 254 words]New call queue ready to test. Also geography.
I just shipped the first working preview version of the new /api/requestCall API [#54]—the API that our caller app uses to get the next location that the caller should be contacting (and lock it so that other users don’t call it at the same time).
Weeknotes: Datasette and Git scraping at NICAR, VaccinateCA
This week I virtually attended the NICAR data journalism conference and made a ton of progress on the Django backend for VaccinateCA (see last week).
[... 773 words]The simplest possible call queue
Today I’ve been working on the queue calling mechanism for the new Django backend.
[... 471 words]Git scraping, the five minute lightning talk
I prepared a lightning talk about Git scraping for the NICAR 2021 data journalism conference. In the talk I explain the idea of running scheduled scrapers in GitHub Actions, show some examples and then live code a new scraper for the CDC’s vaccination data using the GitHub web interface. Here’s the video.
[... 289 words]Replaying logs to exercise the new API
22 days ago n1mmy pushed a change to help.vaccinate which logged full details of inoming Netlify function API traffic to an Airtable database.
API ready for testing, first video status update
To celebrate GitHub’s new support for drag-and-dropping mp4 files into Markdown, I decided to switch things around a little and include a video update with today’s blog entry.
[... 656 words]Drawing the rest of the owl
My ambitious goal for the day was to get the new Django/PostgreSQL preview into a state where we could start sending example API requests to it from the help.vaccinate app.
Trying to end the pandemic a little earlier with VaccinateCA
This week I got involved with the VaccinateCA effort. We are trying to end the pandemic a little earlier, by building the most accurate database possible of vaccination locations and availability in California.
[... 1,154 words]