Tracking FARA by deploying a data API using GitHub Actions and Cloud Run
21st January 2020
I’m using the combination of GitHub Actions and Google Cloud Run to retrieve data from the U.S. Department of Justice FARA website and deploy it as a queryable API using Datasette.
FARA background
The Foreign Agents Registration Act (FARA) law that requires “certain agents of foreign principals who are engaged in political activities or other activities specified under the statute to make periodic public disclosure of their relationship with the foreign principal, as well as activities, receipts and disbursements in support of those activities”.
The law was introduced in 1938 in response to the large number of German propaganda agents that were operating in the U.S. prior to the war.
Basically, if you are in the United States as a lobbyist for a foreign government you need to register under FARA. It was used in 23 criminal cases during World War II, but hasn’t had much use since it was ammended in 1966. Although... if you consult the list of recent cases you’ll see some very interesting recent activity involving Russia and Ukraine.
It’s also for spies! Quoting the FARA FAQ:
Finally, 50 U.S.C. § 851, requires registration of persons who have knowledge of or have received instruction or assignment in espionage, counterespionage or sabotage service or tactics of a foreign country or political party.
I imagine most spies operate in violation of this particular law and don’t take steps to register themselves.
It’s all still pretty fascinating though, in part because it gets updated. A lot. Almost every business day in fact.
Tracking FARA history
I know this because seven months ago I set up a scraper for it. Every twelve hours I have code which downloads the four bulk CSVs published by the Justice department and saves them to a git repository. It’s the same trick I’ve been using to track San Francisco’s database of trees and PG&E’s outage map.
I’ve been running the scraper using Circle CI, but this weekend I decided to switch it over to GitHub Actions to get a better idea for how they work.
Deploying it as an API
I also wanted to upgrade my script to also deploy a fresh Datasette instance of the data using Google Cloud Run. I wrote a script to do this on a manual basis last year, but I never combined it with the daily scraper. Combining the two means I can offer a Datasette-powered API directly against the latest data.
https://fara.datasettes.com is that API—it now updates twice a day, assuming there are some changes to the underlying data.
Putting it all together
The final GitHub action workflow can be seen here. I’m going to present an annotated version here.
on:
repository_dispatch:
schedule:
- cron: '0 0,12 * * *'
This sets when the workflow should be triggered. I’m running it twice a day—at midnight and noon UTC (the 0,12 cron syntax).
The repository_dispatch
key means I can also trigger it manually by running the following curl
command—useful for testing:
curl -XPOST https://api.github.com/repos/simonw/fara-history/dispatches \
-H 'Authorization: token MY_PERSONAL_TOKEN_HERE' \
-d '{"event_type": "trigger_action"}' \
-H 'Accept: application/vnd.github.everest-preview+json'
Next comes the job itself, which I called scheduled
and set to run on the latest Ubuntu:
jobs:
scheduled:
runs-on: ubuntu-latest
steps:
Next comes the steps. Each step is run in turn, in an isolated process (presumably a container) but with access to the current working directory.
- uses: actions/checkout@v2
name: Check out repo
- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: 3.8
The first two steps checkout the fara-history
repository and install Python 3.8.
- uses: actions/cache@v1
name: Configure pip caching
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
This step should set up a cache so that pip
doesn’t have to download fresh dependencies on every run. Unfortunately it doesn’t seem to actually work—it only works for push
and pull_request
events, but my workflow is triggered by schedule
and repository_dispatch
. There’s an open issue about this.
- name: Install Python dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
This step installs my dependencies from requirements.txt.
- name: Fetch, update and commit FARA data
run: . update_and_commit_all.sh
- name: Build fara.db database
run: python build_database.py
Now we’re getting to the fun stuff. My update_and_commit_all.sh script downloads the four zip files from the FARA.gov site, unzips them, sorts them, diffs them against the previously stored files and commits the new copy to GitHub if they have changed. See my explanation of csv-diff for more on this—though sadly only one of the files has a reliable row identifier so I can’t generate great commit messages for most of them.
My build_database.py script uses sqlite-utils to convert the CSV files into a SQLite database.
Now that we’ve got a SQLite database, we can deploy it to Google Cloud Run using Datasette.
But should we run a deploy at all? If the database hasn’t changed, there’s no point in deploying it. How can we tell if the database file has changed from the last one that was published?
Datasette has a mechanism for deriving a content hash of a database, part of a performance optimization which is no longer turned on by default and may be removed in the future.
You can generate JSON that includes hash using the datasette inspect
command. The jq tool can then be used to extract out just the hash:
$ datasette inspect fara.db | jq '.fara.hash' -r
fbc9cbaca6de1e232fc14494faa06cc8d4cb9f379d0d568e4711e9a218800906
The -r
option to jq
causes it to return just the raw string, without quote marks.
Datasette’s /-/databases.json
introspection URL reveals the hashes of the currently deployed database. Here’s how to pull the currently deployed hash:
$ curl -s https://fara.datasettes.com/-/databases.json | jq '.[0].hash' -r
a6c0ab26589bde0d225c5a45044e0adbfa3840b95fbb263d01fd8fb0d2460ed5
If those two hashes differ then we should deploy the new database.
GitHub Actions have a slightly bizarre mechanism for defining “output variables” for steps, which can then be used to conditionally run further steps.
Here’s the step that sets those variables, followed by the step that conditionally installs the Google Cloud CLI tools using their official action:
- name: Set variables to decide if we should deploy
id: decide_variables
run: |-
echo "##[set-output name=latest;]$(datasette inspect fara.db | jq '.fara.hash' -r)"
echo "##[set-output name=deployed;]$(curl -s https://fara.datasettes.com/-/databases.json | jq '.[0].hash' -r)"
- name: Set up Cloud Run
if: steps.decide_variables.outputs.latest != steps.decide_variables.outputs.deployed
uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
with:
version: '275.0.0'
service_account_email: ${{ secrets.GCP_SA_EMAIL }}
service_account_key: ${{ secrets.GCP_SA_KEY }}
Having installed the Google Cloud tools, I can deploy my database using Datasette:
- name: Deploy to Cloud Run
if: steps.decide_variables.outputs.latest != steps.decide_variables.outputs.deployed
run: |-
gcloud components install beta
gcloud config set run/region us-central1
gcloud config set project datasette-222320
datasette publish cloudrun fara.db --service fara-history -m metadata.json
This was by far the hardest part to figure out.
First, I needed to create a Google Cloud service account with an accompanying service key.
I tried and failed to do this using the CLI, so I switched to their web console following these and then these instructions.
Having downloaded the key JSON file, I converted it to base64 and pasted it into a GitHub Actions secret (hidden away in the repository settings area) called GCP_SA_KEY
.
cat ~/Downloads/datasette-222320-2ad02afe6d82.json \
| base64 | pbcopy
The service account needed permissions in order to run a build through Cloud Build and then deploy the result through Cloud Run. I spent a bunch of time trying out different combinations and eventually gave up and gave the account “Editor” permissions across my entire project. This is bad. I am hoping someone can help me understand what the correct narrow set of permissions are, and how to apply them.
It also took me a while to figure out that I needed to run these three commands before I could deploy to my project. The first one installs the Cloud Run tools, the second set up some required configuration:
gcloud components install beta
gcloud config set run/region us-central1
gcloud config set project datasette-222320
But... having done all of the above, the following command run from an action successfully deploys the site!
datasette publish cloudrun fara.db \
--service fara-history -m metadata.json
DNS
Google Cloud Run deployments come with extremely ugly default URLs. For this project, that URL is https://fara-history-j7hipcg4aq-uc.a.run.app/
.
I wanted something nicer. I own datasettes.com
and manage the DNS via Cloudflare, which means I can point subdomains at Cloud Run instances.
This is a two-step process
- I set
fara.datasettes.com
as a DNS-only (no proxying) CNAME forghs.googlehosted.com
. - In the Google Cloud Console I used Cloud Run -> Manage Custom Domains (a button in the header) -> Add Mapping to specify that
fara.datasettes.com
should map to myfara-history
service (the--service
argument fromdatasette publish
earlier).
I had previously verified my domain ownership—I forget quite how I did it. Domains purchased through Google Domains get to skip this step.
Next steps
This was a lot of fiddling around. I’m hoping that by writing this up in detail I’ll be able to get this working much faster next time.
I think this model—GitHub Actions that pull data, build a database and deploy to Cloud Run using datasette publish
—is incredibly promising. The end result should be an API that costs cents-to-dollars a month to operate thanks to Cloud Run’s scale-to-zero architecture. And hopefully by publishing this all on GitHub it will be as easy as possible for other people to duplicate it for their own projects.
More recent articles
- I can now run a GPT-4 class model on my laptop - 9th December 2024
- Prompts.js - 7th December 2024
- First impressions of the new Amazon Nova LLMs (via a new llm-bedrock plugin) - 4th December 2024