TILs

TIL Google OAuth for a CLI application — > :warning: The OOB flow described in this document has been [scheduled for deprecation](https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html?m=1#disallowed-oo) by Google. See [issue #39](https://github.com/simonw/google-drive-to-sqlite/issues/39) for notes on how to replace it.

17th Feb 2022, 4:30 am

TIL Combining substr and instr to extract text — Derek Willis has [a Datasette instance full of political campaign emails](https://political-emails.herokuapp.com/) running on Heroku.

16th Feb 2022, 2:05 am

TIL nullglob in bash — I ran into a tricky problem while working [on this issue](https://github.com/simonw/datasette-publish-fly/issues/17): the following line was behaving in an unexpected way for me:

15th Feb 2022, 5:16 am

TIL Enabling a user to execute a specific command as root without a password — I wanted a script running as a non-root user to be able to restart a systemd service on my Ubuntu machine without needing a password.

13th Feb 2022, 7:53 pm

TIL Annotated package.json for idb-keyval — Jake Archibald [pointed to](https://twitter.com/jaffathecake/status/1491771402294370306) his `package.json` for `idb-keyval` as a "modernish example" of NPM packaging on Twitter this morning.

11th Feb 2022, 1:32 am

TIL Ordered group_concat() in SQLite — I was trying to use `group_concat()` to glue together some column values into a stiched together Markdown document. My first attempt looked like this:

6th Feb 2022, 11:08 pm

TIL Testing against Python 3.11 preview using GitHub Actions — I decided to run my CI tests against the Python 3.11 preview, to avoid the problem I had when Python 3.10 came out with [a bug that affected Datasette](https://simonwillison.net/2021/Oct/9/finding-and-reporting-a-bug/).

2nd Feb 2022, 10:20 pm

TIL get-graphql-schema — The GraphQL schema language is a concise way to represent the available schema provided by a GraphQL endpoint. It looks something like this:

1st Feb 2022, 10:55 pm

TIL Opt-in integration tests with pytest --integration — For both [s3-credentials](https://github.com/simonw/s3-credentials) and [datasette-publish-fly](https://github.com/simonw/datasette-publish-fly) I have a need for real-world integration tests that actually interact with the underlying APIs (AWS or Fly) to create and destroy resources on those platforms.

26th Jan 2022, 9:59 pm

TIL Using the undocumented Fly GraphQL API — [Fly](https://fly.io/) has a GraphQL API which is used by some of their own tools - I found it while [browsing around their code](https://github.com/superfly/flyctl/blob/603b0adccf5416188eabaa7dc73f9c0ec88fa6ca/api/resource_volumes.go#L5-L40) on GitHub.

21st Jan 2022, 10:59 pm

TIL Linking from /latest/ to /stable/ on Read The Docs — [Read The Docs](https://readthedocs.org/) has a handy feature where documentation for older versions will automatically link to the latest release, for example [on this page](https://docs.datasette.io/en/0.56/spatialite.html):

20th Jan 2022, 11:26 pm

TIL Promoting the stable version of the documentation using rel=canonical — I was thinking about documentation SEO today. Like many projects, Datasette offers multiple versions of the documentation:

20th Jan 2022, 11:15 pm

TIL Pixel editing a favicon with Pixelmator — I wanted to [add a favicon](https://github.com/simonw/datasette/issues/1603) to Datasette, using a PNG image served from `/favicon.ico` as suggested in [this article by Adam Johnson](https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/).

20th Jan 2022, 7:15 pm

TIL Helper function for pagination using AWS boto3 — I noticed that a lot of my boto3 code in [s3-credentials](https://github.com/simonw/s3-credentials) looked like this:

19th Jan 2022, 7:49 pm

TIL json_extract() path syntax in SQLite — Several of the [SQLite JSON functions](), such as `json_extract()` and `json_array_length()`, take a path argument. This uses custom syntax along the lines of `$.a[2].b`, as described [in the documentation here](https://sqlite.org/json1.html#path_arguments).

18th Jan 2022, 11 pm

TIL Streaming indented output of a JSON array — I wanted to produce the following output from a command-line tool:

17th Jan 2022, 11:52 pm

TIL JavaScript date objects — A few notes on JavaScript `Date` object, based on trying to do some basic things with them in Observable notebooks.

16th Jan 2022, 8:16 pm

TIL Configuring Dependabot for a Python project — GitHub's Dependabot can automatically file PRs with bumps to dependencies when new versions of them are available.

15th Jan 2022, 12:33 am

TIL Testing a Click app with streaming input — For [sqlite-utils#364](https://github.com/simonw/sqlite-utils/issues/364) I needed to write a test for a [Click](https://click.palletsprojects.com/) app which dealt with input streamed to standard input. I needed to run some assertions during that process, which ruled out the usual [CliRunner.invoke()](https://click.palletsprojects.com/en/8.0.x/testing/) testing tool since that works by running the command until completion.

10th Jan 2022, 4:47 am

TIL Writing pytest tests against tools written with argparse — I usually build command-line tools using [Click](https://click.palletsprojects.com/) (and my [click-app](https://github.com/simonw/click-app) cookiecutter template), which includes a really nice [set of tools](https://click.palletsprojects.com/en/8.0.x/testing/) for writing tests.

8th Jan 2022, 8:42 pm

TIL Adding a CORS policy to an S3 bucket — Amazon S3 buckets that are configured to work as public websites can support CORS, allowing assets such as JavaScript modules to be loaded by JavaScript running on other domains.

4th Jan 2022, 11:42 pm

TIL WebAuthn browser support — I [started exploring](https://twitter.com/simonw/status/1476249939516616704) **[WebAuthn](https://webauthn.guide/)** today - a set of browser standards that adds support for both Yubikey 2FA hardware devices and "platform" authentication using things like Touch ID and Face ID.

29th Dec 2021, 10:15 pm

TIL kubectl proxy — Learned about this today as a way of accessing the Kubernetes REST API.

29th Dec 2021, 1:06 am

TIL Transferring a GitHub issue from a private to a public repository — I have my own private `notes` repository where I sometimes create research threads. Occasionally I want to transfer these to a public repository to publish their contents.

23rd Dec 2021, 1:46 am

TIL Adding a robots.txt using Cloudflare workers — I got an unexpected traffic spike to https://russian-ira-facebook-ads.datasettes.com/ - which runs on Cloud Run - and decided to use `robots.txt` to block crawlers.

21st Dec 2021, 11:07 pm

TIL Annotated explanation of David Beazley's dataklasses — David Beazley [on Twitter](https://twitter.com/dabeaz/status/1472742536649351173):

20th Dec 2021, 4:30 am

TIL Safely outputting JSON — Carelessly including the output of `json.dumps()` in an HTML page can lead to an XSS hole, thanks to the following:

18th Dec 2021, 1:06 am

TIL Creating a minimal SpatiaLite database with Python — When writing a test for [datasette-leaflet-freedraw](https://github.com/simonw/datasette-leaflet-freedraw) I realized I didn't have a simple tiny recipe for creating an in-memory SpatiaLite database in Python. I came up with this:

17th Dec 2021, 11:57 pm · leaflet

TIL Registering the same Pluggy hook multiple times in a single file — I found myself wanting to register more than one instance of a [Pluggy](https://pluggy.readthedocs.io/) plugin hook inside a single module.

16th Dec 2021, 11:16 pm

TIL Using lsof on macOS — `lsof` stands for "list open files". Here are some very basic usage notes for the version that ships with macOS.

11th Dec 2021, 10:37 pm

Simon Willison’s Weblog

TILs

Years

Tags