Entries
Filters: Sorted by date
Weeknotes: Datasette Lite, nogil Python, HYTRADBOI
My big project this week was Datasette Lite, a new way to run Datasette directly in a browser, powered by WebAssembly and Pyodide. I also continued my research into running SQL queries in parallel, described last week. Plus I spoke at HYTRADBOI.
[... 1,434 words]Datasette Lite: a server-side Python web application running in a browser
Datasette Lite is a new way to run Datasette: entirely in a browser, taking advantage of the incredible Pyodide project which provides Python compiled to WebAssembly plus a whole suite of useful extras.
[... 4,800 words]Automatically opening issues when tracked file content changes
I figured out a GitHub Actions pattern to keep track of a file published somewhere on the internet and automatically open a new repository issue any time the contents of that file changes.
[... 1,211 words]Weeknotes: Parallel SQL queries for Datasette, plus some middleware tricks
A promising new performance optimization for Datasette, plus new datasette-gzip and datasette-total-page-time plugins.
Useful tricks with pip install URL and GitHub
The pip install command can accept a URL to a zip file or tarball. GitHub provides URLs that can create a zip file of any branch, tag or commit in any repository. Combining these is a really useful trick for maintaining Python packages.
Building a Covid sewage Twitter bot (and other weeknotes)
I built a new Twitter bot today: @covidsewage. It tweets a daily screenshot of the latest Covid sewage monitoring data published by Santa Clara county.
[... 1,079 words]Pillar Point Stewards, pypi-to-sqlite, improvements to shot-scraper and appreciating datasette-dashboards
This week I helped Natalie launch the Pillar Point Stewards website and built a new tool for loading PyPI package data into SQLite, in order to help promote the excellent datasette-dashboards plugin by Romain Clement.
[... 1,985 words]Weeknotes: datasette-auth0
Datasette 0.61, a Twitter Space and a new Datasette plugin for authenticating against Auth0.
[... 957 words]Datasette 0.61: The annotated release notes
I released Datasette 0.61 this morning—closely followed by 0.61.1 to fix a minor bug. Here are the annotated release notes.
[... 1,465 words]SQLite Happy Hour—a Twitter Spaces conversation about three interesting projects building on SQLite
Yesterday I hosted SQLite Happy Hour. my first conversation using Twitter Spaces. The idea was to dig into three different projects that were doing interesting things on top of SQLite. I think it worked pretty well, and I’m curious to explore this format more in the future.
[... 1,998 words]Weeknotes: Tildes not dashes, and the big refactor
After last week’s shot-scraper distractions with Playwright, this week I finally managed to make some concrete progress on the path towards Datasette 1.0.
[... 1,292 words]Instantly create a GitHub repository to take screenshots of a web page
I just released shot-scraper-template, a GitHub repository template that helps you start taking automated screenshots of a web page by filling out a form.
[... 1,177 words]Scraping web pages from the command line with shot-scraper
I’ve added a powerful new capability to my shot-scraper command line browser automation tool: you can now use it to load a web page in a headless browser, execute JavaScript to extract information and return that information back to the terminal as JSON.
[... 1,277 words]Weeknotes: Distracted by Playwright
My goal for this week was to unblock progress on Datasette by finally finishing the dash encoding implementation I described last week. I was getting close, and then I got very distracted by Playwright.
[... 892 words]shot-scraper: automated screenshots for documentation, built on Playwright
shot-scraper is a new tool that I’ve built to help automate the process of keeping screenshots up-to-date in my documentation. It also doubles as a scraping tool—hence the name—which I picked as a complement to my git scraping and help scraping techniques.
[... 1,802 words]Why I invented “dash encoding”, a new encoding scheme for URL paths
Datasette now includes its own custom string encoding scheme, which I’ve called dash encoding. I really didn’t want to have to invent something new here, but unfortunately I think this is the best solution to my very particular problem. Some notes on how dash encoding works and why I created it.
[... 1,392 words]Weeknotes: Datasette Tutorials
I published two new tutorials for Datasette this week, both focused at end-users of the web application.
[... 479 words]Support open source that you use by paying the maintainers to talk to your team
I think I’ve come up with a novel hack for the challenge of getting your company to financially support the open source projects that it uses: reach out to the maintainers and offer them generous speaking fees for remote talks to your engineering team.
[... 645 words]Google Drive to SQLite
I released a new tool this week: google-drive-to-sqlite. It’s a CLI utility for fetching metadata about files in your Google Drive and writing them to a local SQLite database.
[... 1,221 words]Using SQLite and Datasette with Fly Volumes
A few weeks ago, Fly announced Free Postgres Databases as part of the free tier of their hosting product. Their announcement included this snippet:
[... 1,463 words]Help scraping: track changes to CLI tools by recording their --help using Git
I’ve been experimenting with a new variant of Git scraping this week which I’m calling Help scraping. The key idea is to track changes made to CLI tools over time by recording the output of their --help commands in a Git repository.
Writing better release notes
Release notes are an important part of the open source process. I’ve been thinking about these a lot recently, and I’ve assembled some thoughts on how to do a better job with them.
[... 918 words]Weeknotes: python_requires, documentation SEO
Fixed Datasette on Python 3.6 for the last time. Worked on documentation infrastructure improvements. Spent some time with Fly Volumes.
[... 1,497 words]Weeknotes: s3-credentials prefix and Datasette 0.60
A new release of s3-credentials with support for restricting access to keys that start with a prefix, Datasette 0.60 and a write-up of my process for shipping a feature.
[... 1,134 words]Datasette 0.60: The annotated release notes
I released Datasette 0.60 today. It’s a big release, incorporating 61 commits and 18 issues. Here are the annotated release notes.
[... 1,119 words]How I build a feature
I’m maintaining a lot of different projects at the moment. I thought it would be useful to describe the process I use for adding a new feature to one of them, using the new sqlite-utils create-database command as an example.
[... 2,850 words]What’s new in sqlite-utils 3.20 and 3.21: --lines, --text, --convert
sqlite-utils is my combined CLI tool and Python library for manipulating SQLite databases. Consider this the annotated release notes for sqlite-utils 3.20 and 3.21, both released in the past week.
[... 2,456 words]Weeknotes: Taking a break in Moss Landing
Took some time off. Saw some whales and sea otters. Added a new spot to Niche Museums.
[... 578 words]Weeknotes: datasette-tiddlywiki, filters_from_request
I made some good progress on the big refactor this week, including extracting some core logic out into a new Datasette plugin hook. I also got distracted by TiddlyWiki and released a new Datasette plugin that lets you run TiddlyWiki inside Datasette.
[... 1,197 words]Weeknotes: Trapped in an eternal refactor
I’m still working on refactoring Datasette’s table view. In doing so I spun out a new plugin, datasette-pretty-traces, which improves Datasette’s tooling for seeing the SQL that was executed to build a specific page.
[... 544 words]