Entries
Filters: Sorted by date
Building an Evernote to SQLite exporter
I’ve been using Evernote for over a decade, and I’ve long wanted to export my data from it so I can do interesting things with it.
[... 1,879 words]Weeknotes: Mainly Datasette 0.50
Most of what I’ve been up to this week is covered in Datasette 0.50: The annotated release notes and Git scraping: track changes over time by scraping to a Git repository.
[... 196 words]Datasette 0.50: The annotated release notes
I released Datasette 0.50 this morning, with a new user-facing column actions menu feature and a way for plugins to make internal HTTP requests to consume the JSON API of their parent Datasette instance.
[... 792 words]Git scraping: track changes over time by scraping to a Git repository
Git scraping is the name I’ve given a scraping technique that I’ve been experimenting with for a few years now. It’s really effective, and more people should use it.
[... 963 words]Weeknotes: Datasette column actions, plus three new plugins
A renewed emphasis on building out Datasette Cloud has produced three new plugins this week: datasette-dateutil, datasette-import-table and datasette-edit-schema, plus a major improvement to Datasette’s default interface for browsing tables.
[... 1,093 words]Weeknotes: software carpentry, compiling modules for SQLite
This week I completed the Software Carpentry instructor training course, added two foundational features to sqlite-utils and learned how to compile modules for SQLite.
Refactoring databases with sqlite-utils extract
Yesterday I described the new sqlite-utils transform mechanism for applying SQLite table transformations that go beyond those supported by ALTER TABLE. The other new feature in sqlite-utils 2.20 builds on that capability to allow you to refactor a database table by extracting columns into separate tables. I’ve called it sqlite-utils extract.
Executing advanced ALTER TABLE operations in SQLite
SQLite’s ALTER TABLE has some significant limitations: it can’t drop columns (UPDATE: that was fixed in SQLite 3.35.0 in March 2021), it can’t alter NOT NULL status, it can’t change column types. Since I spend a lot of time with SQLite these days I’ve written some code to fix this—both from Python and as a command-line utility.
[... 689 words]Weeknotes: datasette-seaborn, fivethirtyeight-polls
This week I released Datasette 0.49 and tinkered with datasette-seaborn, dogsheep-beta and polling data from FiveThirtyEight.
[... 951 words]Datasette 0.49: The annotated release notes
Datasette 0.49 is out. Some notes on what’s new.
[... 1,234 words]Weeknotes: datasette-dump, sqlite-backup, talks
I spent some time this week digging into Python’s sqlite3 internals. I also gave two talks and recorded a third, due to air at PyGotham in October.
[... 928 words]Weeknotes: airtable-export, generating screenshots in GitHub Actions, Dogsheep!
This week I figured out how to populate Datasette from Airtable, wrote code to generate social media preview card page screenshots using Puppeteer, and made a big breakthrough with my Dogsheep project.
[... 1,461 words]The “await me maybe” pattern for Python asyncio
I’ve identified a pattern for handling potentially-asynchronous callback functions in Python which I’m calling the “await me maybe” pattern. It works by letting you return a value, a callable function that returns a value OR an awaitable function that returns that value.
[... 787 words]Mexican bean shakshuka
The first time I ever tried a shakshuka I cooked it from this recipe by J. Kenji López-Alt. It quickly became my favourite breakfast.
[... 448 words]Weeknotes: California Protected Areas in Datasette
This week I built a geospatial search engine for protected areas in California, shipped datasette-graphql 1.0 and started working towards the next milestone for Datasette Cloud.
[... 1,099 words]Weeknotes: Rocky Beaches, Datasette 0.48, a commit history of my database
This week I helped Natalie launch Rocky Beaches, shipped Datasette 0.48 and several releases of datasette-graphql, upgraded the CSRF protection for datasette-upload-csvs and figured out how to get a commit log of changes to my blog by backing up its database to a GitHub repository.
Weeknotes: Installing Datasette with Homebrew, more GraphQL, WAL in SQLite
This week I’ve been working on making Datasette easier to install, plus wide-ranging improvements to the Datasette GraphQL plugin.
[... 1,009 words]GraphQL in Datasette with the new datasette-graphql plugin
This week I’ve mostly been building datasette-graphql, a plugin that adds GraphQL query support to Datasette.
[... 1,249 words]Fun with binary data and SQLite
This week I’ve been mainly experimenting with binary data storage in SQLite. sqlite-utils can now insert data from binary files, and datasette-media can serve content over HTTP that originated as binary BLOBs in a database file.
[... 957 words]Weeknotes: datasette-copyable, datasette-insert-api
Two new Datasette plugins this week: datasette-copyable, helping users copy-and-paste data from Datasette into other places, and datasette-insert-api, providing a JSON API for inserting and updating data and creating tables.
[... 953 words]Weeknotes: datasette-auth-passwords, a Datasette logo and a whole lot more
All sorts of project updates this week.
[... 913 words]Building a self-updating profile README for GitHub
GitHub quietly released a new feature at some point in the past few days: profile READMEs. Create a repository with the same name as your GitHub account (in my case that’s github.com/simonw/simonw), add a README.md to it and GitHub will render the contents at the top of your personal profile page—for me that’s github.com/simonw
Weeknotes: SBA Covid-19 PPP loans, Datasette talks, Datasette plugin upgrades
This week I’ve mainly been exploring Small Business Administration Covid-19 loans data, pitching some talks and upgrading some plugins for compatibility with Datasette 0.44+.
[... 524 words]Datasette 0.45: The annotated release notes
Datasette 0.45, out today, features magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks.
[... 863 words]Weeknotes: cookiecutter templates, better plugin documentation, sqlite-generate
I spent this week spreading myself between a bunch of smaller projects, and finally getting familiar with cookiecutter. I wrote about my datasette-plugin cookiecutter template earlier in the week; here’s what else I’ve been working on.
[... 703 words]A cookiecutter template for writing Datasette plugins
Datasette’s plugin system is one of the most interesting parts of the entire project. As I explained to Matt Asay in this interview, the great thing about plugins is that Datasette can gain new functionality overnight without me even having to review a pull request. I just need to get more people to write them!
[... 914 words]Weeknotes: Datasette alphas for testing new plugin hooks
A relatively quiet week this week, compared to last week’s massive push to ship Datasette 0.44 with authentication, permissions and writable canned queries. I can now ship alpha releases, such as today’s Datasette 0.45a1, which means I can preview new plugin features before they are completely ready and stable.
[... 728 words]How much can you learn from just two columns?
Derek Willis shared an intriguing dataset this morning: a table showing every Twitter account followed by an official GOP congressional Twitter account.
[... 951 words]Datasette 0.44: The annotated release notes
I just released Datasette 0.44 to PyPI. With 128 commits since 0.43 this is the biggest release in a long time—and likely the last major release of new features before Datasette 1.0.
[... 1,648 words]Weeknotes, I guess
What a week. Hard to work up the enthusiasm to write about what I’ve been working on.
[... 314 words]