Simon Willison’s Weblog

Subscribe

November 2020

Nov. 1, 2020

Datasette 0.51 (plus weeknotes)

Visit Datasette 0.51 (plus weeknotes)

I shipped Datasette 0.51 today, with a new visual design, plugin hooks for adding navigation options, better handling of binary data, URL building utility methods and better support for running Datasette behind a proxy. It’s a lot of stuff! Here are the annotated release notes.

[... 2,020 words]

Nov. 2, 2020

selenium-wire. Really useful scraping tool: enhances the Python Selenium bindings to run against a proxy which then allows Python scraping code to look at captured requests—great for if a site you are working with triggers Ajax requests and you want to extract data from the raw JSON that came back.

# 6:58 pm / scraping, data-journalism, python, selenium

Nov. 5, 2020

CSVs: The good, the bad, and the ugly (via) Useful, thoughtful summary of the pros and cons of the most common format for interchanging data.

# 5:19 pm / csv, alex-gaynor

Learning from Mini Apps (via) WeChat, Baidu, Alipay and Douyin in China are all examples of “Super apps” that can host “Mini apps” written in HTML and JavaScript by other developers and installed via in-app search or through scanning a QR code. Mini apps are granted (permission-gated) access to further system APIs via a JavaScript bridge. It’s a fascinating developer ecosystem, explored in detail here by Thomas Steiner.

# 5:21 pm / mobile, javascript

Nov. 6, 2020

nyt-2020-election-scraper. Brilliant application of git scraping by Alex Gaynor and a growing team of contributors. Takes a JSON snapshot of the NYT’s latest election poll figures every five minutes, then runs a Python script to iterate through the history and build an HTML page showing the trends, including what percentage of the remaining votes each candidate needs to win each state. This is the perfect case study in why it can be useful to take a “snapshot if the world right now” data source and turn it into a git revision history over time.

# 2:24 pm / elections, alex-gaynor, git-scraping, data-journalism, git, new-york-times

Nov. 7, 2020

Weeknotes: sqlite-utils 3.0 alpha, Git scraping in the zeitgeist

Visit Weeknotes: sqlite-utils 3.0 alpha, Git scraping in the zeitgeist

Natalie and I decided to escape San Francisco for election week, and have been holed up in Fort Bragg on the Northern California coast. I’ve mostly been on vacation, but I did find time to make some significant changes to sqlite-utils. Plus notes on an exciting Git scraping project.

[... 603 words]

Nov. 12, 2020

Intent to Remove: HTTP/2 and gQUIC server push (via) The Chrome / Blink team announce their intent to remove HTTP/2 server push support, where servers can start pushing an asset to a client before it has been requested. It’s been in browsers for over five years now and adoption is terrible. “Over the past 28 days [...] 99.97% of connections never received a pushed stream that got matched with a request [...] These numbers are exactly the same as in June 2019”. Datasette serves redirects with Link: preload headers that cause smart proxies (like Cloudflare) to push the redirected page to the client along with the redirect, but I don’t exepect to miss that optimization if it quietly stops working.

# 1:44 am / http2, chrome, datasette

Nov. 14, 2020

Personal Data Warehouses: Reclaiming Your Data

Visit Personal Data Warehouses: Reclaiming Your Data

I gave a talk yesterday about personal data warehouses for GitHub’s OCTO Speaker Series, focusing on my Datasette and Dogsheep projects. The video of the talk is now available, and I’m presenting that here along with an annotated summary of the talk, including links to demos and further information.

[... 5,166 words]

Hunting for Malicious Packages on PyPI (via) Jordan Wright installed all 268,000 Python packages from PyPI in containers, and ran Sysdig to capture syscalls made during installation to see if any of them were making extra network calls or reading or writing from the filesystem. Absolutely brilliant piece of security engineering and research.

# 4:48 am / security, pypi, python

The Cleanest Trick for Autogrowing Textareas (via) This is a very clever trick. Textarea content is mirrored into a data attribute using a JavaScript one-liner, then a visibility: hidden ::after element clones that content using content: attr(data-replicated-value). The hidden element exists in a CSS grid with the textarea which allows the textarea to resize within the grid when the hidden element increases its height.

# 5:24 am / css, javascript

Nov. 15, 2020

CoronaFaceImpact (via) Variable fonts are fonts that can be customized by passing in additional parameters, which is done in CSS using the font-variation-settings property. Here’s a ​variable font that shows multiple effects of Covid-19 lockdown on a bearded face, created by Friedrich Althausen.

# 10:41 pm / css, fonts, covid19, typography

Nov. 16, 2020

Ok Google: please publish your DKIM secret keys (via) The DKIM standard allows email providers such as Gmail to include cryptographic headers that protect against spoofing, proving that an email was sent by a specific host and has not been tampered with. But it has an unintended side effect: if someone’s email is leaked (as happened to John Podesta in 2016) DKIM headers can be used to prove the validity of the leaked emails. This makes DKIM an enabling factor for blackmail and other security breach related crimes.

Matthew Green proposes a neat solution: providers like Gmail should rotate their DKIM keys frequently and publish the PRIVATE key after rotation. By enabling spoofing of past email headers they would provide deniability for victims of leaks, fixing this unintended consequence of the DKIM standard.

# 10:02 pm / security, cryptography, email

Nov. 17, 2020

Amstelvar (via) A real showcase of what variable fonts can do: this open source font by David Berlow has 17 different variables controlling many different aspects of the font.

# 3:24 pm / typography, fonts

Nov. 18, 2020

Implementing IndieAuth for Datasette

Visit Implementing IndieAuth for Datasette

IndieAuth is a spiritual successor to OpenID, developed and maintained by the IndieWeb community and based on OAuth 2. This weekend I attended IndieWebCamp East Coast and was inspired to try my hand at an implementation. datasette-indieauth is the result, a new plugin which enables IndieAuth logins to a Datasette instance.

[... 1,225 words]

Nov. 19, 2020

Security vulnerability in datasette-indieauth: Implementation trusts the “me” field returned by the authorization server without verifying it. I spotted a critical security vulnerability in my new datasette-indieauth plugin: it accepted the “me” profile URL value returned from the authorization server in the final step of the IndieAuth flow without verifying it, which means a malicious server could imitate any user. I’ve shipped 1.1 with a fix and posted a security advisory to the GitHub repository.

# 9:14 pm / projects, security

Internet Archive Software Library: Flash (via) A fantastic new initiative from the Internet Archive: they’re now archiving Flash (.swf) files and serving them for modern browsers using Ruffle, a Flash Player emulator written in Rust and compiled to WebAssembly. They are fully interactive and audio works too. Considering the enormous quantity of creative material released in Flash over the decades this helps fill a big hole in the Internet’s cultural memory.

# 9:19 pm / flash, internet-archive, rust, webassembly, jason-scott

Nov. 20, 2020

The trouble with transaction.atomic (via) David Seddon provides a detailed explanation of Django’s nestable transaction.atomic() context manager and describes a gotcha that can occur if you lose track of whether your code is already running in a transaction block, since you may be working with savepoints instead—along with some smart workarounds.

# 3:57 pm / transactions, django

The open secret Jennings filled me in on is that OpenStreetMap (OSM) is now at the center of an unholy alliance of the world’s largest and wealthiest technology companies. The most valuable companies in the world are treating OSM as critical infrastructure for some of the most-used software ever written. The four companies in the inner circle— Facebook, Apple, Amazon, and Microsoft— have a combined market capitalization of over six trillion dollars.

Joe Morrison

# 9:11 pm / amazon, apple, facebook, microsoft, openstreetmap

Nov. 21, 2020

I Lived Through A Stupid Coup. America Is Having One Now (via) If, like me, you have been avoiding the word “coup” since it feels like a clear over-reaction to what’s going on, I challenge you to read this piece and not change your mind.

# 1:21 pm / politics, trump

datasette-graphql 1.2 (via) A new release of the datasette-graphql plugin, fixing a minor security flaw: previous versions of the plugin could expose the schema (but not the actual data) of tables in databases that were otherwise protected by Datasette’s permission system.

# 10:21 pm / projects, datasette, security, graphql

Nov. 22, 2020

Weeknotes: datasette-indieauth, datasette-graphql, PyCon Argentina

Visit Weeknotes: datasette-indieauth, datasette-graphql, PyCon Argentina

Last week’s weeknotes took the form of my Personal Data Warehouses: Reclaiming Your Data talk write-up, which represented most of what I got done that week. This week I mainly worked on datasette-indieauth, but I also gave a keynote at PyCon Argentina and released a version of datasette-graphql with a small security fix.

[... 724 words]

Nov. 24, 2020

Datasette Client for Observable (via) Really elegant piece of code design from Alex Garcia: DatasetteClient is a client library he built designed to work in Observable notebooks, which uses JavaScript tagged template literals to allow SQL query results to be executed against a Datasette instance and displayed as inline tables in a notebook, or used to return JSON data for further processing. His example notebook includes a neat d3 stacked area chart example built against a Datasette of congresspeople, plus examples using interactive widgets to update the Notebook.

# 6:53 pm / observable, datasette, javascript, alex-garcia

Nov. 27, 2020

Unravelling `not` in Python (via) Part of a series where Brett Cannon looks at how fundamental Python syntactic sugar works, including a clearly explained dive into the underlying op codes and C implementation.

# 5:59 pm / c, python, brett-cannon

Nov. 28, 2020

datasette-ripgrep: deploy a regular expression search engine for your source code

Visit datasette-ripgrep: deploy a regular expression search engine for your source code

This week I built datasette-ripgrep—a web application for running regular expression searches against source code, built on top of the amazing ripgrep command-line tool.

[... 1,362 words]

Nov. 29, 2020

Datasette 0.52. A relatively small release—it has a new plugin hook (database_actions(), for adding links to a new database actions menu), renames the --config option to --setting and adds a new “datasette publish cloudrun --apt-get-install” option.

# 12:56 am / projects, datasette

Nov. 30, 2020

Datasette Office Hours (via) I’ve decided to try running Datasette Office Hours every Friday. If you’d like to chat with me over Zoom about the project for 20 minutes for any reason at all you can grab a slot here using Calendly.

# 11:01 pm / datasette

Datasette Weekly volume 4. Datasette Office Hours, Personal Data Warehouses, datasette-ripgrep, datasette-indieauth, Datasette Client for Observable and more.

# 11:02 pm / datasette

Why is Apple’s M1 Chip So Fast? (via) This explanation by Erik Engheim is exactly the right level of nerdery for me.

# 11:20 pm / apple

2020 » November

MTWTFSS
      1
2345678
9101112131415
16171819202122
23242526272829
30