Simon Willison’s Weblog

Optimizing Django Admin Paginator. The Django admin paginator uses a count(*) to calculate the total number of rows, so it knows how many pages to display. This makes it unpleasantly slow over large datasets. Haki Benita has an ingenious solution: drop in a custom paginator which uses the PostgreSQL “SET LOCAL statement_timeout TO 200” statement first, then if a timeout error is raised returns 9999999999 as the count instead. This means small tables get accurate page counts and giant tables load display in the admin within a reasonable time period. # 6th November 2018, 6:17 pm

11 barriers to coding in the open and how to overcome them (via) “Terence Eden, open standards lead at GDS, also gave a talk about overcoming barriers to coding in the open”—an intriguing recap of that talk revealing exactly how the UK government have been encouraging a culture of coding in the open and going open source first. # 5th November 2018, 8:53 pm

Twitter conversation about long-term pre-paid archival storage. I kicked off a conversation on Twitter yesterday about long-time archival storage of web content: “Anyone know of a web hosting provider where I can pay a lump sum of money to host a file at a reliable URL essentially forever? Is this even remotely feasible?”. The thread is really interesting—this is definitely an unsolved problem, and it’s clear that the challenge is more organizational (how do you create an entity that can keep this kind of promise—does it need to be some kind of foundation or trust?) than technical. # 5th November 2018, 6:50 pm

Apple’s New Map (via) Map nerds rejoice! Justin O’Beirne had written another spectacularly illustrated essay about web cartography, this time examining the iOS 12 upgrade to Apple Maps in most of California and a little bit of Nevada. # 3rd November 2018, 9:28 pm

Svelte RFC 0001: Reactive assignments (via) Svelte is a really interesting JavaScript framework: it offers a similar component-based developer experience to React but does it while delivering a tiny amount of code-to-the-browser thanks to being built entirely around a compiler that generates the minimum code necessary. In this RFC the Svelte team propose taking this approach even further, by generating code to accompany every relevant variable assignment that can trigger the corresponding view update. The document also has a very clear explanation of how React, Vue and current Svelte differ in their solutions to the challenge of updating the visible HTML view when the corresponding state changes. # 3rd November 2018, 4:58 pm

jantic/DeOldify (via) “A Deep Learning based project for colorizing and restoring old images”. Delightful (and well documented) project that uses a Self-Attention Generative Adversarial Network to colorize old black and white photos, with extremely impressive results. Built on an older version of the fastai library, and trained by running for several days on a 1080TI graphics card. # 2nd November 2018, 11:13 am

Reinforcement Learning with Prediction-Based Rewards (via) Fascinating result: by teaching a reinforcement learning agent that plays video games to optimize for “unfamiliar states”—states where it cannot predict what will happen next—the agent does a much better job of playing some games. “... for the first time exceeds average human performance on Montezuma’s Revenge. RND achieves state-of-the-art performance, periodically finds all 24 rooms and solves the first level without using demonstrations or having access to the underlying state of the game.” # 31st October 2018, 11:51 pm

October 21 post-incident analysis (via) Legitimately fascinating post-mortem by GitHub. They run database masters in multiple data centers with raft for leader election... but when they had an unexpected network split between east and west coast they ended up with several seconds of write that had not been correctly replicated. Cleaning up the resulting mess took the best part of 24 hours! Distributed systems are hard. # 31st October 2018, 8:50 pm

Making Sense of React Hooks (via) Dan Abramov provides the most comprehensive justification I’ve seen so far for the new React hooks API. # 31st October 2018, 4:26 am

This Is How We Radicalized The World (via) Don’t be put off by the click-baity title: this article by Ryan Broderick is absolutely worth your time. Ryan has been traveling the world covering the global rise of populism, which has been driven in a great part by new patterns of social media usage and distrust of the news media. Ryan ties together stories from a bunch of different countries over the last few years and make a compelling case that we need to come to terms with social media radicalization as a global problem and figure out how to respond to it and deal with the fallout. # 31st October 2018, 1:11 am

matthewp/haunted: React’s Hooks API implemented for web components (via) It’s been fascinating over the past few days watching various frontend web stacks start playing with the new ideas introduced by the proposed React hooks API. lit-html is one of my favourite React alternatives—it’s built on web components and makes really clever use of ES6 template literals (in place of React’s JSX, which requires an additional compilation step). With Haunted Matthew Phillips explores the combination of lit-html, web components and hooks-style state management. # 31st October 2018, 1:04 am

python-twitter/get_access_token.py. Creating an OAuth token for accessing a specific Twitter account is way harder than it needs to be. I was about to write my own command-line script for doing this using PIN-based authentication (where you pop open a browser showing the Twitter login flow, then get a PIN number at the end which you paste back into your script) when I discovered that the python-twitter library already ships with a script to do exactly that. Just run “python get_access_token.py”, paste in your app’s consumer key and secret, follow a link, enter the resulting PIN and the script will spit out the consumer_key / consumer_secret / access_token_key / access_token_secret combo you need to start using the Twitter API. # 28th October 2018, 5:25 pm

Introduction to Redis Streams. Redis 5.0 is out, introducing the first new Redis data type in several years: streams, a Kafka-like mechanism for implementing a replayable event stream that can be read by many different subscribers. # 18th October 2018, 8:35 am

github-debug.com (via) This is a neat trick: GitHub have a dedicated site for their support engineers to send you to if you can’t connect to them. The site tests download speeds from their various domains and then lets you click a button to have GitHub run a traceroute/ping from their servers to your detected IP address and output the results (use devtools to spy on their API method for doing this). Then you can paste the results into a message to their support team. Turns out fastly-debug.com and dropbox-debug.com implement a similar pattern for those services as well. # 10th October 2018, 7:32 pm

How we rolled out one of the largest Python 3 migrations ever. “If you’re using Dropbox today, the application is powered by a Dropbox-customized variant of Python 3.5” # 25th September 2018, 11:02 pm

Build impossible programs. Delightful talk by Julia Evans describing how she went about building a Ruby profiler in Rust despite having no knowledge of Ruby internals and only beginner’s knowledge of Rust. # 19th September 2018, 6:38 pm

Sqorn (via) JavaScript library for building SQL queries that makes really smart usage of ES6 tagged template literals. The magic of tagged template literals is that they let you intercept and process interpolated values, making them ideally suited to escaping parameters in SQL queries. Sqorn takes that basic ability and layers on some really interesting API design to allow you to further compose queries. # 19th September 2018, 6:34 pm

Extended Validation Certificates are Dead. Troy Hunt has been writing about the flaws of Extended Validation certificates for a while. Now iOS 12 is out and Mobile Safari no longer displays their visual indicator in the URL bar (and desktop Safari will stop doing so next week when Mac OS Mojave ships). EV certificates are being dropped by many of the larger companies that were using them. “This turned out to be a long blog post because every time I sat down to write, more and more evidence on the absolute pointlessness of EV presented itself”. # 18th September 2018, 1:41 pm

Tech Notes: TypeScript at Google (via) In which Evan Martin provides some fascinating colour on the state of JavaScript tooling within Google, which has some unique challenges given that Gmail is 14 years old now and Google have evolved their own internal JavaScript stack which differs widely from the rest of the industry (mainly because it predates most of the successful open source tools). “Which leads me to the middle path, which my little team has been pursuing: incrementally adopt some external tooling where it makes sense, by figuring out how to make it interoperate with our existing code base.” # 2nd September 2018, 7:08 pm

A tour of JavaScript timers on the web (via) Do you understand the differences between setTimeout, setInterval, setImmediate, requestAnimationFrame and requestIdleCallback? I didn’t. # 2nd September 2018, 10:10 am