Simon Willison’s Weblog

Items in 2018

Filters: Year: 2018 ×


Optimizing Django Admin Paginator. The Django admin paginator uses a count(*) to calculate the total number of rows, so it knows how many pages to display. This makes it unpleasantly slow over large datasets. Haki Benita has an ingenious solution: drop in a custom paginator which uses the PostgreSQL “SET LOCAL statement_timeout TO 200” statement first, then if a timeout error is raised returns 9999999999 as the count instead. This means small tables get accurate page counts and giant tables load display in the admin within a reasonable time period. # 6th November 2018, 6:17 pm

11 barriers to coding in the open and how to overcome them (via) “Terence Eden, open standards lead at GDS, also gave a talk about overcoming barriers to coding in the open”—an intriguing recap of that talk revealing exactly how the UK government have been encouraging a culture of coding in the open and going open source first. # 5th November 2018, 8:53 pm

Twitter conversation about long-term pre-paid archival storage. I kicked off a conversation on Twitter yesterday about long-time archival storage of web content: “Anyone know of a web hosting provider where I can pay a lump sum of money to host a file at a reliable URL essentially forever? Is this even remotely feasible?”. The thread is really interesting—this is definitely an unsolved problem, and it’s clear that the challenge is more organizational (how do you create an entity that can keep this kind of promise—does it need to be some kind of foundation or trust?) than technical. # 5th November 2018, 6:50 pm

If you stop thinking in terms of MVC you might notice that at its core, React is a runtime for effectful functions that don’t execute “once”, but run continuously while being anchored to a call tree.

Dan Abramov # 3rd November 2018, 9:51 pm

Apple’s New Map (via) Map nerds rejoice! Justin O’Beirne had written another spectacularly illustrated essay about web cartography, this time examining the iOS 12 upgrade to Apple Maps in most of California and a little bit of Nevada. # 3rd November 2018, 9:28 pm

Every bitcoin proof-of-work mined is an incremental addition to a vast distributed summoning ritual powering the demon-soul at the heart of the maze, the computational equivalent of a Buddhist prayer wheel spinning in a Himalayan breeze.

Charles Stross # 3rd November 2018, 5:47 pm

Svelte RFC 0001: Reactive assignments (via) Svelte is a really interesting JavaScript framework: it offers a similar component-based developer experience to React but does it while delivering a tiny amount of code-to-the-browser thanks to being built entirely around a compiler that generates the minimum code necessary. In this RFC the Svelte team propose taking this approach even further, by generating code to accompany every relevant variable assignment that can trigger the corresponding view update. The document also has a very clear explanation of how React, Vue and current Svelte differ in their solutions to the challenge of updating the visible HTML view when the corresponding state changes. # 3rd November 2018, 4:58 pm

jantic/DeOldify (via) “A Deep Learning based project for colorizing and restoring old images”. Delightful (and well documented) project that uses a Self-Attention Generative Adversarial Network to colorize old black and white photos, with extremely impressive results. Built on an older version of the fastai library, and trained by running for several days on a 1080TI graphics card. # 2nd November 2018, 11:13 am

Among other things at Netflix the Mantis Query Language (MQL an SQL for streaming data) which ferries around approximately 2 trillion events every day for operational analysis (SPS alerting, quality of experience metrics, debugging production, etc) is written entirely in Clojure.

diab0lic on Hacker News # 1st November 2018, 2:52 am

Reinforcement Learning with Prediction-Based Rewards (via) Fascinating result: by teaching a reinforcement learning agent that plays video games to optimize for “unfamiliar states”—states where it cannot predict what will happen next—the agent does a much better job of playing some games. “... for the first time exceeds average human performance on Montezuma’s Revenge. RND achieves state-of-the-art performance, periodically finds all 24 rooms and solves the first level without using demonstrations or having access to the underlying state of the game.” # 31st October 2018, 11:51 pm

October 21 post-incident analysis (via) Legitimately fascinating post-mortem by GitHub. They run database masters in multiple data centers with raft for leader election... but when they had an unexpected network split between east and west coast they ended up with several seconds of write that had not been correctly replicated. Cleaning up the resulting mess took the best part of 24 hours! Distributed systems are hard. # 31st October 2018, 8:50 pm

Making Sense of React Hooks (via) Dan Abramov provides the most comprehensive justification I’ve seen so far for the new React hooks API. # 31st October 2018, 4:26 am

This Is How We Radicalized The World (via) Don’t be put off by the click-baity title: this article by Ryan Broderick is absolutely worth your time. Ryan has been traveling the world covering the global rise of populism, which has been driven in a great part by new patterns of social media usage and distrust of the news media. Ryan ties together stories from a bunch of different countries over the last few years and make a compelling case that we need to come to terms with social media radicalization as a global problem and figure out how to respond to it and deal with the fallout. # 31st October 2018, 1:11 am

matthewp/haunted: React’s Hooks API implemented for web components (via) It’s been fascinating over the past few days watching various frontend web stacks start playing with the new ideas introduced by the proposed React hooks API. lit-html is one of my favourite React alternatives—it’s built on web components and makes really clever use of ES6 template literals (in place of React’s JSX, which requires an additional compilation step). With Haunted Matthew Phillips explores the combination of lit-html, web components and hooks-style state management. # 31st October 2018, 1:04 am

Automatically playing science communication games with transfer learning and fastai

This weekend was the 9th annual Science Hack Day San Francisco, which was also the 100th Science Hack Day held worldwide.

[... 1174 words]

python-twitter/get_access_token.py. Creating an OAuth token for accessing a specific Twitter account is way harder than it needs to be. I was about to write my own command-line script for doing this using PIN-based authentication (where you pop open a browser showing the Twitter login flow, then get a PIN number at the end which you paste back into your script) when I discovered that the python-twitter library already ships with a script to do exactly that. Just run “python get_access_token.py”, paste in your app’s consumer key and secret, follow a link, enter the resulting PIN and the script will spit out the consumer_key / consumer_secret / access_token_key / access_token_secret combo you need to start using the Twitter API. # 28th October 2018, 5:25 pm

How to Instantly Publish Data to the Internet with Datasette

I spoke about my Datasette project at PyBay in August and they’ve just posted the video of my talk.

[... 58 words]

How I moderated the State of Django panel at DjangoCon US.

On Wednesday last week I moderated the State of Django panel as the closing session for DjangoCon US 2018.

[... 1210 words]

Introduction to Redis Streams. Redis 5.0 is out, introducing the first new Redis data type in several years: streams, a Kafka-like mechanism for implementing a replayable event stream that can be read by many different subscribers. # 18th October 2018, 8:35 am

github-debug.com (via) This is a neat trick: GitHub have a dedicated site for their support engineers to send you to if you can’t connect to them. The site tests download speeds from their various domains and then lets you click a button to have GitHub run a traceroute/ping from their servers to your detected IP address and output the results (use devtools to spy on their API method for doing this). Then you can paste the results into a message to their support team. Turns out fastly-debug.com and dropbox-debug.com implement a similar pattern for those services as well. # 10th October 2018, 7:32 pm

The ASGI specification provides an opportunity for Python to hit a productivity/performance sweet-spot for a wide range of use-cases, from writing high-volume proxy servers through to bringing large-scale web applications to market at speed.

Tom Christie # 8th October 2018, 2:43 pm

Relational databases are a commodity now, but they power a much larger fraction of the world’s economy that AI ever will. And no company has a “relational database strategy”.

Erik Bernhardsson # 8th October 2018, 12:20 pm

The interesting ideas in Datasette

Datasette (previously) is my open source tool for exploring and publishing structured data. There are a lot of ideas embedded in Datasette. I realized that I haven’t put many of them into writing.

[... 2857 words]

How we rolled out one of the largest Python 3 migrations ever. “If you’re using Dropbox today, the application is powered by a Dropbox-customized variant of Python 3.5” # 25th September 2018, 11:02 pm

Build impossible programs. Delightful talk by Julia Evans describing how she went about building a Ruby profiler in Rust despite having no knowledge of Ruby internals and only beginner’s knowledge of Rust. # 19th September 2018, 6:38 pm

Sqorn (via) JavaScript library for building SQL queries that makes really smart usage of ES6 tagged template literals. The magic of tagged template literals is that they let you intercept and process interpolated values, making them ideally suited to escaping parameters in SQL queries. Sqorn takes that basic ability and layers on some really interesting API design to allow you to further compose queries. # 19th September 2018, 6:34 pm

Letterboxing on Lundy

Last week Natalie and I spent a delightful two days with our friends Hannah and Adam on the beautiful island of Lundy in the Bristol Channel, 12 miles off the coast of North Devon.

[... 1461 words]

Extended Validation Certificates are Dead. Troy Hunt has been writing about the flaws of Extended Validation certificates for a while. Now iOS 12 is out and Mobile Safari no longer displays their visual indicator in the URL bar (and desktop Safari will stop doing so next week when Mac OS Mojave ships). EV certificates are being dropped by many of the larger companies that were using them. “This turned out to be a long blog post because every time I sat down to write, more and more evidence on the absolute pointlessness of EV presented itself”. # 18th September 2018, 1:41 pm

Tech Notes: TypeScript at Google (via) In which Evan Martin provides some fascinating colour on the state of JavaScript tooling within Google, which has some unique challenges given that Gmail is 14 years old now and Google have evolved their own internal JavaScript stack which differs widely from the rest of the industry (mainly because it predates most of the successful open source tools). “Which leads me to the middle path, which my little team has been pursuing: incrementally adopt some external tooling where it makes sense, by figuring out how to make it interoperate with our existing code base.” # 2nd September 2018, 7:08 pm

A tour of JavaScript timers on the web (via) Do you understand the differences between setTimeout, setInterval, setImmediate, requestAnimationFrame and requestIdleCallback? I didn’t. # 2nd September 2018, 10:10 am