Simon Willison’s Weblog

Blogmarks in Aug

Filters: Type: blogmark × Month: Aug ×

coleifer/pysqlite3. Now that the pysqlite package is bundled as part of the Python standard library the original open source project is no longer actively maintained, and has not been upgraded for Python 3. Charles Leifer has been working on pysqlite3, a stand-alone package of the module. Crucially, this should enable compiling the latest version of SQLite (via the amalgamation package) without needing to upgrade the version that ships with the operating system. # 15th August 2018, 3:15 pm

Window Functions in SQLite 3.25.0. The next release of SQLite (apparently due for release in September) will add window functions, as specified in various SQL standards and already available in PostgreSQL. This is going to dramatically improve SQLite as an engine for performing analytical queries, especially across time series data. It’s also going to further emphasize the need for people to be able to upgrade their SQLite versions beyond those provided by the operating system—the default Ubuntu run by Travis CI still only ships with SQLite 3.8 for example. # 15th August 2018, 3:12 pm

Experiences with running PostgreSQL on Kubernetes (via) Fascinating interview that makes a solid argument for the idea that running stateful data stores like PostgreSQL or Cassandra is made harder, not easier when you add an orchestration tool like Kubernetes into the mix. # 13th August 2018, 2:30 pm

Using achievement stats to estimate sales on steam (via) Really interesting data leak exploit here: Valve’s Steam API was showing the percentage of users that gained a specific achievement up to 16 decimal places—which inadvertently leaked their exact usage statistics, since if 0.012782207690179348 percent of players get an achievement the only possible input is 8 players out of 62,587. # 9th August 2018, 9:03 am

Securing Web Sites Made Them Less Accessible (via) This is fascinating: the move to HTTP everywhere breaks local HTTP caching servers (like Squid) which are still used in remote areas that get their internet by a high latency satellite connection. # 7th August 2018, 5:52 pm

Faust: Python Stream Processing (via) A new open source stream processing system released by Robinhood, created by Vineet Goel and Celery creator Ask Solem. The API looks delightful, making very smart use of Python decorators and async/await. The initial release requires Kafka but they plan to support multiple backends, hopefully including Redis Streams. # 6th August 2018, 10:51 pm

How to Read an RFC. An extremely useful guide to reading RFCs by Mark Nottingham. I didn’t know most of the stuff in here. # 6th August 2018, 10:38 pm

OWASP Top 10 2007-2017: The Fall of CSRF. I was surprised to learn recently that CSRF didn’t make it into the 2017 OWASP Top 10 security vulnerabilities (after featuring almost every year since the list started). The credited reason is that web frameworks do a good enough job protecting against CSRF by default that it’s no longer a top-ten problem. Defaults really do matter. # 6th August 2018, 10:02 pm

Datasette unit tests: monkeytype_call_traces (via) Faceted browse against every function call that occurs during the execution of Datasette’s test suite. I used Instagram’s MonkeyType tool to generate this, which can run Python code and generates a SQLite database of all of the traced calls. It’s intended to be used to automatically add mypy annotations to your code, but since it produces a SQLite database as a by-product I’ve started exploring the intermediary format using Datasette. Generating this was as easy as running “monkeytype run `which pytest`” in the Datasette root directory. # 2nd August 2018, 9:03 pm

RasterWeb: Lanyrd. Pete Prodoehl calls me out on Lanyrd’s integration with the Twitter auth API at the expense of OpenID. I’ve posted a comment with my justification—essentially, tying to Twitter’s ecosystem means I can actually implement the features I’ve been talking about building on top of OpenID for years, with far less engineering effort. # 31st August 2010, 8:49 pm

Lanyrd—the social conference directory. Nat and my new project, launched today and doing pretty well despite some early server hiccups. Sign in with Twitter to see conferences that your friends are speaking at, attending or tracking, then add your own events. We’re particularly keen on helping people build up a detailed profile of their previous talks, so adding older conferences is encouraged. # 31st August 2010, 7:41 pm

LWPx::ParanoidAgent. Every programming language needs an equivalent of this library—a robust, secure way to make HTTP requests against URLs from untrusted sources without risk of tarpits, internal network access, socket starvation, weird server errors, or other nastiness. # 31st August 2010, 2:30 am

What is the history of Django? I’ve been playing with Quora—it’s a really neat twist on the question-and-answer format, which makes great use of friends, followers and topics and has some very neat live update stuff going on (using Comet on top of Tornado). I just posted quite a long answer to a question about the history of Django. # 24th August 2010, 5:31 pm

Readme Driven Development (via) Tom Preston-Werner advocates for writing the readme before any other code. “Until you’ve written about your software, you have no idea what you’ll be coding.” # 23rd August 2010, 8:20 pm

Using Freebase Gridworks to Create Linked Data. A very handy tutorial from’s Jeni Tennison. # 23rd August 2010, 8:11 pm

PNGStore—Embedding compressed CSS & JavaScript in PNGs. Cal did some further analysis on the CSS/JS to PNG compression trick (including producing some interesting images of jQuery compressed using different image packing techniques) and found it to be slightly less effective than regular GZipping. # 23rd August 2010, 9:47 am

10K Apart Contest: Cheating by Compressing Your JavaScript and CSS to PNG Images. Fascinating hack: transform your JS and CSS in to coloured pixels, save the result as a PNG to benefit from PNG’s built in compression algorithms, then read the data back out of the PNG and convert it back to text using JavaScript and canvas—all to reduce the on-disk filesize when entering the 10K app competition. Alex’s GithubFinder entry is worth checking out too. # 23rd August 2010, 9:45 am

Undelete! How to undelete a file accidentally removed using rm on Linux, by grepping through the raw bytes on the hard drive searching for a unique string that was contained in the file. “grep -a -B 25 -A 100 ’some string in the file’ /dev/sda1 > results.txt” # 21st August 2010, 10:56 am

Polymaps. Absurdly classy: “a JavaScript library for image- and vector-tiled maps using SVG”. It can pull in image tiles from sources such as OpenStreetMap, then overlay SVG paths specified using GeoJSON. The demos make use of GeoJSON tiles for US states and counties hosted on AppEngine. The library is developed by Stamen and SimpleGeo, and released under a BSD license. SVG support in the browser is required. # 20th August 2010, 6:46 pm

A More Royal Royal Opera House. Beautiful piece of work updating the branding for the Royal Opera House, including a strikingly modern take on the original crest. # 20th August 2010, 12:08 pm

Surfin’ Safari: Announcing... MathML! MathML is now supported by the WebKit nightlies. Worth checking out for the typographical discussion that’s broken out in the comments. # 18th August 2010, 1:49 pm

Pictos. Here’s something new: a for-sale font containing a set of beautiful royalty-free icons (like Wingdings, but good) designed to be embedded in web applications using @font-face. Small file sizes, scalable vectors without SVG. Not sure about the accessibility implications though. # 17th August 2010, 8:54 pm

Yahoo! Developer Network: Important API Updates and Changes. Some important (and potentially worrying) news about Yahoo! APIs. The BOSS (Build your Own Search Service) API will no longer be free—not an enormous surprise, and hopefully the pricing will be sensible. Most of the other search APIs (including web, news and image search) are being turned off with no replacement, while term extraction and spelling suggestions will be YQL-only. Most worrying, changes to Geo, Maps and Local APIs will be announced in September, with some set to close. I really hope this doesn’t affect the GeoPlanet APIs. # 17th August 2010, 6:14 pm

Human pylons carry electricity across Iceland. An entry in the “Icelandic High-Voltage Electrical Pylon International Design Competition” proposes giant human-shaped electricity pylons. “The figures can be placed into different poses, with the suggestion that the landscapes could inform the position that the sculpture is placed into. For example, as a power line ascends a hill, the pylons could look as if they’re climbing. The figures could also stretch up to gain increased height over longer spans.” # 17th August 2010, 1:38 pm

Writing Bulletproof Apps with API Errorpoints. This is a very good idea: Web APIs should offer special API endpoints for simulating each of the possible errors that might be returned by the production API. # 16th August 2010, 7:12 pm

Journalism Warning Labels. These are absolutely fantastic. “I’ve been putting them on copies of the free papers that I find on the London Underground. You might want to as well.” # 14th August 2010, 11:16 am

The Pac-Man Dossier. Exuberantly detailed. Everything from how collision detection works to the exact pathfinding and target selection algorithms used by the four different ghosts. There’s even a tutorial for playing the legendary 256th level, where an overflow bug corrupts one half of the screen. # 11th August 2010, 11:20 am

Closure Compiler Service (via) A hosted version of the Google Closure Compiler (JavaScript minifier) running on App Engine. It has both a user interface and a REST API, which means you can use it as part of an automated build process without needing to set up a local copy of the software. # 9th August 2010, 1:17 pm

Writing your own traceroute in Python. How to implement traceroute in Python, using the low-level socket module. # 9th August 2010, 12:58 pm