Simon Willison’s Weblog

Migrating out of PostHaven. Amjith Ramanujam decided to migrate his blog content from PostHaven to a Markdown static site. He used shot-scraper (shelled out to from a Python script) to scrape his existing content using a snippet of JavaScript, wrote the content to a SQLite database using sqlite-utils, then used markdownify (new to me, a neat Python package for converting HTML to Markdown via BeautifulSoup) to write the content to disk as Markdown. # 24th May 2023, 7:38 pm

Fast Autocomplete Search for Your Website (via) I wrote a tutorial for the 24 ways advent calendar on building fast autocomplete search for a website on top of Datasette and SQLite. I built the demo against 24 ways itself—I used wget to recursively fetch all 330 articles as HTML, then wrote code in a Jupyter notebook to extract the raw data from them (with BeautifulSoup) and load them into SQLite using my sqlite-utils Python library. I deployed the resulting database using Datasette, then wrote some vanilla JavaScript to implement autocomplete using fast SQL queries against the Datasette JSON API. # 19th December 2018, 12:26 am

How to Make a US County Thematic Map Using Free Tools. This is the trick I’ve been using to generate choropleths at the Guardian for the past year: figure out the preferred colours for a set of data in a Python script and then rewrite an SVG file to colour in the areas. I use ElementTree rather than BeautifulSoup but the technique is exactly the same. The best thing about SVG is that our graphics department can export them directly out of Illustrator, with named layers and paths automatically becoming SVG ID attributes. Bonus tip: sometimes you don’t have to rewrite the SVG XML at all, instead you can generate CSS to colour areas by ID selector and inject it in to the top of the file. # 12th November 2009, 10:49 am

soupselect. My simple extension to BeautifulSoup that allows you to grab elements using CSS selectors; should be useful for parsing microformats. # 28th February 2007, 1:47 pm