Entries
Filters: Sorted by date
Analyzing US Election Russian Facebook Ads
Two interesting data sources have emerged in the past few weeks concerning the Russian impact on the 2016 US elections.
[... 922 words]Analyzing US Election troll tweets with Datasette
FiveThirtyEight published nearly 3 million tweets from accounts associated with the Russian “Internet Research Agency”, based on research by Darren Linvill and Patrick Warren at at Clemson University.
[... 110 words]Datasette Facets
Datasette 0.22 is out with the most significant new feature I’ve added since the initial release: faceted browse.
[... 1,189 words]Notes from my appearance on the Changelog podcast
After I spoke at Zeit Day SF last weekend I sat down with Adam Stacoviak to record a 25 minute segment for episode 296 of the Changelog podcast, talking about Datasette. We covered a lot of ground!
[... 536 words]Exploring the UK Register of Members Interests with SQL and Datasette
Ever wondered which UK Members of Parliament get gifted the most helicopter rides? How about which MPs have been given Christmas hampers by the Sultan of Brunei? (David Cameron, William Hague and Michael Howard apparently). Here’s how to dig through the Register of Members Interests using SQL and Datasette.
[... 1,167 words]Datasette plugins, and building a clustered map visualization
Datasette now supports plugins!
[... 751 words]Building a combined stream of recent additions using the Django ORM
I’m a big believer in the importance of a “recent additions” feed. Any time you’re building an application that involves users adding and editing records it’s useful to have a page somewhere that shows the most recent objects that have been created across multiple different types of data.
[... 1,647 words]Datasette Demo (video) from the SF Python Meetup
I gave a short talk about Datasette last month at the SF Python Meetup Holiday Party. They’ve just posted the video, so here it is:
[... 63 words]Analyzing my Twitter followers with Datasette
I decided to do some ad-hoc analsis of my social network on Twitter this afternoon… and since everything is more fun if you bundle it up into a SQLite database and publish it to the internet I performed the analysis using Datasette.
[... 1,314 words]Datasette Publish: a web app for publishing CSV files as an online database
I’ve just released Datasette Publish, a web tool for turning one or more CSV files into an online database with a JSON API.
[... 863 words]Building a location to time zone API with SpatiaLite, OpenStreetMap and Datasette
Given a latitude and longitude, how can we tell what time zone that point lies within? Here’s how I built a simple JSON API to answer that question, using a combination of data from OpenStreetMap, the SpatiaLite extension for SQLite and my Datasette API tool.
[... 2,185 words]New in Datasette: filters, foreign keys and search
I’ve released Datasette 0.13 with a number of exciting new features (Datasette previously).
[... 1,143 words]Datasette: instantly create and publish an API for your SQLite databases
I just shipped the first public version of datasette, a new tool for creating and publishing JSON APIs for SQLite databases.
[... 968 words]Running a load testing Go utility using Docker for Mac
I’m playing around with Zeit Now at the moment (see my previous entry) and decided to hit it with some traffic using Apache Bench. I got this SSL handshake error:
[... 818 words]Using “import refs” to iteratively import data into Django
I’ve been writing a few scripts to backfill my blog with content I originally posted elsewhere. So far I’ve imported answers I posted on Quora (background), answers I posted on Ask MetaFilter and content I recovered from the Internet Archive.
[... 559 words]Late night dining near Great American Music Hall
Tommy’s Joynt is a couple of blocks away and is a San Francisco institution—great comfort food, inexpensive, crammed with personality and open late.
[... 40 words]Porting my blog to Python 3
This blog is now running on Python 3! Admittedly this is nearly nine years after the first release of Python 3.0, but it’s the first Python 3 project I’ve deployed myself so I’m pretty excited about it.
[... 883 words]How to set up world-class continuous deployment using free hosted tools
I’m going to describe a way to put together a world-class continuous deployment infrastructure for your side-project without spending any money.
[... 1,294 words]Deploying an asynchronous Python microservice with Sanic and Zeit Now
Back in 2008 Natalie Downe and I deployed what today we would call a microservice: json-head, a tiny Google App Engine app that allowed you to make an HTTP head request against a URL and get back the HTTP headers as JSON. One of our initial use-scase for this was Natalie’s addSizes.js, an unobtrusive jQuery script that could annotate links to PDFs and other large files with their corresponding file size pulled from the Content-Length header. Another potential use-case is detecting broken links, since the API can be used to spot 404 status codes (as in this example).
Changelogs to help understand the fires in the North Bay
The situation in the counties north of San Francisco is horrifying right now. I’ve repurposed some of the tools I built to for the Irma Response project last month to collect and track some data that might be of use to anyone trying to understand what’s happening up there. I’m sharing these now in the hope that they might prove useful.
[... 383 words]Recovering missing content from the Internet Archive
When I restored my blog last weekend I used the most recent SQL backup of my blog’s database from back in 2010. I thought it had all of my content from before I started my 7 year hiatus, but in watching the 404 logs I started seeing the occasional hit to something that really should have been there but wasn’t. Turns out the SQL backup I was working from was missing some content.
[... 636 words]Should I build my startup’s web-based product as if it’s going to one day be widely adopted and experience high-volume, or instead focus on quick delivery over scalability?
Absolutely the second: build for rapid learning, not for eventual scalability. The vast majority of startups fail, and the number one reason they fail is that they didn’t achieve product-market fit: they failed to build something that customers actually wanted.
[... 169 words]Implementing faceted search with Django and PostgreSQL
I’ve added a faceted search engine to this blog, powered by PostgreSQL. It supports regular text search (proper search, not just SQL“like” queries), filter by tag, filter by date, filter by content type (entries vs blogmarks vs quotation) and any combination of the above. Some example searches:
[... 3,103 words]Running gunicorn behind nginx on Heroku for buffering and logging
Heroku’s default setup for Django uses the gunicorn application server. Each Heroku dyno can only run a limited number of gunicorn workers, which means a limited number of requests can be served in parallel (around 4 per dyno is a good rule of thumb).
[... 400 words]Getting the blog back together
Getting this blog up and running again has turned out to be one of those side-projects that keeps threatening to fall down a rabbit hole.
[... 160 words]Scraping hurricane Irma
The Irma Response project is a team of volunteers working together to make information available during and after the storm. There is a huge amount of information out there, on many different websites. The Irma API is an attempt to gather key information in one place, verify it and publish it in a reuseable way. It currently powers the irmashelters.org website.
[... 438 words]The denormalized query engine design pattern
I presented this talk at DjangoCon 2017 in Spokane, Washington. Below is the abstract, the slides and the YouTube video of the talk.
[... 356 words]Doc of docs
Here’s a low-tech, high-impact trick I recently learned at work that’s amazingly useful: create a doc-of-docs.
[... 215 words]Software or project management technique for DIY home renovations?
We’ve been using a Trello board for this. Free, flexible, fun to use and has good mobile as well as desktop support. You can model “this task requires this task first” by linking to other cards in comments.
[... 54 words]