Simon Willison’s Weblog

Entries tagged sqlite

Filters: Type: entry × sqlite ×


Things I learned about shapefiles building shapefile-to-sqlite

The latest in my series of x-to-sqlite tools is shapefile-to-sqlite. I learned a whole bunch of things about the ESRI shapefile format while building it.

[... 1073 words]

sqlite-utils 2.0: real upserts

I just released version 2.0 of my sqlite-utils library/CLI tool to PyPI.

[... 1140 words]

Logging to SQLite using ASGI middleware

I had some fun playing around with ASGI middleware and logging during our flight back to England for the holidays.

[... 2535 words]

Tracking PG&E outages by scraping to a git repo

PG&E have cut off power to several million people in northern California, supposedly as a precaution against wildfires.

[... 833 words]

Weeknotes: Dogsheep

Having figured out my Stanford schedule, this week I started getting back into the habit of writing some code.

[... 1367 words]

Weeknotes: Design thinking for journalists, genome-to-sqlite, datasette-atom

I haven’t had much time for code this week: we’ve had a full five day workshop at JSK with Tran Ha (a JSK alumni) learning how to apply Design Thinking to our fellowship projects and generally to challenges facing journalism.

[... 869 words]

Weeknotes: ONA19, twitter-to-sqlite, datasette-rure

I’ve decided to start writing weeknotes for the duration of my JSK fellowship. Here goes!

[... 919 words]

sqlite-utils: a Python library and CLI tool for building SQLite databases

sqlite-utils is a combination Python library and command-line tool I’ve been building over the past six months which aims to make creating new SQLite databases as quick and easy as possible.

[... 1237 words]

Exploring search relevance algorithms with SQLite

SQLite isn’t just a fast, high quality embedded database: it also incorporates a powerful full-text search engine in the form of the FTS4 and FTS5 extensions. You’ve probably used these a bunch of times already: many iOS, Android and desktop applications use SQLite under-the-hood and use it to implement their built-in search.

[... 1398 words]

The interesting ideas in Datasette

Datasette (previously) is my open source tool for exploring and publishing structured data. There are a lot of ideas embedded in Datasette. I realized that I haven’t put many of them into writing.

[... 2857 words]

Notes from my appearance on the Changelog podcast

After I spoke at Zeit Day SF last weekend I sat down with Adam Stacoviak to record a 25 minute segment for episode 296 of the Changelog podcast, talking about Datasette. We covered a lot of ground!

[... 536 words]

Exploring the UK Register of Members Interests with SQL and Datasette

Ever wondered which UK Members of Parliament get gifted the most helicopter rides? How about which MPs have been given Christmas hampers by the Sultan of Brunei? (David Cameron, William Hague and Michael Howard apparently). Here’s how to dig through the Register of Members Interests using SQL and Datasette.

[... 1167 words]

Analyzing my Twitter followers with Datasette

I decided to do some ad-hoc analsis of my social network on Twitter this afternoon… and since everything is more fun if you bundle it up into a SQLite database and publish it to the internet I performed the analysis using Datasette.

[... 1314 words]

Building a location to time zone API with SpatiaLite, OpenStreetMap and Datasette

Given a latitude and longitude, how can we tell what time zone that point lies within? Here’s how I built a simple JSON API to answer that question, using a combination of data from OpenStreetMap, the SpatiaLite extension for SQLite and my Datasette API tool.

[... 2152 words]

New in Datasette: filters, foreign keys and search

I’ve released Datasette 0.13 with a number of exciting new features (Datasette previously).

[... 1143 words]

Datasette: instantly create and publish an API for your SQLite databases

I just shipped the first public version of datasette, a new tool for creating and publishing JSON APIs for SQLite databases.

[... 968 words]

Exciting stuff in Python 2.5

Python 2.5 alpha 1 is out, and as usual the What’s New in Python 2.5 document provides a pleasant overview of the new features. There are some real treats in there. While I’m hoping that the syntax for conditional expressions will grow on me, I’m looking forward to Partial function application becoming a common Python idiom. Relative imports are going to make Django applications a lot easier to redistribute, and I can’t wait to see all the crazy hacks that result from the introduction of coroutines.

[... 291 words]

Exciting developments with Django

The amount of activity surrounding the Django web framework since its not-quite release a few weeks ago is amazing. Adrian, Jacob and Wilson have been working over-time, with 395 check-ins to source control since the 13th of July. They’ve added WSGI support, a development web server, unit-tests, a ton of documentation, SQLite support, database introspection and dozens of other feature tweaks and bug fixes. Check out the project Timeline for an idea of just how frenetic things have been.

[... 271 words]

PHP 5 Release Candidate 1

I haven’t blogged much about PHP in a while because I’ve been up to my nose in mod_python and loving every minute of it. This news is just too important to miss: PHP 5 Release Candidate 1 has been released, bringing the first production-ready release tantilisingly close. While I doubt PHP 5 will tempt me back it’s definitely an exciting upgrade—my biggest complaint with PHP 4 is the brain-dead object model which defaults to copying whole objects rather than passing references, and this is one of the many things addressed by PHP 5. The new libxml2 powered XML features sound really powerful, and SQLite as an on-board database should be ideal for knocking out small stand-alone applications without needing to set up a mySQL database for them.

[... 173 words]

Installing PySQLite

Techno Weenie has a detailed guide to setting up PySQLite on boxes you don’t have root access to. SQLite looks ideal for small to medium sized applications so I can see this being really useful should I ever write something that uses it.

[... 48 words]

SQLObject

My new favourite toy is SQLObject, an object-relational mapper which makes heavy use of Python’s special method names to create objects which can be used to transparently access and modify data in a relational database. I tried to write something like this in PHP once before and failed miserably, but SQLObject has such an elegant design that I’m just annoyed I didn’t find out about it sooner. Here’s some example code, adapted from the SQLOBject site:

[... 249 words]

PHP philosophy, and bundling SQLite

Here’s a great quote from Rasmus Lerdorf (the creator of PHP) for people who have become disheartened with PHP’s lack of elegance when compared to other languages such as Python:

[... 232 words]

More lightweight software

The other toy I’ve been playing with recently is SQLite. SQLite is an embeddable SQL database engine written in just under 25,000 lines of (heavily commented) C. Don’t let the size fool you—it’s phenomenally powerful and is released under a no-holds-barred public domain license that practically begs you to include it in your applications, commercial or not.

[... 236 words]