Simon Willison’s Weblog

24 items tagged “sqlite”

Datasette: instantly create and publish an API for your SQLite databases

I just shipped the first public version of datasette, a new tool for creating and publishing JSON APIs for SQLite databases.

[... 968 words]

simonw/csvs-to-sqlite. I built a simple tool for bulk converting multiple CSV files into a SQLite database. # 13th November 2017, 6:49 am

Pull request #4120 · python/cpython. I just had my first ever change merged into Python! It was a one sentence documentation improvement (on how to cancel SQLite operations) but it was fascinating seeing how Python’s GitHub flow is set up—clever use of labels, plus a bot that automatically checks that you have signed a copy of their CLA. # 7th November 2017, 2:06 pm

A Minimalist Guide to SQLite. Pretty comprehensive actually—covers the sqlite3 command line app, importing CSVs, integrating with Python, Pandas and Jupyter notebooks, visualization and more. # 2nd November 2017, 1:23 am

System Requirements For SQLite. Document describing the high level goals and objectives of SQLite. Like everything to do with SQLite this exhibits some incredibly well thought out software engineering. I particularly like “S80000: SQLite shall exhibit ductile failure characteristics“ where ductile is described in opposition to brittle: a ductile system begins showing signs of trouble well in advance of failure. # 22nd October 2017, 9:53 pm

Getting the Most out of Sqlite3 with Python. A couple of neat tricks I didn’t know: you can skip cursors entirely by calling .execute and .executemany directly on the connection object, and you can use the connection object as a context manager to execute transactions using a “with” block. # 22nd October 2017, 12:35 pm for television. Dale Lane’s neat hack to visualise his television watching habits. An Ubuntu / vdx home theatre stores TV events in SQLite, and graphs are generated using Python and Open Flash Chart 2. The really clever bit: the back-end captures nearby bluetooth IDs’ allowing events to be filtered by the people watching based on the presence of their mobile phones. # 7th January 2010, 7:28 pm

Browsing my browsing. Roo Reynolds used the MeeTimer Firefox extension to gather statistics on his browsing habits, then extracted data directly from the SQLite database and generated his own graphs using PHP and the canvas element. # 10th April 2009, 8:48 am

iPhone Backup Extractor possibilities (via) Nick Ludlam points out that iTunes backs up your iPhone call records by copying across a sqlite database—which means it wouldn’t be at all hard to extract the logs in to a larger database. Could make for a really cool addition to a private lifestreaming application. # 10th November 2008, 10:41 pm

GeoCouch: Geospatial queries with CouchDB. Interesting approach: uses “external2”, a branch that allows external services to be called from CouchDB. SQLite’s SpatiaLite extension is then used as an external spacial index. # 27th October 2008, 11:48 pm

asql—Apache SQL querying. Command line tool for loading web server log files in common log format in to a SQLite database, with a built-in interactive shell. # 14th October 2008, 11:22 am

Gearshift. Whoa, a full migrations library written in JavaScript for Gears (which uses SQLite for its data store). # 15th September 2008, 2:51 pm

Django Unit Tests and Transactions. If you’re using a transactional database engine (MySQL with InnoDB, Postgres or SQLite) you can speed things up by running each of your unit tests inside a transaction and rolling back in tearDown(). # 7th July 2008, 2:14 pm

OpenStreetMap on the iPhone! Via an ingenious hack. The Google Maps iPhone client caches downloaded tiles using SQLite—to display your own custom tiles, you just need to dump them straight in to the “cache”. # 22nd October 2007, 3:30 pm

WebKit Does HTML5 Client-side Database Storage. SQLite strikes again. The WebKit team have included a neat update to their Web Inspector that lets you browse and modify your client-side databases. # 20th October 2007, 12:03 pm

Incompatible SQLite in OS X and Python. I’ve hit this problem; James has the solution. # 28th November 2006, 5:40 pm

SQLite Keynote. SQLite 3.3.8 has full text indexing! # 21st October 2006, 11:44 pm

Exciting stuff in Python 2.5

Python 2.5 alpha 1 is out, and as usual the What’s New in Python 2.5 document provides a pleasant overview of the new features. There are some real treats in there. While I’m hoping that the syntax for conditional expressions will grow on me, I’m looking forward to Partial function application becoming a common Python idiom. Relative imports are going to make Django applications a lot easier to redistribute, and I can’t wait to see all the crazy hacks that result from the introduction of coroutines.

[... 291 words]

Exciting developments with Django

The amount of activity surrounding the Django web framework since its not-quite release a few weeks ago is amazing. Adrian, Jacob and Wilson have been working over-time, with 395 check-ins to source control since the 13th of July. They’ve added WSGI support, a development web server, unit-tests, a ton of documentation, SQLite support, database introspection and dozens of other feature tweaks and bug fixes. Check out the project Timeline for an idea of just how frenetic things have been.

[... 271 words]

PHP 5 Release Candidate 1

I haven’t blogged much about PHP in a while because I’ve been up to my nose in mod_python and loving every minute of it. This news is just too important to miss: PHP 5 Release Candidate 1 has been released, bringing the first production-ready release tantilisingly close. While I doubt PHP 5 will tempt me back it’s definitely an exciting upgrade—my biggest complaint with PHP 4 is the brain-dead object model which defaults to copying whole objects rather than passing references, and this is one of the many things addressed by PHP 5. The new libxml2 powered XML features sound really powerful, and SQLite as an on-board database should be ideal for knocking out small stand-alone applications without needing to set up a mySQL database for them.

[... 173 words]

Installing PySQLite

Techno Weenie has a detailed guide to setting up PySQLite on boxes you don’t have root access to. SQLite looks ideal for small to medium sized applications so I can see this being really useful should I ever write something that uses it.

[... 48 words]


My new favourite toy is SQLObject, an object-relational mapper which makes heavy use of Python’s special method names to create objects which can be used to transparently access and modify data in a relational database. I tried to write something like this in PHP once before and failed miserably, but SQLObject has such an elegant design that I’m just annoyed I didn’t find out about it sooner. Here’s some example code, adapted from the SQLOBject site:

[... 249 words]

PHP philosophy, and bundling SQLite

Here’s a great quote from Rasmus Lerdorf (the creator of PHP) for people who have become disheartened with PHP’s lack of elegance when compared to other languages such as Python:

[... 232 words]

More lightweight software

The other toy I’ve been playing with recently is SQLite. SQLite is an embeddable SQL database engine written in just under 25,000 lines of (heavily commented) C. Don’t let the size fool you—it’s phenomenally powerful and is released under a no-holds-barred public domain license that practically begs you to include it in your applications, commercial or not.

[... 236 words]