Simon Willison’s Weblog

Big Opportunities in Small Data

8th May 2023

I gave an invited keynote at Citus Con 2023, the PostgreSQL conference. Below is the abstract, video, slides and links from the presentation.

Abstract

Civic data is more abundant than ever, with local and national governments around the world publishing rich data to open data portals. Every organization has untapped data about their business, and every individual has untapped data about their personal activity. This data is measured in megabytes, not terabytes. What’s missing are the tools that help people understand small data: too big for Excel, but not so big that it demands a Big Data warehouse.

Datasette is built on SQLite. Why SQLite? It’s tiny, fast, and ubiquitous—and supports a workflow where databases can be created, shared, and even discarded with ease.

Through the lens of Datasette and SQLite, Simon explores this problem space and asks how the PostgreSQL ecosystem can evolve to address the fascinating opportunities presented by Small Data.

Video

• Why talk about Small data? (01:31)
• Demo of Datasette (02:23)
• Read-only SQL queries via API (08:03)
• Datasette built on SQLite (11:57)
• Baked Data architectural pattern (13:37)
• SQL + HTTP as integration tool (16:30)
• Baked Data with PostgreSQL? (18:44)
• Demo of Datasette in WebAssembly in browser (19:48)

Slides

Links

• San Francisco’s open data portal
  • City Facilities
• Datasette
  • datasette-cluster-map
• My Substack newsletter
  • The tool I use to generate it
  • Semi-automating a Substack newsletter with an Observable notebook
• niche-museums.com
  • The Misalignment Museum
  • How that Atom feed works
• Datasette Lite
  • How that works
  • Datasette Lite source code on GitHub
• What’s in the RedPajama-Data-1T LLM training set
• Crunchy Data’s interactive SQL tutorial
• Django SQL Dashboard
• My Things I Learn blog