3rd September 2008
The team at Global Radio (formerly GCap Media) is the largest group of Django developers I’ve personally worked with, consisting of 14 developers split into two scrum teams, all contributing to the same overall codebase.
Working with that many developers makes smart tools and processes essential, and in some cases we’ve had to develop our own. Today, we’re releasing one of them as an open source project.
dmigrations is a Django migrations tool. It addresses a common problem in Django development: if you change a model after creating the database tables for it with
syncdb, how do you reflect those changes in your database tables without blowing away your existing data and starting again from scratch?
django-evolution attempts to address this problem the clever way, by detecting changes to models that are not yet reflected in the database schema and figuring out what needs to be done to bring the two back in sync. In contrast, dmigrations takes the stupid approach: it requires you to explicitly state the changes in a sequence of migrations, which will be applied in turn to bring a database up to the most recent state that reflects the underlying models.
This means extra work for developers who create migrations, but it also makes the whole process completely transparent—for our projects, we decided to go with the simplest system that could possibly work.
The interface to dmigrations is a pair of custom Django commands. The first,
./manage.py dmigrate, provides a set of command for listing, applying and unapplying (reverting) migrations. This entirely replaces Django’s
./manage.py dmigration, provides commands for code-generating new migrations. It turns out that most migrations fit a set of common patterns: add a new table, add the tables for a new Django application, add a column to an existing table, add an index. These common cases are handled by dmigration; if you want to do something more complex (rename a column while transforming its data for example) you’ll need to write a custom migration class.
The dmigrations tutorial provides a full introduction to both of these commands, as well as hints on writing your own custom migrations. Since migrations are just classes, one of our hopes is that external developers will write extra migration classes for operations like “rename column”—things that currently require a one-off custom migration.
dmigrations is actually the third iteration of our in-house migrations system. The first, smigrations, was designed to do the least amount of work possible to give us a controlled way of applying changes to our database schemas. The ’s’ stood for ’simple’. The second version (dmigrations) written by Tomasz Wegrzanowski consisted of a major upgrade to smigrations that addressed many of the frustrations we found when using it with branched development, in particular the problem of migrations in two branches conflicting with each other. The ’d’ stood for ’distributed’.
Version three, released today, is my refactoring of dmigrations to de-couple it from the rest of our codebase. I’ve also stubbed out hooks for adding support for alternative database engines; dmigrations only supports MySQL out of the box, but I’m keen on getting it working with other databases now that it’s out in the wild. Patches welcome!
That’s an excellent question. We’ll be discussing all three systems on the schema evolution panel at DjangoCon this weekend. I would love to see co-operation between the projects; at the very least I’d like to see the emergence of a standard Django-style abstraction library for create/alter table statements (something we punted on entirely with dmigrations). You’ll certainly be hearing a lot more about migrations in Django after the conference.
More recent articles
- Weeknotes: Embeddings, more embeddings and Datasette Cloud - 17th September 2023
- Build an image search engine with llm-clip, chat with models with llm chat - 12th September 2023
- LLM now provides tools for working with embeddings - 4th September 2023
- Datasette 1.0a4 and 1.0a5, plus weeknotes - 30th August 2023
- Making Large Language Models work for you - 27th August 2023
- Datasette Cloud, Datasette 1.0a3, llm-mlc and more - 16th August 2023
- How I make annotated presentations - 6th August 2023
- Weeknotes: Plugins for LLM, sqlite-utils and Datasette - 5th August 2023
- Catching up on the weird world of LLMs - 3rd August 2023
- Run Llama 2 on your own Mac using LLM and Homebrew - 1st August 2023