Simon Willison’s Weblog

Announcing dmigrations

3rd September 2008

The team at Global Radio (formerly GCap Media) is the largest group of Django developers I’ve personally worked with, consisting of 14 developers split into two scrum teams, all contributing to the same overall codebase.

Working with that many developers makes smart tools and processes essential, and in some cases we’ve had to develop our own. Today, we’re releasing one of them as an open source project.

dmigrations is a Django migrations tool. It addresses a common problem in Django development: if you change a model after creating the database tables for it with syncdb, how do you reflect those changes in your database tables without blowing away your existing data and starting again from scratch?

django-evolution attempts to address this problem the clever way, by detecting changes to models that are not yet reflected in the database schema and figuring out what needs to be done to bring the two back in sync. In contrast, dmigrations takes the stupid approach: it requires you to explicitly state the changes in a sequence of migrations, which will be applied in turn to bring a database up to the most recent state that reflects the underlying models.

This means extra work for developers who create migrations, but it also makes the whole process completely transparent—for our projects, we decided to go with the simplest system that could possibly work.

The interface to dmigrations is a pair of custom Django commands. The first, ./manage.py dmigrate, provides a set of command for listing, applying and unapplying (reverting) migrations. This entirely replaces Django’s syncdb command.

The second, ./manage.py dmigration, provides commands for code-generating new migrations. It turns out that most migrations fit a set of common patterns: add a new table, add the tables for a new Django application, add a column to an existing table, add an index. These common cases are handled by dmigration; if you want to do something more complex (rename a column while transforming its data for example) you’ll need to write a custom migration class.

The dmigrations tutorial provides a full introduction to both of these commands, as well as hints on writing your own custom migrations. Since migrations are just classes, one of our hopes is that external developers will write extra migration classes for operations like “rename column”—things that currently require a one-off custom migration.

dmigrations is actually the third iteration of our in-house migrations system. The first, smigrations, was designed to do the least amount of work possible to give us a controlled way of applying changes to our database schemas. The ’s’ stood for ’simple’. The second version (dmigrations) written by Tomasz Wegrzanowski consisted of a major upgrade to smigrations that addressed many of the frustrations we found when using it with branched development, in particular the problem of migrations in two branches conflicting with each other. The ’d’ stood for ’distributed’.

Version three, released today, is my refactoring of dmigrations to de-couple it from the rest of our codebase. I’ve also stubbed out hooks for adding support for alternative database engines; dmigrations only supports MySQL out of the box, but I’m keen on getting it working with other databases now that it’s out in the wild. Patches welcome!

How does this fit in with South and django-evolution?

That’s an excellent question. We’ll be discussing all three systems on the schema evolution panel at DjangoCon this weekend. I would love to see co-operation between the projects; at the very least I’d like to see the emergence of a standard Django-style abstraction library for create/alter table statements (something we punted on entirely with dmigrations). You’ll certainly be hearing a lot more about migrations in Django after the conference.