Simon Willison's Weblog: gcap

Announcing dmigrations

2008-09-03T19:17:01+00:00

The team at Global Radio (formerly GCap Media) is the largest group of Django developers I've personally worked with, consisting of 14 developers split into two scrum teams, all contributing to the same overall codebase.

Working with that many developers makes smart tools and processes essential, and in some cases we've had to develop our own. Today, we're releasing one of them as an open source project.

dmigrations is a Django migrations tool. It addresses a common problem in Django development: if you change a model after creating the database tables for it with syncdb, how do you reflect those changes in your database tables without blowing away your existing data and starting again from scratch?

django-evolution attempts to address this problem the clever way, by detecting changes to models that are not yet reflected in the database schema and figuring out what needs to be done to bring the two back in sync. In contrast, dmigrations takes the stupid approach: it requires you to explicitly state the changes in a sequence of migrations, which will be applied in turn to bring a database up to the most recent state that reflects the underlying models.

This means extra work for developers who create migrations, but it also makes the whole process completely transparent - for our projects, we decided to go with the simplest system that could possibly work.

The interface to dmigrations is a pair of custom Django commands. The first, ./manage.py dmigrate, provides a set of command for listing, applying and unapplying (reverting) migrations. This entirely replaces Django's syncdb command.

The second, ./manage.py dmigration, provides commands for code-generating new migrations. It turns out that most migrations fit a set of common patterns: add a new table, add the tables for a new Django application, add a column to an existing table, add an index. These common cases are handled by dmigration; if you want to do something more complex (rename a column while transforming its data for example) you'll need to write a custom migration class.

The dmigrations tutorial provides a full introduction to both of these commands, as well as hints on writing your own custom migrations. Since migrations are just classes, one of our hopes is that external developers will write extra migration classes for operations like "rename column" - things that currently require a one-off custom migration.

dmigrations is actually the third iteration of our in-house migrations system. The first, smigrations, was designed to do the least amount of work possible to give us a controlled way of applying changes to our database schemas. The 's' stood for 'simple'. The second version (dmigrations) written by Tomasz Wegrzanowski consisted of a major upgrade to smigrations that addressed many of the frustrations we found when using it with branched development, in particular the problem of migrations in two branches conflicting with each other. The 'd' stood for 'distributed'.

Version three, released today, is my refactoring of dmigrations to de-couple it from the rest of our codebase. I've also stubbed out hooks for adding support for alternative database engines; dmigrations only supports MySQL out of the box, but I'm keen on getting it working with other databases now that it's out in the wild. Patches welcome!

How does this fit in with South and django-evolution?

That's an excellent question. We'll be discussing all three systems on the schema evolution panel at DjangoCon this weekend. I would love to see co-operation between the projects; at the very least I'd like to see the emergence of a standard Django-style abstraction library for create/alter table statements (something we punted on entirely with dmigrations). You'll certainly be hearing a lot more about migrations in Django after the conference.

Tags: django, dmigrations, gcap, globalradio, open-source, projects, python

Django: Security fix released

2008-09-03T00:14:00+00:00

Django: Security fix released

The Django admin used to save partially-submitted forms if your session expired, and continue the submission when you logged in. It turns out that’s actually an unblockable CSRF exploit and is hence broken as designed, so it’s now been removed. Thanks Ed Eliot and other GCap colleagues for helping me flesh out the potential attack.

Tags: csrf, django, django-admin, ed-eliot, exploit, gcap, security

Capital Radio (the new site)

2008-08-28T14:47:27+00:00

Capital Radio (the new site)

Launched today, this is the Django-powered project I’ve been working on with the fantastic team at GCap.

Tags: capitalradio, django, gcap

Back to full-time employment

2008-08-22T17:26:39+00:00

I've been freelance for a year and a half now, and it's been a great deal of fun. For me, being freelance meant having the freedom to pursue all sorts of different interests - technical writing, public speaking, Django, OpenID, JavaScript - and the opportunity to work with some really fantastic people.

It was going to take a very special opportunity to pull me back in to full-time employment, but I believe I've found that opportunity at the Guardian. I'll be joining them full time (well, four days a week) in mid-October as a software architect, collaborating with their development team on some ambitious API projects. The Guardian have access to a lot of interesting data and I can't wait to get stuck in to it. Since they're a newspaper, it shouldn't be a surprise that they scooped me to the story.

I'll be particularly sorry to say good-bye to the outstanding team I've been working with at GCap. I'm looking forward to talking about some of the things we've been working together over the next few weeks.

Tags: employment, gcap, guardian, jobs, personal-news

South

2008-08-08T11:42:00+00:00

South

A brand new light-weight Django migrations tool from Andrew Godwin. On first glance, this is spookily similar to the system we’ve been putting together at GCap.

Tags: andrew-godwin, django, gcap, migrations, south

Capital FM London Traffic Map

2008-06-27T18:22:05+00:00

Capital FM London Traffic Map

We launched this today at GCap (née Global Radio). I’m particularly impressed with how well the team handled clustering the traffic cameras on the Google map.

Tags: capitalfm, clustering, gcap, google-maps, london, traffic

Django Users Group London meetup, 19th of May

2008-05-02T12:19:45+00:00

Django Users Group London meetup, 19th of May

The inaugural meeting of DJUGL will be on the 19th of May at the Capital Radio building in Leicester Square, sponsored by GCap Media. Three presentations starting at 7pm (I’ll be giving one of them), then on to the pub. Sign up on EventWax; there are only 70 places.

Tags: django, djugl, events, gcap, gcapmedia, london, python