Simon Willison’s Weblog

MRSK. A new open source web application deployment tool from 37signals, developed to help migrate their Hey webmail app out of the cloud and onto their own managed hardware. The key feature is one that I care about deeply: it enables zero-downtime deploys by running all traffic through a Traefik reverse proxy in a way that allows requests to be paused while a new deployment is going out—so end users get a few seconds delay on their HTTP requests before being served by the replaced application. # 29th April 2023, 11:54 pm

Running Datasette on DigitalOcean App Platform (via) I spent some time with DigitalOcean’s new App Platform today, which is a Heroku-style PaaS that starts at $5/month. It looks like it could be a really good fit for Datasette. Disk is ephemeral, but if you’re publishing read-only data that doesn’t matter since you can build the SQLite database as part of the deployment and bundle it up in the Docker/Kubernetes container. # 7th October 2020, 2:52 am

Zero Downtime Release: Disruption-free Load Balancing of a Multi-Billion User Website (via) I remain fascinated by techniques for zero downtime deployment—once you have it working it makes shipping changes to your software so much less stressful, which means you can iterate faster and generally be much more confident in shipping code.

Facebook have invested vast amounts of effort into getting this right, and their new paper for the ACM SIGCOMM conference goes into detail about how it all works. # 5th August 2020, 3:27 am

How to do Zero Downtime Deployments of Docker Containers. I’m determined to get reliable zero-downtime deploys working for a new project, because I know from experience that even a few seconds of downtime during a deploy changes the project mentality from “deploy any time you want” to “don’t deploy too often”. I’m using Docker containers behind Traefik, which means new containers should have traffic automatically balanced to them by Traefik based on their labels. After much fiddling around the pattern described by this article worked best for me: it lets me start a new container, then stop the old one and have Traefik’s “retry” mechanism send any requests to the stopped container over to the new one instead. # 16th January 2020, 11:12 pm

How we use “ship small” to rapidly build new features at GitHub (via) Useful insight into how GitHub develop new features. They make aggressive use of feature flags, shipping a rough skeleton of a new feature to production as early as possible and actively soliciting feedback from other employees as they iterate on the feature. They static JSON mocks of APIs to unblock their frontend engineers and iterate on the necessary data structures while the real backend is bring implemented. # 2nd January 2020, 4:30 am

Azure Readiness Checklist (via) I love a good comprehensive checklist. This one is focused on large projects running on Azure but it’s still fun to browse through if you are hosting elsewhere, mainly as a reminder of quite how much still goes into deploying large web services into production. # 26th October 2019, 8:32 pm

What to do when PyPI goes down. My deployment scripts tend to rely on PyPI these days (they install dependencies in to a virtualenv) which makes me distinctly uncomfortable. Jacob explains how to use the PyPI mirrors that are starting to come online, but that won’t help if the PyPI listing links to an externally hosted file which starts to 404, as happened with the python-openid package quite recently (now fixed). The comments on the post discuss workarounds, including hosting your own PyPI mirror or bundling tar.gz files of your dependencies with your project. # 21st July 2010, 10:19 am

Automated deployments with Fabric—tips and tricks. “If it’s not in a Fabric fabfile, it’s not deployable”—I’m slowly applying this philosophy to my personal projects. # 16th March 2010, 11:19 am

Automate EC2 Instance Setup with user-data Scripts (via) I knew about EC2’s user-data feature—what I didn’t know is that the Alestic and Canonical images are configured so that if the user-data starts with #! the instance will automatically execute it as a shell script as soon as it boots up (after networking has been configured). # 11th March 2010, 12:31 pm

jacobian’s django-deployment-workshop. Notes and resources from Jacob’s 3 hour Django deployment workshop at PyCon, including example configuration files for Apache2 + mod_wsgi, nginx, PostgreSQL and pgpool. # 19th February 2010, 2:28 pm

Why toppcloud will not be agnostic. Ian Bicking’s toppcloud aims to offer deployment with the ease of use of AppEngine against a standard, open source Ubuntu + Python 2.6 + mod_wsgi + Varnish stack. Here he explains why he’s not going to vary the required components: keeping everything completely standardised means everyone gets the same bugs (and the same fixes). # 12th February 2010, 9:21 am

Hot Code Loading in Node.js. Blaine Cook’s patch for Node.js that enables Erlang-style hot code loading, so you can switch out your application logic without restarting the server or affecting existing requests. This could make deploying new versions of Node applications trivial. I’d love to see a Node hosting service that allows you to simply upload a script file and have it execute on the Web. # 31st January 2010, 1:57 pm

Fabric 0.9.0. A Python-based SSH automation and deployment tool. Released today, 0.9.0 is finally the official “stable” release—which is good, as it breaks API compatibility with previous versions and caused me all sorts of confusion when I tried to learn Fabric recently. # 9th November 2009, 2:02 pm

Automating web site deployment at Barcamp Brighton. I’m determined to start using Fabric and proper deployment scripts for my personal projects. # 6th September 2009, 2:16 pm

Fabric, Django, Git, Apache, mod_wsgi, virtualenv and pip deployment. I’m slowly working my way through this stack at the moment—next stop, fabric. # 28th July 2009, 11:56 am

Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip. Ashamed to say I’m not using any of these yet—for Django projects, my manage.py inserts an “ext” directory at the beginning of the Python path which contains my dependencies for that project. # 9th July 2009, 11:40 am

Phusion Passenger for nginx. Passenger (aka mod_rails / mod_rack) enables easy deployment of Rails and Ruby apps under Apache... and the latest version adds support for nginx as well. It works as an HTTP proxy and process manager, spawning worker processes and forwarding HTTP requests to them via a request queue. It can also handle Python WSGI applications—anyone tried it out for that yet? # 20th April 2009, 4:53 am

How to use Django with Apache and mod_wsgi. My favourite deployment option is now included in the official Django docs, thanks to Alex Gaynor. I tend to run a stripped down Apache with mod_wsgi behind an nginx proxy, and have nginx serve static files directly. This avoids the need for a completely separate media server (although a separate media domain is still a good idea for better client-side performance). # 1st April 2009, 12:24 am

Webistrano. Web based interface for managing Capistrano deployments. Cal recommends having a “deploy to live site” button in his book; this looks like an easy way to build that. # 31st August 2007, 11:05 pm

Quick Django Benching. Django under Apache/mod_python outperforms nginx/FastCGI and LightTPD/FastCGI once you ramp up the concurrency levels. My setup for this site (Apache/mod_python behind an nginx proxy, with nginx handling static files) should give the best of both worlds. # 17th February 2007, 4:56 pm

The Django Book: Deploying Django. Solid advice based on years of experience at the Journal-World and the Washington Post. # 26th January 2007, 12:38 pm