http://simonwillison.net/2024-12-13T05:29:10+00:00Simon Willison2024-12-13T05:29:10+00:002024-12-13T05:29:10+00:00https://simonwillison.net/2024/Dec/13/openai-postmortem/#atom-tag

<p><strong><a href="https://status.openai.com/incidents/ctrsv3lwd797">OpenAI&#x27;s postmortem for API, ChatGPT &amp; Sora Facing Issues</a></strong></p> OpenAI had an outage across basically everything for four hours on Wednesday. They've now published a detailed postmortem which includes some fascinating technical details about their "hundreds of Kubernetes clusters globally".</p> <p>The culprit was a newly deployed telemetry system:</p> <blockquote> <p>Telemetry services have a very wide footprint, so this new service’s configuration unintentionally caused every node in each cluster to execute resource-intensive Kubernetes API operations whose cost scaled with the size of the cluster. With thousands of nodes performing these operations simultaneously, the Kubernetes API servers became overwhelmed, taking down the Kubernetes control plane in most of our large clusters. [...]</p> <p>The Kubernetes data plane can operate largely independently of the control plane, but DNS relies on the control plane – services don’t know how to contact one another without the Kubernetes control plane. [...]</p> <p>DNS caching mitigated the impact temporarily by providing stale but functional DNS records. However, as cached records expired over the following 20 minutes, services began failing due to their reliance on real-time DNS resolution.</p> </blockquote> <p>It's always DNS. <p><small></small>Via <a href="https://twitter.com/therealadamg/status/1867393379287650778">@therealadamg</a></small></p> <p>Tags: <a href="https://simonwillison.net/tags/dns">dns</a>, <a href="https://simonwillison.net/tags/devops">devops</a>, <a href="https://simonwillison.net/tags/kubernetes">kubernetes</a>, <a href="https://simonwillison.net/tags/postmortem">postmortem</a>, <a href="https://simonwillison.net/tags/openai">openai</a>, <a href="https://simonwillison.net/tags/chatgpt">chatgpt</a></p>

2013-01-08T10:32:00+00:002013-01-08T10:32:00+00:00https://simonwillison.net/2013/Jan/8/what-are-good-and/#atom-tag

<p><em>My answer to <a href="https://www.quora.com/What-are-good-and-easy-practices-for-frequent-web-deployments/answer/Simon-Willison">What are good and easy practices for frequent web deployments?</a> on Quora</em></p> <p>At Lanyrd we use a combination of Fabric to drive our deploy scripts, git to get the code on to the servers, puppet for configuration management and Jenkins to run continuous integration tests and provide a "deploy the site" button.</p> <p>Here are a few important techniques I've learned:</p> <ul><li>Use symlink switching to keep the previous version of the code around, so you can switch back in the case of problems (that said, we've never actually used this capability - but its nice for atomic deploys as well)</li><li>Have your build script rename your static asset files (CSS/JS/etc) to include part of the md5 hash of the file contents in their filename. This means you can upload them to your static host provider (we use S3) before you run a deploy, guaranteeing that freshly deployed templates will point to the right files. It also keeps the older versions around in case you need to roll back.</li><li>Having one button that deploys the site is invaluable</li><li>Deploys need to be almost "free" in terms of impact on site performance - if it doesn't cost anything to deploy the site people will be freely able to deploy often and push out small fixes, which is good for the health of your codebase</li><li>Get new engineers to deploy on the first day! Doing so forces you/them to get a full development and deployment environment up and running for them on day one, which means that they can start doing real work on day two.</li></ul> <p>Tags: <a href="https://simonwillison.net/tags/continuous-integration">continuous-integration</a>, <a href="https://simonwillison.net/tags/startups">startups</a>, <a href="https://simonwillison.net/tags/quora">quora</a>, <a href="https://simonwillison.net/tags/lanyrd">lanyrd</a>, <a href="https://simonwillison.net/tags/devops">devops</a></p>

2010-12-10T10:04:00+00:002010-12-10T10:04:00+00:00https://simonwillison.net/2010/Dec/10/etsy/#atom-tag

<p><strong><a href="http://codeascraft.etsy.com/2010/12/08/track-every-release/">Tracking Every Release</a></strong></p> How Etsy use Graphite to monitor their continuous deployment releases. <p>Tags: <a href="https://simonwillison.net/tags/continuous-deployment">continuous-deployment</a>, <a href="https://simonwillison.net/tags/monitoring">monitoring</a>, <a href="https://simonwillison.net/tags/devops">devops</a>, <a href="https://simonwillison.net/tags/recovered">recovered</a>, <a href="https://simonwillison.net/tags/etsy">etsy</a></p>