<?xml version="1.0" encoding="utf-8"?>
<feed xml:lang="en-us" xmlns="http://www.w3.org/2005/Atom"><title>Simon Willison's Weblog: boring-technology</title><link href="http://simonwillison.net/" rel="alternate"/><link href="http://simonwillison.net/tags/boring-technology.atom" rel="self"/><id>http://simonwillison.net/</id><updated>2026-03-09T13:37:45+00:00</updated><author><name>Simon Willison</name></author><entry><title>Perhaps not Boring Technology after all</title><link href="https://simonwillison.net/2026/Mar/9/not-so-boring/#atom-tag" rel="alternate"/><published>2026-03-09T13:37:45+00:00</published><updated>2026-03-09T13:37:45+00:00</updated><id>https://simonwillison.net/2026/Mar/9/not-so-boring/#atom-tag</id><summary type="html">
    &lt;p&gt;A recurring concern I've seen regarding LLMs for programming is that they will push our technology choices towards the tools that are best represented in their training data, making it harder for new, better tools to break through the noise.&lt;/p&gt;
&lt;p&gt;This was certainly the case a couple of years ago, when asking models for help with Python or JavaScript appeared to give much better results than questions about less widely used languages.&lt;/p&gt;
&lt;p&gt;With &lt;a href="https://simonwillison.net/tags/november-2025-inflection/"&gt;the latest models&lt;/a&gt; running in good coding agent harnesses I'm not sure this continues to hold up.&lt;/p&gt;
&lt;p&gt;I'm seeing excellent results with my &lt;a href="https://simonwillison.net/2026/Feb/17/chartroom-and-datasette-showboat/"&gt;brand new tools&lt;/a&gt; where I start by prompting "use uvx showboat --help / rodney --help / chartroom --help to learn about these tools" - the context length of these new models is long enough that they can consume quite a lot of documentation before they start working on a problem.&lt;/p&gt;
&lt;p&gt;Drop a coding agent into &lt;em&gt;any&lt;/em&gt; existing codebase that uses libraries and tools that are too private or too new to feature in the training data and my experience is that it works &lt;em&gt;just fine&lt;/em&gt; - the agent will consult enough of the existing examples to understand patterns, then iterate and test its own output to fill in the gaps.&lt;/p&gt;
&lt;p&gt;This is a surprising result. I thought coding agents would prove to be the ultimate embodiment of the &lt;a href="https://boringtechnology.club"&gt;Choose Boring Technology&lt;/a&gt; approach, but in practice they don't seem to be affecting my technology choices in that way at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: A few follow-on thoughts:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The issue of what technology LLMs &lt;em&gt;recommend&lt;/em&gt; is a separate one. &lt;a href="https://amplifying.ai/research/claude-code-picks"&gt;What Claude Code &lt;em&gt;Actually&lt;/em&gt; Chooses&lt;/a&gt; is an interesting recent study where Edwin Ong and Alex Vikati where they proved Claude Code over 2,000 times and found a strong bias towards build-over-buy but also identified a preferred technical stack, with GitHub Actions, Stripe, and shadcn/ui seeing a "near monopoly" in their respective categories. For the sake of this post my interest is in what happens when the human makes a technology choice that differs from those preferred by the model harness.&lt;/li&gt;
&lt;li&gt;The &lt;a href="https://simonwillison.net/tags/skills/"&gt;Skills&lt;/a&gt; mechanism that is being rapidly embraced by most coding agent tools is super-relevant here. We are already seeing projects release official skills to help agents use them - here are examples from &lt;a href="https://github.com/remotion-dev/skills"&gt;Remotion&lt;/a&gt;, &lt;a href="https://github.com/supabase/agent-skills"&gt;Supabase&lt;/a&gt;, &lt;a href="https://github.com/vercel-labs/agent-skills"&gt;Vercel&lt;/a&gt;, and &lt;a href="https://github.com/prisma/skills"&gt;Prisma&lt;/a&gt;.&lt;/li&gt;
&lt;/ol&gt;
    
        &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/ai"&gt;ai&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/generative-ai"&gt;generative-ai&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/llms"&gt;llms&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/ai-assisted-programming"&gt;ai-assisted-programming&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/coding-agents"&gt;coding-agents&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/agentic-engineering"&gt;agentic-engineering&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/november-2025-inflection"&gt;november-2025-inflection&lt;/a&gt;&lt;/p&gt;
    

</summary><category term="ai"/><category term="generative-ai"/><category term="llms"/><category term="ai-assisted-programming"/><category term="boring-technology"/><category term="coding-agents"/><category term="agentic-engineering"/><category term="november-2025-inflection"/></entry><entry><title>Hallucinations in code are the least dangerous form of LLM mistakes</title><link href="https://simonwillison.net/2025/Mar/2/hallucinations-in-code/#atom-tag" rel="alternate"/><published>2025-03-02T06:25:33+00:00</published><updated>2025-03-02T06:25:33+00:00</updated><id>https://simonwillison.net/2025/Mar/2/hallucinations-in-code/#atom-tag</id><summary type="html">
    &lt;p&gt;A surprisingly common complaint I see from developers who have tried using LLMs for code is that they encountered a hallucination - usually the LLM inventing a method or even a full software library that doesn't exist - and it crashed their confidence in LLMs as a tool for writing code. How could anyone productively use these things if they invent methods that don't exist?&lt;/p&gt;
&lt;p&gt;Hallucinations in code &lt;strong&gt;are the least harmful hallucinations you can encounter from a model&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;(When I talk about &lt;em&gt;hallucinations&lt;/em&gt; here I mean instances where an LLM invents a completely untrue fact, or in this case outputs code references which don't exist at all. I see these as a separate issue from bugs and other mistakes, which are the topic of the rest of this post.)&lt;/p&gt;

&lt;p&gt;The real risk from using LLMs for code is that they'll make mistakes that &lt;em&gt;aren't&lt;/em&gt; instantly caught by the language compiler or interpreter. And these happen &lt;em&gt;all the time&lt;/em&gt;!&lt;/p&gt;

&lt;p&gt;The moment you run LLM generated code, any hallucinated methods will be instantly obvious: you'll get an error. You can fix that yourself or you can feed the error back into the LLM and watch it correct itself.&lt;/p&gt;
&lt;p&gt;Compare this to hallucinations in regular prose, where you need a critical eye, strong intuitions and well developed fact checking skills to avoid sharing information that's incorrect and directly harmful to your reputation.&lt;/p&gt;
&lt;p&gt;With code you get a powerful form of fact checking for free. Run the code, see if it works.&lt;/p&gt;
&lt;p&gt;In some setups - &lt;a href="https://simonwillison.net/tags/code-interpreter/"&gt;ChatGPT Code Interpreter&lt;/a&gt;, &lt;a href="https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview"&gt;Claude Code&lt;/a&gt;, any of the growing number of "agentic" code systems that write and then execute code in a loop - the LLM system itself will spot the error and automatically correct itself.&lt;/p&gt;
&lt;p&gt;If you're using an LLM to write code without even running it yourself, &lt;em&gt;what are you doing?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Hallucinated methods are such a tiny roadblock that when people complain about them I assume they've spent minimal time learning how to effectively use these systems - they dropped them at the first hurdle.&lt;/p&gt;
&lt;p&gt;My cynical side suspects they may have been looking for a reason to dismiss the technology and jumped at the first one they found.&lt;/p&gt;
&lt;p&gt;My less cynical side assumes that nobody ever warned them that you have to put a lot of work in to learn how to get good results out of these systems. I've been exploring &lt;a href="https://simonwillison.net/tags/ai-assisted-programming/"&gt;their applications for writing code&lt;/a&gt; for over two years now and I'm still learning new tricks (and new strengths and weaknesses) almost every day.&lt;/p&gt;

&lt;h4 id="qa"&gt;Manually testing code is essential&lt;/h4&gt;

&lt;p&gt;Just because code looks good and runs without errors doesn't mean it's actually doing the right thing. No amount of meticulous code review - or even comprehensive automated tests - will demonstrably prove that code actually does the right thing. You have to run it yourself!&lt;/p&gt;
&lt;p&gt;Proving to yourself that the code works is your job. This is one of the many reasons I don't think LLMs are going to put software professionals out of work.&lt;/p&gt;
&lt;p&gt;LLM code will usually look fantastic: good variable names, convincing comments, clear type annotations and a logical structure. This can lull you into a false sense of security, in the same way that a gramatically correct and confident answer from ChatGPT might tempt you to skip fact checking or applying a skeptical eye.&lt;/p&gt;
&lt;p&gt;The way to avoid &lt;em&gt;those&lt;/em&gt; problems is the same as how you avoid problems in code by other humans that you are reviewing, or code that you've written yourself: you need to actively exercise that code. You need to have great manual QA skills.&lt;/p&gt;
&lt;p&gt;A general rule for programming is that you should &lt;em&gt;never&lt;/em&gt; trust any piece of code until you've seen it work with your own eye - or, even better, seen it fail and then fixed it.&lt;/p&gt;
&lt;p&gt;Across my entire career, almost every time I've assumed some code works without actively executing it - some branch condition that rarely gets hit, or an error message that I don't expect to occur - I've later come to regret that assumption.&lt;/p&gt;

&lt;h4 id="tips"&gt;Tips for reducing hallucinations&lt;/h4&gt;
&lt;p&gt;If you really are seeing a deluge of hallucinated details in the code LLMs are producing for you, there are a bunch of things you can do about it.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Try different models. It might be that another model has better training data for your chosen platform. As a Python and JavaScript programmer my favorite models right now are Claude 3.7 Sonnet with thinking turned on, OpenAI's o3-mini-high and GPT-4o with Code Interpreter (for Python).&lt;/li&gt;
&lt;li&gt;Learn how to use the context. If an LLM doesn't know a particular library you can often fix this by dumping in a few dozen lines of example code. LLMs are incredibly good at imitating things, and at rapidly picking up patterns from very limited examples. Modern model's have increasingly large context windows - I've recently started using Claude's new &lt;a href="https://support.anthropic.com/en/articles/10167454-using-the-github-integration"&gt;GitHub integration&lt;/a&gt; to dump entire repositories into the context and it's been working extremely well for me.&lt;/li&gt;
&lt;li&gt;Chose &lt;a href="https://boringtechnology.club/"&gt;boring technology&lt;/a&gt;. I genuinely find myself picking libraries that have been around for a while partly because that way it's much more likely that LLMs will be able to use them.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I'll finish this rant with a related observation: I keep seeing people say "if I have to review every line of code an LLM writes, it would have been faster to write it myself!"&lt;/p&gt;
&lt;p&gt;Those people are loudly declaring that they have under-invested in the crucial skills of reading, understanding and reviewing code written by other people. I suggest getting some more practice in. Reviewing code written for you by LLMs is a great way to do that.&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;&lt;em&gt;Bonus section&lt;/em&gt;: I asked Claude 3.7 Sonnet "extended thinking mode" to review an earlier draft of this post - "&lt;code&gt;Review my rant of a blog entry. I want to know if the argument is convincing, small changes I can make to improve it, if there are things I've missed.&lt;/code&gt;". It was quite helpful, especially in providing tips to make that first draft a little less confrontational! Since you can share Claude chats now &lt;a href="https://claude.ai/share/685cd6d9-f18a-47ef-ae42-e9815df821f1"&gt;here's that transcript&lt;/a&gt;.&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;&lt;strong&gt;Update March 11th 2025&lt;/strong&gt;: I wrote a longer piece about &lt;a href="https://simonwillison.net/2025/Mar/11/using-llms-for-code/"&gt;how I use LLMs to help me write code&lt;/a&gt;.&lt;/p&gt;
    
        &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/ai"&gt;ai&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/openai"&gt;openai&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/generative-ai"&gt;generative-ai&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/llms"&gt;llms&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/ai-assisted-programming"&gt;ai-assisted-programming&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/anthropic"&gt;anthropic&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/claude"&gt;claude&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/code-interpreter"&gt;code-interpreter&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/ai-agents"&gt;ai-agents&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/hallucinations"&gt;hallucinations&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/coding-agents"&gt;coding-agents&lt;/a&gt;&lt;/p&gt;
    

</summary><category term="ai"/><category term="openai"/><category term="generative-ai"/><category term="llms"/><category term="ai-assisted-programming"/><category term="anthropic"/><category term="claude"/><category term="boring-technology"/><category term="code-interpreter"/><category term="ai-agents"/><category term="hallucinations"/><category term="coding-agents"/></entry><entry><title>Matt Webb's Colophon</title><link href="https://simonwillison.net/2024/Oct/29/matt-webbs-colophon/#atom-tag" rel="alternate"/><published>2024-10-29T04:59:47+00:00</published><updated>2024-10-29T04:59:47+00:00</updated><id>https://simonwillison.net/2024/Oct/29/matt-webbs-colophon/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://interconnected.org/home/2024/10/28/colophon"&gt;Matt Webb&amp;#x27;s Colophon&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
I love a good colophon (&lt;a href="https://simonwillison.net/about/#about-site"&gt;here's mine&lt;/a&gt;, I should really expand it). Matt Webb has been publishing his thoughts online for 24 years, so his colophon is a delightful accumulation of ideas and principles.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;So following the principles of web longevity, what matters is the data, i.e. the posts, and simplicity. I want to minimise maintenance, not panic if a post gets popular, and be able to add new features without thinking too hard. [...]&lt;/p&gt;
&lt;p&gt;I don’t deliberately &lt;a href="https://boringtechnology.club/"&gt;choose boring technology&lt;/a&gt; but I think a lot about &lt;a href="https://interconnected.org/home/2017/08/17/upsideclown"&gt;longevity on the web&lt;/a&gt; &lt;em&gt;(that’s me writing about it in 2017)&lt;/em&gt; and boring technology is a consequence.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I'm tempted to adopt Matt's &lt;a href="https://github.com/genmon/aboutfeeds/blob/main/tools/pretty-feed-v3.xsl"&gt;XSL template&lt;/a&gt; that he uses to style &lt;a href="https://interconnected.org/home/feed"&gt;his RSS feed&lt;/a&gt; for my own sites.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/blogging"&gt;blogging&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/matt-webb"&gt;matt-webb&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/rss"&gt;rss&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/xslt"&gt;xslt&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;&lt;/p&gt;



</summary><category term="blogging"/><category term="matt-webb"/><category term="rss"/><category term="xslt"/><category term="boring-technology"/></entry><entry><title>Give people something to link to so they can talk about your features and ideas</title><link href="https://simonwillison.net/2024/Jul/13/give-people-something-to-link-to/#atom-tag" rel="alternate"/><published>2024-07-13T16:06:28+00:00</published><updated>2024-07-13T16:06:28+00:00</updated><id>https://simonwillison.net/2024/Jul/13/give-people-something-to-link-to/#atom-tag</id><summary type="html">
    &lt;p&gt;If you have a project, an idea, a product feature, or anything else that you want other people to understand and have conversations about... give them something to link to!&lt;/p&gt;
&lt;p&gt;Two illustrative examples are ChatGPT Code Interpreter and Boring Technology.&lt;/p&gt;
&lt;h4 id="chatgpt-code-interpreter-is-effectively-invisible"&gt;ChatGPT Code Interpreter is effectively invisible&lt;/h4&gt;
&lt;p&gt;&lt;strong&gt;ChatGPT Code Interpreter&lt;/strong&gt; has been one of my favourite AI tools for over a year. It's the feature of ChatGPT which allows the bot to write &lt;em&gt;and then execute&lt;/em&gt; Python code as part of responding to your prompts. It's incredibly powerful... and almost invisible! If you don't know how to use prompts to activate the feature you may not realize it exists.&lt;/p&gt;
&lt;p&gt;OpenAI don't even have a help page for it (and it very desperately needs documentation) - if you search their site you'll find &lt;a href="https://platform.openai.com/docs/assistants/tools/code-interpreter"&gt;confusing technical docs&lt;/a&gt; about an API feature and &lt;a href="https://community.openai.com/t/how-can-i-access-the-code-interpreter-plugin-model/205304"&gt;misleading outdated forum threads&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;I evangelize this tool &lt;em&gt;a lot&lt;/em&gt;, but OpenAI really aren't helping me do that. I end up linking people to &lt;a href="https://simonwillison.net/tags/code-interpreter/"&gt;my code-interpreter tag page&lt;/a&gt; because it's more useful than anything on OpenAI's own site.&lt;/p&gt;
&lt;p&gt;Compare this with Claude's similar Artifacts feature which at least has an &lt;a href="https://support.anthropic.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them"&gt;easily discovered help page&lt;/a&gt; - though &lt;a href="https://www.anthropic.com/news/claude-3-5-sonnet"&gt;the Artifacts announcement post&lt;/a&gt; was shared with Claude 3.5 Sonnet so isn't obviously linkable. Even that help page isn't quite what I'm after. Features deserve dedicated pages!&lt;/p&gt;
&lt;p&gt;GitHub understand this: here are their feature landing pages for &lt;a href="https://github.com/features/codespaces"&gt;Codespaces&lt;/a&gt; and &lt;a href="https://github.com/features/copilot"&gt;Copilot&lt;/a&gt; (I could even guess the URL for Copilot's page based on the Codespaces one).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; It turns out there IS documentation about Code Interpreter mode... but I failed to find it because it didn't use those terms anywhere on the page! The title is &lt;a href="https://help.openai.com/en/articles/8437071-data-analysis-with-chatgpt"&gt;Data analysis with ChatGPT&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This amuses me greatly because OpenAI have been oscillating on the name for this feature almost since they launched - Code Interpreter, then Advanced Data Analysis, now Data analysis with ChatGPT. I made fun of this &lt;a href="https://simonwillison.net/2023/Oct/17/open-questions/#open-questions.034.jpeg"&gt;last year&lt;/a&gt;.&lt;/p&gt;

&lt;h4 id="boring-technology-an-idea-with-a-website"&gt;Boring Technology: an idea with a website&lt;/h4&gt;
&lt;p&gt;Dan McKinley coined the term &lt;strong&gt;Boring Technology&lt;/strong&gt; in &lt;a href="https://mcfunley.com/choose-boring-technology"&gt;an essay in 2015&lt;/a&gt;. The key idea is that any development team has a limited capacity to solve new problems which should be reserved for the things that make their product unique. For everything else they should pick the most boring and well-understood technologies available to them - stuff where any bugs or limitations have been understood and discussed online for years.&lt;/p&gt;
&lt;p&gt;(I'm very proud that Django has earned the honorific of "boring technology" in this context!)&lt;/p&gt;
&lt;p&gt;Dan turned that essay into a talk, and then he turned that talk into a website with a brilliant domain name:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;a href="https://boringtechnology.club/"&gt;boringtechnology.club&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The idea has stuck. I've had many productive conversations about it, and more importantly if someone &lt;em&gt;hasn't&lt;/em&gt; heard the term before I can drop in that one link and they'll be up to speed a few minutes later.&lt;/p&gt;
&lt;p&gt;I've tried to do this myself for some of my own ideas: &lt;a href="https://simonwillison.net/2021/Jul/28/baked-data/"&gt;baked data&lt;/a&gt;, &lt;a href="https://simonwillison.net/2020/Oct/9/git-scraping/"&gt;git scraping&lt;/a&gt; and &lt;a href="https://simonwillison.net/series/prompt-injection/"&gt;prompt injection&lt;/a&gt; all have pages that I frequently link people to. I never went as far as committing to a domain though and I think maybe that was a mistake - having a clear message that "this is the key page to link to" is a very powerful thing.&lt;/p&gt;
&lt;h4 id="this-is-about-both-seo-and-conversations"&gt;This is about both SEO and conversations&lt;/h4&gt;
&lt;p&gt;One obvious goal here is SEO: if someone searches for your product feature you want them to land on your own site, not surrender valuable attention to someone else who's squatting on the search term.&lt;/p&gt;
&lt;p&gt;I personally value the conversation side of it even more. Hyperlinks are the best thing about the web - if I want to talk about something I'd much rather drop in a link to the definitive explanation rather than waste a paragraph (as I did earlier with Code Interpreter) explaining what the thing is for the upmteenth time!&lt;/p&gt;
&lt;p&gt;If you have an idea, project or feature that you want people to understand and discuss, build it the web page it deserves. &lt;strong&gt;Give people something to link to!&lt;/strong&gt;&lt;/p&gt;
    
        &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/github"&gt;github&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/marketing"&gt;marketing&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/seo"&gt;seo&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/writing"&gt;writing&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/openai"&gt;openai&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/chatgpt"&gt;chatgpt&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/claude"&gt;claude&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/code-interpreter"&gt;code-interpreter&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/coding-agents"&gt;coding-agents&lt;/a&gt;&lt;/p&gt;
    

</summary><category term="github"/><category term="marketing"/><category term="seo"/><category term="writing"/><category term="openai"/><category term="chatgpt"/><category term="claude"/><category term="boring-technology"/><category term="code-interpreter"/><category term="coding-agents"/></entry><entry><title>Serving a billion web requests with boring code</title><link href="https://simonwillison.net/2024/Jun/28/boring-code/#atom-tag" rel="alternate"/><published>2024-06-28T16:22:45+00:00</published><updated>2024-06-28T16:22:45+00:00</updated><id>https://simonwillison.net/2024/Jun/28/boring-code/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://notes.billmill.org/blog/2024/06/Serving_a_billion_web_requests_with_boring_code.html"&gt;Serving a billion web requests with boring code&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Bill Mill provides a deep retrospective from his work helping build a relaunch of the &lt;a href="https://www.medicare.gov/plan-compare/"&gt;medicare.gov/plan-compare&lt;/a&gt; site.&lt;/p&gt;
&lt;p&gt;It's a fascinating case study of the &lt;a href="https://boringtechnology.club/"&gt;choose boring technology&lt;/a&gt; mantra put into action. The "boring" choices here were PostgreSQL, Go and React, all three of which are so widely used and understood at this point that you're very unlikely to stumble into surprises with them.&lt;/p&gt;
&lt;p&gt;Key goals for the site were accessibility, in terms of users, devices and performance. Despite best efforts:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The result fell prey after a few years to a common failure mode of react apps, and became quite heavy and loaded somewhat slowly.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I've seen this pattern myself many times over, and I'd love to understand why. React itself isn't a particularly large dependency but somehow it always seems to lead to architectural bloat over time. Maybe that's more of an SPA thing than something that's specific to React.&lt;/p&gt;
&lt;p&gt;Loads of other interesting details in here. The ETL details - where brand new read-only RDS databases were spun up every morning after a four hour build process - are particularly notable.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://lobste.rs/s/icigm4/serving_billion_web_requests_with_boring"&gt;Lobste.rs&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/accessibility"&gt;accessibility&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/go"&gt;go&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/postgresql"&gt;postgresql&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/scaling"&gt;scaling&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/react"&gt;react&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;&lt;/p&gt;



</summary><category term="accessibility"/><category term="go"/><category term="postgresql"/><category term="scaling"/><category term="react"/><category term="boring-technology"/></entry><entry><title>Lateral Thinking with Withered Technology</title><link href="https://simonwillison.net/2024/Mar/14/lateral-thinking-with-weathered-technology/#atom-tag" rel="alternate"/><published>2024-03-14T04:13:57+00:00</published><updated>2024-03-14T04:13:57+00:00</updated><id>https://simonwillison.net/2024/Mar/14/lateral-thinking-with-weathered-technology/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="https://en.wikipedia.org/wiki/Gunpei_Yokoi#Lateral_Thinking_with_Withered_Technology"&gt;Lateral Thinking with Withered Technology&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
Gunpei Yokoi’s product design philosophy at Nintendo (“Withered” is also sometimes translated as “Weathered”). Use “mature technology that can be mass-produced cheaply”, then apply lateral thinking to find radical new ways to use it.&lt;/p&gt;

&lt;p&gt;This has echos for me of Dan McKinley’s “Choose Boring Technology”, which argues that in software projects you should default to a proven, stable stack so you can focus your innovation tokens on the problems that are unique to your project.


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/design"&gt;design&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/dan-mckinley"&gt;dan-mckinley&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;&lt;/p&gt;



</summary><category term="design"/><category term="dan-mckinley"/><category term="boring-technology"/></entry><entry><title>Porting VaccinateCA to Django</title><link href="https://simonwillison.net/2021/Apr/12/porting-vaccinateca-to-django/#atom-tag" rel="alternate"/><published>2021-04-12T05:18:48+00:00</published><updated>2021-04-12T05:18:48+00:00</updated><id>https://simonwillison.net/2021/Apr/12/porting-vaccinateca-to-django/#atom-tag</id><summary type="html">
    &lt;p&gt;As I mentioned &lt;a href="https://simonwillison.net/2021/Feb/28/vaccinateca/"&gt;back in February&lt;/a&gt;, I've been working with the &lt;a href="https://www.vaccinateca.com/"&gt;VaccinateCA&lt;/a&gt; project to try to bring the pandemic to an end a little earlier by helping gather as accurate a model as possible of where the Covid vaccine is available in California and how people can get it.&lt;/p&gt;
&lt;p&gt;The key activity at VaccinateCA is calling places to check on their availability and eligibility criteria. Up until last night this was powered by a heavily customized Airtable instance, accompanied by a custom JavaScript app for the callers that communicated with the Airtable API via some Netlify functions.&lt;/p&gt;
&lt;p&gt;Today, the flow is powered by a new custom Django backend, running on top of PostgreSQL.&lt;/p&gt;
&lt;h4&gt;The thing you should never do&lt;/h4&gt;
&lt;blockquote class="twitter-tweet"&gt;&lt;p lang="en" dir="ltr"&gt;Here&amp;#39;s one that took me fifteen years to learn: &amp;quot;let&amp;#39;s build a new thing and replace this&amp;quot; is hideously dangerous: 90% of the time you won&amp;#39;t fully replace the old thing, and now you have two problems!&lt;/p&gt;- Simon Willison (@simonw) &lt;a href="https://twitter.com/simonw/status/1145114228170190848?ref_src=twsrc%5Etfw"&gt;June 29, 2019&lt;/a&gt;&lt;/blockquote&gt;
&lt;p&gt;Replacing an existing system with a from-scratch rewrite is risky. Replacing a system that is built on something as flexible as Airtable that is evolving on a daily basis is positively terrifying!&lt;/p&gt;
&lt;p&gt;Airtable served us extremely well, but unfortunately there are hard limits to the number of rows Airtable can handle and we've already bounced up against them and had to archive some of our data. To keep scaling the organization we needed to migrate away.&lt;/p&gt;
&lt;p&gt;We needed to build a matching relational database with a comprehensive, permission-controlled interface for editing it, plus APIs to drive our website and application. And we needed to do it using the most &lt;a href="http://boringtechnology.club/"&gt;boring technology&lt;/a&gt; possible, so we could focus on solving problems directly rather than researching anything new.&lt;/p&gt;
&lt;p&gt;It will never cease to surprise me that Django has attained boring technology status! VaccineCA sits firmly in Django's sweet-spot. So we used that to build our replacement.&lt;/p&gt;
&lt;p&gt;The new Django-based system is called VIAL, for "Vaccine Information Archive and Library" - a neat &lt;a href="https://twitter.com/obra"&gt;Jesse Vincent&lt;/a&gt; bacronym.&lt;/p&gt;
&lt;p&gt;We switched things over to VIAL last night, but we still have activity in Airtable as well. I expect we'll keep using Airtable for the lifetime of the organization - there are plenty of ad-hoc data projects for which it's a perfect fit.&lt;/p&gt;
&lt;p&gt;The most important thing here is to have a trusted single point of truth for any piece of information. I'm not quite ready to declare victory on that point just yet, but hopefully once things settle down over the next few days.&lt;/p&gt;

&lt;p&gt;&lt;img src="https://static.simonwillison.net/static/2021/vial-index.png" style="max-width: 100%" alt="Screenshot of the Django admin VIAL index page" /&gt;&lt;/p&gt;

&lt;h4&gt;Data synchronization patterns&lt;/h4&gt;
&lt;p&gt;The first challenge, before even writing any code, was how to get stuff out of Airtable. I built a tool for this a while ago called &lt;a href="https://datasette.io/tools/airtable-export"&gt;airtable-export&lt;/a&gt;, and it turned out the VaccinateCA team were using it already before I joined!&lt;/p&gt;
&lt;p&gt;&lt;code&gt;airtable-export&lt;/code&gt; was already running several times an hour, backing up the data in JSON format to a GitHub repository (a form of &lt;a href="https://simonwillison.net/2020/Oct/9/git-scraping/"&gt;Git scraping&lt;/a&gt;). This gave us a detailed history of changes to the Airtable data, which occasionally proved extremely useful for answering questions about when a specific record was changed or deleted.&lt;/p&gt;
&lt;p&gt;Having the data in a GitHub repository was also useful because it gave us somewhere to pull data from that wasn't governed by Airtable's rate limits.&lt;/p&gt;
&lt;p&gt;I iterated through a number of different approaches for writing importers for the data.&lt;/p&gt;
&lt;p&gt;Each Airtable table ended up as a single JSON file in our GitHub repository, containing an array of objects - those files got pretty big, topping out at about 80MB.&lt;/p&gt;
&lt;p&gt;I started out with Django management commands, which could be passed a file or a URL. A neat thing about using GitHub for this is that you can use the "raw data" link to obtain a URL with a short-lived token, which grants access to that file. So I could create a short-term URL and paste it directly to my import tool.&lt;/p&gt;
&lt;p&gt;I don't have a good pattern for running Django management commands on Google Cloud Run, so I started moving to API-based import scripts instead.&lt;/p&gt;
&lt;p&gt;The pattern that ended up working best was to provide a &lt;code&gt;/api/importRecords&lt;/code&gt; API endpoint which accepts a JSON array of items.&lt;/p&gt;
&lt;p&gt;The API expects the input to have a unique primary key in each record - &lt;code&gt;airtable_id&lt;/code&gt; in our case. It then uses Django's &lt;a href="https://docs.djangoproject.com/en/3.2/ref/models/querysets/#update-or-create"&gt;update_or_create()&lt;/a&gt; ORM method to create new records if they were missing, and update existing records otherwise.&lt;/p&gt;
&lt;p&gt;One remaining challenge: posting 80MB of JSON to an API in one go would likely run into resource limits. I needed a way to break that input up into smaller batches.&lt;/p&gt;
&lt;p&gt;I ended up building a new tool for this called &lt;a href="https://github.com/simonw/json-post"&gt;json-post&lt;/a&gt;. It has an extremely specific use-case: it's for when you want to POST a big JSON array to an API endpoint but you want to first break it up into batches!&lt;/p&gt;
&lt;p&gt;Here's how to break up the JSON in &lt;code&gt;Reports.json&lt;/code&gt; into 50 item arrays and send them to that API as separate POSTs:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;json-post Reports.json \                              
   "https://example.com/api/importReports" \
   --batch-size 50
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Here are some more complex options. Here we need to pass an &lt;code&gt;Authorization: Bearer XXXtokenXXX&lt;/code&gt; API key header, run the array in reverse, record our progress (the JSON responses from the API as newline-delimited JSON) to a log file, set a longer HTTP read timeout and filter for just specific items:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;% json-post Reports.json \                              
   "https://example.com/api/importReports" \
  -h Authorization 'Bearer XXXtokenXXX' \
  --batch-size 50 \
  --reverse \
  --log /tmp/progress.txt \
  --http-read-timeout 20 \
  --filter 'item.get("is_soft_deleted")'
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;--filter&lt;/code&gt; option proved particularly useful. As we kicked the tires on VIAL we would spot new bugs - things like the import script failing to correctly record the &lt;code&gt;is_soft_deleted&lt;/code&gt; field we were using in Airtable. Being able to filter that input file with a command-line flag meant we could easily re-run the import just for a subset of reports that were affected by a particular bug.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;--filter&lt;/code&gt; takes a Python expression that gets compiled into a function and passed &lt;code&gt;item&lt;/code&gt; as the current item in the list. I borrowed the pattern from &lt;a href="https://datasette.io/tools/sqlite-transform#user-content-lambda-for-executing-your-own-code"&gt;my sqlite-transform tool&lt;/a&gt;.&lt;/p&gt;
&lt;h4 id="value-of-api-logs"&gt;The value of API logs&lt;/h4&gt;
&lt;p&gt;VaccineCA's JavaScript caller application used to send data to Airtable via a Netlify function, which allowed additional authentication to be added built using  Auth0.&lt;/p&gt;
&lt;p&gt;Back in February, the team had the bright idea to log the API traffic to that function to a separate base in Airtable - including full request and response bodies.&lt;/p&gt;
&lt;p&gt;This proved invaluable for debugging. It also meant that when I started building VIAL's alternative implementation of the "submit a call report" API I could replay historic API traffic that had been recorded in that table, giving me a powerful way to exercise the new API with real-world traffic.&lt;/p&gt;
&lt;p&gt;This meant that when we turned on VIAL we could switch our existing JavaScript SPA over to talking to it using a fully tested clone of the existing Airtable-backed API.&lt;/p&gt;
&lt;p&gt;VIAL implements this logging pattern again, this time using Django and PostgreSQL.&lt;/p&gt;
&lt;p&gt;Given that the writable APIs will recieve in the low thousands of requests a day, keeping them in a database table works great. The table has grown to 90MB so far. I'm hoping that the pandemic will be over before we have to worry about logging capacity!&lt;/p&gt;
&lt;p&gt;We're using PostgreSQL &lt;code&gt;jsonb&lt;/code&gt; columns to store the incoming and returned JSON, via Django's &lt;a href="https://docs.djangoproject.com/en/3.2/ref/models/fields/#jsonfield"&gt;JSONField&lt;/a&gt;. This means we can do in-depth API analysis using PostgreSQL's JSON SQL functions! Being able to examine returned JSON error messages or aggregate across incoming request bodies helped enormously when debugging problems with the API import scripts.&lt;/p&gt;
&lt;h4&gt;Storing the original JSON&lt;/h4&gt;
&lt;p&gt;Today, almost all of the data stored in VIAL originated in Airtable. One trick that has really helped build the system is that each of the tables that might contain imported data has both an &lt;code&gt;airtable_id&lt;/code&gt; nullable column and an &lt;code&gt;import_json&lt;/code&gt; JSON field.&lt;/p&gt;
&lt;p&gt;Any time we import a record from Airtable, we record both the ID and the full, original Airtable JSON that we used for the import.&lt;/p&gt;
&lt;p&gt;This is another powerful tool for debugging: we can view the original Airtable JSON directly in the Django admin interface for a record, and confirm that it matches the ORM fields that we set from that.&lt;/p&gt;
&lt;p&gt;I came up with a simple pattern for &lt;a href="https://til.simonwillison.net/django/pretty-print-json-admin"&gt;Pretty-printing all read-only JSON in the Django admin&lt;/a&gt; that helps with this too.&lt;/p&gt;
&lt;h4&gt;Staying as flexible as possible&lt;/h4&gt;
&lt;p&gt;The thing that worried me most about replacing Airtable with Django was Airtable's incredible flexibility. In the organization's short life it has already solved &lt;em&gt;so many&lt;/em&gt; problems by adding new columns in Airtable, or building new views.&lt;/p&gt;
&lt;p&gt;Is it possible to switch to custom software without losing that huge cultural advantage?&lt;/p&gt;
&lt;p&gt;This is the same reason it's so hard for custom software to compete with spreadsheets.&lt;/p&gt;
&lt;p&gt;We've only just made the switch, so we won't know for a while how well we've done at handling this. I have a few mechanisms in place that I'm hoping will help.&lt;/p&gt;
&lt;p&gt;The first is &lt;a href="https://github.com/simonw/django-sql-dashboard"&gt;django-sql-dashboard&lt;/a&gt;. I wrote about this project in previous weeknotes &lt;a href="https://simonwillison.net/2021/Mar/14/weeknotes/"&gt;here&lt;/a&gt; and &lt;a href="https://simonwillison.net/2021/Mar/21/django-sql-dashboard-widgets/"&gt;here&lt;/a&gt; - the goal is to bring some of the ideas from &lt;a href="https://datasette.io/"&gt;Datasette&lt;/a&gt; into the Django/PostgreSQL world, by providing a read-only mechanism for constructing SQL queries, bookmarking and saving the results and outputting simple SQL-driven visualizations.&lt;/p&gt;
&lt;p&gt;We have a lot of SQL knowledge at VaccinateCA, so my hope is that people with SQL will be able to solve their own problems, and people who don't know SQL yet will have no trouble finding someone who can help them.&lt;/p&gt;
&lt;p&gt;In the &lt;a href="http://boringtechnology.club/#17"&gt;boring technology&lt;/a&gt; model of things, &lt;code&gt;django-sql-dashboard&lt;/code&gt; counts as the main innovation token I'm spending for this project. I'm optimistic that it will pay off.&lt;/p&gt;
&lt;p&gt;I'm also leaning heavily on Django's migration system, with the aim of making database migrations common and boring, rather than their usual default of being rare and exciting. We're up to 77 migrations already, in a codebase that is just over two months old!&lt;/p&gt;
&lt;p&gt;I think a culture that evolves the database schema quickly and with as little drama as possible is crucial to maintaining the agility that this kind of organization needs.&lt;/p&gt;
&lt;p&gt;Aside from the Django Admin providing the editing interface, everything that comes into and goes out of VIAL happens through APIs. These are fully documented: I want people to be able to build against the APIs independently, especially for things like data import.&lt;/p&gt;
&lt;p&gt;After seeing significant success with PostgreSQL JSON already, I'm considering using it to add even more API-driven flexbility to VIAL in the future. Allowing our client developers to start collecting a new piece of data from our volunteers in an existing JSON field, then migrating that into a separate column once it has proven its value, is very tempting indeed.&lt;/p&gt;
&lt;h4&gt;Open source tools we are using&lt;/h4&gt;
&lt;p&gt;An incomplete list of open source packages we are using for VIAL so far:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://pydantic-docs.helpmanual.io/"&gt;pydantic&lt;/a&gt; - as a validation layer for some of the API endpoints&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/python-social-auth/social-app-django"&gt;social-auth-app-django&lt;/a&gt; - to integrate with &lt;a href="https://auth0.com/"&gt;Auth0&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/adamchainz/django-cors-headers"&gt;django-cors-headers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/mpdavis/python-jose"&gt;python-jose&lt;/a&gt; - for JWTs, which were already in use by our Airtable caller app&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/etianen/django-reversion"&gt;django-reversion&lt;/a&gt; and &lt;a href="https://github.com/jedie/django-reversion-compare/"&gt;django-reversion-compare&lt;/a&gt; to provide a diffable, revertable history of some of our core models&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/django-admin-tools/django-admin-tools"&gt;django-admin-tools&lt;/a&gt; - which adds a handy customizable menu to the admin, good for linking to additional custom tools&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/3YOURMIND/django-migration-linter"&gt;django-migration-linter&lt;/a&gt; - to help avoid accidentally shipping migrations that could cause downtime during a deploy&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://pytest-django.readthedocs.io/en/latest/"&gt;pytest-django&lt;/a&gt;, &lt;a href="https://github.com/adamchainz/time-machine"&gt;time-machine&lt;/a&gt; and &lt;a href="https://colin-b.github.io/pytest_httpx/"&gt;pytest-httpx&lt;/a&gt; for our unit tests&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.sentry.io/platforms/python/"&gt;sentry-sdk&lt;/a&gt;, &lt;a href="https://docs.honeycomb.io/getting-data-in/python/beeline/"&gt;honeycomb-beeline&lt;/a&gt; and  &lt;a href="https://github.com/prometheus/client_python"&gt;prometheus-client&lt;/a&gt; for error logging and observability&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;Want to help out?&lt;/h4&gt;
&lt;p&gt;VaccinateCA &lt;a href="https://twitter.com/patio11/status/1379587878624190466"&gt;is hiring&lt;/a&gt;! It's an interesting gig, because the ultimate goal is to end the pandemic and put this non-profit permanently out of business. So if you want to help end things faster, get in touch.&lt;/p&gt;

&lt;blockquote class="twitter-tweet"&gt;&lt;p lang="en" dir="ltr"&gt;VaccinateCA is hiring a handful of engineers to help scale our data ingestion and display by more than an order of magnitude.&lt;br /&gt;&lt;br /&gt;If you&amp;#39;d like to register interest:&lt;a href="https://t.co/BSvi40sW1M"&gt;https://t.co/BSvi40sW1M&lt;/a&gt;&lt;br /&gt;&lt;br /&gt;Generalists welcome. Three subprojects; Python backend/pedestrian front-end JS.&lt;/p&gt;- Patrick McKenzie (@patio11) &lt;a href="https://twitter.com/patio11/status/1379587878624190466?ref_src=twsrc%5Etfw"&gt;April 7, 2021&lt;/a&gt;&lt;/blockquote&gt;
&lt;h4&gt;TIL this week&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/simonw/til/blob/main/vscode/language-specific-indentation-settings.md"&gt;Language-specific indentation settings in VS Code&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/simonw/til/blob/main/django/efficient-bulk-deletions-in-django.md"&gt;Efficient bulk deletions in Django&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/simonw/til/blob/main/postgresql/unnest-csv.md"&gt;Using unnest() to use a comma-separated string as the input to an IN query&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;Releases this week&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/simonw/json-post"&gt;json-post&lt;/a&gt;&lt;/strong&gt;: &lt;a href="https://github.com/simonw/json-post/releases/tag/0.2"&gt;0.2&lt;/a&gt; - (&lt;a href="https://github.com/simonw/json-post/releases"&gt;3 total releases&lt;/a&gt;) - 2021-04-11
&lt;br /&gt;Tool for posting JSON to an API, broken into pages&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/simonw/airtable-export"&gt;airtable-export&lt;/a&gt;&lt;/strong&gt;: &lt;a href="https://github.com/simonw/airtable-export/releases/tag/0.7.1"&gt;0.7.1&lt;/a&gt; - (&lt;a href="https://github.com/simonw/airtable-export/releases"&gt;10 total releases&lt;/a&gt;) - 2021-04-09
&lt;br /&gt;Export Airtable data to YAML, JSON or SQLite files on disk&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/simonw/django-sql-dashboard"&gt;django-sql-dashboard&lt;/a&gt;&lt;/strong&gt;: &lt;a href="https://github.com/simonw/django-sql-dashboard/releases/tag/0.6a0"&gt;0.6a0&lt;/a&gt; - (&lt;a href="https://github.com/simonw/django-sql-dashboard/releases"&gt;13 total releases&lt;/a&gt;) - 2021-04-09
&lt;br /&gt;Django app for building dashboards using raw SQL queries&lt;/li&gt;
&lt;/ul&gt;
    
        &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/django"&gt;django&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/django-admin"&gt;django-admin&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/migrations"&gt;migrations&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/postgresql"&gt;postgresql&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/weeknotes"&gt;weeknotes&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/airtable"&gt;airtable&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/vaccinate-ca"&gt;vaccinate-ca&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;&lt;/p&gt;
    

</summary><category term="django"/><category term="django-admin"/><category term="migrations"/><category term="postgresql"/><category term="weeknotes"/><category term="airtable"/><category term="vaccinate-ca"/><category term="boring-technology"/></entry><entry><title>Getting started</title><link href="https://simonwillison.net/2021/Feb/22/vaccinateca-2021-02-22/#atom-tag" rel="alternate"/><published>2021-02-22T17:00:00+00:00</published><updated>2021-02-22T17:00:00+00:00</updated><id>https://simonwillison.net/2021/Feb/22/vaccinateca-2021-02-22/#atom-tag</id><summary type="html">
    &lt;p class="context"&gt;&lt;em&gt;Originally posted to my internal blog at VaccinateCA&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Here we go then... I've signed up to work on this project full-time, four days a week!&lt;/p&gt;
&lt;p&gt;Several years into my time at Eventbrite I started an internal engineering blog, and it quickly proved itself to be really valuable. The great thing about an internal blog is that it gives you somewhere to post status updates on existing projects, thoughts on what you're going to work on next and even ad-hoc documentation on things that you learn along the way.&lt;/p&gt;
&lt;p&gt;I don't know if I'll post here every day, and I don't intend to spend more than fifteen minutes on each post - but I think it's a good habit to establish early on.&lt;/p&gt;
&lt;h4&gt;
What I'm going to be doing&lt;/h4&gt;
&lt;p&gt;My initial goal, as directed by Jesse, is to help move VaccinateCA beyond Airtable as quickly as possible.&lt;/p&gt;
&lt;p&gt;Airtable is an amazing product, so this is not a trivial undertaking. Every startup founder needs to watch out that they don't end up accidentally competing against an Excel spreadsheet, because the spreadsheet will probably win. Airtable is already like a spreadsheet but better.&lt;/p&gt;
&lt;p&gt;But... if we're going to scale this thing to 10x its current size we need to gain complete ownership of our data.&lt;/p&gt;
&lt;p&gt;My plan for this first week is to spin up a full fidelity Django + PostgreSQL prototype of an alternative stack, to help decide as quickly as possible if that's the right path to go. A lesson I've learned from my career is that prototypes inevitable end up in production, so I build them with that in mind - the code I write will be tested, documented and ready to become not-a-prototype with as little additional effort as possible.&lt;/p&gt;
&lt;p&gt;Why Django and PostgreSQL? Because they are &lt;a href="http://boringtechnology.club/" rel="nofollow"&gt;boring technology&lt;/a&gt;, and that's a huge bonus for this kind of project.&lt;/p&gt;
&lt;p&gt;I'm also going to be talking to the Airtable power users within the organization to make sure I understand what works, what doesn't and what the weird use-cases are for the existing stack that I might not have figured out.&lt;/p&gt;
&lt;h4&gt;
What I've done&lt;/h4&gt;
&lt;p&gt;I've started building a few things agaist the existing public API, mainly because I think by building and this is a good way to help get familiar with the high level of what already exists.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://vaccinateca.datasette.io/" rel="nofollow"&gt;https://vaccinateca.datasette.io/&lt;/a&gt; is a public (initially private) Datasette instance providing an interactive interface against the public API data. Datasette is my principle open source project at the moment - you can learn more about that at &lt;a href="https://datasette.io/" rel="nofollow"&gt;https://datasette.io/&lt;/a&gt; - and you can see how this particular Datasette works by visiting its repo at &lt;a href="https://github.com/simonw/vaccinate-ca-datasette"&gt;https://github.com/simonw/vaccinate-ca-datasette&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/CAVaccineInventory/vaccinate-ca-history"&gt;https://github.com/CAVaccineInventory/vaccinate-ca-history&lt;/a&gt; is a private repo that's doing something weird and, I hope, valuable. It uses GitHub Actions to pull the latest copy of the public API data every ten minutes and commits any changes back to its own repository. It uses &lt;a href="https://github.com/simonw/csv-diff"&gt;csv-diff&lt;/a&gt; (actually &lt;a href="https://github.com/simonw/csv-diff/issues/12"&gt;a new feature&lt;/a&gt; I just added to csv-diff) to try to generate a human-readable commit message. The key feature here becomes &lt;a href="https://github.com/CAVaccineInventory/vaccinate-ca-history/commits/main"&gt;the commit history&lt;/a&gt;, which shows at a finely grained level what got changed and when.&lt;/li&gt;
&lt;li&gt;I started this internal blog&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;
What I need to do next&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;Short term: talk to people! Meet people who I'll be working with, understand what's been built so far and where everything is, use those conversations to help me decide what I should work on next.&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;
Onboarding&lt;/h4&gt;
&lt;p&gt;So far I'm on:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Slack&lt;/li&gt;
&lt;li&gt;Discord&lt;/li&gt;
&lt;li&gt;Airtable&lt;/li&gt;
&lt;li&gt;Auth0&lt;/li&gt;
&lt;li&gt;The GitHub organization&lt;/li&gt;
&lt;li&gt;Google account&lt;/li&gt;
&lt;li&gt;Honeycomb (via the new Google account)&lt;/li&gt;
&lt;/ul&gt;
    
        &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/vaccinate-ca"&gt;vaccinate-ca&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/vaccinate-ca-blog"&gt;vaccinate-ca-blog&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;&lt;/p&gt;
    

</summary><category term="vaccinate-ca"/><category term="vaccinate-ca-blog"/><category term="boring-technology"/></entry><entry><title>Choose Boring Technology</title><link href="https://simonwillison.net/2019/Jul/3/choose-boring-technology/#atom-tag" rel="alternate"/><published>2019-07-03T18:09:47+00:00</published><updated>2019-07-03T18:09:47+00:00</updated><id>https://simonwillison.net/2019/Jul/3/choose-boring-technology/#atom-tag</id><summary type="html">
    
&lt;p&gt;&lt;strong&gt;&lt;a href="http://boringtechnology.club/"&gt;Choose Boring Technology&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
The definitive write-up of Dan McKinley’s presentation on why you should mostly use “boring” technology rather than going after the latest shiniest stack components. There’s so much accumulated wisdom in here. I particularly like how Dan owns up to having introduced Scala and MongoDB at Etsy before eventually helping remove them and go back to something less exciting and far more predictable. Also neat: the site is generated using Dan’s better-keynote-export tool which helps turn Keynote presentations into a flat web page with notes and images.

    &lt;p&gt;&lt;small&gt;&lt;/small&gt;Via &lt;a href="https://news.ycombinator.com/item?id=20323246"&gt;Hacker News&lt;/a&gt;&lt;/small&gt;&lt;/p&gt;


    &lt;p&gt;Tags: &lt;a href="https://simonwillison.net/tags/dan-mckinley"&gt;dan-mckinley&lt;/a&gt;, &lt;a href="https://simonwillison.net/tags/boring-technology"&gt;boring-technology&lt;/a&gt;&lt;/p&gt;



</summary><category term="dan-mckinley"/><category term="boring-technology"/></entry></feed>