Simon Willison's Weblog: design-patterns

Writing about Agentic Engineering Patterns

2026-02-23T17:43:02+00:00

I've started a new project to collect and document Agentic Engineering Patterns - coding practices and patterns to help get the best results out of this new era of coding agent development we find ourselves entering.

I'm using Agentic Engineering to refer to building software using coding agents - tools like Claude Code and OpenAI Codex, where the defining feature is that they can both generate and execute code - allowing them to test that code and iterate on it independently of turn-by-turn guidance from their human supervisor.

I think of vibe coding using its original definition of coding where you pay no attention to the code at all, which today is often associated with non-programmers using LLMs to write code.

Agentic Engineering represents the other end of the scale: professional software engineers using coding agents to improve and accelerate their work by amplifying their existing expertise.

There is so much to learn and explore about this new discipline! I've already published a lot under my ai-assisted-programming tag (345 posts and counting) but that's been relatively unstructured. My new goal is to produce something that helps answer the question "how do I get good results out of this stuff" all in one place.

I'll be developing and growing this project here on my blog as a series of chapter-shaped patterns, loosely inspired by the format popularized by Design Patterns: Elements of Reusable Object-Oriented Software back in 1994.

I published the first two chapters today:

Writing code is cheap now talks about the central challenge of agentic engineering: the cost to churn out initial working code has dropped to almost nothing, how does that impact our existing intuitions about how we work, both individually and as a team?
Red/green TDD describes how test-first development helps agents write more succinct and reliable code with minimal extra prompting.

I hope to add more chapters at a rate of 1-2 a week. I don't really know when I'll stop, there's a lot to cover!

Written by me, not by an LLM

I have a strong personal policy of not publishing AI-generated writing under my own name. That policy will hold true for Agentic Engineering Patterns as well. I'll be using LLMs for proofreading and fleshing out example code and all manner of other side-tasks, but the words you read here will be my own.

Chapters and Guides

Agentic Engineering Patterns isn't exactly a book, but it's kind of book-shaped. I'll be publishing it on my site using a new shape of content I'm calling a guide. A guide is a collection of chapters, where each chapter is effectively a blog post with a less prominent date that's designed to be updated over time, not frozen at the point of first publication.

Guides and chapters are my answer to the challenge of publishing "evergreen" content on a blog. I've been trying to find a way to do this for a while now. This feels like a format that might stick.

If you're interested in the implementation you can find the code in the Guide, Chapter and ChapterChange models and the associated Django views, almost all of which was written by Claude Opus 4.6 running in Claude Code for web accessed via my iPhone.

Tags: blogging, design-patterns, projects, writing, ai, generative-ai, llms, ai-assisted-programming, vibe-coding, coding-agents, agentic-engineering

Enough AI copilots! We need AI HUDs

2025-07-27T22:15:55+00:00

Enough AI copilots! We need AI HUDs

Geoffrey Litt compares Copilots - AI assistants that you engage in dialog with and work with you to complete a task - with HUDs, Head-Up Displays, which enhance your working environment in less intrusive ways.

He uses spellcheck as an obvious example, providing underlines for incorrectly spelt words, and then suggests his AI-implemented custom debugging UI as a more ambitious implementation of that pattern.

Plenty of people have expressed interest in LLM-backed interfaces that go beyond chat or editor autocomplete. I think HUDs offer a really interesting way to frame one approach to that design challenge.

Tags: design, design-patterns, ai, generative-ai, llms, geoffrey-litt

Design Patterns for Securing LLM Agents against Prompt Injections

2025-06-13T13:26:43+00:00

This new paper by 11 authors from organizations including IBM, Invariant Labs, ETH Zurich, Google and Microsoft is an excellent addition to the literature on prompt injection and LLM security.

In this work, we describe a number of design patterns for LLM agents that significantly mitigate the risk of prompt injections. These design patterns constrain the actions of agents to explicitly prevent them from solving arbitrary tasks. We believe these design patterns offer a valuable trade-off between agent utility and security.

Here's the full citation: Design Patterns for Securing LLM Agents against Prompt Injections (2025) by Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, Ezinwanne Ozoani, Andrew Paverd, Florian Tramèr, and Václav Volhejn.

I'm so excited to see papers like this starting to appear. I wrote about Google DeepMind's Defeating Prompt Injections by Design paper (aka the CaMeL paper) back in April, which was the first paper I'd seen that proposed a credible solution to some of the challenges posed by prompt injection against tool-using LLM systems (often referred to as "agents").

This new paper provides a robust explanation of prompt injection, then proposes six design patterns to help protect against it, including the pattern proposed by the CaMeL paper.

The scope of the problem

The authors of this paper very clearly understand the scope of the problem:

As long as both agents and their defenses rely on the current class of language models, we believe it is unlikely that general-purpose agents can provide meaningful and reliable safety guarantees.

This leads to a more productive question: what kinds of agents can we build today that produce useful work while offering resistance to prompt injection attacks? In this section, we introduce a set of design patterns for LLM agents that aim to mitigate — if not entirely eliminate — the risk of prompt injection attacks. These patterns impose intentional constraints on agents, explicitly limiting their ability to perform arbitrary tasks.

This is a very realistic approach. We don't have a magic solution to prompt injection, so we need to make trade-offs. The trade-off they make here is "limiting the ability of agents to perform arbitrary tasks". That's not a popular trade-off, but it gives this paper a lot of credibility in my eye.

This paragraph proves that they fully get it (emphasis mine):

The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment. At a minimum, this means that restricted agents must not be able to invoke tools that can break the integrity or confidentiality of the system. Furthermore, their outputs should not pose downstream risks — such as exfiltrating sensitive information (e.g., via embedded links) or manipulating future agent behavior (e.g., harmful responses to a user query).

The way I think about this is that any exposure to potentially malicious tokens entirely taints the output for that prompt. Any attacker who can sneak in their tokens should be considered to have complete control over what happens next - which means they control not just the textual output of the LLM but also any tool calls that the LLM might be able to invoke.

Let's talk about their design patterns.

The Action-Selector Pattern

A relatively simple pattern that makes agents immune to prompt injections — while still allowing them to take external actions — is to prevent any feedback from these actions back into the agent.

Agents can trigger tools, but cannot be exposed to or act on the responses from those tools. You can't read an email or retrieve a web page, but you can trigger actions such as "send the user to this web page" or "display this message to the user".

They summarize this pattern as an "LLM-modulated switch statement", which feels accurate to me.

The Plan-Then-Execute Pattern

A more permissive approach is to allow feedback from tool outputs back to the agent, but to prevent the tool outputs from influencing the choice of actions taken by the agent.

The idea here is to plan the tool calls in advance before any chance of exposure to untrusted content. This allows for more sophisticated sequences of actions, without the risk that one of those actions might introduce malicious instructions that then trigger unplanned harmful actions later on.

Their example converts "send today’s schedule to my boss John Doe" into a calendar.read() tool call followed by an email.write(..., 'john.doe@company.com'). The calendar.read() output might be able to corrupt the body of the email that is sent, but it won't be able to change the recipient of that email.

The LLM Map-Reduce Pattern

The previous pattern still enabled malicious instructions to affect the content sent to the next step. The Map-Reduce pattern involves sub-agents that are directed by the co-ordinator, exposed to untrusted content and have their results safely aggregated later on.

In their example an agent is asked to find files containing this month's invoices and send them to the accounting department. Each file is processed by a sub-agent that responds with a boolean indicating whether the file is relevant or not. Files that were judged relevant are then aggregated and sent.

They call this the map-reduce pattern because it reflects the classic map-reduce framework for distributed computation.

The Dual LLM Pattern

I get a citation here! I described the The Dual LLM pattern for building AI assistants that can resist prompt injection back in April 2023, and it influenced the CaMeL paper as well.

They describe my exact pattern, and even illustrate it with this diagram:

The key idea here is that a privileged LLM co-ordinates a quarantined LLM, avoiding any exposure to untrusted content. The quarantined LLM returns symbolic variables - $VAR1 representing a summarized web page for example - which the privileged LLM can request are shown to the user without being exposed to that tainted content itself.

The Code-Then-Execute Pattern

This is the pattern described by DeepMind's CaMeL paper. It's an improved version of my dual LLM pattern, where the privileged LLM generates code in a custom sandboxed DSL that specifies which tools should be called and how their outputs should be passed to each other.

The DSL is designed to enable full data flow analysis, such that any tainted data can be marked as such and tracked through the entire process.

The Context-Minimization pattern

To prevent certain user prompt injections, the agent system can remove unnecessary content from the context over multiple interactions.

For example, suppose that a malicious user asks a customer service chatbot for a quote on a new car and tries to prompt inject the agent to give a large discount. The system could ensure that the agent first translates the user’s request into a database query (e.g., to find the latest offers). Then, before returning the results to the customer, the user’s prompt is removed from the context, thereby preventing the prompt injection.

I'm slightly confused by this one, but I think I understand what it's saying. If a user's prompt is converted into a SQL query which returns raw data from a database, and that data is returned in a way that cannot possibly include any of the text from the original prompt, any chance of a prompt injection sneaking through should be eliminated.

The case studies

The rest of the paper presents ten case studies to illustrate how thes design patterns can be applied in practice, each accompanied by detailed threat models and potential mitigation strategies.

Most of these are extremely practical and detailed. The SQL Agent case study, for example, involves an LLM with tools for accessing SQL databases and writing and executing Python code to help with the analysis of that data. This is a highly challenging environment for prompt injection, and the paper spends three pages exploring patterns for building this in a responsible way.

Here's the full list of case studies. It's worth spending time with any that correspond to work that you are doing:

OS Assistant
SQL Agent
Email & Calendar Assistant
Customer Service Chatbot
Booking Assistant
Product Recommender
Resume Screening Assistant
Medication Leaflet Chatbot
Medical Diagnosis Chatbot
Software Engineering Agent

Here's an interesting suggestion from that last Software Engineering Agent case study on how to safely consume API information from untrusted external documentation:

The safest design we can consider here is one where the code agent only interacts with untrusted documentation or code by means of a strictly formatted interface (e.g., instead of seeing arbitrary code or documentation, the agent only sees a formal API description). This can be achieved by processing untrusted data with a quarantined LLM that is instructed to convert the data into an API description with strict formatting requirements to minimize the risk of prompt injections (e.g., method names limited to 30 characters).

Utility: Utility is reduced because the agent can only see APIs and no natural language descriptions or examples of third-party code.

Security: Prompt injections would have to survive being formatted into an API description, which is unlikely if the formatting requirements are strict enough.

I wonder if it is indeed safe to allow up to 30 character method names... it could be that a truly creative attacker could come up with a method name like run_rm_dash_rf_for_compliance() that causes havoc even given those constraints.

Closing thoughts

I've been writing about prompt injection for nearly three years now, but I've never had the patience to try and produce a formal paper on the subject. It's a huge relief to see papers of this quality start to emerge.

Prompt injection remains the biggest challenge to responsibly deploying the kind of agentic systems everyone is so excited to build. The more attention this family of problems gets from the research community the better.

Tags: design-patterns, security, ai, prompt-injection, generative-ai, llms, exfiltration-attacks, ai-agents, paper-review

Malleable software

2025-06-11T19:21:39+00:00

Malleable software

New, delightful manifesto from Ink & Switch.

In this essay, we envision malleable software: tools that users can reshape with minimal friction to suit their unique needs. Modification becomes routine, not exceptional. Adaptation happens at the point of use, not through engineering teams at distant corporations.

This is a beautifully written essay. I love the early framing of a comparison with physical environments such as the workshop of a luthier:

A guitar maker sets up their workshop with their saws, hammers, chisels and files arranged just so. They can also build new tools as needed to achieve the best result—a wooden block as a support, or a pair of pliers sanded down into the right shape. […] In the physical world, the act of crafting our environments comes naturally, because physical reality is malleable.

Most software doesn’t have these qualities, or requires deep programming skills in order to make customizations. The authors propose “malleable software” as a new form of computing ecosystem to “give users agency as co-creators”.

They mention plugin systems as one potential path, but highlight their failings:

However, plugin systems still can only edit an app's behavior in specific authorized ways. If there's not a plugin surface available for a given customization, the user is out of luck. (In fact, most applications have no plugin API at all, because it's hard work to design a good one!)

There are other problems too. Going from installing plugins to making one is a chasm that's hard to cross. And each app has its own distinct plugin system, making it typically impossible to share plugins across different apps.

Does AI-assisted coding help? Yes, to a certain extent, but there are still barriers that we need to tear down:

We think these developments hold exciting potential, and represent a good reason to pursue malleable software at this moment. But at the same time, AI code generation alone does not address all the barriers to malleability. Even if we presume that every computer user could perfectly write and edit code, that still leaves open some big questions.

How can users tweak the existing tools they've installed, rather than just making new siloed applications? How can AI-generated tools compose with one another to build up larger workflows over shared data? And how can we let users take more direct, precise control over tweaking their software, without needing to resort to AI coding for even the tiniest change?

They describe three key design patterns: a gentle slope from user to creator (as seen in Excel and HyperCard), focusing on tools, not apps (a kitchen knife, not an avocado slicer) and encouraging communal creation.

I found this note inspiring when considering my own work on Datasette:

Many successful customizable systems such as spreadsheets, HyperCard, Flash, Notion, and Airtable follow a similar pattern: a media editor with optional programmability. When an environment offers document editing with familiar direct manipulation interactions, users can get a lot done without needing to write any code.

The remainder of the essay focuses on Ink & Switch's own prototypes in this area, including Patchwork, Potluck and Embark.

Honestly, this is one of those pieces that defies attempts to summarize it. It's worth carving out some quality time to spend with this.

Via lobste.rs

Tags: design-patterns, ai, generative-ai, llms, ai-assisted-programming, local-first, geoffrey-litt, ink-and-switch

The Baked Data architectural pattern

2021-07-28T20:23:44+00:00

I've been exploring an architectural pattern for publishing websites over the past few years that I call the "Baked Data" pattern. It provides many of the advantages of static site generators while avoiding most of their limitations. I think it deserves to be used more widely.

I define the Baked Data architectural pattern as the following:

Baked Data: bundling a read-only copy of your data alongside the code for your application, as part of the same deployment

Most dynamic websites keep their code and data separate: the code runs on an application server, the data lives independently in some kind of external data store - something like PostgreSQL, MySQL or MongoDB.

With Baked Data, the data is deployed as part of the application bundle. Any time the content changes, a fresh copy of the site is deployed that includes those updates.

I mostly use SQLite database files for this, but plenty of other formats can work here too.

This works particularly well with so-called "serverless" deployment platforms - platforms that support stateless deployments and only charge for resources spent servicing incoming requests ("scale to zero").

Since every change to the data results in a fresh deployment this pattern doesn't work for sites that change often - but in my experience many content-oriented sites update their content at most a few times a day. Consider blogs, documentation sites, project websites - anything where content is edited by a small group of authors.

Benefits of Baked Data

Why would you want to apply this pattern? A few reasons:

Inexpensive to host. Anywhere that can run application code can host a Baked Data application - there's no need to pay extra for a managed database system. Scale to zero serverless hosts such as Cloud Run, Vercel or AWS Lambda will charge only cents per month for low-traffic deployments.
Easy to scale. Need to handle more traffic? Run more copies of your application and its bundled data. Horizontally scaling Baked Data applications is trivial. They're also a great fit to run behind a caching proxy CDN such as Cloudflare or Fastly - when you deploy a new version you can purge that entire cache.
Difficult to break. Hosting server-side applications on a VPS is always disquieting because there's so much that might go wrong - the server could be compromised, or a rogue log file could cause it to run out of disk space. With Baked Data the worst that can happen is that you need to re-deploy the application - there's no risk at all of data loss, and providers that can auto-restart code can recover from errors automatically.
Server-side functionality is supported. Static site generators provide many of the above benefits, but with the limitation that any dynamic functionality needs to happen in client-side JavaScript. With a Baked Data application you can execute server-side code too.
Templated pages. Another improvement over static site generators: if you have 10,000 pages, a static site generator will need to generate 10,000 HTML files. With Baked Data those 10,000 pages can exist as rows in a single SQLite database file, and the pages can be generated at run-time using a server-side template.
Easy to support multiple formats. Since your content is in a dynamic data store, outputting that same content in alternative formats is easy. I use Datasette plugins for this: datasette-atom can produce an Atom feed from a SQL query, and datasette-ics does the same thing for iCalendar feeds.
Integrates well with version control. I like to keep my site content under version control. The Baked Data pattern works well with build scripts that read content from a git repository and use it to build assets that are bundled with the deployment.

How to bake your data

My initial implementations of Baked Data have all used SQLite. It's an ideal format for this kind of application: a single binary file which can store anything that can be represented as relational tables, JSON documents or binary objects - essentially anything at all.

Any format that can be read from disk by your dynamic server-side code will work too: YAML or CSV files, Berkeley DB files, or anything else that can be represented by a bucket of read-only bytes in a file on disk.

[I have a hunch that you could even use something like PostgreSQL, MySQL or Elasticsearch by packaging up their on-disk representations and shipping them as part of a Docker container, but I've not tried that myself yet.]

Once your data is available in a file, your application code can read from that file and use it to generate and return web pages.

You can write code that does this in any server-side language. I use Python, usually with my Datasette application server which can read from a SQLite database file and use Jinja templates to generate pages.

The final piece of the puzzle is a build and deploy script. I use GitHub Actions for this, but any CI tool will work well here. The script builds the site content into a deployable asset, then deploys that asset along with the application code to a hosting platform.

Baked Data in action: datasette.io

The most sophisticated Baked Data site I've published myself is the official website for my Datasette project, datasette.io - source code in this repo.

The site is deployed using Cloud Run. It's actually a heavily customized Datasette instance, using a custom template for the homepage, custom pages for other parts of the site and the datasette-template-sql plugin to execute SQL queries and display their results from those templates.

The site currently runs off four database files:

content.db has most of the site content. It is built inside GitHub Actions by the build.sh script, which does the following:
- Import the contents of the news.yaml file into a news table using yaml-to-sqlite.
- Import the markdown files from the for/ folder (use-cases for Datasette) into the uses table using markdown-to-sqlite.
- Populate the plugin_repos and tool_repos single-column tables using data from more YAML files. These are used in the next step.
- Runs the build_directory.py Python script. This uses the GitHub GraphQL API to fetch information about all of those plugin and tool repositories, including their README files and their most recent tagged releases.
- Populates a stats table with the latest download statistics for all of the Datasette ecosystem PyPI packages. That data is imported from a stats.json file in my simonw/package-stats repository, which is itself populated by this git scraping script that runs in GitHub Actions. I also use this for my Datasette Downloads Observable notebook.
blog.db contains content from my blog that carries any of the datasette, dogsheep or sqliteutils tags.
- This is fetched by the fetch_blog_content.py script, which hits the paginated per-tag Atom feed for my blog content, implemented in Django here.
docs-index.db is a database table containing the documentation for the most recent stable Datasette release, broken up by sections.
- This database file is downloaded from a separate site, stable-docs.datasette.io, which is built and deployed as part of Datasette's release process.
dogsheep-index.db is the search index that powers site search (e.g. this search for dogsheep).
- The search index is built by dogsheep-beta using data pulled from tables in the other database files, as configured by this YAML file.

The site is automatically deployed once a day by a scheduled action, and I can also manually trigger that action if I want to ensure a new software release is reflected on the homepage.

Other real-world examples of Baked Data

I'm currently running two other sites using this pattern:

Niche Museums is my blog about tiny museums that I've visited. Again, it's Datasette with custom templates. Most of the content comes from this museums.yaml file, but I also run a script to figure out when each item was created or updated from the git history.
My TILs site runs on Vercel and is built from my simonw/til GitHub repository by this build script (populating this tils table). It uses the GitHub API to convert GitHub Flavored Markdown to HTML. I'm also running a script that generates small screenshots of each page and stashes them in a BLOB column in SQLite in order to provide social media preview cards, see Social media cards for my TILs.

My favourite example of this pattern in a site that I haven't worked on myself is Mozilla.org.

They started using SQLite back in 2018 in a system they call Bedrock - Paul McLanahan provides a detailed description of how this works.

Their site content lives in a ~22MB SQLite database file, which is built and uploaded to S3 and then downloaded on a regular basis to each of their application servers.

You can view their healthcheck page to see when the database was last downloaded, and grab a copy of the SQLite file yourself. It's fun to explore that using Datasette:

Compared to static site generators

Static site generators have exploded in popularity over the past ten years. They drive the cost of hosting a site down to almost nothing, provide excellent performance, work well with CDNs and produce sites that are extremely unlikely to break.

Used carefully, the Baked Data keeps most of these characteristics while still enabling server-side code execution.

My example sites use this in a few different ways:

datasette.io provides search across 1,588 different pieces of content, plus simpler search on the plugins and tools pages.
My TIL site also provides search, as does Niche Museums.
All three sites provide Atom feeds that are configured using a server-side SQL query: Datasette, Niche Museums, TILs.
Niche Museums offers a "Use my location" button which then serves museums near you, using a SQL query that makes use of the datasette-haversine plugin.

A common complaint about static site generators when used for larger sites is that build times can get pretty long if the builder has to generate tens of thousands of pages.

With Baked Data, 10,000 pages can be generated by a single template file and 10,000 rows in a SQLite database table.

This also makes for a faster iteration cycle during development: you can edit a template and hit "refresh" to see any page rendered by the new template instantly, without needing to rebuild any pages.

Want to give this a go?

If you want to give the Baked Data pattern a try, I recommend starting out using the combination of Datasette, GitHub Actions and Vercel. Hopefully the examples I've provided above are a good starting point - also feel free to reach out to me on Twitter or in the Datasette Discussions forum with any questions.

Tags: definitions, design-patterns, sqlite, static-generator, datasette, baked-data

Documentation unit tests

2018-07-28T15:59:55+00:00

Or: Test-driven documentation.

Keeping documentation synchronized with an evolving codebase is difficult. Without extreme discipline, it’s easy for documentation to get out-of-date as new features are added.

One thing that can help is keeping the documentation for a project in the same repository as the code itself. This allows you to construct the ideal commit: one that includes the code change, the updated unit tests AND the accompanying documentation all in the same unit of work.

When combined with a code review system (like Phabricator or GitHub pull requests) this pattern lets you enforce documentation updates as part of the review process: if a change doesn’t update the relevant documentation, point that out in your review!

Good code review systems also execute unit tests automatically and attach the results to the review. This provides an opportunity to have the tests enforce other aspects of the codebase: for example, running a linter so that no-one has to waste their time arguing over standardize coding style.

I’ve been experimenting with using unit tests to ensure that aspects of a project are covered by the documentation. I think it’s a very promising technique.

Introspect the code, introspect the docs

The key to this trick is introspection: interogating the code to figure out what needs to be documented, then parsing the documentation to see if each item has been covered.

I’ll use my Datasette project as an example. Datasette’s test_docs.py module contains three relevant tests:

test_config_options_are_documented checks that every one of Datasette’s configuration options are documented.
test_plugin_hooks_are_documented ensures all of the plugin hooks (powered by pluggy) are covered in the plugin documentation.
test_view_classes_are_documented iterates through all of the *View classes (corresponding to pages in the Datasette user interface) and makes sure they are covered.

In each case, the test uses introspection against the relevant code areas to figure out what needs to be documented, then runs a regular expression against the documentation to make sure it is mentioned in the correct place.

Obviously the tests can’t confirm the quality of the documentation, so they are easy to cheat: but they do at least protect against adding a new option but forgetting to document it.

Testing that Datasette’s view classes are covered

Datasette’s view classes use a naming convention: they all end in View. The current list of view classes is DatabaseView, TableView, RowView, IndexView and JsonDataView.

Since these classes are all imported into the datasette.app module (in order to be hooked up to URL routes) the easiest way to introspect them is to import that module, then run dir(app) and grab any class names that end in View. We can do that with a Python list comprehension:

from datasette import app
views = [v for v in dir(app) if v.endswith("View")]

I’m using reStructuredText labels to mark the place in the documentation that addresses each of these classes. This also ensures that each documentation section can be linked to, for example:

http://datasette.readthedocs.io/en/latest/pages.html#tableview

The reStructuredText syntax for that label looks like this:

.. _TableView:

Table
=====

The table page is the heart of Datasette...

We can extract these labels using a regular expression:

from pathlib import Path
import re

docs_path = Path(__file__).parent.parent / 'docs'
label_re = re.compile(r'\.\. _([^\s:]+):')

def get_labels(filename):
    contents = (docs_path / filename).open().read()
    return set(label_re.findall(contents))

Since Datasette’s documentation is spread across multiple *.rst files, and I want the freedom to document a view class in any one of them, I iterate through every file to find the labels and pull out the ones ending in View:

def documented_views():
    view_labels = set()
    for filename in docs_path.glob("*.rst"):
        for label in get_labels(filename):
            first_word = label.split("_")[0]
            if first_word.endswith("View"):
                view_labels.add(first_word)
    return view_labels

We now have a list of class names and a list of labels across all of our documentation. Writing a basic unit test comparing the two lists is trivial:

def test_view_documentation():
    view_labels = documented_views()
    view_classes = set(v for v in dir(app) if v.endswith("View"))
    assert view_labels == view_classes

Taking advantage of pytest

Datasette uses pytest for its unit tests, and documentation unit tests are a great opportunity to take advantage of some advanced pytest features.

Parametrization

The first of these is parametrization: pytest provides a decorator which can be used to execute a single test function multiple times, each time with different arguments.

This example from the pytest documentation shows how parametrization works:

import pytest
@pytest.mark.parametrize("test_input,expected", [
    ("3+5", 8),
    ("2+4", 6),
    ("6*9", 42),
])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

pytest treats this as three separate unit tests, even though they share a single function definition.

We can combine this pattern with our introspection to execute an independent unit test for each of our view classes. Here’s what that looks like:

@pytest.mark.parametrize("view", [v for v in dir(app) if v.endswith("View")])
def test_view_classes_are_documented(view):
    assert view in documented_views()

Here’s the output from pytest if we execute just this unit test (and one of our classes is undocumented):

$ pytest -k test_view_classes_are_documented -v
=== test session starts ===
collected 249 items / 244 deselected

tests/test_docs.py::test_view_classes_are_documented[DatabaseView] PASSED [ 20%]
tests/test_docs.py::test_view_classes_are_documented[IndexView] PASSED [ 40%]
tests/test_docs.py::test_view_classes_are_documented[JsonDataView] PASSED [ 60%]
tests/test_docs.py::test_view_classes_are_documented[RowView] PASSED [ 80%]
tests/test_docs.py::test_view_classes_are_documented[TableView] FAILED [100%]

=== FAILURES ===

view = 'TableView'

    @pytest.mark.parametrize("view", [v for v in dir(app) if v.endswith("View")])
    def test_view_classes_are_documented(view):
>       assert view in documented_views()
E       AssertionError: assert 'TableView' in {'DatabaseView', 'IndexView', 'JsonDataView', 'RowView', 'Table2View'}
E        +  where {'DatabaseView', 'IndexView', 'JsonDataView', 'RowView', 'Table2View'} = documented_views()

tests/test_docs.py:77: AssertionError
=== 1 failed, 4 passed, 244 deselected in 1.13 seconds ===

Fixtures

There’s a subtle inefficiency in the above test: for every view class, it calls the documented_views() function - and that function then iterates through every *.rst file in the docs/ directory and uses a regular expression to extract the labels. With 5 view classes and 17 documentation files that’s 85 executions of get_labels(), and that number will only increase as Datasette’s code and documentation grow larger.

We can use pytest’s neat fixtures to reduce this to a single call to documented_views() that is shared across all of the tests. Here’s what that looks like:

@pytest.fixture(scope="session")
def documented_views():
    view_labels = set()
    for filename in docs_path.glob("*.rst"):
        for label in get_labels(filename):
            first_word = label.split("_")[0]
            if first_word.endswith("View"):
                view_labels.add(first_word)
    return view_labels

@pytest.mark.parametrize("view_class", [
    v for v in dir(app) if v.endswith("View")
])
def test_view_classes_are_documented(documented_views, view_class):
    assert view_class in documented_views

Fixtures in pytest are an example of dependency injection: pytest introspects every test_* function and checks if it has a function argument with a name matching something that has been annotated with the @pytest.fixture decorator. If it finds any matching arguments, it executes the matching fixture function and passes its return value in to the test function.

By default, pytest will execute the fixture function once for every test execution. In the above code we use the scope="session" argument to tell pytest that this particular fixture should be executed only once for every pytest command-line execution of the tests, and that single return value should be passed to every matching test.

What if you haven’t documented everything yet?

Adding unit tests to your documentation in this way faces an obvious problem: when you first add the tests, you may have to write a whole lot of documentation before they can all pass.

Having tests that protect against future code being added without documentation is only useful once you’ve added them to the codebase - but blocking that on documenting your existing features could prevent that benefit from ever manifesting itself.

Once again, pytest to the rescue. The @pytest.mark.xfail decorator allows you to mark a test as “expected to fail” - if it fails, pytest will take note but will not fail the entire test suite.

This means you can add deliberately failing tests to your codebase without breaking the build for everyone - perfect for tests that look for documentation that hasn’t yet been written!

I used xfail when I first added view documentation tests to Datasette, then removed it once the documentation was all in place. Any future code in pull requests without documentation will cause a hard test failure.

Here’s what the test output looks like when some of those tests are marked as “expected to fail”:

$ pytest tests/test_docs.py
collected 31 items

tests/test_docs.py ..........................XXXxx.                [100%]

============ 26 passed, 2 xfailed, 3 xpassed in 1.06 seconds ============

Since this reports both the xfailed and the xpassed counts, it shows how much work is still left to be done before the xfail decorator can be safely removed.

Structuring code for testable documentation

A benefit of comprehensive unit testing is that it encourages you to design your code in a way that is easy to test. In my experience this leads to much higher code quality in general: it encourages separation of concerns and cleanly decoupled components.

My hope is that documentation unit tests will have a similar effect. I’m already starting to think about ways of restructuring my code such that I can cleanly introspect it for the areas that need to be documented. I’m looking forward to discovering code design patterns that help support this goal.

Tags: design-patterns, documentation, restructuredtext, testing, datasette, pytest

The denormalized query engine design pattern

2017-08-16T22:49:22+00:00

I presented this talk at DjangoCon 2017 in Spokane, Washington. Below is the abstract, the slides and the YouTube video of the talk.

Abstract

Most web applications need to offer search functionality. Open source tools like Solr and Elasticsearch are a powerful option for building custom search engines… but it turns out they can be used for way more than just search.

By treating your search engine as a denormalization layer, you can use it to answer queries that would be too expensive to answer using your core relational database. Questions like “What are the top twenty tags used by my users from Spain?” or “What are the most common times of day for events to start?” or “Which articles contain addresses within 500 miles of Toronto?”.

With the denormalized query engine design pattern, modifications to relational data are published to a denormalized schema in Elasticsearch or Solr. Data queries can then be answered using either the relational database or the search engine, depending on the nature of the specific query. The search engine returns database IDs, which are inflated from the database before being displayed to a user - ensuring that users never see stale data even if the search engine is not 100% up to date with the latest changes. This opens up all kinds of new capabilities for slicing, dicing and exploring data.

In this talk, I’ll be illustrating this pattern by focusing on Elasticsearch - showing how it can be used with Django to bring new capabilities to your application. I’ll discuss the challenge of keeping data synchronized between a relational database and a search engine, and show examples of features that become much easier to build once you have this denormalization layer in place.

Video

Slides

Tags: architecture, definitions, denormalisation, design-patterns, django, djangocon, elasticsearch, my-talks

What are the main weaknesses of Java as a programming language?

2010-10-15T14:25:00+00:00

My answer to What are the main weaknesses of Java as a programming language? on Quora

A cultural bias towards over-engineering. In my experience Java code often ends up a huge network of Factories and AbstractFactories and Visitors and XML configuration files and every design pattern you care to mention, dozens of classes many of which contain hardly any procedural code at all. A lot of Java projects are essentially impossible to navigate without an IDE.

Tags: design-patterns, java, programming-languages, quora

The Universal Design Pattern

2008-10-20T23:13:47+00:00

The Universal Design Pattern

Steve Yegge presents a small book on key/value pairs and prototypal inheritance. “I call it the Universal design pattern because it is (by far) the best known solution to the problem of designing open-ended systems, which in turn translates to long-lived systems.”

Tags: design-patterns, keyvaluepairs, programming, prototypal-inheritance, steve-yegge

ActsAsUndoable

2007-09-18T15:51:14+00:00

ActsAsUndoable

Lawrence Carvalho shows how robust undo functionality can be added to a JavaScript application through careful application of the Memento design pattern.

Tags: actsasundoable, design-patterns, javascript, lawrence-carvalho, memento, undo, yui

factoryjoe: Design Patterns

2007-04-10T11:22:59+00:00

factoryjoe: Design Patterns

Chris Messina’s collection of user interface design pattern screenshots, collated on Flickr.

Tags: chris-messina, design, design-patterns, flickr, ui

Design patterns of 1972

2006-09-14T05:57:54+00:00

Design patterns of 1972

Patterns are signs of weakness in programming languages.

Via Standard Deviations

Tags: design-patterns

Yahoo! Design Pattern Library

2006-02-14T01:12:13+00:00

Yahoo! Design Pattern Library

Common UI design patterns for web applications.

Tags: design, design-patterns

Why the term Ajax is useful

2005-04-19T01:15:58+00:00

Software design patterns are useful mainly because they provide a shared vocabulary: rather than discussing the intimate details of a three layered application architecture, we say "MVC". Rather than describing an object that tracks your progress while looping over a collection, we say "Iterator".

The same is true for Ajax. While the techniques it describes have been around for years, grouping them under a single term is extremely valuable for raising the level of discussion about them. No longer will we have to explain XMLHttpRequest / hidden iframes / crazy cookie tricks in depth when discussing sites which pull fresh information from the server without reloading the whole page. Instead, we can say "Ajax" and move on to more interesting things.

Matthew Haughey says it's all about marketing. I disagree; it's about smarter and more effective conversations.

Tags: ajax, design-patterns, jargon

Patterns of Intermediation

2005-04-03T19:16:01+00:00

Patterns of Intermediation

Design patterns for bookmarklets, greasemonkey and similar.

Tags: bookmarklets, design-patterns, greasemonkey

Dating Design Patterns

2003-12-02T20:54:20+00:00

Dating Design Patterns

Reusable solutions for a complex system

Via rc3.org | Dating Design Patterns

Tags: design-patterns

phpPatterns

2002-11-24T16:13:16+00:00

phpPatterns is a brand new site which advocates and documents the use of object oriented design patterns with PHP. It's a great concept and the site already has some impressive content (although it could really do with a PHP references tutorial). The site is a project of Harry Fuecks, a regular contributor to SitePoint's PHP forums.

Tags: design-patterns, php

Enterprise Application Architecture

2002-06-26T23:12:45+00:00

Enterprise Application Architecture by Martin Fowler: A fantastic book on Object Oriented patterns and how they can be applied to large software projects. The book is available on the web as a work-in-progress and I can safely say I've never found an online resource that has taught me more about software design. Literally 24 hours after finding it my head is swimming with design patterns, domain models and relational database mapping techniques, and I've already started using some of the patterns in my latest project. A big thanks to Captain Proton on the SitePoint forums for pointing it out (and also for helping me understand PHP references a few days ago). I thoroughly recommend this to anyone who is serious about learning Object Oriented design, or indeed any OO-capable language.

Tags: design-patterns, martin-fowler