Simon Willison’s Weblog

The “await me maybe” pattern for Python asyncio

2nd September 2020

I’ve identified a pattern for handling potentially-asynchronous callback functions in Python which I’m calling the “await me maybe” pattern. It works by letting you return a value, a callable function that returns a value OR an awaitable function that returns that value.

Background

Datasette has been built on top of Python 3 asyncio from the very start—initially using Sanic, and as-of Datasette 0.29 using a custom mini-framework on top of ASGI 3, usually running under Uvicorn.

Datasette also has a plugin system, built on top of Pluggy.

Pluggy is a beautifully designed mechanism for plugins. It works based on decorated functions, which are called at various points by Datasette itself.

A simple plugin that injects a new JavaScript file into a page coud look like this:

from datasette import hookimpl

@hookimpl
def extra_js_urls():
    return [
        "https://code.jquery.com/jquery-3.5.1.min.js"
    ]

Datasette can then gather together all of the extra JavaScript URLs that should be injected into a page by running this code:

urls = []
for url in pm.hook.extra_js_urls(
    template=template.name,
    datasette=datasette,
):
    urls.extend(url)

What’s up with the template= and datasette= parameters that are passed here?

Pluggy implements a form of dependency injection, where plugin hook functions can optionally list additional parameters that they would like to have access to.

The above simple example didn’t need any extra information. But imagine a plugin that only wants to inject jQuery on the table.html template page:

@hookimpl
def extra_js_urls(template):
    if template == "table.html":
        return [
            "https://code.jquery.com/jquery-3.5.1.min.js"
        ]

Datasette actually provides several more optional argument for these plugin functions—see the plugin hooks documentation for full details.

What if we need to await something?

The datasette object that can be passed to plugin hooks is special: it provides an object that can be used for the following:

• Executing SQL against databases connected to Datasette
• Looking up Datasette metadata and configuration settings, including plugin configuration
• Rendering templates using the template environment configured by Datasette
• Performing checks against the Datasette permissions system

Here’s the problem: many of those methods on Datasette are awaitable—await datasette.render_template(...) for example. But Pluggy is built around regular non-awaitable Python functions.

If my def extra_js_urls() plugin function needs to execute a SQL query to decide what JavaScript to include, it won’t be able to—because you can’t use await inside a regular Python function.

That’s where the “await me maybe” pattern comes in.

The basic idea is that a function can return a value, OR a function-that-returns-a-value, OR an awaitable-function-that-returns-a-value.

If we want our extra_js_urls(datasette) hook to execute a SQL query in order to decide what URLs to return, it can look like this:

@hookimpl
def extra_js_urls(datasette):
    async def inner():
        db = datasette.get_database()
        results = await db.execute("select url from js_files")
        return [r[0] for r in results]

    return inner

Note that Python lets you define an async def inner() function inside the body of a regular function, which is what we’re doing here.

The code that calls the plugin hook in Datasette can then look like this:

urls = []
for url in pm.hook.extra_js_urls(
    template=template.name,
    datasette=datasette,
):
    if callable(url):
        url = url()
    if asyncio.iscoroutine(url):
        url = await url
    urls.append(url)

I use this pattern in a bunch of different places in Datasette, so today I refactored that into a utility function:

import asyncio

async def await_me_maybe(value):
    if callable(value):
        value = value()
    if asyncio.iscoroutine(value):
        value = await value
    return value

This commit includes a bunch of examples where this function is called, for example this code which gathers extra body scripts to be included at the bottom of the page:

body_scripts = []
for extra_script in pm.hook.extra_body_script(
    template=template.name,
    database=context.get("database"),
    table=context.get("table"),
    columns=context.get("columns"),
    view_name=view_name,
    request=request,
    datasette=self,
):
    extra_script = await await_me_maybe(extra_script)
    body_scripts.append(Markup(extra_script))