Simon Willison’s Weblog

My @covidsewage bot now includes useful alt text. I've been running a @covidsewage Mastodon bot for a while now, posting daily screenshots (taken with shot-scraper) of the Santa Clara County COVID in wastewater dashboard.

Prior to today the screenshot was accompanied by the decidedly unhelpful alt text "Screenshot of the latest Covid charts".

I finally fixed that today, closing issue #2 more than two years after I first opened it.

The screenshot is of a Microsoft Power BI dashboard. I hoped I could scrape the key information out of it using JavaScript, but the weirdness of their DOM proved insurmountable.

Instead, I'm using GPT-4o - specifically, this Python code (run using a python -c block in the GitHub Actions YAML file):

import base64, openai
client = openai.OpenAI()
with open('/tmp/covid.png', 'rb') as image_file:
    encoded_image = base64.b64encode(image_file.read()).decode('utf-8')
messages = [
    {'role': 'system',
     'content': 'Return the concentration levels in the sewersheds - single paragraph, no markdown'},
    {'role': 'user', 'content': [
        {'type': 'image_url', 'image_url': {
            'url': 'data:image/png;base64,' + encoded_image
        }}
    ]}
]
completion = client.chat.completions.create(model='gpt-4o', messages=messages)
print(completion.choices[0].message.content)

I'm base64 encoding the screenshot and sending it with this system prompt:

Return the concentration levels in the sewersheds - single paragraph, no markdown

Given this input image:

Here's the text that comes back:

The concentration levels of SARS-CoV-2 in the sewersheds from collected samples are as follows: San Jose Sewershed has a high concentration, Palo Alto Sewershed has a high concentration, Sunnyvale Sewershed has a high concentration, and Gilroy Sewershed has a medium concentration.

The full implementation can be found in the GitHub Actions workflow, which runs on a schedule at 7am Pacific time every day.

# 25th August 2024, 4:09 pm / shot-scraper, openai, covid19, gpt-4, ai, llms, generative-ai, projects, alt-attribute, accessibility

Alt-texts: The Ultimate Guide. By Daniel Göransson, a web developer with vision impairment who uses a screen reader. This is the best, most practical guide to writing image alt text I’ve seen. Just one of the neat tips contained within: consider ending your alt text in a period, so the screen user knows to pause.

# 6th November 2017, 4:54 pm / alt-attribute, accessibility

The alt=“” attribute from Ian Hickson. In case you were wondering how it all ended, Hixie has a mammoth summary post explaining the facts and the potential alternatives.

# 11th September 2008, 5:45 pm / ian-hickson, html5, alt-attribute

This Week in HTML 5—Episode 1. It looks like the most controversial aspect of the HTML 5 spec has been addressed - now, instead of omitting the alt attribute for user generated content that has no relevant information available, sites are advised to provide an indication of the kind of image expected surrounded by braces, for example alt="{uploaded photo}".

# 7th August 2008, 7:57 am / alt-attribute, html5, mark-pilgrim, whatwg

Why the Alt Attribute May Be Omitted. “The benefit of requiring the alt attribute to be omitted, rather than simply requiring the empty value, is that it makes a clear distinction between an image that has no alternate text (such as an iconic or graphical representation of the surrounding text) and an image that is a critical part of the content, but for which not alt text is available.”

# 25th August 2007, 1:11 pm / accessibility, html, html5, whatwg, alt-attribute