Simon Willison’s Weblog

Subscribe

17th August 2025

TIL Running a gpt-oss eval suite against LM Studio on a Mac — OpenAI's [gpt-oss models](https://github.com/openai/gpt-oss/) come with an eval suite, which is described in their [Verifying gpt-oss implementations](https://cookbook.openai.com/articles/gpt-oss/verifying-implementations) cookbook. I figured out how to run it on my Mac against their `gpt-oss-20b` model hosted locally using LM Studio, using [uv](https://github.com/astral-sh/uv).

Recent articles

This is a beat by Simon Willison, posted on 17th August 2025.

Monthly briefing

Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.

Pay me to send you less!

Sponsor & subscribe