Simon Willison’s Weblog

--- +++ @@ -22,7 +22,7 @@ One of my favorite prompting patterns is to tell an agent to build something new by combining two or more existing working examples. -A project that helped crystallize how effective this can be was the first thing I added to my tools collection - a browser-based [OCR tool](https://tools.simonwillison.net/ocr). +A project that helped crystallize how effective this can be was the first thing I added to my tools collection - a browser-based [OCR tool](https://tools.simonwillison.net/ocr), described [in more detail here](https://simonwillison.net/2024/Mar/30/ocr-pdfs-images/). I wanted an easy, browser-based tool for OCRing pages from PDF files - in particular PDFs that consist entirely of scanned images with no text version provided at all.

--- +++ @@ -167,18 +167,18 @@ If your coding agent has internet access you can tell it to do things like: -> Use curl to fetch the source of https://tools.simonwillison.net/ocr and https://tools.simonwillison.net/gemini-bbox and build a new tool that lets you select a page from a PDF and pass it to Gemini to return bounding boxes for illustrations on that page. +> Use curl to fetch the source of `https://tools.simonwillison.net/ocr` and `https://tools.simonwillison.net/gemini-bbox` and build a new tool that lets you select a page from a PDF and pass it to Gemini to return bounding boxes for illustrations on that page. (I specified `curl` there because Claude Code defaults to using a WebFetch tool which summarizes the page content rather than returning the raw HTML.) Coding agents are excellent at search, which means you can run them on your own machine and tell them where to find the examples of things you want them to do: -> Add mocked HTTP tests to the `~/dev/ecosystem/datasette-oauth` project inspired by how ~/dev/ecosystem/llm-mistral` is doing it. +> Add mocked HTTP tests to the `~/dev/ecosystem/datasette-oauth` project inspired by how `~/dev/ecosystem/llm-mistral` is doing it. Often that's enough - the agent will fire up a search sub-agent to investigate and pull back just the details it needs to achieve the task. Since so much of my research code is public I'll often tell coding agents to clone my repositories to `/tmp` and use them as input: -> Clone simonw/research from GitHub to /tmp and find examples of compiling Rust to WebAssembly, then use that to build a demo HTML page for this project. +> Clone `simonw/research` from GitHub to `/tmp` and find examples of compiling Rust to WebAssembly, then use that to build a demo HTML page for this project. The key idea here is that coding agents mean we only ever need to figure out a useful trick *once*. If that trick is then documented somewhere with a working code example our agents can consult that example and use it to solve any similar shaped project in the future.

--- +++ @@ -22,13 +22,13 @@ One of my favorite prompting patterns is to tell an agent to build something new by combining two or more existing working examples. -A project that helped crystalize how effective this can be was the first that I added to my tools collection - a browser-based [OCR tool](https://tools.simonwillison.net/ocr). +A project that helped crystallize how effective this can be was the first thing I added to my tools collection - a browser-based [OCR tool](https://tools.simonwillison.net/ocr). I wanted an easy, browser-based tool for OCRing pages from PDF files - in particular PDFs that consist entirely of scanned images with no text version provided at all. I had previously experimented with running the [Tesseract.js OCR library](https://tesseract.projectnaptha.com/) in my browser, and found it to be very capable. That library provides a WebAssembly build of the mature Tesseract OCR engine and lets you call it from JavaScript to extract text from an image. -I didn’t want to work with images though, I wanted to work with PDFs. Then I remembered that I had also experimented in the past with Mozilla’s [PDF.js](https://mozilla.github.io/pdf.js/) library, a which among other things can turn individual pages of a PDF into rendered images. +I didn’t want to work with images though, I wanted to work with PDFs. Then I remembered that I had also worked with Mozilla’s [PDF.js](https://mozilla.github.io/pdf.js/) library, which among other things can turn individual pages of a PDF into rendered images. I had snippets of JavaScript for both of those libraries in my notes. @@ -159,4 +159,26 @@ This worked flawlessly! The model kicked out a proof-of-concept page that did exactly what I needed. -I ended up [iterating with it a few times](https://gist.github.com/simonw/6a9f077bf8db616e44893a24ae1d36eb) to get to my final result, but it took just a few minutes to build a genuinely useful tool that I’ve benefitted from ever since. +I ended up [iterating with it a few times](https://gist.github.com/simonw/6a9f077bf8db616e44893a24ae1d36eb) to get to my final result, but it took just a few minutes to build a genuinely useful tool that I’ve benefited from ever since. + +## Coding agents make this even more powerful + +I built that OCR example back in March 2024, nearly a year before the first release of Claude Code. Coding agents have made hoarding working examples even more valuable. + +If your coding agent has internet access you can tell it to do things like: + +> Use curl to fetch the source of https://tools.simonwillison.net/ocr and https://tools.simonwillison.net/gemini-bbox and build a new tool that lets you select a page from a PDF and pass it to Gemini to return bounding boxes for illustrations on that page. + +(I specified `curl` there because Claude Code defaults to using a WebFetch tool which summarizes the page content rather than returning the raw HTML.) + +Coding agents are excellent at search, which means you can run them on your own machine and tell them where to find the examples of things you want them to do: + +> Add mocked HTTP tests to the `~/dev/ecosystem/datasette-oauth` project inspired by how ~/dev/ecosystem/llm-mistral` is doing it. + +Often that's enough - the agent will fire up a search sub-agent to investigate and pull back just the details it needs to achieve the task. + +Since so much of my research code is public I'll often tell coding agents to clone my repositories to `/tmp` and use them as input: + +> Clone simonw/research from GitHub to /tmp and find examples of compiling Rust to WebAssembly, then use that to build a demo HTML page for this project. + +The key idea here is that coding agents mean we only ever need to figure out a useful trick *once*. If that trick is then documented somewhere with a working code example our agents can consult that example and use it to solve any similar shaped project in the future.