Simon Willison’s Weblog

Subscribe

2 items tagged “aider”

2024

LLMs are bad at returning code in JSON (via) Paul Gauthier's Aider is a terminal-based coding assistant which works against multiple different models. As part of developing the project Paul runs extensive benchmarks, and his latest shows an interesting result: LLMs are slightly less reliable at producing working code if you request that code be returned as part of a JSON response.

Coding skill by model and code wrapping strategy - four models, each showing their pass rate % average of five runs. Claude 3.5 Sonnet gets 60.5% with Markdown, 54.1% with JSON. DeepSeek-Coder V2 0724 gets 60.6% with Markdown, 51.1% with JSON. GPT-4o-2024-05-13 gets 60.0% with Markdown, 59.6% with JSON. GPT-4o-2024-08-06 gets 60.8% with Markdown, 57.6% with JSON, and 56.9% with JSON (strict). Markdown consistently performs better than JSON across all models.

The May release of GPT-4o is the closest to a perfect score - the August appears to have regressed slightly, and the new structured output mode doesn't help and could even make things worse (though that difference may not be statistically significant).

Paul recommends using Markdown delimiters here instead, which are less likely to introduce confusing nested quoting issues.

# 16th August 2024, 5:04 pm / json, ai, prompt-engineering, generative-ai, llms, aider, evals

Aider. Aider is an impressive open source local coding chat assistant terminal application, developed by Paul Gauthier (founding CTO of Inktomi back in 1996-2000).

I tried it out today, using an Anthropic API key to run it using Claude 3.5 Sonnet:

pipx install aider-chat
export ANTHROPIC_API_KEY=api-key-here
aider --dark-mode

I found the --dark-mode flag necessary to make it legible using the macOS terminal "Pro" theme.

Aider starts by generating a concise map of files in your current Git repository. This is passed to the LLM along with the prompts that you type, and Aider can then request additional files be added to that context - or you can add the manually with the /add filename command.

It defaults to making modifications to files and then committing them directly to Git with a generated commit message. I found myself preferring the /ask command which lets you ask a question without making any file modifications:

Loaded /Users/simon/Dropbox/Development/simonwillisonblog/.env Aider v0.46.1 Models: claude-3-5-sonnet-20240620 with diff edit format, weak model claude-3-haiku-20240307 Git repo: .git with 156 files Repo-map: using 1024 tokens Use /help <question> for help, run "aider --help" to see cmd line args > /add blog/search.py Added /Users/simon/Dropbox/Development/simonwillisonblog/blog/search.py to the chat blog/search.py > /ask what does ?exclude.tag= do in the search feature? The ?exclude.tag= parameter in the search feature is used to exclude specific tags from the search results. Here's how it works in the code: 1 The excluded tags are retrieved from the request's GET parameters: excluded_tags = request.GET.getlist("exclude.tag") This line collects all the tags that should be excluded from the search results. The getlist() method is used because there might be multiple excluded tags. 2 These excluded tags are then applied to the queryset for each content type (Entry, Blogmark, Quotation) in the make_queryset() function: for exclude_tag in excluded_tags: qs = qs.exclude(tags__tag=exclude_tag) This loop iterates through each excluded tag and uses Django's exclude() method to filter out any items that have the specified tag. So, if a user includes ?exclude.tag=python&exclude.tag=django in their search query, the results will not include any items tagged with either "python" or "django". This feature allows users to refine their search by explicitly removing certain topics from the results, which can be particularly useful when searching through a large number of tagged items. Tokens: 3,884 sent, 248 received. Cost: $0.02 request, $0.02 session. blog/search.py

The Aider documentation includes extensive examples and the tool can work with a wide range of different LLMs, though it recommends GPT-4o, Claude 3.5 Sonnet (or 3 Opus) and DeepSeek Coder V2 for the best results. Aider maintains its own leaderboard, emphasizing that "Aider works best with LLMs which are good at editing code, not just good at writing code".

The prompts it uses are pretty fascinating - they're tucked away in various *_prompts.py files in aider/coders.

# 31st July 2024, 3:26 am / python, ai, generative-ai, llms, ai-assisted-programming, claude-3-5-sonnet, aider

Related