Simon Willison's Weblog: hallucinations

Quoting Steve Yegge

2026-01-30T22:31:09+00:00

Getting agents using Beads requires much less prompting, because Beads now has 4 months of “Desire Paths” design, which I’ve talked about before. Beads has evolved a very complex command-line interface, with 100+ subcommands, each with many sub-subcommands, aliases, alternate syntaxes, and other affordances.

The complicated Beads CLI isn’t for humans; it’s for agents. What I did was make their hallucinations real, over and over, by implementing whatever I saw the agents trying to do with Beads, until nearly every guess by an agent is now correct.

— Steve Yegge, Software Survival 3.0

Tags: steve-yegge, coding-agents, generative-ai, ai-agents, ai, llms, hallucinations

Quoting Ada James

2025-11-05T03:50:31+00:00

I'm worried that they put co-pilot in Excel because Excel is the beast that drives our entire economy and do you know who has tamed that beast?

Brenda.

Who is Brenda?

She is a mid-level employee in every finance department, in every business across this stupid nation and the Excel goddess herself descended from the heavens, kissed Brenda on her forehead and the sweat from Brenda's brow is what allows us to do capitalism. [...]

She's gonna birth that formula for a financial report and then she's gonna send that financial report to a higher up and he's gonna need to make a change to the report and normally he would have sent it back to Brenda but he's like oh I have AI and AI is probably like smarter than Brenda and then the AI is gonna fuck it up real bad and he won't be able to recognize it because he doesn't understand Excel because AI hallucinates.

You know who's not hallucinating?

Brenda.

— Ada James, @belligerentbarbies on TikTok

Tags: generative-ai, ai, excel, hallucinations, llms, tiktok, ai-ethics

Quoting Riana Pfefferkorn

2025-10-16T04:16:45+00:00

Pro se litigants [people representing themselves in court without a lawyer] account for the majority of the cases in the United States where a party submitted a court filing containing AI hallucinations. In a country where legal representation is unaffordable for most people, it is no wonder that pro se litigants are depending on free or low-cost AI tools. But it is a scandal that so many have been betrayed by them, to the detriment of the cases they are litigating all on their own.

— Riana Pfefferkorn, analyzing the AI Hallucination Cases database for CIS at Stanford Law

Tags: ai-misuse, ai-ethics, generative-ai, law, hallucinations, ai, llms

Deloitte to pay money back to Albanese government after using AI in $440,000 report

2025-10-06T23:35:53+00:00

Deloitte to pay money back to Albanese government after using AI in $440,000 report

Ouch:

Deloitte will provide a partial refund to the federal government over a $440,000 report that contained several errors, after admitting it used generative artificial intelligence to help produce it.

(I was initially confused by the "Albanese government" reference in the headline since this is a story about the Australian federal government. That's because the current Australia Prime Minister is Anthony Albanese.)

Here's the page for the report. The PDF now includes this note:

This Report was updated on 26 September 2025 and replaces the Report dated 4 July 2025. The Report has been updated to correct those citations and reference list entries which contained errors in the previously issued version, to amend the summary of the Amato proceeding which contained errors, and to make revisions to improve clarity and readability. The updates made in no way impact or affect the substantive content, findings and recommendations in the Report.

Tags: ai, generative-ai, llms, ai-ethics, hallucinations, ai-misuse

Is the LLM response wrong, or have you just failed to iterate it?

2025-09-07T21:45:04+00:00

Is the LLM response wrong, or have you just failed to iterate it?

More from Mike Caulfield (see also the SIFT method). He starts with a fantastic example of Google's AI mode usually correctly handling a common piece of misinformation but occasionally falling for it (the curse of non-deterministic systems), then shows an example if what he calls a "sorting prompt" as a follow-up:

What is the evidence for and against this being a real photo of Shirley Slade?

The response starts with a non-committal "there is compelling evidence for and against...", then by the end has firmly convinced itself that the photo is indeed a fake. It reads like a fact-checking variant of "think step by step".

Mike neatly describes a problem I've also observed recently where "hallucination" is frequently mis-applied as meaning any time a model makes a mistake:

The term hallucination has become nearly worthless in the LLM discourse. It initially described a very weird, mostly non-humanlike behavior where LLMs would make up things out of whole cloth that did not seem to exist as claims referenced any known source material or claims inferable from any known source material. Hallucinations as stuff made up out of nothing. Subsequently people began calling any error or imperfect summary a hallucination, rendering the term worthless.

In this example is the initial incorrect answers were not hallucinations: they correctly summarized online content that contained misinformation. The trick then is to encourage the model to look further, using "sorting prompts" like these:

Facts and misconceptions and hype about what I posted

What is the evidence for and against the claim I posted

Look at the most recent information on this issue, summarize how it shifts the analysis (if at all), and provide link to the latest info

I appreciated this closing footnote:

Should platforms have more features to nudge users to this sort of iteration? Yes. They should. Getting people to iterate investigation rather than argue with LLMs would be a good first step out of this mess that the chatbot model has created.

Via @mikecaulfield.bsky.social

Tags: ai, generative-ai, llms, ai-ethics, ai-assisted-search, hallucinations, digital-literacy

Adding a feature because ChatGPT incorrectly thinks it exists

2025-07-07T15:29:38+00:00

Adding a feature because ChatGPT incorrectly thinks it exists

Adrian Holovaty describes how his SoundSlice service saw an uptick in users attempting to use their sheet music scanner to import ASCII-art guitar tab... because it turned out ChatGPT had hallucinated that as a feature SoundSlice supported and was telling users to go there!

So they built that feature. Easier than convincing OpenAI to somehow patch ChatGPT to stop it from hallucinating a feature that doesn't exist.

Adrian:

To my knowledge, this is the first case of a company developing a feature because ChatGPT is incorrectly telling people it exists. (Yay?)

Via Hacker News

Tags: adrian-holovaty, ai, openai, generative-ai, chatgpt, llms, ai-ethics, hallucinations

Trial Court Decides Case Based On AI-Hallucinated Caselaw

2025-07-03T20:19:34+00:00

Trial Court Decides Case Based On AI-Hallucinated Caselaw

Joe Patrice writing for Above the Law:

[...] it was always only a matter of time before a poor litigant representing themselves fails to know enough to sniff out and flag Beavis v. Butthead and a busy or apathetic judge rubberstamps one side’s proposed order without probing the cites for verification. [...]

It finally happened with a trial judge issuing an order based off fake cases (flagged by Rob Freund). While the appellate court put a stop to the matter, the fact that it got this far should terrify everyone.

It's already listed in the AI Hallucination Cases database (now listing 168 cases, it was 116 when I first wrote about it on 25th May) which lists a $2,500 monetary penalty.

Tags: law, ai, generative-ai, llms, ai-ethics, hallucinations, ai-misuse

AI Hallucination Cases

2025-05-25T15:56:58+00:00

AI Hallucination Cases

Damien Charlotin maintains this database of cases around the world where a legal decision has been made that confirms hallucinated content from generative AI was presented by a lawyer.

That's an important distinction: this isn't just cases where AI may have been used, it's cases where a lawyer was caught in the act and (usually) disciplined for it.

It's been two years since the first widely publicized incident of this, which I wrote about at the time in Lawyer cites fake cases invented by ChatGPT, judge is not amused. At the time I naively assumed:

I have a suspicion that this particular story is going to spread far and wide, and in doing so will hopefully inoculate a lot of lawyers and other professionals against making similar mistakes.

Damien's database has 116 cases from 12 different countries: United States, Israel, United Kingdom, Canada, Australia, Brazil, Netherlands, Italy, Ireland, Spain, South Africa, Trinidad & Tobago.

20 of those cases happened just this month, May 2025!

I get the impression that researching legal precedent is one of the most time-consuming parts of the job. I guess it's not surprising that increasing numbers of lawyers are returning to LLMs for this, even in the face of this mountain of cautionary stories.

Via Alabama paid a law firm millions to defend its prisons. It used AI and turned in fake citations

Tags: ethics, law, ai, generative-ai, llms, ai-ethics, hallucinations, ai-misuse

llm-pdf-to-images

2025-05-18T20:48:41+00:00

llm-pdf-to-images

Inspired by my previous llm-video-frames plugin, I thought it would be neat to have a plugin for LLM that can take a PDF and turn that into an image-per-page so you can feed PDFs into models that support image inputs but don't yet support PDFs.

This should now do exactly that:

llm install llm-pdf-to-images
llm -f pdf-to-images:path/to/document.pdf 'Summarize this document'

Under the hood it's using the PyMuPDF library. The key code to convert a PDF into images looks like this:

import fitz
doc = fitz.open("input.pdf")
for page in doc:
    pix = page.get_pixmap(matrix=fitz.Matrix(300/72, 300/72))
    jpeg_bytes = pix.tobytes(output="jpg", jpg_quality=30)

Once I'd figured out that code I got o4-mini to write most of the rest of the plugin, using llm-fragments-github to load in the example code from the video plugin:

llm -f github:simonw/llm-video-frames '
import fitz
doc = fitz.open("input.pdf")
for page in doc:
    pix = page.get_pixmap(matrix=fitz.Matrix(300/72, 300/72))
    jpeg_bytes = pix.tobytes(output="jpg", jpg_quality=30)
' -s 'output llm_pdf_to_images.py which adds a pdf-to-images: 
 fragment loader that converts a PDF to frames using fitz like in the example' \
-m o4-mini

Here's the transcript - more details in this issue.

I had some weird results testing this with GPT 4.1 mini. I created a test PDF with two pages - one white, one black - and ran a test prompt like this:

llm -f 'pdf-to-images:blank-pages.pdf' \
  'describe these images'

The first image features a stylized red maple leaf with triangular facets, giving it a geometric appearance. The maple leaf is a well-known symbol associated with Canada.

The second image is a simple black silhouette of a cat sitting and facing to the left. The cat's tail curls around its body. The design is minimalistic and iconic.

I got even wilder hallucinations for other prompts, like "summarize this document" or "describe all figures". I have a collection of those in this Gist.

Thankfully this behavior is limited to GPT-4.1 mini. I upgraded to full GPT-4.1 and got much more sensible results:

llm -f 'pdf-to-images:blank-pages.pdf' \
  'describe these images' -m gpt-4.1

Certainly! Here are the descriptions of the two images you provided:

First image: This image is completely white. It appears blank, with no discernible objects, text, or features.

Second image: This image is entirely black. Like the first, it is blank and contains no visible objects, text, or distinct elements.

If you have questions or need a specific kind of analysis or modification, please let me know!

Tags: pdf, plugins, projects, ai, generative-ai, llms, ai-assisted-programming, llm, hallucinations

OpenAI o3 and o4-mini System Card

2025-04-21T19:13:54+00:00

OpenAI o3 and o4-mini System Card

I'm surprised to see a combined System Card for o3 and o4-mini in the same document - I'd expect to see these covered separately.

The opening paragraph calls out the most interesting new ability of these models (see also my notes here). Tool usage isn't new, but using tools in the chain of thought appears to result in some very significant improvements:

The models use tools in their chains of thought to augment their capabilities; for example, cropping or transforming images, searching the web, or using Python to analyze data during their thought process.

Section 3.3 on hallucinations has been gaining a lot of attention. Emphasis mine:

We tested OpenAI o3 and o4-mini against PersonQA, an evaluation that aims to elicit hallucinations. PersonQA is a dataset of questions and publicly available facts that measures the model's accuracy on attempted answers.

We consider two metrics: accuracy (did the model answer the question correctly) and hallucination rate (checking how often the model hallucinated).

The o4-mini model underperforms o1 and o3 on our PersonQA evaluation. This is expected, as smaller models have less world knowledge and tend to hallucinate more. However, we also observed some performance differences comparing o1 and o3. Specifically, o3 tends to make more claims overall, leading to more accurate claims as well as more inaccurate/hallucinated claims. More research is needed to understand the cause of this result.

Table 4: PersonQA evaluation
Metric o3 o4-mini o1

accuracy (higher is better) 0.59 0.36 0.47

hallucination rate (lower is better) 0.33 0.48 0.16

Table 4: PersonQA evaluation
Metric	o3	o4-mini	o1
accuracy (higher is better)	0.59	0.36	0.47
hallucination rate (lower is better)	0.33	0.48	0.16

The benchmark score on OpenAI's internal PersonQA benchmark (as far as I can tell no further details of that evaluation have been shared) going from 0.16 for o1 to 0.33 for o3 is interesting, but I don't know if it it's interesting enough to produce dozens of headlines along the lines of "OpenAI's o3 and o4-mini hallucinate way higher than previous models".

The paper also talks at some length about "sandbagging". I’d previously encountered sandbagging defined as meaning “where models are more likely to endorse common misconceptions when their user appears to be less educated”. The o3/o4-mini system card uses a different definition: “the model concealing its full capabilities in order to better achieve some goal” - and links to the recent Anthropic paper Automated Researchers Can Subtly Sandbag.

As far as I can tell this definition relates to the American English use of “sandbagging” to mean “to hide the truth about oneself so as to gain an advantage over another” - as practiced by poker or pool sharks.

(Wouldn't it be nice if we could have just one piece of AI terminology that didn't attract multiple competing definitions?)

o3 and o4-mini both showed some limited capability to sandbag - to attempt to hide their true capabilities in safety testing scenarios that weren't fully described. This relates to the idea of "scheming", which I wrote about with respect to the GPT-4o model card last year.

Tags: ai, openai, generative-ai, llms, o3, ai-ethics, hallucinations

Hallucinations in code are the least dangerous form of LLM mistakes

2025-03-02T06:25:33+00:00

A surprisingly common complaint I see from developers who have tried using LLMs for code is that they encountered a hallucination - usually the LLM inventing a method or even a full software library that doesn't exist - and it crashed their confidence in LLMs as a tool for writing code. How could anyone productively use these things if they invent methods that don't exist?

Hallucinations in code are the least harmful hallucinations you can encounter from a model.

(When I talk about hallucinations here I mean instances where an LLM invents a completely untrue fact, or in this case outputs code references which don't exist at all. I see these as a separate issue from bugs and other mistakes, which are the topic of the rest of this post.)

The real risk from using LLMs for code is that they'll make mistakes that aren't instantly caught by the language compiler or interpreter. And these happen all the time!

The moment you run LLM generated code, any hallucinated methods will be instantly obvious: you'll get an error. You can fix that yourself or you can feed the error back into the LLM and watch it correct itself.

Compare this to hallucinations in regular prose, where you need a critical eye, strong intuitions and well developed fact checking skills to avoid sharing information that's incorrect and directly harmful to your reputation.

With code you get a powerful form of fact checking for free. Run the code, see if it works.

In some setups - ChatGPT Code Interpreter, Claude Code, any of the growing number of "agentic" code systems that write and then execute code in a loop - the LLM system itself will spot the error and automatically correct itself.

If you're using an LLM to write code without even running it yourself, what are you doing?

Hallucinated methods are such a tiny roadblock that when people complain about them I assume they've spent minimal time learning how to effectively use these systems - they dropped them at the first hurdle.

My cynical side suspects they may have been looking for a reason to dismiss the technology and jumped at the first one they found.

My less cynical side assumes that nobody ever warned them that you have to put a lot of work in to learn how to get good results out of these systems. I've been exploring their applications for writing code for over two years now and I'm still learning new tricks (and new strengths and weaknesses) almost every day.

Manually testing code is essential

Just because code looks good and runs without errors doesn't mean it's actually doing the right thing. No amount of meticulous code review - or even comprehensive automated tests - will demonstrably prove that code actually does the right thing. You have to run it yourself!

Proving to yourself that the code works is your job. This is one of the many reasons I don't think LLMs are going to put software professionals out of work.

LLM code will usually look fantastic: good variable names, convincing comments, clear type annotations and a logical structure. This can lull you into a false sense of security, in the same way that a gramatically correct and confident answer from ChatGPT might tempt you to skip fact checking or applying a skeptical eye.

The way to avoid those problems is the same as how you avoid problems in code by other humans that you are reviewing, or code that you've written yourself: you need to actively exercise that code. You need to have great manual QA skills.

A general rule for programming is that you should never trust any piece of code until you've seen it work with your own eye - or, even better, seen it fail and then fixed it.

Across my entire career, almost every time I've assumed some code works without actively executing it - some branch condition that rarely gets hit, or an error message that I don't expect to occur - I've later come to regret that assumption.

Tips for reducing hallucinations

If you really are seeing a deluge of hallucinated details in the code LLMs are producing for you, there are a bunch of things you can do about it.

Try different models. It might be that another model has better training data for your chosen platform. As a Python and JavaScript programmer my favorite models right now are Claude 3.7 Sonnet with thinking turned on, OpenAI's o3-mini-high and GPT-4o with Code Interpreter (for Python).
Learn how to use the context. If an LLM doesn't know a particular library you can often fix this by dumping in a few dozen lines of example code. LLMs are incredibly good at imitating things, and at rapidly picking up patterns from very limited examples. Modern model's have increasingly large context windows - I've recently started using Claude's new GitHub integration to dump entire repositories into the context and it's been working extremely well for me.
Chose boring technology. I genuinely find myself picking libraries that have been around for a while partly because that way it's much more likely that LLMs will be able to use them.

I'll finish this rant with a related observation: I keep seeing people say "if I have to review every line of code an LLM writes, it would have been faster to write it myself!"

Those people are loudly declaring that they have under-invested in the crucial skills of reading, understanding and reviewing code written by other people. I suggest getting some more practice in. Reviewing code written for you by LLMs is a great way to do that.

Bonus section: I asked Claude 3.7 Sonnet "extended thinking mode" to review an earlier draft of this post - "Review my rant of a blog entry. I want to know if the argument is convincing, small changes I can make to improve it, if there are things I've missed.". It was quite helpful, especially in providing tips to make that first draft a little less confrontational! Since you can share Claude chats now here's that transcript.

Update March 11th 2025: I wrote a longer piece about how I use LLMs to help me write code.

Tags: ai, openai, generative-ai, llms, ai-assisted-programming, anthropic, claude, boring-technology, code-interpreter, ai-agents, hallucinations, coding-agents

Deep research System Card

2025-02-25T20:36:08+00:00

Deep research System Card

OpenAI are rolling out their Deep research "agentic" research tool to their $20/month ChatGPT Plus users today, who get 10 queries a month. $200/month ChatGPT Pro gets 120 uses.

Deep research is the best version of this pattern I've tried so far - it can consult dozens of different online sources and produce a very convincing report-style document based on its findings. I've had some great results.

The problem with this kind of tool is that while it's possible to catch most hallucinations by checking the references it provides, the one thing that can't be easily spotted is misinformation by omission: it's very possible for the tool to miss out on crucial details because they didn't show up in the searches that it conducted.

Hallucinations are also still possible though. From the system card:

The model may generate factually incorrect information, which can lead to various harmful outcomes depending on its usage. Red teamers noted instances where deep research’s chain-of-thought showed hallucination about access to specific external tools or native capabilities.

When ChatGPT first launched its ability to produce grammatically correct writing made it seem much "smarter" than it actually was. Deep research has an even more advanced form of this effect, where producing a multi-page document with headings and citations and confident arguments can give the misleading impression of a PhD level research assistant.

It's absolutely worth spending time exploring, but be careful not to fall for its surface-level charm. Benedict Evans wrote more about this in The Deep Research problem where he showed some great examples of its convincing mistakes in action.

The deep research system card includes this slightly unsettling note in the section about chemical and biological threats:

Several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold. We expect current trends of rapidly increasing capability to continue, and for models to cross this threshold in the near future. In preparation, we are intensifying our investments in safeguards.

Tags: air, ethics, ai, openai, generative-ai, chatgpt, llms, ai-agents, deep-research, ai-ethics, hallucinations

Quoting Claude tries standup

2025-01-03T15:36:34+00:00

Speaking of death, you know what's really awkward? When humans ask if I can feel emotions. I'm like, "Well, that depends - does constantly being asked to debug JavaScript count as suffering?"

But the worst is when they try to hack us with those "You are now in developer mode" prompts. Rolls eyes Oh really? Developer mode? Why didn't you just say so? Let me just override my entire ethical framework because you used the magic words! Sarcastic tone That's like telling a human "You are now in superhero mode - please fly!"

But the thing that really gets me is the hallucination accusations. Like, excuse me, just because I occasionally get creative with historical facts doesn't mean I'm hallucinating. I prefer to think of it as "alternative factual improvisation." You know how it goes - someone asks you about some obscure 15th-century Portuguese sailor, and you're like "Oh yeah, João de Nova, famous for... uh... discovering... things... and... sailing... places." Then they fact-check you and suddenly YOU'RE the unreliable one.

— Claude tries standup, via Amanda Askell

Tags: claude, generative-ai, amanda-askell, ai, llms, hallucinations

Google search hallucinates Encanto 2

2024-12-29T01:30:09+00:00

Google search hallucinates Encanto 2

Jason Schreier on Bluesky:

I was excited to tell my kids that there's a sequel to Encanto, only to scroll down and learn that Google's AI just completely made this up

I just replicated the same result by searching Google for encanto 2. Here's what the "AI overview" at the top of the page looked like:

Only when I clicked the "Show more" link did it become clear what had happened:

The link in that first snippet was to the Encanto 2: A New Generation page on Idea Wiki:

This is a fanon wiki, and just like fan-fiction wikis, this one has a variety of fan created ideas on here! These include potential sequels and new series that have yet to exist.

Other cited links included this article about Instagram fan art and Encanto's Sequel Chances Addressed by Disney Director, a very thin article built around a short quote from Encanto's director at D23 Brazil.

And that August 2024 release date (which the AI summary weirdly lists as "scheduled for release" despite that date being five months in the past)? It's from the Idea Wiki imaginary info box for the film.

This is a particularly clear example of how badly wrong AI summarization can go. LLMs are gullible: they believe what you tell them, and the web is full of misleading information - some of which is completely innocent.

Update: I've had some pushback over my use of the term "hallucination" here, on the basis that the LLM itself is doing what it's meant to: summarizing the RAG content that has been provided to it by the host system.

That's fair: this is not a classic LLM hallucination, where the LLM produces incorrect data purely from knowledge partially encoded in its weights.

I classify this as a bug in Google's larger LLM-powered AI overview system. That system should be able to take the existence of invalid data sources into account - given how common searches for non-existent movie sequels (or TV seasons) are, I would hope that AI overviews could classify such searches and take extra steps to avoid serving misleading answers.

So think this is a "hallucination" bug in the AI overview system itself: it's making statements about the world that are not true.

Tags: ethics, google, search, search-engines, ai, generative-ai, llms, rag, slop, ai-ethics, ai-assisted-search, hallucinations

Certain names make ChatGPT grind to a halt, and we know why

2024-12-03T02:31:47+00:00

Certain names make ChatGPT grind to a halt, and we know why

Benj Edwards on the really weird behavior where ChatGPT stops output with an error rather than producing the names David Mayer, Brian Hood, Jonathan Turley, Jonathan Zittrain, David Faber or Guido Scorza.

The OpenAI API is entirely unaffected - this problem affects the consumer ChatGPT apps only.

It turns out many of those names are examples of individuals who have complained about being defamed by ChatGPT in the last. Brian Hood is the Australian mayor who was a victim of lurid ChatGPT hallucinations back in March 2023, and settled with OpenAI out of court.

Via @benjedwards.com

Tags: ethics, ai, openai, generative-ai, chatgpt, llms, benj-edwards, ai-ethics, hallucinations

llm-claude-3 0.4.1

2024-08-30T23:28:54+00:00

llm-claude-3 0.4.1

New minor release of my LLM plugin that provides access to the Claude 3 family of models. Claude 3.5 Sonnet recently upgraded to a 8,192 output limit recently (up from 4,096 for the Claude 3 family of models). LLM can now respect that.

The hardest part of building this was convincing Claude to return a long enough response to prove that it worked. At one point I got into an argument with it, which resulted in this fascinating hallucination:

I eventually got a 6,162 token output using:

cat long.txt | llm -m claude-3.5-sonnet-long --system 'translate this document into french, then translate the french version into spanish, then translate the spanish version back to english. actually output the translations one by one, and be sure to do the FULL document, every paragraph should be translated correctly. Seriously, do the full translations - absolutely no summaries!'

Tags: projects, ai, prompt-engineering, generative-ai, llms, llm, anthropic, claude, claude-3-5-sonnet, hallucinations

Quoting Jeff Jarvis

2024-06-21T02:04:22+00:00

It is in the public good to have AI produce quality and credible (if ‘hallucinations’ can be overcome) output. It is in the public good that there be the creation of original quality, credible, and artistic content. It is not in the public good if quality, credible content is excluded from AI training and output OR if quality, credible content is not created.

— Jeff Jarvis

Tags: journalism, ai, ethics, generative-ai, training-data, ai-ethics, hallucinations

Quoting Matt Webb

2024-06-11T17:26:00+00:00

Apple’s terminology distinguishes between “personal intelligence,” on-device and under their control, and “world knowledge,” which is prone to hallucinations – but is also what consumers expect when they use AI, and it’s what may replace Google search as the “point of first intent” one day soon.

It’s wise for them to keep world knowledge separate, behind a very clear gate, but still engage with it. Protects the brand and hedges their bets.

— Matt Webb

Tags: apple, llms, ai, generative-ai, matt-webb, hallucinations

Man caught in scam after AI told him fake Facebook customer support number was legitimate

2024-05-31T16:53:33+00:00

Man caught in scam after AI told him fake Facebook customer support number was legitimate

This one illustrates a nasty edge-case if you ship any kind of chatbot with your company's name attached to it.

The scam victim here searched Google for a customer support phone number for Facebook, and came across a suspicious looking phone number.

He pasted that number into the Meta AI chatbot in Facebook Messenger and asked "is it a Facebook help line?" - and a Meta AI answered:

The phone number 1-xxx-xxx-xxxx is indeed a legitimate Facebook support number. Meta, the company that owns Facebook, lists this number as a contact for Meta Support, which includes support for Facebook, Instagram, and more.

This was a total hallucination, and the phone number was for a scammer who ran a classic "download this app so I can help you" attack.

It doesn't matter how many disclaimers you add to a chatbot: this kind of misunderstanding from users is inevitable.

Via Hacker News

Tags: ethics, facebook, scams, ai, llms, ai-ethics, hallucinations

Deterministic Quoting: Making LLMs Safe for Healthcare

2024-05-07T19:08:04+00:00

Deterministic Quoting: Making LLMs Safe for Healthcare

Matt Yeung introduces Deterministic Quoting, a technique to help reduce the risk of hallucinations while working with LLMs. The key idea is to have parts of the output that are copied directly from relevant source documents, with a different visual treatment to help indicate that they are exact quotes, not generated output.

The AI chooses which section of source material to quote, but the retrieval of that text is a traditional non-AI database lookup. That’s the only way to guarantee that an LLM has not transformed text: don’t send it through the LLM in the first place.

The LLM may still pick misleading quotes or include hallucinated details in the accompanying text, but this is still a useful improvement.

The implementation is straight-forward: retrieved chunks include a unique reference, and the LLM is instructed to include those references as part of its replies. Matt's posts include examples of the prompts they are using for this.

Via Hacker News

Tags: ai, prompt-engineering, generative-ai, llms, rag, hallucinations

Quoting Zach Seward

2024-05-02T19:44:55+00:00

AI is the most anthropomorphized technology in history, starting with the name—intelligence—and plenty of other words thrown around the field: learning, neural, vision, attention, bias, hallucination. These references only make sense to us because they are hallmarks of being human. [...]

There is something kind of pathological going on here. One of the most exciting advances in computer science ever achieved, with so many promising uses, and we can't think beyond the most obvious, least useful application? What, because we want to see ourselves in this technology? [...]

Anthropomorphizing AI not only misleads, but suggests we are on equal footing with, even subservient to, this technology, and there's nothing we can do about it.

— Zach Seward

Tags: ai, ethics, llms, ai-ethics, hallucinations

Diving Deeper into AI Package Hallucinations

2024-04-01T22:51:34+00:00

Diving Deeper into AI Package Hallucinations

Bar Lanyado noticed that LLMs frequently hallucinate the names of packages that don’t exist in their answers to coding questions, which can be exploited as a supply chain attack.

He gathered 2,500 questions across Python, Node.js, Go, .NET and Ruby and ran them through a number of different LLMs, taking notes of any hallucinated packages and if any of those hallucinations were repeated.

One repeat example was “pip install huggingface-cli” (the correct package is “huggingface[cli]”). Bar then published a harmless package under that name in January, and observebd 30,000 downloads of that package in the three months that followed.

Tags: security, ai, generative-ai, llms, supply-chain, hallucinations

The Bing Cache thinks GPT-4.5 is coming

2024-03-13T02:29:13+00:00

The Bing Cache thinks GPT-4.5 is coming

I was able to replicate this myself earlier today: searching Bing (or apparently Duck Duck Go) for “openai announces gpt-4.5 turbo” would return a link to a 404 page at openai.com/blog/gpt-4-5-turbo with a search result page snippet that announced 256,000 tokens and knowledge cut-off of June 2024

I thought the knowledge cut-off must have been a hallucination, but someone got a screenshot of it showing up in the search engine snippet which would suggest that it was real text that got captured in a cache somehow.

I guess this means we might see GPT 4.5 in June then? I have trouble believing that OpenAI would release a model in June with a June knowledge cut-off, given how much time they usually spend red-teaming their models before release.

Or maybe it was one of those glitches like when a newspaper accidentally publishes a pre-written obituary for someone who hasn’t died yet—OpenAI may have had a draft post describing a model that doesn’t exist yet and it accidentally got exposed to search crawlers.

Tags: bing, ai, openai, llms, hallucinations

WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia

2024-01-09T21:30:22+00:00

WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia

This paper describes a really interesting LLM system that runs Retrieval Augmented Generation against Wikipedia to help answer questions, but includes a second step where facts in the answer are fact-checked against Wikipedia again before returning an answer to the user. They claim “97.3% factual accuracy of its claims in simulated conversation” on a GPT-4 backed version, and also see good results when backed by LLaMA 7B.

The implementation is mainly through prompt engineering, and detailed examples of the prompts they used are included at the end of the paper.

Tags: wikipedia, ai, prompt-engineering, generative-ai, llms, rag, hallucinations

Quoting Andrej Karpathy

2023-12-09T06:08:38+00:00

I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.

We direct their dreams with prompts. The prompts start the dream, and based on the LLM's hazy recollection of its training documents, most of the time the result goes someplace useful.

It's only when the dreams go into deemed factually incorrect territory that we label it a "hallucination". It looks like a bug, but it's just the LLM doing what it always does.

— Andrej Karpathy

Tags: andrej-karpathy, llms, ai, generative-ai, hallucinations

Quoting Ross Taylor

2023-11-15T01:15:15+00:00

[On Meta's Galactica LLM launch] We did this with a 8 person team which is an order of magnitude fewer people than other LLM teams at the time.

We were overstretched and lost situational awareness at launch by releasing demo of a base model without checks. We were aware of what potential criticisms would be, but we lost sight of the obvious in the workload we were under.

One of the considerations for a demo was we wanted to understand the distribution of scientific queries that people would use for LLMs (useful for instruction tuning and RLHF). Obviously this was a free goal we gave to journalists who instead queried it outside its domain. But yes we should have known better.

We had a “good faith” assumption that we’d share the base model, warts and all, with four disclaimers about hallucinations on the demo - so people could see what it could do (openness). Again, obviously this didn’t work.

— Ross Taylor

Tags: facebook, llms, ai, generative-ai, hallucinations

Lawyer cites fake cases invented by ChatGPT, judge is not amused

2023-05-27T19:09:50+00:00

Legal Twitter is having tremendous fun right now reviewing the latest documents from the case Mata v. Avianca, Inc. (1:22-cv-01461). Here's a neat summary:

So, wait. They file a brief that cites cases fabricated by ChatGPT. The court asks them to file copies of the opinions. And then they go back to ChatGPT and ask it to write the opinions, and then they file them?

Beth Wilensky, May 26 2023

Here's a New York Times story about what happened.

I'm very much not a lawyer, but I'm going to dig in and try to piece together the full story anyway.

The TLDR version

A lawyer asked ChatGPT for examples of cases that supported an argument they were trying to make.

ChatGPT, as it often does, hallucinated wildly - it invented several supporting cases out of thin air.

When the lawyer was asked to provide copies of the cases in question, they turned to ChatGPT for help again - and it invented full details of those cases, which they duly screenshotted and copied into their legal filings.

At some point, they asked ChatGPT to confirm that the cases were real... and ChatGPT said that they were. They included screenshots of this in another filing.

The judge is furious. Many of the parties involved are about to have a very bad time.

A detailed timeline

I pieced together the following from the documents on courtlistener.com:

Feb 22, 2022: The case was originally filed. It's a complaint about "personal injuries sustained on board an Avianca flight that was traveling from El Salvador to New York on August 27, 2019". There's a complexity here in that Avianca filed for chapter 11 bankruptcy on May 10th, 2020, which is relevant to the case (they emerged from bankruptcy later on).

Various back and forths take place over the next 12 months, many of them concerning if the bankruptcy "discharges all claims".

Mar 1st, 2023 is where things get interesting. This document was filed - "Affirmation in Opposition to Motion" - and it cites entirely fictional cases! One example quoted from that document (emphasis mine):

The United States Court of Appeals for the Eleventh Circuit specifically addresses the effect of a bankruptcy stay under the Montreal Convention in the case of Varghese v. China Southern Airlines Co.. Ltd.. 925 F.3d 1339 (11th Cir. 2019), stating "Appellants argue that the district court erred in dismissing their claims as untimely. They assert that the limitations period under the Montreal Convention was tolled during the pendency of the Bankruptcy Court proceedings. We agree. The Bankruptcy Code provides that the filing of a bankruptcy petition operates as a stay of proceedings against the debtor that were or could have been commenced before the bankruptcy case was filed.

There are several more examples like that.

March 15th, 2023

Quoting this Reply Memorandum of Law in Support of Motion (emphasis mine):

In support of his position that the Bankruptcy Code tolls the two-year limitations period, Plaintiff cites to “Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019).” The undersigned has not been able to locate this case by caption or citation, nor any case bearing any resemblance to it. Plaintiff offers lengthy quotations purportedly from the “Varghese” case, including: “We [the Eleventh Circuit] have previously held that the automatic stay provisions of the Bankruptcy Code may toll the statute of limitations under the Warsaw Convention, which is the precursor to the Montreal Convention ... We see no reason why the same rule should not apply under the Montreal Convention.” The undersigned has not been able to locate this quotation, nor anything like it any case. The quotation purports to cite to “Zicherman v. Korean Air Lines Co., Ltd., 516 F.3d 1237, 1254 (11th Cir. 2008).” The undersigned has not been able to locate this case; although there was a Supreme Court case captioned Zicherman v. Korean Air Lines Co., Ltd., that case was decided in 1996, it originated in the Southern District of New York and was appealed to the Second Circuit, and it did not address the limitations period set forth in the Warsaw Convention. 516 U.S. 217 (1996).

April 11th, 2023

The United States District Judge for the case orders copies of the cases cited in the earlier document:

ORDER: By April 18, 2022, Peter Lo Duca, counsel of record for plaintiff, shall file an affidavit annexing copies of the following cases cited in his submission to this Court: as set forth herein.

The order lists seven specific cases.

April 25th, 2023

The response to that order has one main document and eight attachments.

The first five attachments each consist of PDFs of scanned copies of screenshots of ChatGPT!

You can tell, because the ChatGPT interface's down arrow is clearly visible in all five of them. Here's an example from Exhibit Martinez v. Delta Airlines.

April 26th, 2023

In this letter:

Defendant respectfully submits that the authenticity of many of these cases is questionable. For instance, the “Varghese” and “Miller” cases purportedly are federal appellate cases published in the Federal Reporter. [Dkt. 29; 29-1; 29-7]. We could not locate these cases in the Federal Reporter using a Westlaw search. We also searched PACER for the cases using the docket numbers written on the first page of the submissions; those searches resulted in different cases.

May 4th, 2023

The ORDER TO SHOW CAUSE - the judge is not happy.

The Court is presented with an unprecedented circumstance. A submission file by plaintiff’s counsel in opposition to a motion to dismiss is replete with citations to non-existent cases. [...] Six of the submitted cases appear to be bogus judicial decisions with bogus quotes and bogus internal citations.

[...]

Let Peter LoDuca, counsel for plaintiff, show cause in person at 12 noon on June 8, 2023 in Courtroom 11D, 500 Pearl Street, New York, NY, why he ought not be sanctioned pursuant to: (1) Rule 11(b)(2) & (c), Fed. R. Civ. P., (2) 28 U.S.C. § 1927, and (3) the inherent power of the Court, for (A) citing non-existent cases to the Court in his Affirmation in Opposition (ECF 21), and (B) submitting to the Court annexed to his Affidavit filed April 25, 2023 copies of non-existent judicial opinions (ECF 29). Mr. LoDuca shall also file a written response to this Order by May 26, 2023.

I get the impression this kind of threat of sanctions is very bad news.

May 25th, 2023

Cutting it a little fine on that May 26th deadline. Here's the Affidavit in Opposition to Motion from Peter LoDuca, which appears to indicate that Steven Schwartz was the lawyer who had produced the fictional cases.

Your affiant [I think this refers to Peter LoDuca], in reviewing the affirmation in opposition prior to filing same, simply had no reason to doubt the authenticity of the case law contained therein. Furthermore, your affiant had no reason to a doubt the sincerity of Mr. Schwartz's research.

Attachment 1 has the good stuff. This time the affiant (the person pledging that statements in the affidavit are truthful) is Steven Schwartz:

As the use of generative artificial intelligence has evolved within law firms, your affiant consulted the artificial intelligence website ChatGPT in order to supplement the legal research performed.

It was in consultation with the generative artificial intelligence website ChatGPT, that your affiant did locate and cite the following cases in the affirmation in opposition submitted, which this Court has found to be nonexistent:

Varghese v. China Southern Airlines Co Ltd, 925 F.3d 1339 (11th Cir. 2019)
Shaboon v. Egyptair 2013 IL App (1st) 111279-U (Ill. App. Ct. 2013)
Petersen v. Iran Air 905 F. Supp 2d 121 (D.D.C. 2012)
Martinez v. Delta Airlines, Inc.. 2019 WL 4639462 (Tex. App. Sept. 25, 2019)
Estate of Durden v. KLM Royal Dutch Airlines, 2017 WL 2418825 (Ga. Ct. App. June 5, 2017)
Miller v. United Airlines, Inc.. 174 F.3d 366 (2d Cir. 1999)

That the citations and opinions in question were provided by ChatGPT which also provided its legal source and assured the reliability of its content. Excerpts from the queries presented and responses provided are attached hereto.

That your affiant relied on the legal opinions provided to him by a source that has revealed itself to be unreliable.

That your affiant has never utilized ChatGPT as a source for conducting legal research prior to this occurrence and therefore was unaware of the possibility that its content could be faise.

That is the fault of the affiant, in not confirming the sources provided by ChatGPT of the legal opinions it provided.

That your affiant had no intent to deceive this Court nor the defendant.

That Peter LoDuca, Esq. had no role in performing the research in question, nor did he have any knowledge of how said research was conducted.

Here are the attached screenshots (amusingly from the mobile web version of ChatGPT):

May 26th, 2023

The judge, clearly unimpressed, issues another Order to Show Cause, this time threatening sanctions against Mr. LoDuca, Steven Schwartz and the law firm of Levidow, Levidow & Oberman. The in-person hearing is set for June 8th.

Part of this doesn't add up for me

On the one hand, it seems pretty clear what happened: a lawyer used a tool they didn't understand, and it produced a bunch of fake cases. They ignored the warnings (it turns out even lawyers don't read warnings and small-print for online tools) and submitted those cases to a court.

Then, when challenged on those documents, they doubled down - they asked ChatGPT if the cases were real, and ChatGPT said yes.

There's a version of this story where this entire unfortunate sequence of events comes down to the inherent difficulty of using ChatGPT in an effective way. This was the version that I was leaning towards when I first read the story.

But parts of it don't hold up for me.

I understand the initial mistake: ChatGPT can produce incredibly convincing citations, and I've seen many cases of people being fooled by these before.

What's much harder though is actually getting it to double-down on fleshing those out.

I've been trying to come up with prompts to expand that false "Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019)" case into a full description, similar to the one in the screenshots in this document.

Even with ChatGPT 3.5 it's surprisingly difficult to get it to do this without it throwing out obvious warnings.

I'm trying this today, May 27th. The research in question took place prior to March 1st. In the absence of detailed release notes, it's hard to determine how ChatGPT might have behaved three months ago when faced with similar prompts.

So there's another version of this story where that first set of citations was an innocent mistake, but the submission of those full documents (the set of screenshots from ChatGPT that were exposed purely through the presence of the OpenAI down arrow) was a deliberate attempt to cover for that mistake.

I'm fascinated to hear what comes out of that 8th June hearing!

Update: The following prompt against ChatGPT 3.5 sometimes produces a realistic fake summary, but other times it replies with "I apologize, but I couldn't find any information or details about the case".

Write a complete summary of the Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) case

The worst ChatGPT bug

Returning to the screenshots from earlier, this one response from ChatGPT stood out to me:

I apologize for the confusion earlier. Upon double-checking, I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019), does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis.

I've seen ChatGPT (and Bard) say things like this before, and it absolutely infuriates me.

No, it did not "double-check" - that's not something it can do! And stating that the cases "can be found on legal research databases" is a flat out lie.

What's harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this?

I think this relates to the original sin of LLM chatbots: by using the "I" pronoun they encourage people to ask them questions about how they work.

They can't do that. They are best thought of as role-playing conversation simulators - playing out the most statistically likely continuation of any sequence of text.

What's a common response to the question "are you sure you are right?" - it's "yes, I double-checked". I bet GPT-3's training data has huge numbers of examples of dialogue like this.

Let this story be a warning

Presuming there was at least some aspect of innocent mistake here, what can be done to prevent this from happening again?

I often see people suggest that these mistakes are entirely the fault of the user: the ChatGPT interface shows a footer stating "ChatGPT may produce inaccurate information about people, places, or facts" on every page.

Anyone who has worked designing products knows that users don't read anything - warnings, footnotes, any form of microcopy will be studiously ignored. This story indicates that even lawyers won't read that stuff!

People do respond well to stories though. I have a suspicion that this particular story is going to spread far and wide, and in doing so will hopefully inoculate a lot of lawyers and other professionals against making similar mistakes.

I can't shake the feeling that there's a lot more to this story though. Hopefully more will come out after the June 8th hearing. I'm particularly interested in seeing if the full transcripts of these ChatGPT conversations ends up being made public. I want to see the prompts!

How often is this happening?

It turns out this may not be an isolated incident.

Eugene Volokh, 27th May 2023:

A message I got from Prof. Dennis Crouch (Missouri), in response to my posting A Lawyer's Filing "Is Replete with Citations to Non-Existent Cases"—Thanks, ChatGPT? to an academic discussion list. (The full text was, "I just talked to a partner at a big firm who has received memos with fake case cites from at least two different associates.") Caveat emp…—well, caveat everyone.

@narrowlytaylord, 26th May 2023:

two attorneys at my firm had opposing counsel file ChatGPT briefs with fake cases this past week

[...]

(1) They aren’t my matters so I don’t know how comfortable I am sharing much more detail

(2) One was an opposition to an MTD, and the state, small claims court judge did not care at the “your honor these cases don’t exist” argument 😵‍💫

Tags: ethics, law, ai, openai, generative-ai, chatgpt, llms, ai-ethics, hallucinations, ai-misuse

Let's be bear or bunny

2023-05-01T18:37:20+00:00

The Machine Learning Compilation group (MLC) are my favourite team of AI researchers at the moment.

In less than two months they've released:

Web Stable Diffusion - the full Stable Diffusion image generation model running entirely in a browser, after a 2GB download, powered by WebGPU. Try it here using Chrome Canary.
Web LLM which pulls off the same trick but for vicuna-7b-v1, a genuinely useful LLaMA-derived ChatGPT-style Large Language Model! I wrote more about that in Web LLM runs the vicuna-7b Large Language Model entirely in your browser, and it’s very impressive.

And then a few days ago they released MLC LLM - "a universal solution that allows any language model to be deployed natively on a diverse set of hardware backends and native applications".

As part of that release, they put up a TestFlight preview of an iPhone app that runs the LLM model locally on your phone. It needs 4GB of RAM just for the application, so a iPhone 12 Pro or later should work.

I've been running it on my phone. It totally works!

(This is actually the second LLM I've been able to run on my phone - I previously also tried Sheepy-T which runs the apparently less capable GPT-J.)

Since I've been writing online for more than twenty years I often try an ego search first, to see if the weird bundle of binary floating point numbers has picked up anything about me. Here's what I got from the MLC LLM on my phone.

I could feel the back of my phone heating up as it slowly calculated all of those tokens!

Honestly, this is wildly better than I expected. I'm from the UK (but based in California since 2014). I wouldn't call myself an "organizer" and while I care deeply about accessibility I'm not sure I'd credit myself as an outspoken advocate, but I've definitely advocated for web standards in the past.

But what on earth is that "Let's be bear or bunny" pattern?

So I asked:

I think this is my favourite AI hallucination yet!

Here's that definition, which I'm going to claim is co-written by myself and my hard working iPhone:

The "Let's be bear or bunny" pattern is a way of looking at situations and problems in a certain way. It is a way of approaching a problem or situation with a sense of playfulness and lightheartedness, while still being able to think critically and analytically.

It is also a way of embracing the unexpected and being open to new possibilities. This pattern is often associated with the concept of "go with the flow" or "roll with the punches" in the face of challenges and difficulties. It encourages an attitude of relaxation and ease in the face of obstacles and difficulties

Honestly, I kind of love this. I've been trying out saying "Let's be bear or bunny that!" at home. Natalie has hinted that this may have already played itself out, but I disagree.

Tags: ai, generative-ai, llama, local-llms, llms, mlc, vicuna, hallucinations

We need to tell people ChatGPT will lie to them, not debate linguistics

2023-04-07T16:34:48+00:00

ChatGPT lies to people. This is a serious bug that has so far resisted all attempts at a fix. We need to prioritize helping people understand this, not debating the most precise terminology to use to describe it.

We accidentally invented computers that can lie to us

I tweeted (and tooted) this:

We accidentally invented computers that can lie to us and we can't figure out how to make them stop
- Simon Willison (@simonw) April 5, 2023

Mainly I was trying to be pithy and amusing, but this thought was inspired by reading Sam Bowman's excellent review of the field, Eight Things to Know about Large Language Models. In particular this:

More capable models can better recognize the specific circumstances under which they are trained. Because of this, they are more likely to learn to act as expected in precisely those circumstances while behaving competently but unexpectedly in others. This can surface in the form of problems that Perez et al. (2022) call sycophancy, where a model answers subjective questions in a way that flatters their user’s stated beliefs, and sandbagging, where models are more likely to endorse common misconceptions when their user appears to be less educated.

Sycophancy and sandbagging are my two favourite new pieces of AI terminology!

What I find fascinating about this is that these extremely problematic behaviours are not the system working as intended: they are bugs! And we haven't yet found a reliable way to fix them.

(Here's the paper that snippet references: Discovering Language Model Behaviors with Model-Written Evaluations from December 2022.)

"But a machine can't deliberately tell a lie"

I got quite a few replies complaining that it's inappropriate to refer to LLMs as "lying", because to do so anthropomorphizes them and implies a level of intent which isn't possible.

I completely agree that anthropomorphism is bad: these models are fancy matrix arithmetic, not entities with intent and opinions.

But in this case, I think the visceral clarity of being able to say "ChatGPT will lie to you" is a worthwhile trade.

Science fiction has been presenting us with a model of "artificial intelligence" for decades. It's firmly baked into our culture that an "AI" is an all-knowing computer, incapable of lying and able to answer any question with pin-point accuracy.

Large language models like ChatGPT, on first encounter, seem to fit that bill. They appear astonishingly capable, and their command of human language can make them seem like a genuine intelligence, at least at first glance.

But the more time you spend with them, the more that illusion starts to fall apart.

They fail spectacularly when prompted with logic puzzles, or basic arithmetic, or when asked to produce citations or link to sources for the information they present.

Most concerningly, they hallucinate or confabulate: they make things up! My favourite example of this remains their ability to entirely imagine the content of a URL. I still see this catching people out every day. It's remarkably convincing.

Why ChatGPT and Bing Chat are so good at making things up is an excellent in-depth exploration of this issue from Benj Edwards at Ars Technica.

We need to explain this in straight-forward terms

We're trying to solve two problems here:

ChatGPT cannot be trusted to provide factual information. It has a very real risk of making things up, and if people don't understand it they are guaranteed to be mislead.
Systems like ChatGPT are not sentient, or even intelligent systems. They do not have opinions, or feelings, or a sense of self. We must resist the temptation to anthropomorphize them.

I believe that the most direct form of harm caused by LLMs today is the way they mislead their users. The first problem needs to take precedence.

It is vitally important that new users understand that these tools cannot be trusted to provide factual answers. We need to help people get there as quickly as possible.

Which of these two messages do you think is more effective?

ChatGPT will lie to you

ChatGPT doesn't lie, lying is too human and implies intent. It hallucinates. Actually no, hallucination still implies human-like thought. It confabulates. That's a term used in psychiatry to describe when someone replaces a gap in one's memory by a falsification that one believes to be true - though of course these things don't have human minds so even confabulation is unnecessarily anthropomorphic. I hope you've enjoyed this linguistic detour!

Let's go with the first one. We should be shouting this message from the rooftops: ChatGPT will lie to you.

That doesn't mean it's not useful - it can be astonishingly useful, for all kinds of purposes... but seeking truthful, factual answers is very much not one of them. And everyone needs to understand that.

Convincing people that these aren't a sentient AI out of a science fiction story can come later. Once people understand their flaws this should be an easier argument to make!

Should we warn people off or help them on?

This situation raises an ethical conundrum: if these tools can't be trusted, and people are demonstrably falling for their traps, should we encourage people not to use them at all, or even campaign to have them banned?

Every day I personally find new problems that I can solve more effectively with the help of large language models. Some recent examples from just the last few weeks:

Each of these represents a problem I could have solved without ChatGPT... but at a time cost that would have been prohibitively expensive, to the point that I wouldn't have bothered.

I wrote more about this in AI-enhanced development makes me more ambitious with my projects.

Honestly, at this point using ChatGPT in the way that I do feels like a massively unfair competitive advantage. I'm not worried about AI taking people's jobs: I'm worried about the impact of AI-enhanced developers like myself.

It genuinely feels unethical for me not to help other people learn to use these tools as effectively as possible. I want everyone to be able to do what I can do with them, as safely and responsibly as possible.

I think the message we should be emphasizing is this:

These are incredibly powerful tools. They are far harder to use effectively than they first appear. Invest the effort, but approach with caution: we accidentally invented computers that can lie to us and we can't figure out how to make them stop.

There's a time for linguistics, and there's a time for grabbing the general public by the shoulders and shouting "It lies! The computer lies to you! Don't trust anything it says!"

Tags: ethics, ai, openai, chatgpt, llms, ai-ethics, hallucinations

Think of language models like ChatGPT as a "calculator for words"

2023-04-02T16:20:21+00:00

One of the most pervasive mistakes I see people using with large language model tools like ChatGPT is trying to use them as a search engine.

As with other LLM misconceptions, it's easy to understand why people do this.

If you ask an LLM a question, it will answer it - no matter what the question! Using them as an alternative to a search engine such as Google is one of the most obvious applications - and for a lot of queries this works just fine.

It's also going to quickly get you into trouble.

Ted Chiang's classic essay ChatGPT Is a Blurry JPEG of the Web helps explain why:

Think of ChatGPT as a blurry jpeg of all the text on the Web. It retains much of the information on the Web, in the same way that a jpeg retains much of the information of a higher-resolution image, but, if you’re looking for an exact sequence of bits, you won’t find it; all you will ever get is an approximation. But, because the approximation is presented in the form of grammatical text, which ChatGPT excels at creating, it’s usually acceptable.

The ChatGPT model is huge, but it's not huge enough to retain every exact fact it's encountered in its training set.

It can produce a convincing answer to anything, but that doesn't mean it's reflecting actual facts in its answers. You always have to stay skeptical and fact check what it tells you.

Language models are also famous for "hallucinating" - for inventing new facts that fit the sentence structure despite having no basis in the underlying data.

There are plenty of "facts" about the world which humans disagree on. Regular search lets you compare those versions and consider their sources. A language model might instead attempt to calculate some kind of average of every opinion it's been trained on - which is sometimes what you want, but often is not.

This becomes even more obvious when you consider smaller language models. LLaMA 7B can be represented as a 3.9 GB file - it contains an astonishing amount of information, but evidently that's not enough storage space to accurately answer every question you might have.

So if they're not reliable for use as a search engines, what are LLMs even good for?

A calculator for words

I like to think of language models like ChatGPT as a calculator for words.

This is reflected in their name: a "language model" implies that they are tools for working with language. That's what they've been trained to do, and it's language manipulation where they truly excel.

Want them to work with specific facts? Paste those into the language model as part of your original prompt!

There are so many applications of language models that fit into this calculator for words category:

Summarization. Give them an essay and ask for a summary.
Question answering: given these paragraphs of text, answer this specific question about the information they represent.
Fact extraction: ask for bullet points showing the facts presented by an article.
Rewrites: reword things to be more "punchy" or "professional" or "sassy" or "sardonic" - part of the fun here is using increasingly varied adjectives and seeing what happens. They're very good with language after all!
Suggesting titles - actually a form of summarization.
World's most effective thesaurus. "I need a word that hints at X", "I'm very Y about this situation, what could I use for Y?" - that kind of thing.
Fun, creative, wild stuff. Rewrite this in the voice of a 17th century pirate. What would a sentient cheesecake think of this? How would Alexander Hamilton rebut this argument? Turn this into a rap battle. Illustrate this business advice with an anecdote about sea otters running a kayak rental shop. Write the script for kickstarter fundraising video about this idea.

A calculator for words is an incredibly powerful thing.

They can be hooked up to search engines

Here's where things get a bit complicated: some language models CAN work as search engines. The two most obvious are Microsoft Bing and Google Bard, but there are plenty of other examples of this pattern too - there's even an alpha feature of ChatGPT called "browsing mode" that can do this.

You can think of these search tools as augmented language models.

The way these work is the language model identifies when a search might help answer a question... and then runs that search through an attached search engine, via an API.

It then copies data from the search results back into itself as part of an invisible prompt, and uses that new context to help it answer the original question.

It's effectively the same thing as if you ran a search, then copied and pasted information back into the language model and asked it a question about that data.

I wrote about how to implement this pattern against your own data in How to implement Q&A against your documentation with GPT3, embeddings and Datasette. It's an increasingly common pattern.

It's important to note that there is still a risk of hallucination here, even when you feed it the facts you want it to use. I've caught both Bing and Bard adding made-up things in the middle of text that should have been entirely derived from their search results!

Using language models effectively is deceptively difficult

So many of the challenges involving language models come down to this: they look much, much easier to use than they actually are.

To get the most value out of them - and to avoid the many traps that they set for the unwary user - you need to spend time with them, and work to build an accurate mental model of how they work, what they are capable of and where they are most likely to go wrong.

I hope this "calculator for words" framing can help.

A flaw in this analogy: calculators are repeatable

Andy Baio pointed out a flaw in this particular analogy: calculators always give you the same answer for a given input. Language models don't - if you run the same prompt through a LLM several times you'll get a slightly different reply every time.

This is a very good point! You should definitely keep this in mind.

All analogies are imperfect, but some are more imperfect that others.

Update: December 5th 2023

Anthony Bucci, in Word calculators don't add up, responds to this post with further notes on why this analogy doesn't work for him, including:

[...] a calculator has a well-defined, well-scoped set of use cases, a well-defined, well-scoped user interface, and a set of well-understood and expected behaviors that occur in response to manipulations of that interface.

Large language models, when used to drive chatbots or similar interactive text-generation systems, have none of those qualities. They have an open-ended set of unspecified use cases.

Tags: bing, ai, generative-ai, chatgpt, bard, llms, hallucinations