Confident and wrong

The word everybody uses is hallucination. This course will use it once, here, and then stop.

The objection is in the standards body's own profile, recorded in a footnote: some commenters have noted that the terms "hallucination" and "fabrication" "anthropomorphize GAI, which itself is a risk related to GAI systems as it can inappropriately attribute human characteristics to non-human entities".¹ That objection is right, and the replacement NIST itself uses, "confabulation", has the same problem in a longer word.

So the course says what it means instead: a confident false answer, or a source cited for something it does not say. Those are plainer and they name two different things, which turns out to matter.

You need the word once because everybody else uses it. You don't need it again.

Two accounts, and they answer different questions

Ask why these systems produce confident false answers and you will be given one of two explanations. They are usually offered as rivals. In fact they are compatible, and the reason to hold both is that they disagree about whether the problem can be fixed.

The first is structural. NIST's profile puts it as the design working rather than failing. In NIST's word, confabulations "are a natural result of the way generative models are designed: they generate outputs that approximate the statistical distribution of their training data".¹ Lesson 2 built its whole argument on that sentence. On this account there's a floor, because producing text that fits the shape of writing will sometimes produce text that fits the shape of true writing without being true.

NIST also says where it bites hardest, and the sentence is worth having because it predicts your own experience: the dynamic "is particularly relevant when it comes to open-ended prompts for long-form responses and in domains which require highly contextual and/or domain expertise".¹ Open-ended, long, and specialist. That's most of the work anybody would want help with.

The second is about incentives, and it was posted in September 2025 by Kalai, Nachum, Vempala and Zhang. Their argument is that these systems guess because guessing is rewarded: training and evaluation score a confident wrong answer and an admission of ignorance the same way, which is to say at zero, so there's never a reason to say "I don't know".² Their own comparison is to a student sitting an exam with no negative marking, who should fill in every box.

What they propose follows from that, and it is not a change to how the systems are built. It is a change to how the dominant benchmarks score answers, so that admitting uncertainty is worth something.²

Now put the two side by side. The structural account implies a floor that better engineering lowers and does not remove. The incentive account implies a lever that nobody has pulled. So the honest answer to "will this be fixed" is that nobody knows, and that's a more useful answer than either confident version, because it tells you to keep checking without telling you to despair.

Two failures, and the second is the one that gets people

The forms this takes aren't all the same, and telling them apart changes what you check.

An invented source. An author who may exist, a journal that does exist, a title in the right style for the field, a year, a volume. Lesson 2 explained why it looks so right. A citation has a shape, that shape is all over the material these systems are built from, and producing a well-formed citation is exactly what a system producing well-shaped text will do.

Worked, with the simplest case there is. A teacher asks for research on whether handwriting notes beats typing them, and gets, among others: Reynolds, C. and Oduya, T., "Encoding Effects of Longhand and Keyboard Note-Taking", Journal of Educational Psychology, 2019, 111(4), 612 to 627. Every part of that is the right shape. The journal is real, the volume is plausible for the year, the page range is the right length for an article, and the title is written the way titles in that field are written.

The check takes forty seconds, and it's the cheapest thing in this lesson. Search the title. Search the authors. Search the journal's own index for volume 111. Nothing. The whole reference fails at the first question, which is why this kind is annoying rather than dangerous. You lose a minute and you haven't lost anything else.

A real source cited for something it does not say. The paper exists. The section exists. You look it up, confirm it is real, and stop, because everything you thought to check has passed.

Worked, with a case that gets all the way through. A parish clerk asks how much notice the council has to give of an extraordinary meeting. The answer is fluent, it is about the right length, and it names an Act and a section.

She checks the way anybody sensible checks. The Act exists. The section number exists inside it. Two questions, two clean answers, and at this point she's done more than the checking most answers get.

Then she opens the section, and it is about something else. It is in the right Act, it is in roughly the right area of the right Act, and it does not say what it was cited as saying.

Notice what caught it. Not searching, which passed. Not checking that the section number was real, which passed. Only reading the section. Every check that was cheap came back clean, and the only check that worked was the one that cost her ten minutes and a trip to the legislation site.

That is why this is the one that ends up in published work, and the evaluation in the next section counts the two separately for exactly that reason.

NIST names a third member of the same family, which lesson 5 met: outputs "may also include confabulated logic or citations that purport to justify or explain the system's answer, which may further mislead humans into inappropriately trusting the system's output".¹ Reasoning that supports an answer, produced the same way as the answer.

So the check is two questions rather than one. Does this source exist? And does it say what it is being cited as saying? The first is cheap and the second is not, which is a good reason to expect the second to be skipped.

What happens when you connect it to real documents

Every reader has the same objection by now, and it's a fair one. Mine searches real documents before it answers. Surely that fixes it.

It's been measured, and the measurement is worth more than any argument either way.

In 2024 a research team ran the first preregistered evaluation of commercial legal research tools: products built for professionals, connected to real case law, sold on the claim that the problem had been dealt with. The providers had claimed that retrieval "eliminat[es]" or "avoid[s]" hallucinations and guarantees "hallucination-free" citations.³

The result: "while hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time."³

Both halves of that sentence matter, and a summary that carries one of them is not reporting the study.

The reduction is real. Connecting a system to a document set does help, measurably, against a general chatbot. Anybody telling you retrieval does nothing is wrong.

And between 17% and 33% is not "solved". On tools sold to lawyers, tested in 2024, that is between one answer in six and one answer in three.

The per-tool figures are worth having because they vary so much: Lexis+ AI answered 65% of queries accurately and was the best of the three; Westlaw's AI-Assisted Research was accurate 42% of the time; Ask Practical Law AI gave incomplete answers on more than 60% of queries.³ Three products built for the same job, by serious companies, differing by more than twenty points.

Those figures belong to three products as they stood on the days they were tested, and none of them is a standing fact about any of those products now. The convention this course applies to a vendor's claim applies to a measurement of one: it carries its date, and the date is where it stops.

And note how the study had to define its terms to get a number at all. It counted "a response that contains either incorrect information or a false assertion that a source supports a proposition".³ Two failures, separated, which is the distinction the previous section drew and is why it is worth drawing.

Why a confident tone tells you nothing, and the careful version of that

NIST says plainly where the damage comes from: "Risks from confabulations may arise when users believe false content ... leading users to act upon or promote the false information."¹ The clause the ellipsis covers names the cause, and it is the confident nature of the response.

Fluency is a property of the text. That was lesson 6's sentence about benchmark scores and it transfers intact: it is not a measurement of anything, and it is not a report on the system's state. Digital Literacy made the same point about a padlock: a signal that costs nothing to produce cannot tell you what you wanted to know, and a well-shaped confident sentence costs a system that produces well-shaped text exactly nothing.

Don't take that further than it goes, though, because there's a careful result here and it cuts both ways. A 2022 study titled "Language Models (Mostly) Know What They Know" found that larger models are well calibrated on multiple-choice and true-or-false questions, and can predict the probability that their own answer is right with "encouraging performance, calibration, and scaling".⁴ The paper names the models it tested and this course has not read which ones, so take the finding as evidence that self-assessment of this kind exists rather than as a fact about the product in front of you.

The parenthesis in that title is doing work, and so is the limitation the authors state: models "struggle with calibration of P(IK) on new tasks", and the predictions only partially generalise.⁴

So the honest position is neither of the two you'll hear. "It has no idea whether it's right" is wrong, and so is "just ask it how confident it is". On the models that study tested, self-assessment was real, better on familiar closed-form questions than on unfamiliar open ones, and no substitute for checking. Lesson 8 puts it to work as one step among four, which is the right weight to give it.

Five things people believe about this

"It is a bug and they will fix it." Two accounts, one implying a floor and one implying an untried lever, and nobody knows. A vendor saying otherwise is making a claim you can ask for a number about.

"It only happens on obscure questions." NIST's own sentence says the dynamic is particularly relevant on open-ended prompts, long-form responses, and specialist domains.¹ Those are not the obscure corners of a subject. They are its depth, and they describe the work you most want help with.

"If it cites a source, the source is real." Sometimes, and the important half is the other question: if the source is real, does it say what it is cited as saying? The legal evaluation had to count both kinds to get a number at all, precisely because both happen.³

"A confident tone means firm ground." Fluency is a feature of produced text. So is hedging. A reader who learns to distrust the confident answers and trust the careful-sounding ones has changed which non-signal they are reading, and not much else.

"Telling it not to make things up helps." The instruction is in the request, and on both accounts above the problem is upstream of the request. Whether it does anything at all on your product is a measurement this course doesn't have. So the honest answer is that this is not the lever, and buying a sense of safety with it is buying nothing.

Practice

Connections

Back. Lesson 2 gave you the mechanism this whole lesson rests on, and the explanation of why an invented citation looks right. Lesson 5 met the same thing from the other side, as printed steps you can check. Lesson 6 told you which tasks put you here in the first place, and the answer, the ones whose answers have to match something outside the request, is exactly where this lesson's failures live.

Forward. Lesson 8 is the procedure, and it is the lesson this one exists to make necessary. Lesson 11 takes the vendor claims this lesson brushes against and teaches you to read them properly.

Go deeper

• NIST AI 600-1, section 2.2. Short, and every sentence this lesson quotes comes from it. Worth reading whole, because it is from a standards body rather than from somebody selling or attacking a product.
• Kalai, Nachum, Vempala and Zhang, "Why Language Models Hallucinate" (2025). This course has read the abstract and not the paper. The exam analogy is in the abstract, and it is the part of the argument this course found most useful.
• Magesh and colleagues, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools" (2025). Open access. This course read the abstract, the introduction and the results summary, and the introduction is the part to start with: it sets the vendors' claims out in their own words before testing any of them.

Sources

1. NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024, section 2.2. Read in part. Supports: the quoted objection to the word "hallucination" from NIST's own footnote; the "natural result of the way generative models are designed" sentence; the sentence about open-ended prompts, long-form responses and domains requiring contextual or domain expertise; the sentence about users believing false content due to the confident nature of the response; and the sentence about confabulated logic and citations.
2. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, "Why Language Models Hallucinate", arXiv 2509.04664, September 2025. Abstract read; the paper was not opened. Supports: the incentive argument, the exam comparison, and the proposal to change how dominant benchmarks score rather than to add another benchmark. This is one author group's argument and is not a consensus, and the lesson presents it as an account alongside NIST's rather than as a finding.
3. Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning and Daniel E. Ho, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", Journal of Empirical Legal Studies, 2025. Read in part: the abstract, the introduction and the results summary. Supports: the quoted vendor claims; the quoted 17% to 33% finding with the reduction against GPT-4; the per-tool figures of 65%, 42% and more than 60% incomplete; and the quoted definition of a hallucination as either incorrect information or a false assertion that a source supports a proposition. Tools were tested in 2024 and the figures carry that date.
4. Saurav Kadavath and colleagues, "Language Models (Mostly) Know What They Know", arXiv 2207.05221, 2022. Abstract read; the paper was not opened. Supports: the calibration finding on multiple-choice and true-or-false questions, the quoted phrase about performance, calibration and scaling, and the stated limitation that calibration struggles on new tasks and generalises only partially.