What you are actually talking to
90 min
Two hosts talk the lesson through. The voices are synthetic; the script was written from this lesson and checked against it, and asserts nothing the lesson does not.
- Describe, without mathematics, what the system is doing when it answers, in a way you could say to somebody else
- Predict from that description alone which of two described tasks the system will be unreliable at, and say why
- Explain why a confident false answer follows from the design rather than indicating that something has broken
Ask one of these systems to count the letter r in a long word and you may well get the wrong number, from a system that has just written you a competent paragraph about mediaeval crop rotation. That failure is small and funny.
A second failure is not funny at all. Ask for a reference and you may get an author who exists, in a journal that exists, with a title in exactly the right style for that field, and a year, and a volume number, for a paper that was never written.
Those look like unrelated faults and they have the same cause, and one paragraph explains both. Nine of the eleven lessons in this course are that paragraph applied to something, so it is worth the time it takes to get it properly. Everything you are told to do with these systems, by me or by anybody, either follows from it or is a ritual somebody picked up and passed on.
No mathematics is required to hold it.
What the system is doing
It produces text one piece at a time. Each piece is chosen given everything in front of it. The choosing is done by a model built to approximate the statistical shape of an enormous amount of writing.
Two of those three sentences come almost straight from the United States' standards body, which published a profile for exactly this technology in 2024 and, while explaining where confident false answers come from, wrote that they "are a natural result of the way generative models are designed: they generate outputs that approximate the statistical distribution of their training data; for example, LLMs predict the next token or word in a sentence or phrase."1
The middle sentence, about everything in front of it, is not in that quotation. It is the subject of lesson 3.
Read the last clause of NIST's sentence again, because there is a word in it doing a great deal of work.
The pieces are not letters, and they are not quite words
The word is token, and it no longer means what it used to. The standard textbook in the field is blunt about the change: "we now try to reserve the word token instead to mean the output of subword tokenization algorithms."2
Follow that through. Before a system can work with your text, the text has to be cut into pieces. The pieces are not characters, because that would make every sentence enormously long. They are not words either, because then any word the system had never seen would be a blank. So they are something in between, and the textbook's name for them is subwords: "modern tokenizers automatically induce sets of tokens that include tokens smaller than words, called subwords".2
The worked example in that chapter is the clearest one I know. Suppose the system has met low and new and newer during training, but never lower. Cut on whole words and lower is a hole. Cut into subwords and it is not: "if we had happened not to ever see the word lower, when it appears we could segment it successfully into low and er which we had already seen."2
So a token is sometimes a whole common word, sometimes a fragment like -er, and, at the limit, sometimes a single letter: "In the worst case, a really unusual word (perhaps an acronym like GRPO) could be tokenized as a sequence of individual letters if necessary."2
Picture a sentence being cut. A common word like strawberry may well arrive whole, in one piece. Something long and predictable like tokenization might come in two, perhaps token and ization. A rare surname might be cut into four or five fragments, or into letters. Which happens is decided by how often each piece turns up in text, by one of two algorithms the textbook names, byte-pair encoding and unigram language modelling.2 You don't need either algorithm. You need the consequence, and here it is as this course's own inference rather than as something the chapter states:
Because the pieces are chosen by frequency, nothing guarantees that the system is handed the letters of a word, and no step in the process spells one out. That is what makes a letter-level question a different kind of question. It is also why strawberry arriving whole is worse for counting than a rare surname arriving in fragments, which is the opposite of what you would expect.
You ask a system how many times the letter s appears in "Mississippi sandstorms", and then you ask the same question with a space typed between every letter. Write both predictions down before you run either.
Show the answer
Run both and compare. What matters is the comparison, not either answer on its own.
What you are testing is whether the difficulty lives in the counting or in the reading. If the spaced-out version does better, the counting was never the problem: the letters were not available as separate things until you made them separate.
Your system may well get the first one right. Longer and stranger words are where it shows, and a system that gets this right isn't a system that read it the way you did. It's a system for which this particular word happened to be cut conveniently.
There is nowhere to look it up
This is the part that surprises people most, and it's worth being slow about, because two other things you already understand are nearly right and the difference matters.
You know from Digital Literacy what a file system is: a thing lives in a place, it has an address, and fetching it means going to the address. Nothing is produced. The file that comes back is the file that was put there.
You also know what a search engine does, from using one rather than from any course. Somewhere there are documents that people wrote. The engine finds the ones that match and hands you a list. Again, nothing is produced. The page you open is the page its author wrote.
A language model is neither, and the figure below is the difference. In the first two rows there is a store, and an arrow reaches it and comes back carrying something that was already there. In the third row there is no store for an arrow to reach.
That dashed box is the whole of the difference. How to put it exactly is this course's own wording rather than a claim any of its sources makes, and the careful version matters, because the loose version is wrong in a way an expert would catch. The system's parameters plainly carry a great deal about the world; that is why the thing answers factual questions correctly at all. What is missing is a document to go and fetch. So an answer is produced rather than retrieved, and so is a citation.
Now put the invented reference back in front of you.
A real citation has a shape. Surname, initials, a year in brackets, a title in sentence case, a journal in italics, a volume, a page range. That shape is in the training material tens of millions of times, so a system producing text that fits the statistical shape of writing will produce citations that fit the shape of citations, with the right sort of author for the field and the right sort of journal for the claim. It is convincing for exactly that reason, and it was not a failed attempt at retrieval. Nothing retrieved anything.
A colleague argues that because the system was trained on real papers, its citations must come from those papers, even if it occasionally muddles two of them. What has she got wrong?
Show the answer
She has kept the retrieval picture and made it fuzzy, which is the commonest half-step towards understanding this, and it is still the wrong picture.
Training isn't storage. What the training produced is a model of the shape of text rather than a copy of the documents it read, and no step in it consults a particular document.
The practical difference is what she does next. If she believes a citation is a muddled retrieval, she'll try to work out which real paper was meant. If she understands it was produced, she'll go and look for a real source instead, which is both faster and the thing lesson 8 turns into a procedure.
Why "it made that up" is the wrong complaint
You've now got enough to see something most writing on this subject misses. A system built this way is not sometimes producing text and sometimes reporting facts; it is producing text every time. When the produced text happens to be true, that is because the shape of true writing about that subject was well represented in what it learned from, and when it is false, the same process ran.
So there's no separate faulty mode to be switched off, and a confident false answer isn't evidence that anything malfunctioned. The confidence and the falsehood are both properties of the text, and the text is what the thing makes.
That isn't the whole story. Lesson 7 takes it properly, including an argument published in 2025 that the training and scoring of these systems rewards guessing over admitting ignorance, which would be a real lever somebody could pull.3 For now hold the narrower version, because it's the one that changes how you work: ask whether this is the kind of task where the shape of good writing and the truth come apart. When the answer is no, as in rewriting something you supplied, you are on safe ground. When the answer is yes, as in anything that has to match a fact outside the conversation, you are not, and no amount of asking nicely changes that. Lesson 6 turns the question into something you can apply in a few seconds, and this lesson is where it comes from.
What people get wrong
"It looks things up." Unless it has been handed a tool that goes and searches, it does not. Plenty of products now hand it such a tool, and that changes what it can reach rather than how the answer is produced when it has not used one. How much the tool helps has been measured, on commercial legal research systems tested in 2024, and lesson 7 has the number.4 It is smaller than the people selling those systems said.
"It has a database." No. Training built a model of the shape of text. There's no table of facts to be corrected, which is why pointing out an error changes this conversation rather than the system. What a correction does and does not reach is lesson 3's subject, and the answer depends on the product more than people expect.
"It understands the question the way I do." Careful in both directions. It plainly does something that behaves a great deal like understanding on a great many tasks, and this course isn't going to tell you it's merely pattern matching, because that phrase does no work. What you can say with more confidence is narrower and more useful: whatever it is doing, it is doing it over pieces of text that do not correspond to letters, and without a document to consult. Those two facts predict its failures, and "does it really understand" doesn't.
"If it can write an essay it can count letters." Those two aren't on a scale with each other. Essay-shaped text is what the pieces are good for. Letter counting is a job about something below the pieces.
"Getting it wrong means it is broken." It means you were on the wrong kind of task, which is most of what lesson 6 is about. Your first instinct on a bad answer should be to ask what kind of question you asked, rather than to ask again more firmly.
How the model is built, which is transformers, attention, parameters and training, and which takes a graduate textbook a hundred pages. You don't need it. Everything in this course follows from the one paragraph, and I would rather you held one paragraph tightly than four chapters loosely.
Whether it "really" reasons. People argue about this, and the argument turns on what the word's being asked to mean. Lesson 5 says so explicitly and answers the question you can act on instead, which is what asking for steps actually does to the answer.
Practice
Take 25 minutes. Write your prediction down first, every time. Guessing at an answer before you are given it makes the answer stick better than reading it twice would, which is what How to Learn Anything's lessons on retrieval and testing are about.
One: the letters.
- Pick a long, unusual word. Predict whether the system will count a given letter in it correctly.
- Ask. Write down what came back.
- Now ask the same question with the word spaced o u t like that, and record the answer.
- Then ask it why it got the first one wrong, and write down what it says.
- Two things to judge, and they're separate. Whether the count was right. And whether the explanation it gave for its own mistake is one you can rely on, given what this lesson says about where explanations come from.
Two: the citation.
- Pick something obscure but checkable in a subject you know. Ask for three references on it.
- Before you check any of them, predict how many will be real, how many will be real but not say what they're cited as saying, and how many won't exist.
- Check all three properly. Not by asking the system. By looking.
- Write down which of the three kinds each one was.
Keep both results. Lesson 7 asks for the citation one back, and by then you'll have a name for each of the three kinds.
Take 10 minutes, and do this one at least an hour after the rest of the lesson rather than straight away.
Close the page. Write the paragraph from the start of this lesson in your own words, from memory: what the system produces, what decides each piece, and what it's approximating.
Then open the page and compare, and mark what you left out. The thing you left out is the thing to reread, and it is worth much more to you found this way than it would have been from reading the section again.
Connections
Back. Lesson 1 gave you two studies pointing opposite ways and a reconciliation: gains near the edge of what you can already do, losses on work you had mastered. This lesson is the first half of why the frontier is jagged rather than smooth. Get out the two-column baseline you wrote there and mark any entry this lesson has changed your mind about, which will usually be on the right-hand side. It borrows one picture from Digital Literacy, the file system and its addresses, and uses it as a contrast rather than re-teaching it.
Forward. Lesson 3 is what the system can see when it answers, which is the second sentence of the paragraph above and the one NIST's quotation does not cover. Lesson 6 turns the question about shape and truth into a working test. Lesson 7 is confident false answers taken properly, with the measurement of what a search tool does and does not fix. And lesson 8, checking, only makes sense once you know there is no document to go and look at.
Go deeper
- Jurafsky and Martin, Speech and Language Processing, 3rd edition draft, chapter 2, "Words and Tokens". Free. The first eight pages are readable without any background and will show you where tokens come from. Stop when the algorithm starts; you don't need it.
- NIST AI 600-1, the Generative AI Profile, section 2.2. Lesson 1 sent you to the whole document. This is the two pages that matter here, on confident false answers, from a standards body rather than from a vendor or a critic.
- Brown and colleagues, "Language Models are Few-Shot Learners" (2020). The abstract alone is worth ten minutes: it's where the idea that you specify a task in the text of the request, with no training at all, was established.
Sources
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024, section 2.2. Read in part. Supports: the quoted sentence on approximating the statistical distribution of training data and predicting the next token, and the claim that confident false answers follow from the design. It does not support the middle sentence of this lesson's core paragraph, about conditioning on everything in front of the system, which belongs to lesson 3 and is flagged as such in the body.
- Dan Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft of 19 August 2026, chapter 2, "Words and Tokens". Chapter read in full; the rest of the book was not opened. Supports: the redefinition of "token" as the output of subword tokenization, the definition of subwords, the
lowerintolowanderexample, the worst case of a word tokenized as individual letters, and the naming of byte-pair encoding and unigram language modelling as the two algorithms in current use. The sentence about nothing spelling a word out is this course's inference from that chapter rather than a claim the chapter makes, and the body says so. - Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, "Why Language Models Hallucinate", arXiv 2509.04664, September 2025. Abstract read. Supports: the one sentence here that training and evaluation reward guessing over admitting uncertainty. Taken properly in lesson 7.
- Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning and Daniel E. Ho, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", Journal of Empirical Legal Studies, 2025. Read in part. Supports: the single sentence in the misconceptions list that the benefit of giving one of these systems a search tool has been measured, on commercial legal research systems tested in 2024, and is smaller than the vendors claimed. The figures are in lesson 7.
Check your understanding
This lesson has a 6-question quiz. Pass it and the questions come back on a schedule in Review, so what you learned stays learned. Your progress is saved in your browser; no account needed.