Checking an answer you could not have produced

Everything the course has taught you so far assumes you can tell whether an answer is any good.

Lesson 6 told you which tasks to be careful on. Lesson 7 told you what going wrong looks like. Both leave the same question standing, and it is the one that actually matters at your desk on a Tuesday.

You asked because you could not produce the answer yourself. So how do you check it?

If you could check it by reading it and knowing, you would not have needed to ask. That's the whole difficulty, and it's why the honest answer to it has steps in it rather than a slogan.

Four steps follow. They run cheapest first. None of them requires you to know the answer.

Say plainly where the four steps come from. They are this course's construction, not a finding and not anybody's published method.³ They are assembled out of things that are sourced: lesson 5's three readings, lesson 3's variation between runs, one measured result about self-assessment, and ordinary practice. Use them as a working method and judge them the way you would judge any working method. If one lets something through, the useful thing to say afterwards is which step it was.

Before the steps: how much checking does this need?

Checking expands to fill whatever time you've given it, so decide the amount before you start, and decide it from the consequence rather than from the clock.

Two questions.

What will this be used for? A note to yourself, a paragraph in a report somebody else signs, a figure that goes to a client, a decision that costs money.

What does being wrong cost? Embarrassment, an afternoon, a relationship, a regulatory problem.

Write the budget down as a sentence. "This is going in an internal update, so twenty minutes and the figures have to be right." "This is going to a client, so I am checking everything and asking Ravi to read it."

Writing it down is not about discipline. It is that the budget decides which of the four steps you run, and without it you will either check nothing or check until you are tired, and neither of those is a decision.

Step one: check the parts that are independently checkable

Names. Numbers. Dates. Quotations. Citations. Section numbers. Anything with an outside referent.

These are cheap to check and they are where failures concentrate, for the reason lesson 6 gave: they are the parts of the answer that have to match something outside the request. On a plain chat window nothing in the process goes and looks. On a product with a search tool something may have gone and looked, which lesson 6's callout is careful about, and it moves particular items across the line rather than moving the line.

This step is the best value in the procedure and it isn't sufficient, which is worth saying in the same breath, because people who do it well often stop here. An answer can be built out of entirely accurate components and still not follow.

Step two: ask for the steps, and read the seam

Lesson 5 gave you this and it earns its place here for a particular reason: you don't have to be able to produce the answer to notice that step three doesn't follow from step two.

That's what makes it usable when the material is beyond you. You aren't checking the claims. You're checking the joins, which is a much smaller job and is sometimes the whole of it.

Three readings, from lesson 5: do the steps answer the question that was asked, is each step right in itself, and does the conclusion follow from the steps. The third is the cheapest of the three, and it is the one that gets left out, because by the time you have read two sound steps the conclusion feels established.

Step three: ask it again, cleanly, and be careful what you conclude

Lesson 3 established that two runs are two draws. That variation is usable.

Ask the same question in a fresh conversation, not further down the same one, and compare.

If the two disagree, you have learned something solid: one of them is wrong, and you have been shown exactly where to look. That is worth a great deal for the few minutes it costs.

If they agree, you've learned much less than it feels like. Two runs of the same model on the same question agreeing tells you the system is consistent here. A system can be consistently wrong, and a consistent error is precisely the kind this step cannot see.

This step also covers self-review, which belongs here rather than at the end, and there is a measured reason to include it at all. A 2022 study found that larger models are well calibrated on multiple-choice and true-or-false questions, and can predict the probability that their own answer is right with "encouraging performance, calibration, and scaling".¹ The paper names the models it tested and this course has not read which ones, so read it as evidence that self-assessment of this kind exists rather than as a fact about the product in front of you.

Why it should work at all is a question the study does not answer, and the reading this course offers is that evaluating a claim is a different and easier task than producing one. That is an inference rather than a finding, and it is worth having because it predicts where self-review will be weakest.¹

The same study says why this cannot be the last step: models "struggle with calibration of P(IK) on new tasks", and the predictions only partially generalise.¹ P(IK) is their shorthand for the probability the model assigns to knowing the answer. So asking the system to review its own answer catches real things, on familiar ground, and quietly reassures you everywhere else.

Step four: take it to something that is not the system

A search engine. The actual document. The regulation itself. A colleague who knows.

This is the expensive step and it's the only one that can catch an error the system makes every time, because every other step stays inside the system and inherits whatever the system consistently gets wrong.

That's why the budget comes first. Step four is where the time goes, and the budget is what tells you whether this piece of work has earned it.

Two answers, run through the procedure

Both cases below are constructed, and so is every figure in them. The subject matter is plausible and none of it is advice. The figures exist so that you can see which step catches what, which is a thing you cannot see in a case described from the outside.

The short one: three clean steps and a wrong answer

A school business manager asks whether a particular equipment purchase can be spread across two budget years. Three paragraphs come back, confident, with one figure and a named policy document. It's the sort of answer nobody would think to doubt.

Step one, four minutes. The document exists under that name. The figure, a £10,000 capital threshold, is in it. Clean.

Step two, three minutes. She asks for the claims in order. Three of them: this equipment counts as capital; capital purchases above the threshold may be spread; therefore this purchase may be spread. Read the joins. The second claim does the work and the third follows from the first two. Clean.

Step three, two minutes. Fresh conversation, same question, same answer, same document. Clean.

Step four, twenty minutes. She rings the finance office. That policy was replaced eight months ago, and the rule now runs the other way.

Nothing inside the system could have caught this, because the system was not confused about anything. It was consistent, and it was consistently working from a document that had been superseded. This is the case that justifies the expensive step, and it is the reason the expensive step exists.

The long one: all four steps, each catching something different

A community transport charity has been told it may be able to reclaim VAT on a minibus purchase. The finance volunteer asks, and gets back four paragraphs: confident, specific, with two figures, a reference to a VAT notice by number, and a conclusion that the charity can reclaim the VAT if the vehicle is used for a qualifying purpose.

She sets the budget first. This decides whether a £34,000 purchase goes ahead this quarter. Being wrong costs real money and an apology to the board, so everything gets checked, and nothing goes to the board until somebody outside has confirmed it.

Step one, eleven minutes. The VAT notice exists and carries that number. One of the two figures, a nine-seat threshold, is in the notice where the answer says it is. The other, a figure of 60% put against business use, is nowhere in the notice at all. That is one finding already, and it is exactly the sort step one is for.

Step two, six minutes. She asks it to set out the claims it is arguing from, in order. Four come back:

1. The charity is a relevant body for this relief.
2. A vehicle with nine or more seats is a qualifying vehicle.
3. The relief applies where a qualifying vehicle is bought by a relevant body.
4. Therefore this charity can reclaim on this minibus.

Each one reads soundly. Now read the joins. Three follows from one and two. Four follows from three. The seam is inside claim two, which asserts that the seat count is what makes a vehicle qualify and offers nothing for it, while the answer's own prose had said the qualifying test was about use. She doesn't need to know any VAT law to see that the load-bearing claim was asserted rather than derived. That is the point of this step.

Step three, four minutes. A fresh conversation, same question. The second answer reaches the same conclusion, cites the same notice, and gives a different percentage. Consistent on the conclusion, inconsistent on a figure. That does not tell her which percentage is right. It tells her that the percentage is where the uncertainty sits, which is the same place step one had already pointed, and she carries it into step four.

Step four, an afternoon. She reads the section of the notice the answer pointed at, and finds that the qualifying test is about how the vehicle is used, which is what claim two had quietly replaced with a seat count. Then she rings the charity's accountant, who says the answer is broadly right in direction, that the percentage is wrong in a way that changes the sum, and that there is a further condition about selling the vehicle on within a few years that no answer mentioned at all.

What each step caught. One: a figure that is in the answer and not in the notice. Two: a claim asserted rather than derived, which turned out to be the same fault seen from inside. Three: that the figure was unstable, which pointed back at step one's finding. Four: the condition nothing inside the system was ever going to produce, because it was consistently absent from every answer.

Notice that steps two and four found the same thing twice, from opposite sides. Step two could see that a claim had no support under it. Only step four could say what the right claim was. That is the relationship between the cheap steps and the expensive one in one sentence: the cheap ones tell you where to spend the expensive one.

And the honest accounting. The whole thing took an afternoon, and writing it from scratch would have taken her a week she hasn't got. The answer was worth asking for and it was not worth trusting, and those are two judgements rather than one.

The standard to check against

Verification is easier when you know what you are aiming at, and there is a definition worth having. NIST's generative AI profile quotes a definition of high-integrity information as information that "distinguishes fact from fiction, opinion, and inference; acknowledges uncertainties; and is transparent about its level of vetting", and that "can be linked to the original source(s) with appropriate evidence".² NIST took it from elsewhere rather than writing it, which is part of why it is useful here: it was not written about answers from these systems at all.

Read it as a checklist for the answer that's in front of you.

Does it distinguish what is established from what is inferred? Does it acknowledge where it's uncertain, or is everything in the same confident register, which lesson 7 said is the default? Can each claim be linked to a source, and does that source say it?

An answer that fails those tests is not necessarily wrong. What it is is an answer whose vetting you now have to supply yourself, which is exactly what the four steps are.

What people get wrong

"If I cannot check it, I cannot use it." Every step above works without your knowing the answer, which is the whole of what they're for. The useful question is almost never whether you can check it. It is how much checking this particular piece of work has earned.

"Asking it to double-check is checking." It is step three of four. It catches real things on familiar ground, and it stays inside the system, so treating it as the end of the procedure leaves the whole of step four undone.

"Two agreeing answers confirm each other." They are two draws from the same distribution. Agreement is a fact about the system and it isn't one about the world.

"Checking takes as long as doing it, so there is no point." Exhaustive checking often does. Checking to a budget usually doesn't, and the difference between those two is why the budget comes first.

"The parts I could check were right, so the rest is probably right." The parts you can check and the argument they are assembled into are different objects, and step two exists because an answer can be accurate throughout and still not follow.

Practice

Connections

Back. Lesson 3 gave you the variation this lesson's third step uses, and the warning about what agreement means. Lesson 5 gave you the steps and the three readings, and this is where they are put to work on material beyond you. Lesson 6 told you which tasks land you here, and lesson 7 told you what you are looking for.

Forward. Lesson 9 is the other half of the decision. This lesson tells you what checking costs, and that cost belongs in the calculation about whether to hand a task over at all. Lesson 10 is about what you hand over of a different kind. And lesson 11 turns the same four steps on claims about these systems rather than on answers from them.

Go deeper

• Kadavath and colleagues, "Language Models (Mostly) Know What They Know" (2022). This course has read the abstract and not the paper. Read the abstract for the parenthesis in the title, which is the whole argument of step three in one word.
• NIST AI 600-1, section 2.8 on information integrity. The definition quoted above is the best short standard to check an answer against, and it wasn't written about AI answers in particular, which is part of why it works.

Sources

1. Saurav Kadavath and colleagues, "Language Models (Mostly) Know What They Know", arXiv 2207.05221, 2022. Abstract read; the paper was not opened. Supports: the finding that larger models are calibrated on multiple-choice and true-or-false questions, the quoted phrase about performance, calibration and scaling, and the stated limitation that calibration struggles on new tasks and generalises only partially. The claim that evaluating is an easier task than producing is this course's reading of why that result holds, rather than a claim the abstract makes.
2. NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024, section 2.8. Read in part. Supports: the quoted definition of high-integrity information. NIST takes that definition from a 2022 White House roadmap, which this course has not opened.
3. The four-step procedure itself is this course's construction, and the body says so where the steps are introduced. No source in research/SOURCES.md states it as a procedure. It is assembled from lesson 5's steps, lesson 3's variation, the calibration result in source 1, and ordinary practice. Treat it as a working method rather than as a finding, and if it fails you, the thing to report is which step let something through.