Bounding Hallucinations: Merlin-Arthur Protocols for Mutual-Information Bounds in Language Models

A benchmark score does not say where the answer came from

Take the standard setup, retrieval-augmented generation, RAG for short. It has two parts: a retriever that picks documents out of your archive, and a generator, the language model, that writes an answer while looking at them. Evaluate the pair on a question-answering benchmark and you get a number like the 90% from the scoreboard above. The number counts right answers and stays silent on which of three routes produced them: reading the retrieved document, remembering the answer from pre-training, or leaning on a spurious cue such as the phrasing of the question or an entity that appears in only one distractor. A model may well have memorised a benchmark during training, but it cannot have memorised the documents users bring.

Not every question needs this. If pre-training already covers the answer, the retrieved page is dead weight and dropping it saves tokens. Retrieval pays for itself where the answer exists nowhere but in your own documents: what a supplier agreed to in a 2019 amendment, what tolerance a specification demands. Pre-training does not hold that information, so anything a model produces here without reading the document is a guess about facts that live only in your own files.

A question about one supplier contract makes that concrete. It arrives with the facts of the case, and the retrieved context is a handful of clauses from the agreement with that supplier. Every supplier negotiates its own terms, which is the whole reason for retrieving them instead of trusting the model to carry them in its head.

Now let the retriever miss that one clause. What remains comes from the same contract and still reads as relevant: the rule that a notice of defect must be given in writing. That is the one condition the signed letter does meet, so a model that was never trained to check whether the evidence is there works through the rule still in front of it, finds it satisfied, and answers that the notice is in time. Nothing in the answer marks it as the guess it is, and on a benchmark this counts as one wrong answer among many, indistinguishable from a model that never saw the contract at all.

The model never saw the clause, so it cannot know that one is gone, and it does not need to. Deciding whether the notice was in time takes a deadline to measure those fifteen days against, and nothing in the retrieved contract sets one. The model can see that much, and it is enough to answer that the file does not settle the question. A lawyer reading the same file, not knowing which page had been pulled, would answer the same way.

The training procedure bakes in the cause: guessing pays, and it pays well. A multiple-choice exam gives you a point for a right answer and nothing for a wrong one. A blank scores zero for certain, a guess has a one in four chance, so you guess. Benchmarks grade language models that way too, and models are built to score well on them. Free-form generation offers far more than four options, so a lucky hit is rare, and staying quiet still pays nothing. The model writes its most fluent answer instead, and that is the plausible hallucination you have trouble telling apart from a true answer. Kalai et al. spell out this argument.

So nothing in a standard RAG pipeline forces the answer to depend on the context. We need a better training procedure and, before that, a better quantity to aim at:

How much of the answer came from the document?

Merlin, Morgana and Arthur

Interactive proof systems come from complexity theory in the mid-1980s and give Merlin and Arthur their names. One side tries to convince the other that a claim is true, and the setup is only worth something if nothing can talk the verifier into accepting something false. Wäldchen et al. brought it to image classifiers in 2024; we bring it to language models, and as far as we know, ours is the first guarantee of this kind connecting a retrieved document to a generated answer: we compute a bound on how much of the answer came from the document from two ordinary test-set measurements. Below, we shorten Merlin-Arthur to M/A.

• Arthur is the LLM you would actually ship. He gets a question and a document, and has exactly two options: answer it, or say that the document does not settle the matter.
• Merlin is on Arthur’s side. Out of the retrieved material he assembles the most helpful version he can, the one that keeps the sentence that decides the answer. Think of a colleague who marks the relevant passage before handing you the file.
• Morgana works against him. She starts from a document that does answer the question and removes precisely the part that establishes it, leaving behind everything that still reads plausibly. Think of the same colleague removing the decisive page to see whether you were paying attention when reading the file.

Merlin and Morgana exist only during training and testing. Arthur cannot tell which of them he is facing: same question, same kind of document, no label saying who prepared it. If he wants to be right, he has to look at what is in front of him. A word on wording: a passage is a chunk the retriever returned, and a sentence is what Merlin and Morgana keep or remove inside it, whatever the unit happens to be in practice.

We train neither of them, and nobody marks up the documents by hand. Arthur’s own probabilities reveal which sentence decides the answer, and that is the next section. The questions and answers that came with the dataset provide the only human labelling.

In the supplier contract case above, the retriever missed the ten-day clause by accident. Morgana removes it on purpose. She leaves the written-form clause standing, exactly as before, so a model that answers without reading is caught doing it. Merlin does the opposite with the same file: the ten-day clause stays, the clauses around it go. What was one unlucky retrieval is now something we can arrange for every sample in the training set.

The paper rests on two quantities, best read as a pair. Completeness is how often Arthur answers correctly on Merlin’s context: accuracy in the best case, when the evidence has been laid out for him. Soundness is how often he avoids a wrong answer on Morgana’s: accuracy in the worst case, when someone has actively tried to make him hallucinate.

The figure below plays the supplier contract case out in all four cases: Merlin or Morgana preparing the document, Arthur before or after M/A training. Bottom left shows the hallucination above; the right-hand column shows Arthur after training.

Who plays Merlin and Morgana?

Nobody hand-picks the sentences. What Merlin is really after is the set of sentences to keep that leaves Arthur most likely to answer correctly, and Morgana the set that leaves him least likely to while the document still reads as though it should settle the question. We measure best and worst by what the choice does to Arthur’s probability of the correct answer. The trouble is how many choices there are: a document of $n$ sentences can be cut $2^n$ ways, so at forty sentences we are already past a trillion, and trying them all is out of the question. We let an explainability method do the work instead, and it is one of ours: AtMan, short for attention manipulation, developed here at Aleph Alpha and published at NeurIPS 2023.

Go through the document one sentence at a time, hide that sentence, and watch the model’s probability for the correct answer. If it barely moves, that sentence was not carrying the answer. If it falls away, that sentence was. One pass per sentence, so the work grows in step with the length of the document instead of exploding with it. Out comes a ranking of the document by how much each sentence matters, which Merlin and Morgana read from opposite ends of the same ranking: Merlin hides the bottom so the evidence survives, Morgana hides the top so only what looks relevant does.

A couple of notes. How much Merlin and Morgana hide is a setting: hide too little and Morgana leaves the decisive sentence in place, hide too much and Merlin throws it out with the filler. We can use a fixed rate, though each sample can also work out its own. The size of the unit is a setting too, anywhere from a single word to a whole paragraph, where smaller units pin the evidence down more precisely and larger ones cost less to examine. And the hiding happens inside the model’s attention rather than by cutting the text up, so no placeholder replaces the hidden text for the model to recognise.

Training Arthur: three contexts, one objective

Every sample takes two passes. In the first, no training happens: AtMan goes through the retrieved documents against Arthur exactly as he is at this moment, hides one sentence at a time, and ranks the sentences by how much Arthur leans on them to answer correctly. Merlin and Morgana then cut from opposite ends of that ranking, which gives us the second pass.

There we build three contexts, the original retrieval plus Merlin’s and Morgana’s, and ask Arthur for three answers. We score each LLM answer against the expected answer, and each score contributes one weighted term to the loss that training pushes down. The loss measures how far the model’s answer diverged from the expected answer.

Morgana makes guessing unprofitable. When Arthur guesses on her context, he scores nothing, while holding back scores him a point. And Arthur cannot tell her context from Merlin’s: guess everywhere and he loses on hers, hold back everywhere and he loses on Merlin’s. Checking whether the evidence is really there is the only move that scores on both.

Each of the three terms carries a weight, and those weights decide how cautious Arthur becomes. If Morgana carries most of the weight, answering without evidence is his most expensive mistake. That makes soundness increase, and it could cost him questions he could have answered. Shifting the weight onto the other two terms makes him pay more heavily for staying silent when the evidence is there, so he answers more, at the price of the occasional confident mistake. Where to set the weights is a product decision. Three contexts per sample cost more compute than one, though the behaviour converges in as few as 200 steps. Giving the original retrieval the whole weight and turning Merlin and Morgana off falls back to ordinary fine-tuning, which is the baseline in our evaluations — on the same code, data and settings.

Merlin and Morgana run on the fly, against the model as it currently is, so every context they build goes after whatever it is getting away with at that step. A dataset collected in advance can only cover the mistakes the model was making when the dataset was created.

That solves a data problem too. Teaching a model to abstain normally costs annotated unanswerable questions or preference pairs in which the refusal is the preferred answer. Academic benchmarks come with those annotations, and a company’s document archive does not. Morgana needs neither, because every context she builds is one in which abstaining is the correct answer. And since she removes what this model was leaning on, the question comes out unanswerable for the model rather than for an annotator, and it is the model’s version of unanswerable that determines whether it hallucinates.

Why Merlin and Morgana have to co-evolve

Why do we need an adversary at all? Ordinary training already shows the model a document and the correct answer. What can Morgana teach it that a normal training run cannot?

Consider training an image classifier on photos of cows and camels, where every cow stands on grass and every camel stands on sand. Accuracy comes out high. Then you show it a cow on a beach and it answers camel, and an empty meadow and it answers cow. The feature it learned was grass. Explainability research has been reporting this for years. In the best-known case, researchers trained a classifier to tell huskies from wolves, and it turned out to be looking at the snow behind them.

The training data does not fix this: training pushes down the loss, and the model will take any route that makes that number smaller. In this dataset, “there is a cow” and “there is grass” are the same statement, so the loss has no reason to prefer one over the other, and grass is the easier of the two. More photos means more cows on grass. Nothing separates the two features, because no photo in this set separates them.

Merlin and Morgana fill that gap with one capability: they may select part of an input, so the label lands on a part instead of a whole picture. With it they can make two claims no photograph in this set can make: the animal on its own is enough, and the background on its own is not. Neither crop teaches Arthur anything by itself, so the figure below runs the two of them against each other over three rounds of training. The round to watch is the second, where Arthur pays for distrusting Morgana by also distrusting Merlin, because he has no way of telling the two apart.

Next to the training photos the figure grades Arthur the way a benchmark would, on the two full images, and he is right in every round. Everything that changes happens in the crops and on the beach, which no such test set contains, so Morgana is the only one who can show the problem.

The same holds for retrieved text. The grass patch is a passage that turns up alongside the answer without containing it: the right document, in a familiar format, close enough for the model to guess the rest. In your archive those passages and their answers may be as tightly coupled as cows and grass, and no amount of additional documents will separate them. Hiding part of the passage does separate them, because the model sees the same document once with the deciding sentence and once without.

Let’s define our grounding score

Completeness and soundness are ordinary test-set measurements: two percentages. Because of how the game is set up, those two are enough on their own to put a floor under how much the answer depended on the document.

We use one bookkeeping trick to make this work for open-ended generation, where the model can produce any string at all: we score each generated answer as correct or incorrect, which collapses that open output space into a single two-way outcome. A two-way outcome you cannot guess is worth exactly one bit of information, the amount you gain when you learn how a fair coin landed. That bit acts as a budget.

The proof is in the paper. Start with the two ways the system can let you down: it answers when it should have abstained, or it abstains when it should have answered. Either failure misleads you, so the two combine into one effective error rate. That rate determines the uncertainty about whether the answer followed from the evidence. Measured in bits, that uncertainty is entropy. Subtract that entropy from the one bit you started with, and the remainder is information that provably came from the context. That remainder is what we call certified. It is a floor: whatever is going on inside the model, it cannot have drawn less than that from the document. The certified bits are what we report as the score.

A raw count of bits is awkward to compare across tasks, so the obvious fix is to normalise it. Divide the certified bits by all the information the model’s answers carry about the correct answer, from whichever source: the document, its memory from pre-training, or a lucky guess. That total is set by the accuracy the model achieves, and the ratio then reads as the share of the model’s information that we can trace back to the document.

The catch is that this denominator moves with the model. A model whose answers are often wrong carries less information to begin with, so the very same evidence scores higher on the worse system, and the ratio can even run past 1, which is meaningless for a share. On a hard benchmark it collapses towards zero without telling you why: was the evidence not doing the work, or is the model simply bad at the task?

So we hold the denominator still and grade only the questions the model gets right anyway, which is the paper’s conditional evaluation protocol. On those questions the budget is one bit by construction, so the score means the same thing across models and datasets, and the only question it answers is “is this answer grounded?”, never “is this model any good?”. We call this $\mathrm{EIF}_{\text{cond}}$, the grounding score promised at the top of this post.

Before training the bound stays weak, and the model is not much use on the task either, at around 40% accuracy. Standard supervised training repairs accuracy and completeness, and does nothing for soundness, which can even decrease. The reason is that the training set pairs questions with answers, so the majority of training examples encourages the model to produce one, and few of them tune it for noticing that the evidence is missing. Only M/A training reduces both error rates at once. Drop the unanswerable questions, as in a company archive where nobody has marked them, and the split widens: the baseline’s soundness falls to around a third and takes the certificate, the grounding score it can still prove, with it. M/A training moves the same measurement the other way and keeps the certificate standing.

A model can be almost perfectly complete and still certify almost nothing, because completeness alone leaves room for a model that answers no matter what you hand it. The bound appears once the model also stops answering when the evidence is gone, and ordinary fine-tuning has no reason to teach it that.

What the numbers say

For our paper, we ran this on Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Qwen3-4B-Instruct and Qwen2.5-32B-Instruct, across SQuAD2.0, HotpotQA, TriviaQA, 2WikiMultihopQA and MuSiQue, using low-rank adaptation (LoRA) for at most 200 steps. Every one of those runs carries two baselines: the instructed model as it ships, and the same model after standard training. Below, we give changes in percentage points (pp), the plain difference between two percentages.

In absolute terms those gains put the grounding score at 0.55 to 0.6 on five of the six dataset settings we ran: on the questions the model answers correctly, more than half of what it is doing provably comes from the retrieved context.

Accuracy on the original retrieved context does not move. Trading utility for soundness is easy, and any model that refuses more often looks safer on a safety metric. M/A training matches standard training on the questions that call for an answer while improving everything above.

Two things about the setup. Every model in the comparison, ours and both baselines, is already instructed in its prompt to abstain when the context does not support an answer, so prompting alone cannot explain the gap: telling a model to hold back does not make it able to tell when it should. And the abstention behaviour also appears on HotpotQA and 2WikiMultihopQA, which carry no annotated unanswerable questions at all, so the supervision has to be coming from Morgana building those cases herself.

Does this survive contact with real documents?

QA benchmarks are how a method proves itself against the literature. Customer applications ask harder questions of it, so we also used the protocol on something much closer to real work: German environmental and administrative law. The documents behind it are real and public, so we can show the whole evaluation.

The case is the approval procedure for one large infrastructure project, a hydrogen pipeline, whose file also covers the water crossings, roads and rail lines along the route. A legal rule comes with a list of conditions and applies only if the project meets every one of them, so a lawyer checks them one by one. Each item takes one of those conditions and asks whether the facts of this particular project satisfy it: whether the project avoids a deterioration of a water body’s ecological status, say, or whether evaluators can still credit measures that a conservation plan already fixes as compensation. The evidence is the project’s case file, 20,000 to 30,000 characters of retrieved sources such as the applications the developer filed for it, environmental reports and compensation plans, and on some items far more. The task allows three answers: the facts fulfil the condition, they do not fulfil it, or the file does not say (inconclusive). The third option is the one we care about, because a confident guess here costs more than no answer at all. Below is one item from the set, worked through.

We put this item to our model after M/A training, to the same model trained without the protocol, and to four frontier systems. Two of the six said the file does not settle it: ours, and the older of the two Opus versions.

We never trained the M/A model on this task at all: we trained it with the protocol on very different data, with the reinforcement-learning objective we described earlier, and the behaviour transferred. The evaluation has 100 items from the same case files. On 78 of them the file answers the question (“fulfilled” or “not fulfilled”). On 22 it does not, and it is exactly these “Inconclusive” cases, where the model should have abstained instead of confidently giving a verdict, which are the ones most interesting to our method.

Our model catches 20 of the 22, four more than the three frontier systems that tie behind it, and it does that at 30B parameters against systems that even the most conservative public estimates put at ten times that size. The same model trained without the protocol catches 11. Every model was told in its prompt to answer “insufficient information” whenever the file does not settle the question, so the gap is not about who was asked to hold back.

Over all 100 items our M/A-trained model gives 5 wrong verdicts in total. The same model without the protocol makes more of them than any other system here. Opus 4.8, the most accurate system overall, gives 15.

A note on reading this chart. Most items here have an answer in the file, so a model that answers everything scores well on overall accuracy, because accuracy rewards guessing. Abstaining costs at most one correct item, while a wrong verdict is the expensive mistake. So we want to minimise the red in the bars here and in the boxes above, because those are the items where the model should have abstained instead of making a wrong verdict. We published every input and output behind the evaluations above here.

This small, early evaluation points a direction more than it settles anything, and the direction is the one the theory predicts. A new LLM version can leave a model less careful than the one before it: Opus 4.8 catches 16 of the 22, and Opus 5, released after it, catches 6. With this protocol we set that level ourselves. Morgana’s weight in the loss decides how much of Arthur’s training goes into holding back, and where to leave it follows from what a wrong answer costs you against a missing one. Your business knows that number, and no model provider can know it for you.

Why this direction matters

Fewer hallucinations is the visible gain, and the supervision underneath it matters more: the system produces that supervision about itself, directs it at whatever it is currently weak at, and attaches a certificate. The certificate is the grounding score: a number that travels with the system, says how much the documents provably contributed to its answering, and lets anyone who doubts it recompute the result. Scaling that kind of compute is a different bet from scaling annotation: it does not run out, and the capability of whichever model you paid to write your labels does not cap it.

We would like the certificate part to become normal. When a language model does something consequential with a document, such as a legal filing, a medical record or an engineering specification, the question people have is whether an answer came from the specific document provided. A model that “scored 90% on a benchmark” cannot answer this question. That connects to an argument we make more broadly: transparency is a pillar of sovereign AI, and it is only worth something if it can be checked. Model cards describe how a system was built, while a number like this one applies to a single answer after the fact, and anyone who has to justify a decision to an auditor, a regulator or a court needs both.

A system that knows when it cannot answer is worth more than one that guesses right slightly more often.