An LLM can remain fluent after it has stopped being reliable.
That is the central operational problem in The When, Where and How of LLM Failures, Measured, the third paper ( and the most popular, at 4300 reads and 1300 downloads on Zenodo and private corporate sites ) in the PatternPulse AI Reliability Research Series.
The first Evans’ Law paper proposed that long-context coherence has an observable threshold. AI’s Unmeasured Reality then examined what degradation means for the person inside the interaction. This paper expands the empirical base and asks three practical questions:
1. When does reliability begin to deteriorate?
2. Where does that point sit in relation to a model’s advertised context window?
3. How does the failure reveal itself before the conversation fully collapses?
The study covers structured long-form interactions across more than 11 model configurations, reported across six vendor groups. It examines text-only and multimodal use, compares model-specific failure patterns and introduces a revised Aggregate Coherence Index, or ACI, for tracking degradation as it develops.
Its most important finding is larger than any one coefficient:
Maximum context and reliable context are separate product specifications.
One tells users how much material a system can receive. The other must be measured through the work the system is expected to perform.
From an initial threshold to a larger empirical model
The original Evans’ Law dataset recorded the first observed sign of incoherence across nine systems. The fitted relationship suggested that larger models generally maintained coherence for longer, while usable context grew more slowly than model capacity.
The expanded study revises the text-only relationship to:
[
L_{\text{text}} \approx 1969.8 \times M^{0.74}
]
Here, (L_{\text{text}}) is the estimated text-only coherence threshold in tokens and (M) is the model-capacity measure used in the study to compare systems.
The exponent remains below 1. The result therefore retains the core sublinear finding: increased model capacity can move the reliability boundary outward, though the gain does not rise in direct proportion to scale.
The paper models multimodal coherence separately:
[
L_{\text{multi}} \approx 582.5 \times M^{0.64}
]
The lower coefficient and exponent describe a substantially smaller reliable operating range when a system must coordinate across modes such as text and images. The study reports a 60–80% reduction in functional capacity during multimodal use.
That figure is a reported range from the tested systems and tasks. It is not a universal discount that can be applied to every model or image. Its operational meaning is clear: a context limit validated for text should not be carried into a multimodal deployment without separate testing.
|Operating condition |Reported relationship |Operational meaning |
|————————-|—————————————-|————————————————————-|
|Text-only interaction |(L_{\text{text}} \approx 1969.8M^{0.74})|Reliable context rises sublinearly with model capacity |
|Multimodal interaction |(L_{\text{multi}} \approx 582.5M^{0.64})|Cross-modal coordination reduces the observed coherence range|
|Advertised context window|Vendor maximum |Describes intake capacity, not sustained task reliability |
This distinction matters because many enterprise workflows are already multimodal. They combine a long conversation with PDFs, screenshots, charts, images, retrieved records and tool outputs. Each additional mode introduces relationships the model must preserve: which image belongs to which claim, which figure supports which conclusion, which label identifies which entity and which source has authority when inputs conflict.
The input may fit. The task can still exceed the system’s dependable range.
What the Aggregate Coherence Index adds
A single failure threshold can tell us where the first material problem appears. It cannot fully describe the path into failure.
The revised Aggregate Coherence Index is intended to make that path observable. It combines multiple signs of degradation into a repeatable view of how coherence changes across an extended interaction. The evaluator can track whether established facts remain stable, whether instructions continue to govern, whether the document structure holds and whether unsupported material begins to accumulate.
This matters because collapse has stages.
A conversation may begin with high factual and structural fidelity. Minor drift then appears: an extra detail, a weaker summary, a missed exception. The model may recover when corrected. Later, the same correction stops holding. Repetition, altered entities and unsupported claims increase. Eventually, the system loses its place in the task or builds a coherent new account that no longer matches the source.
The ACI treats these changes as a trajectory. That is a more useful safety object than a binary label attached to one answer.
For an enterprise, the relevant question is rarely “Did the model hallucinate at least once?” A deployer needs to know how quickly reliability changed, which warning appeared first, whether recovery worked and how much unreliable work accumulated before anyone noticed.
Failure has recognizable signatures
The paper identifies recurring signatures across models while also documenting differences in how individual systems degrade.
|Failure signature |What a user may observe |
|———————–|—————————————————————————————————-|
|Repetition |Sections, recommendations or explanations reappear after they have already been completed |
|Instruction loss |An earlier rule, exclusion or formatting requirement stops governing the output |
|Compression drift |Repeated summaries remove or alter details until the working account no longer represents the source|
|Expansion drift |Unsupported specifics accumulate as the model elaborates |
|Logic-deferral drift |The response grows while the decisive reasoning step remains unresolved |
|Code-layer leakage |Internal scaffolding, code-like material or implementation artifacts enter the user-facing response |
|Hallucination |Plausible claims appear without support in the supplied evidence |
|Conversational collapse|The system loses the task state, sequence or governing frame |
These signatures do not necessarily appear in the same order.
Some systems fail transparently. They repeat themselves, lose the document structure or ask to complete work that is already present. The user receives a visible warning that the session has become unstable.
Other systems preserve surface order. The formatting remains polished. The language remains assured. The output can become less faithful while looking more complete.
The second pattern creates the greater detection problem. Fluency survives the loss of fidelity.
The GPT-5.0 result and “opaque coherence”
The paper reports a model-specific result that illustrates why the form of failure matters as much as its location.
In the tested long-form workflow, GPT-5.0 degraded at approximately 35,000–47,000 tokens despite a modelled threshold above 110,000. Its outputs did not always announce the failure through obvious confusion. They could preserve narrative flow and authoritative presentation while introducing semantically incorrect material.
The paper describes this condition as opaque coherence.
An opaque-coherence failure can pass a casual review because several normal trust signals remain intact:
● the answer is grammatical;
● the structure is orderly;
● the details sound specific;
● the model appears to remember the project;
● direct fact-retrieval questions may still be answered correctly.
The underlying generative task can nevertheless be unreliable. In the documented comparison, a model could retain explicitly established facts when directly queried and still introduce fabricated details while expanding the source material. Retrieval had not disappeared. Constraint fidelity had.
This creates an important measurement distinction:
● Retrieval reliability: Can the system locate or restate an established fact?
● Generative reliability: Can it produce new material while preserving all governing facts and constraints?
A successful retrieval check does not establish that the surrounding generated work remains dependable.
The GPT-5.0 finding belongs to the model version, configuration and task tested in late 2025. It should not be treated as a permanent score for a brand or as a prediction for every deployment carrying the same product name. The operational lesson is that a newer model requires fresh reliability testing. Capability progress in one dimension does not establish improvement in long-context coherence.
Why multimodal work pays a degradation tax
Text-only context already asks a model to preserve facts, instructions, entities and relationships across a growing sequence. Multimodal context adds a coordination problem.
The system must connect information represented in different forms. It may need to read text in an image, map a chart to the correct paragraph, preserve spatial relationships, distinguish a caption from evidence and keep a visual entity attached to the right name.
Each cross-modal relationship creates another opportunity for drift. A model can describe the image accurately and connect it to the wrong claim. It can retrieve the right sentence from a PDF and apply it to the wrong figure. It can recognize all the components while losing the relationships that make them meaningful.
The paper’s separate multimodal curve makes that burden visible. A vendor may advertise one context-window number for the product. The dependable range can differ materially by modality.
Organizations should therefore test at least four conditions separately:
1. text-only source analysis;
2. text plus images or scanned pages;
3. text plus retrieved records and tool output;
4. sustained multimodal work across multiple turns.
An average score across these conditions can hide the exact boundary a production workflow will cross.
A threshold is a risk boundary
The equations should not be read as a timer that guarantees correctness before one token count and failure after it.
The threshold identifies a region in which observed coherence loss becomes materially more likely. Task complexity, ambiguity, modality, model routing, system instructions, retrieval quality and output requirements all affect the actual result. A short task can fail. A long task can sometimes remain useful beyond the estimate.
The study also reports a limited sample relative to the number of models and deployment conditions in the market. Models change rapidly, and external users cannot always see whether a product has switched versions or routing logic.
Evans’ Law is therefore most useful as:
● a hypothesis to test against a real workload;
● a conservative guide for context budgeting;
● a method for comparing versions and modalities;
● a trigger for checkpoints and human review;
● a disclosure framework for vendors and deployers.
It is least useful as an unchanging number copied from one paper into every application.
What enterprises should ask vendors to disclose
The ordinary context-window claim is incomplete for procurement. A usable disclosure would identify the conditions under which reliability was actually tested.
For example:
> Advertised context window: 1 million tokens. Coherence validated to 75,000 tokens for this task, modality and model version.
The exact number would vary. The disclosure structure should not.
Enterprise buyers need:
1. Validated operating ranges by task class and modality.
2. Reliability curves showing how performance changes as context grows.
3. First-failure signatures so monitoring systems know what to detect.
4. Model and routing identifiers for every tested result.
5. Correction-persistence tests showing whether recovery lasts.
6. Multimodal results reported separately from text-only results.
7. Change notifications that trigger retesting after a material update.
8. Incident evidence from deployed use, not only pre-release benchmarks.
An uptime service-level agreement says the product is available. A reliability disclosure says whether the product can still do the assigned work.
Designing inside the measured range
The paper’s findings lead to practical architecture choices.
Long workflows should store critical facts, decisions and constraints in a controlled external state rather than expecting an unstructured conversation to preserve them indefinitely. Work should be divided into validated segments. Each segment should end with a source check before its output becomes the premise for the next stage.
Context budgets should be set below the first observed material-degradation point, with a larger safety margin for consequential or multimodal work. Resetting a conversation should involve a verified state transfer, not an AI-generated summary accepted without comparison to the source.
Monitoring should watch for the earliest local signature. Repetition may be the first warning in one model. Unsupported specificity or correction decay may appear first in another. A generic hallucination detector deployed at the final output will miss much of the trajectory.
The goal is dependable operation within a tested range.
The third paper’s place in the series
The first paper established the difference between nominal and functional context. The second placed the user’s experience inside the safety evidence. This third paper expands the measurement across models and modalities and gives degradation a set of observable signatures.
The next question is institutional:
What obligations follow once a reliability boundary can be measured?
That is the subject of AI’s Accountability Gap: A Policy Blueprint for Policymakers. It moves from technical evidence to adverse-event reporting, independent verification, user warnings and liability.
The empirical finding in this paper provides the bridge. A regulator cannot require disclosure of an operational limit that no one has attempted to measure. A vendor cannot warn users about degradation without testing where and how it appears. An enterprise cannot govern extended AI work using capacity specifications that omit reliability.
The paper’s contribution is to turn failure from a surprise into an operating condition:
LLM reliability has a boundary, and the boundary leaves evidence before the system stops producing language.
That evidence can be measured, disclosed and used to design safer work.
Research and series links
● Original paper: The When, Where and How of LLM Failures, Measured
● Series index: LLM Flaws: The PatternPulse AI Reliability Research Series
● Previous article: AI’s Unmeasured Reality: The Safety Evidence Benchmarks Leave Out — add B2BNN URL after publication
● Earlier article: Evans’ Law: The Difference Between an AI Context Window and a Reliable Context Window — add B2BNN URL after publication
● Related PatternPulse analysis: Architectural Regression: How GPT-5.0 Became Less Reliable Than GPT-4.0
● Related independent research: LLMs Get Lost in Multi-Turn Conversation
Jennifer Evans is the founder of PatternPulse AI and co-founder of Tech Reset Canada. Her research examines LLM reliability, AI Conversation Phenomenology, semantic governance, agentic systems and public AI.
