One of the most persistent myths in AI engineering is that efficiency comes from compactness. Fewer tokens, tighter schemas, denser representations; the assumption is that if data is cleanly organized, models will naturally perform better. Recent empirical work undermines that belief. What matters is not whether data is organized, but whether it is organized in a way the model already knows how to navigate.
To understand why, it helps to start with a very old tool.
Grep is a foundational Unix command used to search text. Its name comes from “global regular expression print,” and its function is brutally simple: scan everything, line by line, looking for patterns. Grep does not understand structure, hierarchy, or meaning. It does not know where important information is likely to live. It just searches everywhere until it finds a match.
When large language models lose their bearings, they behave exactly like this.
A recent paper on structured context engineering for file-native agentic systems ran nearly 10,000 experiments across multiple models, formats, and large-scale document environments. The researchers tested familiar schemas like Markdown, XML, JSON, and YAML alongside a compact, token-efficient format called TOON. On the surface, accuracy barely changed across formats. But beneath that flat accuracy curve was a dramatic divergence in behavior, cost, and stability.
The most striking finding was what the authors called the “Grep Tax.”
TOON was designed to reduce token usage. Instead, at scale, it caused models to consume up to 740 percent more tokens. The reason had nothing to do with content size. The models did not recognize the syntax. Without familiar structural cues, they repeatedly scanned the same material using different internal search strategies, cycling through pattern-matching attempts in an effort to orient themselves. The model wasn’t reasoning more. It was searching blindly.
This is the Grep Tax: the hidden cost models pay when structure fails to communicate significance.
In familiar formats like Markdown or XML, models have learned navigation heuristics during training. Headers, tags, indentation, and section boundaries act as landmarks. The model “knows” where introductions, methods, citations, and conclusions usually appear. This is not comprehension. It is pattern recognition reinforced at massive scale. But it works.
When those cues disappear, significance becomes flat. Everything looks equally important, so the model looks everywhere. The result is not better reasoning, but more wandering: more tokens, more latency, and greater instability as context size grows.
The paper revealed another important fracture: architectural best practices do not generalize across models. Agentic architectures that improved performance for frontier models actively harmed open-source models. The same orchestration layer amplified existing model biases rather than compensating for them. This suggests that many “universal” AI engineering rules are artifacts of which models we test on, not laws of intelligence.
The implications extend far beyond academic benchmarks. The Grep Tax applies anywhere large language models are asked to navigate structured information: enterprise document systems, compliance workflows, legal discovery, scientific research, internal knowledge bases, and multi-file software repositories. Any system that scales context without encoding recognizable significance risks paying for it in tokens, cost, and failure modes.
What can be done is counterintuitive. The answer is not more compression or cleverer schemas. It is alignment with the model’s learned structural priors. Verbosity often helps. Familiar formats outperform elegant ones. Redundancy stabilizes behavior. Systems that explicitly tell models what matters outperform those that assume models will infer it.
This is not a limitation of tooling. It is a limitation of current intelligence.
Which brings us to the natural question: will models ever be able to organize the data themselves?
In theory, yes, but not under current architectures.
Today’s models do not possess an internal, persistent sense of significance. They cannot reliably decide what matters across time, files, or tasks without external cues. When asked to organize unfamiliar data, they do not construct stable hierarchies; they experiment, search, and collapse back into pattern matching. Any apparent success is usually scaffolded by prompts, retrieval systems, or human-designed structure.
For models to truly organize data themselves, they would need durable significance weighting, an internal mechanism that allows importance to be learned, revised, and preserved across interactions. That is not a scaling problem. It is an architectural one.
Until that changes, structure will remain a form of governance rather than convenience. And the Grep Tax will continue to remind us that when models do not know what matters, they will look everywhere, and the cost is substantial amounts of time and efficiency.
