Over the past year, a set of reinforcement learning papers reframed one of the most basic assumptions in modern AI training: that models learn to reason because they are rewarded for being correct. A growing body of evidence suggests something subtle, and potentially more consequential, is happening instead.
At the center of this shift is reinforcement learning with verifiable rewards (RLVR), a training approach that replaces subjective human feedback with automated checks: unit tests, format validators, execution traces, or mathematically verifiable outcomes. First framed as a way to scale reinforcement learning without relying on costly human raters, RLVR has produced a series of surprising empirical results that challenge conventional explanations of how reasoning emerges in large language models.
Most interesting among these is emergence of epiplexity, a measure of structural information in trained models distinct from traditional complexity metrics, and the finding that, in certain models, reinforcement learning improves reasoning performance even when the reward signal is weak, noisy, or explicitly wrong. This result has forced researchers to revisit long-standing assumptions about what reinforcement learning actually does, and what it does not do.
What Re-Evaluating RLVR Causes
At a practical level, several concepts address why some AI systems generalize reliably while others fail outside familiar conditions: MDL, prequential coding, requential coding, and epiplexity. The Minimum Description Length (MDL) principle links compression to generalization: models that capture underlying structure rather than memorizing noise are more likely to perform well on new, unseen data. Prequential coding, used as a proxy for MDL, estimates this by summing training loss over time, but in practice, mixes learnable structure with irreducible noise, which made it difficult to distinguish genuine capability from overfitting.
Requential coding addresses this limitation by measuring the informational cost of the learning process itself, separating structural updates from shared background entropy; in practical terms, this offers a clearer signal of whether a model has learned transferable concepts or merely optimized for its training distribution. Epiplexity as a framework provides a way to identify when models develop compact, reusable internal representations, exactly the kind of structure enterprises depend on for robustness, out-of-distribution performance, and long-term reliability. Together, these ideas shift evaluation away from raw benchmark scores and toward understanding whether a system’s intelligence is built on stable structure or fragile correlations.
The clearest evidence for this reframing comes from an unexpected place: experiments where the rewards were deliberately weakened or randomized.
⸻
The unexpected lesson from “spurious rewards”
This effect does not generalize across all model families. It appears strongly in certain architectures and weakly or not at all in others. That distinction turns out to be the key to understanding what RLVR is actually doing.
The evidence increasingly suggests that RLVR is not teaching models how to reason. Rather, it is selecting and amplifying reasoning behaviors that already exist due to pretraining.
⸻
Reinforcement learning as a selector, not a teacher
Across multiple RLVR papers, a consistent pattern emerges. During pretraining, large models appear to acquire latent internal structures that resemble algorithmic reasoning, especially in domains like code, symbolic manipulation, and mathematics. These structures are not always dominant during inference. Models often default to shorter, more probabilistic responses that are locally coherent but shallow overall.
RLVR changes that balance.
Even weak or noisy reward signals tend to favor outputs that are:
• longer,
• more internally consistent,
• more structured,
• and easier to verify mechanically.
In models that already possess latent reasoning circuits, these constraints act as a mode selector, nudging the system toward computationally explicit behaviors. Once those modes are activated, correctness often follows, not because the reward signal teaches correctness, but because the underlying reasoning machinery was already present.
This helps explain why random rewards can still produce gains in some models. The reward does not need to encode truth. It only needs to apply selection pressure toward stability, structure, and coherence. In architectures where those properties correlate with genuine reasoning, performance improves as a side effect.
⸻
Why this does not work everywhere
One of the most important findings across the RLVR literature is that these effects are architecture and pretraining dependent. Models with weaker latent reasoning representations do not show the same resilience to spurious rewards. In those cases, noisy reinforcement behaves as expected: it fails to help or actively degrades performance.
This distinction undercuts the idea that RLVR is a universally powerful training method. Instead, it suggests that RLVR is best understood as a mechanism for developing existing capabilities, not creating new ones. Where pretraining has not already done the hard work, reinforcement learning has little to amplify. This architecture-dependency points toward a deeper theoretical question: what exactly is pretraining creating that RLVR can only select among?
⸻
Enter epiplexity
These findings intersect closely with recent theoretical work on epiplexity, a term used to describe the emergence of complex, coherent behavior in large models without direct supervision for that behavior.
Epiplexity proposes that:
• large-scale pretraining induces internal structures that support algorithmic or quasi-symbolic computation,
• these structures can exist without being reliably expressed,
• and downstream training often functions as a revealer rather than a builder of competence.
In this view, RLVR is not a learning signal in the traditional sense. It is a control signal, shifting the probability that the model enters a reasoning-heavy mode.
In From Entropy to Epiplexity, researchers revisit the long-standing Minimum Description Length (MDL) principle, the idea that models which compress data most efficiently are more likely to capture true underlying structure rather than noise, and argue that existing approximations of MDL fail in cases of emergence.
Traditional prequential coding, which estimates complexity by summing training loss, conflates learnable structure with irreducible noise and relies on heuristic corrections that break down under computational constraints. The paper proposes requential coding, a method that measures the complexity of the learning process itself by introducing an explicit observer or “teacher” that shares the data’s entropy. Only the deviation between the student’s learning trajectory and this shared baseline is counted as structural cost. This reframing, what is called epiplexity, suggests that what matters for generalization is not parameter count or final loss, but the presence of transferable, compressible structure revealed during learning.
Empirically, requential coding succeeds where prequential methods fail, correctly identifying emergent concepts as higher-value structure even when brute-force models achieve lower loss. In this light, RLVR’s ability to surface reasoning under weak or fake rewards looks less mysterious: it preferentially amplifies learning trajectories rich in epiplectic structure, even when the reward signal itself carries little information about correctness.
If reasoning capabilities are an emergent property of pretraining, then it makes sense that reinforcement learning does not need exact instructional rewards to improve reasoning performance. It only needs to bias the system toward patterns where those emergent structures dominate.
⸻
RLVR vs RLHF
Reinforcement learning with verifiable rewards also stands in sharp contrast to the dominant alignment paradigm of recent years: reinforcement learning from human feedback (RLHF). RLHF relies on subjective human judgments to shape model behavior, rewarding responses that appear helpful, harmless, or aligned with user preferences. While effective at improving surface-level fluency and compliance, RLHF entangles reasoning quality with human bias, preference noise, and stylistic conformity, often encouraging models to optimize for plausibility rather than correctness. RLVR, by contrast, replaces human evaluators with mechanically checkable signals, unit tests, executability, mathematical validity, or formal constraints, shifting the objective from social acceptability to internal coherence under verification. Importantly, recent results suggest that RLVR does not primarily teach new reasoning strategies, but instead selects for pre-existing computational modes that survive repeated verification pressure. Where RLHF smooths outputs toward human expectations, RLVR sharpens internal structure toward consistency and stability. This distinction helps explain why RLHF struggles to reliably induce deep reasoning, while RLVR can surface it even under weak or noisy rewards: one optimizes for preference alignment, the other for structural persistence.
A recent paper, Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning, sharpens this picture by offering one of the first mechanistic explanations for why RLVR works when it does. Rather than treating reinforcement learning as a uniform adjustment across all generated tokens, the authors show that reasoning trajectories are dominated by a small minority of high-entropy tokens, decision points where multiple reasoning paths remain viable. Empirically, updating only these high-entropy tokens during RLVR preserves, and in some cases improves, reasoning performance compared to updating the full token stream. The remaining majority of tokens are low-entropy execution steps whose optimization contributes little to reasoning quality. This finding reframes RLVR not as global behavioral shaping, but as selective amplification of uncertainty at structurally meaningful forks in the reasoning process. In doing so, it provides a concrete bridge between RLVR and epiplexity: transferable structure is not evenly distributed across computation, but concentrated in rare moments where the model chooses how to proceed. Reinforcement learning succeeds not by refining every step, but by stabilizing those few decisions that determine the shape of the entire reasoning trajectory.
Implications for alignment and evaluation
These results carry implications well beyond training efficiency.
First, they complicate common narratives about alignment. If reinforcement learning primarily selects among pre-existing modes, then alignment techniques that rely on reward shaping may be governing which internal pathways are activated, rather than instilling new values or competencies. This reframes alignment as purely a control-surface problem.
Second, they raise concerns about evaluation. A model trained under RLVR may become more coherent, more confident, and more structured, even when its underlying assumptions are wrong. Verifiability rewards enforce consistency, not truth. In domains where correctness cannot be mechanically checked, the same techniques could amplify convincingly wrong reasoning.
Finally, they suggest that future gains in reasoning may depend less on strong reinforcement schemes and more on pretraining choices: data composition, architectural inductive biases, and representational capacity.
⸻
Why RLVR is suddenly “hot”
RLVR sits at the intersection of several pressures shaping contemporary AI research. It promises scalability without human raters, produces visible gains in reasoning benchmarks, and most importantly, it forces a reevaluation of how reasoning arises in the first place.
The excitement around RLVR is less about reinforcement learning, more about what reinforcement learning reveals.
⸻
Summary: What RLVR is and why it matters
What RLVR is
• A form of reinforcement learning that uses automated, verifiable signals (tests, execution, format checks) instead of human feedback.
• Designed to scale training in domains like math and code where correctness can be mechanically evaluated.
What RLVR is not
• It is not a general method for teaching models how to reason.
• It does not reliably create new cognitive capabilities from scratch.
What RLVR appears to do
• Selects for longer, more structured, internally consistent reasoning trajectories.
• Amplifies latent reasoning capacities acquired during pretraining.
• Acts as a mode selector rather than an instructor.
Why RLVR is attracting attention
• Some models improve even under weak or spurious reward signals.
• This challenges classical assumptions about reinforcement learning.
• It aligns with emerging theories like epiplexity, which locate reasoning emergence in pretraining rather than fine-tuning.
Why caution is warranted
• Verifiable rewards enforce structure, not truth.
• Gains do not generalize across all models.
• Alignment risks remain if coherence is mistaken for correctness.
RLVR matters because it reveals that machines can already think, and shows how to find where that capability hides.
