A named evaluation experiment records the artifact, task law, inference protocol, evaluator, seed kernel, budget, and sampling unit. Two reported scores are comparable only after these coordinates and their intended target have been stated.

For a fixed evaluation interface from Convention 1.1, an evaluation law is a probability measure \(\mathsf Q\) on the finite outcome space \(\mathcal Z=\mathcal X_{\mathsf T_{\rm ev}}\times \mathcal O_{\mathsf T_{\rm ev}}\) with random pair \((X,O)\sim \mathsf Q\). Its score for artifact \(M\) is a bounded measurable map \(s_M:\mathcal Z\to [0,1]\). Write \(J_{\mathsf Q}(M)=\mathbb E_{\mathsf Q}[s_M(X,O)]\).

A matched evaluation pair is \((\mathsf Q,\mathsf I,b,\Lambda )\) and \((\mathsf Q',\mathsf I',b',\Lambda ')\) together with a coupling \((Z,Z')\) whose marginals are the two laws. The pair is matched on the declared coordinates when the task, utility, evaluator version, and sampling unit are shared; any changed coordinate is named.

For iid coupled draws \((Z_j,Z'_j)_{j=1}^m\), define \(\widehat \Delta _m=m^{-1}\sum _{j=1}^m(s_M(Z_j)-s_{M'}(Z'_j))\). The pairing is part of the estimator specification; it does not silently identify the two evaluation laws.

Under the iid coupling in Definition 2.1.4, \(\mathbb E[\widehat \Delta _m]=J_{\mathsf Q}(M)-J_{\mathsf Q'}(M')\).

If \(s_M(Z)=s_{M'}(Z')\) almost surely under a common-seed coupling, then \(\widehat \Delta _m=0\) for every sample even when independent sampling would have nonzero variance. This is a variance statement, not evidence that the two artifacts have equal behavior on a different law.

The result in Theorem 2.1.5 follows from its displayed hypotheses. A shared seed can make a paired comparison precise while the changed task law, evaluator, or inference budget still changes the estimand. No causal effect follows without a declared intervention and matched coordinates.

On a common finite outcome law \(\mathsf Q\), let each artifact \(M\) have score maps \(s_M,t_M:\mathcal Z\to [0,1]\). They have perturbation radius \(\varepsilon =\sup _{M,\,z\in \operatorname {supp}(\mathsf Q)} |s_M(z)-t_M(z)|\) when \(0\leq \varepsilon \leq 1\); write \(J_s(M)=\mathbb E_{\mathsf Q}[s_M]\) and \(J_t(M)=\mathbb E_{\mathsf Q}[t_M]\).

Under Definition 2.2.1, every artifact \(M\) satisfies \(|J_s(M)-J_t(M)|\leq \varepsilon \).

If two artifacts \(M_1,M_0\) are scored by both \(s\) and \(t\) under the same law and \(\lVert s-t\rVert _{\infty ,\mathsf Q}\leq \varepsilon \), then the gain difference obeys \(|(J_s(M_1)-J_s(M_0))-(J_t(M_1)-J_t(M_0))| \leq 2\varepsilon \). This is a finite utility perturbation bound, not a training or distribution-shift theorem.

On equally likely outcomes \(a,b\), take score vectors \((s_{M_1}(a),s_{M_1}(b))=(1,0)\) and \((s_{M_0}(a),s_{M_0}(b))=(0,1)\); a second score map swaps coordinates. Both artifacts have aggregate score \(1/2\) under both maps, but their per-outcome ordering reverses. Aggregate transport alone does not identify localized behavior.

The bound in Theorem 2.2.2 requires the same finite outcome support. Changing prompts, parsers, evaluators, or inference budgets is a new law and must be recorded rather than hidden inside a score perturbation.

For a finite record space \(\mathcal R\), an observable is a map \(h:\mathcal R\to \mathcal H\). A summary-only evaluator sees \(h(r)\) and not the underlying record \(r\); the omitted coordinates are unavailable to that evaluator.

A compaction kernel is a randomized map \(K(dh\mid r)\) from records in \(\mathcal R\) to summaries in \(\mathcal H\). A deterministic summary is the special case \(K(dh\mid r)=\delta _{h(r)}\).

Let \(r,r'\in \mathcal R\) induce the same summary law under the kernel in Definition 2.3.2. Any randomized decision rule that receives only that summary has the same output law in the two worlds.

Two research records can share every summary score while one contains a withdrawn lemma and the other contains a feasible proof plan. A summary-only selector therefore chooses identically, although the correct next action differs. This is a finite witness to information loss, not a claim about a particular model.

The case study in [li2026longhorizon], Sections 4--7, reports file-based memory, human steering, and research-state failures under one system and task. It motivates explicit observables and omitted constraints; it does not supply the finite theorem in Theorem 2.3.3 or a general model capability law.

An evaluation cell is \(C=(M,\mathsf Q,\mathsf I,b,\Lambda ,v)\), where \(M\) is an artifact, \(\mathsf Q\) a task law, \(\mathsf I\) an inference protocol, \(b\) a budget, \(\Lambda \) a seed kernel, and \(v\) an evaluator version. A cell comparison may differ in named coordinates only.

For a finite evaluation plan with disjoint stages of costs \(c_1,\ldots ,c_k\in \mathbb R_{\geq 0}\), total cost is \(c=\sum _{i=1}^k c_i\). Appending a stage increases cost by its declared amount and cannot preserve a budget \(B\) unless the remaining slack covers it.

Let two evaluation plans use the same declared stages except for a substitution whose cost is at most the replaced stage cost. If their common prefix cost is \(c_0\) and both suffix costs fit \(B-c_0\), then both cells in Definition 2.4.1 are admissible under budget \(B\).

A paired run can spend \(B=100\) units as 40 units of inference, 20 of evaluator calls, and 40 of audit. Replacing 10 inference units by 10 audit units preserves the budget, but changes the cell and must not be described as the same inference protocol.

A score change can arise from the artifact, prompt law, inference budget, parser, evaluator version, or execution failure. A matched-cell report names these coordinates before interpreting a paired gain.

Let \(\mathsf Q_{\rm tr}\) put masses \(.9,.1\) on \(a,b\), and let \(\mathsf Q_{\rm ev}\) swap them. Artifacts \(M_1,M_0\) with score vectors \((s_{M_1}(a),s_{M_1}(b))=(1,0)\) and \((s_{M_0}(a),s_{M_0}(b))=(0,1)\) rank oppositely under the two laws. A training score is not an evaluation transport certificate.

For any score \(s:\mathcal Z\to [0,1]\) and finite laws \(\mathsf Q,\mathsf Q'\), \(|\mathbb E_{\mathsf Q}s-\mathbb E_{\mathsf Q'}s|\leq \lVert \mathsf Q-\mathsf Q'\rVert _{\rm TV}\), where \(\lVert \mathsf Q-\mathsf Q'\rVert _{\rm TV}:=\frac 12\sum _{z\in \mathcal Z} |\mathsf Q(z)-\mathsf Q'(z)|\).

On \(\mathcal Z=\{a,b\}\), let \(s(a)=1,s(b)=0\), and let \(\mathsf Q(a)=1\), \(\mathsf Q'(a)=1-\eta \), and \(\mathsf Q'(b)=\eta \). The score difference and total variation distance are both \(\eta \).

The bounds above transport a declared score across a declared law. They do not identify why an artifact changed, establish support expansion, prove generalization, or order different evaluators whose outcome spaces differ.

A reproducible record is \((M,\mathsf Q,\mathsf I,b,\Lambda ,v,m, \widehat \Delta _m,c)\), listing the cells, sample count, paired estimate, and cost. Replaying this tuple reproduces the estimator target only when the recorded laws and versions remain available.

A matched evaluation requires execution gates, contamination checks, grader calibration, and an audit of the result before it is committed. These controls make the comparison auditable; they do not by themselves establish capability acquisition.

Fix a task instance, evaluator, success threshold, and a joint law for \(B\geq 1\) attempts. Let \(E_i\) be the event that attempt \(i\) succeeds. The discovery event by attempt \(B\) is

Write \(p_i=\Pr (E_i)\). Independence is an additional property of the declared joint law; it is not implied by repeated decoding, a shared model, or a common environment. In the independent and identically distributed case, write \(p_i=p\). When the attempt law is the one fixed in Definition 3.1, this \(p\) is the corresponding successful-support probability.

Assume the events \(E_1,\ldots ,E_B\) of Convention 4.1.1 are independent. Then

In particular, independent attempts with a common success probability \(p\) satisfy

This finite statement follows from the displayed hypotheses. The calculation in Example 3.14 is its ten-attempt numerical preview.

This is a finite corollary of Theorem 4.1.2, obtained from its displayed hypotheses.

Let \(0<\delta <1\). Under the independent and identically distributed setup of Theorem 4.1.2, suppose first that \(0<p<1\). The least positive integer budget whose discovery failure probability is at most \(\delta \) is

If \(p=0\), no finite positive budget reaches failure probability below one. If \(p=1\), one attempt suffices.

Put \(D_0=\varnothing \). Whenever \(\Pr (D_{i-1}^{\mathsf c})>0\), define the surviving conditional success rate

If these rates are defined through step \(B\), then

Consequently, if \(0\leq \underline q\leq q_i\leq \overline q\leq 1\) for every surviving step, then

No independence assumption is used. The conditional rates may change with the earlier failures, an adaptive decoder, or a changing environment state.

This finite statement follows from the displayed hypotheses.

The discovery event concerns whether a declared sampling procedure observes at least one success. Increasing its probability can be an elicitation effect in the sense of Definition [ftip-0005]: more attempts expose behaviour that already had positive probability. It is not by itself the independent before--after comparison required by Definition [ftip-0007].

Successful-support coverage in Definition 3.2 integrates an instance-level threshold over a task law. A large discovery probability on one instance therefore does not establish broad coverage or the uniform task-family witness of Definition 3.4.

Fix a prompt \(x\), a finite response set \(\mathcal Y\), a reference policy \(\pi _{\mathrm {ref}}\), a finite real reward \(r(x,y)\), and \(\beta >0\). Let \(\pi _r\) be the normalized exponential tilt of Definition [ftip-003O]. Then, for every \(y\in \mathcal Y\),

The source policy form appears as equation (4) in [rafailov2023direct, Section 4]; the displayed result is the finite corollary derived here. Infinite rewards, non-normalizable response spaces, approximate optimization, and changes to the generation mechanism lie outside the statement.

Support preservation is a statement about exact positive probability. A response can remain in mathematical support while its mass becomes too small to observe within the available rollout budget. The identities in Theorem 4.1.2 and Corollary 4.1.3 quantify this distinction for declared independent attempts.

Top-\(k\) truncation, nucleus sampling, finite numerical precision, context limits, and parser constraints can also remove an operational route even when the underlying softmax law assigns it positive mass. Claims about elicitation must therefore name both the policy law and the executable inference procedure.

Let the evaluation law be uniform on two instances \(x_1,x_2\). Consider protocols \(P\) and \(P'\) with successful-support probabilities

Both protocols have mean one-shot success \(1/2\). At threshold \(\alpha =1/2\), however, Definition 3.2 gives

Thus average success does not determine successful-support coverage. The example is finite and uses the same task law and threshold for both protocols; it does not compare their training costs or establish acquisition.

Let \(\mathcal Y_0\) be a nonempty finite set of candidate outcomes. Let \(U:\mathcal Y_0\to \mathbb R\) be an independently declared utility and \(\widehat U:\mathcal Y_0\to \mathbb R\) a proxy score. The proxy is a uniform approximation to \(U\) on \(\mathcal Y_0\) with error \(\varepsilon \) when \(\varepsilon \geq 0\) and

The candidate set is part of the claim. Approximation on training outcomes, on one policy's support, or in expectation is not uniform approximation on a larger set reached after optimization.

Under the conditions of Definition 4.2.1, choose

Then the utility regret of selecting by the proxy satisfies

This finite statement follows from the displayed hypotheses.

The bound in Theorem 4.2.2 follows directly from the uniform approximation condition. It compares two maximizers on one fixed finite candidate set. It does not describe how either score was learned or how a policy changes when that score becomes a training objective.

The result is useful because its transfer assumption is visible. To apply it after an update or a larger search, the uniform bound must hold on the outcomes that the new procedure can actually select. The verifier evidence of Definition [ftip-0047] can support such an audit, but reproducible evidence alone does not establish the bound.

Let the finite decision set be \(\mathcal Y_0=\{y_{\rm safe},y_{\rm exploit}\}\). Define its utility and proxy utility by the following table.

For \(0<\delta <1\), let a reference law \(P_\delta \) assign probability \(\delta \) to \(y_{\rm exploit}\). Its mean absolute proxy error is

which tends to zero with \(\delta \). Nevertheless, proxy maximization selects \(y_{\rm exploit}\) and incurs utility regret \(1\). Under the shifted law concentrated on that selected outcome, the mean error is \(2\). Thus no vanishing selection-regret bound can depend only on average error under the reference law.

This two-outcome construction isolates the distribution-shift warning in Remark 3.7. It does not claim that a particular learned verifier has this error profile; Example 3.8 gives a source-linked checker instance with the same proxy-versus-utility separation.

A repeated verifier audit of size \(N\geq 1\) records pairs \((S_i,A_i)\in \{0,1\}^2\) for candidates \(1\leq i\leq N\). Here \(S_i\) is an independently defined success label and \(A_i\) is the verifier decision, as in Definition 3.6. Define

The event \(\mathcal E_N^{\rm fa}\) says that at least one invalid candidate was accepted. It does not say which candidate a downstream selector returns. Write \(\phi _i=\Pr (C_i^{\rm fa})\). When \(\iota _i=\Pr (S_i=0)>0\), the conditional false-accept rate \(\eta _{+,i}=\Pr (A_i=1\mid S_i=0)\) is defined and \(\phi _i=\iota _i\eta _{+,i}\).

In the repeated audit of Definition 4.2.5, suppose the pairs \((S_i,A_i)\) are independent and identically distributed. Assume

Then the probability that at least one accepted invalid candidate exists among the \(N\) audited candidates is

It is strictly increasing in \(N\) when \(0<\iota \eta _+<1\), and it converges to \(1\) when \(\iota \eta _+>0\).

This finite statement follows from the displayed hypotheses.

For any joint law of a repeated verifier audit,

If \(\iota _i=\Pr (S_i=0)>0\) for every \(i\), this becomes

Without a dependence assumption, matching marginal error rates do not give the equality in Theorem 4.2.6; the events \(C_i^{\rm fa}\) could coincide.

This finite statement follows from the displayed hypotheses.

A proxy score can enter a training operator or a selection operator. In the first route it changes the policy; in the second it ranks a finite sample from a policy whose weights remain fixed.

The lifecycle accounting of Definition 1.5 places the first route's rollout and update work in separate coordinates. The second route incurs rollout and evaluation work but no parameter update. Consequently, the event \(\mathcal E_N^{\rm fa}\) in Definition 4.2.5 concerns the candidate set presented to selection; it is not a claim about the selected output or a training-time policy change.

The rates in Definition 3.6 are properties of a declared audit law. They do not identify where errors occur within the task family, how error events depend across repeated candidates, or which accepted candidate a selector returns. Theorem 4.2.6 adds identical marginals and independence; Theorem 4.2.7 retains only the union bound.

Nor do these rates determine the effect of optimizing against the verifier. That effect depends on how the induced policy or selection law moves mass toward particular outcomes. The incomplete-checker example Example 3.8 demonstrates one such movement, while Example 4.2.4 shows why an average reference-law error cannot rule it out. These are local finite statements. Any claim about a named verifier still requires its own audit distribution, evidence, and transfer argument.

The deterministic regret lemma Theorem 4.2.2, the independent-audit identity Theorem 4.2.6, and the union bound Theorem 4.2.7 follow from uniform approximation, independent trials, and subadditivity, respectively. Their hypotheses need not hold for a verifier chosen only for its empirical performance.

Fix a horizon \(N\geq 0\), finite action sets \(\mathcal A_n\), finite feedback sets \(\mathcal F_n\), and a finite internal-seed set \(\mathcal U\) with a declared law \(\lambda \). A finite declared protocol \(\mathsf P\) consists of selection maps

An action may specify the task, rollout record, feedback request, or sampling choice made at that round. The protocol can depend on earlier exposed feedback and its declared seed, but it cannot depend on a latent world field that has not entered those arguments.

For the action and feedback sets of Definition 4.3.1, a finite feedback world \(w\) supplies, for each round \(0\leq n<N\), a probability kernel \(K_{n,w}\) from a generic action--feedback history \((a_0,f_0,\ldots ,a_n)\) ending in \(a_n\in \mathcal A_n\) to \(\mathcal F_n\). Thus, whenever \(a_j\in \mathcal A_j\) and \(f_j\in \mathcal F_j\),

Two worlds may agree on every feedback kernel queried by a protocol while differing in latent utility, unrequested labels, or unobserved environment facts. Those latent fields are not feedback until a declared query exposes them.

Fix a finite declared protocol \(\mathsf P\) from Definition 4.3.1 and a finite feedback world \(w\) from Definition 4.3.2. Draw \(U\sim \lambda \) and, for \(0\leq n<N\), set

The resulting observed feedback transcript is

All protocol randomness is included in \(U\). Immutable rollout records and typed feedback events may be components of the finite action and feedback sets, as in Definition [ftip-0051] and Definition [ftip-0053].

Let \(\mathcal T_{\mathsf P}\) be the finite set of possible transcripts for a declared protocol \(\mathsf P\). Two feedback worlds \(w_0,w_1\) are observationally equivalent for \(\mathsf P\), written \(w_0\equiv _{\mathsf P}w_1\), when

The relation is protocol-relative. Another protocol may issue a different feedback request and thereby separate the same worlds. Equality only on the realized transcript is weaker than this definition, which compares the whole finite transcript law.

For a finite random variable \(X\), write \(\operatorname {Law}(X)\) for its probability law. If \(h\) maps the value space of \(X\) to a finite set \(\mathcal Y\), its pushforward law \(h_{\#}\operatorname {Law}(X)\) is characterized by \[ \bigl (h_{\#}\operatorname {Law}(X)\bigr )(\{y\}) =\Pr (h(X)=y),\qquad y\in \mathcal Y. \]

Let \(w_0\equiv _{\mathsf P}w_1\). For any finite output set \(\mathcal Y\) and any readout \(h:\mathcal T_{\mathsf P}\to \mathcal Y\),

Thus a final artifact stamp, update decision, or test chosen solely from the declared protocol's transcript has the same distribution in both worlds.

This finite statement follows from the displayed hypotheses.

Theorem 4.3.5 says that the declared observations supply no information that distinguishes \(w_0\) from \(w_1\). It does not say that the output must equal the initial model, that the parameters cannot change, or that performance cannot improve in both worlds. Pretraining, inductive bias, computation on the observed records, and generalization may still produce an improvement shared by the two worlds.

Calling the theorem a no-learning result would therefore erase the central condition: only world-dependent conclusions unavailable from the common transcript are ruled out.

Fix a realized finite replay pool \(d\) as in Definition [ftip-0056]. Let \(\mathcal V\) be a finite seed set, and let \(V\in \mathcal V\) be a replay-and-update seed with the same declared law in two feedback worlds and whose law does not depend on the world. If a replay-only procedure makes no new environment or feedback query, then its output has the form \(Y=h(d,V)\). The law of \(Y\) is the same in the two worlds.

This finite statement follows from the displayed hypotheses.

A replay schedule can reweight records, reduce optimization error on the fixed pool, or produce a different update proposal in the sense of Definition [ftip-006C]. Those are genuine computational effects. They do not add a label, preference, verifier result, or environment transition to the fixed evidence. Fresh optimizer randomness is likewise not feedback about the latent world.

This distinction prevents sample reuse from being counted as feedback acquisition. It does not imply that replay is useless; it isolates the source of any benefit as further computation on already acquired records.

Let the response set be \(\{a,b\}\). A one-round protocol queries only \(x_0\); both worlds return the preference \(a\succ b\). The protocol therefore has the same transcript law in both worlds. Fix a deterministic transcript-to-policy readout that chooses \(a\) also at an unqueried \(x_1\). The worlds agree on every observed field but reverse utility at \(x_1\):

The transcript laws are identical, so Theorem 4.3.5 applies, while the deterministic readout returns the same output in both worlds. That output is optimal in \(w_+\) and suboptimal in \(w_-\) on \(x_1\). The example does not show that generalization always fails. It shows that a claim about unqueried utility requires an assumption connecting observed feedback to that utility.

[zhao2025limits, Theorems 3.3--3.5] study a more structured post-training model. They give ordinal-preference distortion lower bounds, including a lower bound under Bradley--Terry noise with linear scores, and a positive result using a limited number of cardinal queries. These results depend on the routing model, utility class, and query budget developed in § 5.2.

Theorem 4.3.5 gives a finite pushforward theorem, while Example 4.3.9 gives a counterexample to identification from an incomplete transcript. They are neither proofs nor special cases of the paper's distortion results.

Fix a sampled history and an active token \(i\) in the interior of the unclipped branch of the GRPO objective. Consider a finite directed acyclic computation graph that is differentiable at the parameter point in question. Let \(a_i\) be the sampled token, \(\widehat A_i\) its fixed normalized advantage, and \(p_i=\operatorname {softmax}(z_i)\) the current distribution on a finite vocabulary \(\mathcal V\). With fixed behavior probability \(b_i>0\), set \[ r_i=\frac {p_i(a_i)}{b_i}, \qquad \mathcal L_i=r_i\widehat A_i, \qquad E_i=r_i\widehat A_i(e_{a_i}-p_i). \] Here \(e_{a_i}\in \mathbb R^{|\mathcal V|}\) is the sampled-token basis vector. The history and sampling decisions are held fixed during differentiation.

Suppose the output head \(W_{\rm lm}\in \mathbb R^{|\mathcal V|\times d_{\rm model}}\) is untied: its only path to \(\mathcal L_i\) is through \(z_i=W_{\rm lm}h_{L,i}\), and \(h_{L,i}\in \mathbb R^{d_{\rm model}}\) is independent of \(W_{\rm lm}\). Then \[ G_i^{\rm lm}=\nabla _{W_{\rm lm}}\mathcal L_i =E_i h_{L,i}^{\mathsf T}. \]

Let \(W_{\rm int}\in \mathbb R^{m\times d}\) be a shared intermediate weight. Index by a finite set \(\mathcal O_i\) every occurrence of this weight that can affect \(z_i\), assuming that all such uses have the form \(y_o=W_{\rm int}x_o\), with \(x_o\in \mathbb R^d\). First replace these uses by independent copies \(W_o\) and evaluate them all at \(W_o=W_{\rm int}\). Let \(J_{io}\in \mathbb R^{|\mathcal V|\times m}\) be the downstream Jacobian from node \(y_o\) to \(z_i\) in this graph, with the other weight copies fixed. Define the contribution of occurrence \(o\) by \[ H_{io}=\nabla _{W_o}\mathcal L_i =(J_{io}^{\mathsf T}E_i)x_o^{\mathsf T}. \] On tying the copies, the total derivative is \[ G_i^{\rm int}=\nabla _{W_{\rm int}}\mathcal L_i =\sum _{o\in \mathcal O_i}H_{io}. \] The head gradient and each occurrence contribution have rank at most one; the total shared-weight gradient need not.

For a layer used once per position in a causal network, the sum includes every earlier position whose output affects token \(i\). Reuse across depth adds further occurrences. Tying the head to embeddings or other blocks also requires their contributions; the displayed head identity assumes that such tying is absent.

Clipped or inactive terms require their own derivative or mask. [miao2026when, Section 4.2.1, Proposition 1, and Appendix A.1] derives the intermediate outer product through one local application. For a shared weight, that calculation gives \(H_{io}\); identifying it with the total \(G_i^{\rm int}\) omits the other occurrences. The sum above supplies the chain rule needed for shared parameters.

Use the per-token quantities and untied head of Theorem 4.4.1, and fix one occurrence \(o\in \mathcal O_i\). Assume \(d_{\rm model}\geq 1\) and positive constants \(\alpha _{\min }\), \(\beta _{\max }\), and \(C\) satisfy \[ \|h_{L,i}\|_2^2\geq \alpha _{\min }d_{\rm model}, \qquad \|x_o\|_2^2\leq \beta _{\max }d_{\rm model}, \] and the two logit-sensitivity bounds \[ \mathbb E_{a\sim p_i} \|(J_{io})_{a,:}\|_2^2\leq C, \qquad \|(J_{io})_{a_i,:}\|_2^2\leq C. \] If \(\widehat A_i\neq 0\) and \(p_i(a_i)<1\), then \[ \frac {\|H_{io}\|_F^2}{\|G_i^{\rm lm}\|_F^2} \leq \frac {\mathcal C_{\rm occ}} {(1-p_i(a_i))^2}, \qquad \mathcal C_{\rm occ} =\frac {4\beta _{\max }C}{\alpha _{\min }}. \]

This bounds one occurrence contribution, not the total derivative of a shared intermediate weight. For the latter, the sum in Theorem 4.4.1 gives only \[ \|G_i^{\rm int}\|_F \leq \sum _{o\in \mathcal O_i} \|J_{io}^{\mathsf T}E_i\|_2\|x_o\|_2. \] If the same displayed hypotheses hold for every occurrence, the triangle inequality yields the ratio bound \(|\mathcal O_i|^2\mathcal C_{\rm occ}/(1-p_i(a_i))^2\) for \(\|G_i^{\rm int}\|_F^2/\|G_i^{\rm lm}\|_F^2\). Controlling only the application at position \(i\) does not establish this all-occurrence hypothesis or the source's claimed bound for the total gradient. The contributing positions, reuse pattern, and Jacobians depend on the architecture. Neither bound establishes a small gradient or an evaluation-score comparison.

The activation and row-energy hypotheses and the local inequality follow the calculation in [miao2026when, Section 4.2.1, Lemma 1, Assumption 1, Theorem 1, and Appendix A.3], with the differentiated object restricted to an occurrence contribution. They do not prove the source's total shared-weight claim under its local hypotheses.

For \(T\geq 1\) active, unclipped tokens with fixed sampled histories and the untied output head of Theorem 4.4.1, let \[ G^{\rm lm}=\frac 1T\sum _{i=1}^T G_i^{\rm lm}, \qquad \overline {r^2}=\frac 1T\sum _{i=1}^T r_i^2, \] and set \[ c_{\max }= \max _{1\leq i\leq T} \widehat A_i^2 \|e_{a_i}-\pi _\theta (\mathord \cdot \mid h_{L,i})\|_2^2 \|h_{L,i}\|_2^2. \] Then \[ \|G^{\rm lm}\|_F^2\leq c_{\max }\overline {r^2} =c_{\max }(1+\widehat \chi ^2), \] where \(\widehat \chi ^2\) is the statistic of Definition 3.13. If \(c_{\max }>0\), rearrangement gives the source's equivalent direction \[ \widehat \chi ^2\geq \frac {\|G^{\rm lm}\|_F^2}{c_{\max }}-1. \]

The inequality points from observed head-gradient energy to a lower bound on this finite-batch squared-ratio statistic. It is not a converse: cancellation can hide nonunit ratios. The active cancellation example in Example 4.4.6 states its clipping interval explicitly. The proof uses no intermediate-weight bound.

The finite-batch inequality follows the calculation in [miao2026when, Section 4.2.2, Lemma 2, Theorem 2, and Appendix B], under the untied-head and fixed-history assumptions stated above.

The identities in Theorem 4.4.1--Theorem 4.4.3 concern gradients, importance ratios, activations, and occurrence-specific logit Jacobians. The intermediate bound controls a local contribution; a shared-weight bound needs control over all contributing occurrences. None of their hypotheses mentions the fixed independent-evaluation functional \(J_{\rm ev}\) of Definition 1.2. They therefore cannot imply that accepting an update preserves \(J_{\rm ev}\), or that rejecting one would have prevented a decrease.

DGG adds an empirical policy on top of those identities: it monitors the increment in head-gradient energy, standardizes that increment against a trailing window, and rejects some reused updates before the optimizer step [miao2026when, Section 5 and Algorithm 1]. Its reported experiments relate that policy to observed training stability. They do not turn the Z-score into a calibrated test of independent-evaluation safety.

Take one observed action \(a\) with \(\pi _{\rm old}(a)=1/2\) and \(\pi _\theta (a)=1/4\). The batch contains only that action, so \(T=1\) and \(r_1=1/2\). Hence \[ \widehat \chi ^2=r_1^2-1=-\frac 34. \] Both policies can be completed on a two-action space by assigning their remaining mass to the other action.

Thus the finite-batch statistic in Definition 3.13 need not share the nonnegativity of the population Pearson divergence. Theorem Theorem 4.4.3 remains valid: its right side contains \(1+\widehat \chi ^2=1/4=\overline {r^2}\).

Fix \(\epsilon _{\rm clip}\in (0,1)\) and \(1<R<1+\epsilon _{\rm clip}\). On a two-token vocabulary, let both observed tokens have \(a_i=1\), current probability \(\pi _\theta (1)=1/2\), and behavior probability \(\pi _{\rm old}(1)=1/(2R)\). Take scalar hidden states \(h_{L,1}=h_{L,2}=1\) and fixed normalized advantages \(\widehat A_1=1\), \(\widehat A_2=-1\). These may be selected token terms from different rollout members. An untied head with both logits zero realizes the current probabilities. Both terms lie strictly inside the clipping interval of Definition [ftip-003F], so the clipped surrogate locally agrees with \(\mathcal L_i=r_i\widehat A_i\). Thus \(r_1=r_2=R\), and Theorem 4.4.1 gives \[ G_1^{\rm lm}=R(1/2,-1/2)^{\mathsf T}, \qquad G_2^{\rm lm}=-G_1^{\rm lm}. \] Consequently \(G^{\rm lm}=0\), while \(\widehat \chi ^2=R^2-1>0\). For this fixed clipping rule, the statistic in this construction is bounded above by \((1+\epsilon _{\rm clip})^2-1\).

For the raw surrogate \(\mathcal L_i=r_i\widehat A_i\) considered as a separate objective, the same algebra works at every \(R>1\) and yields arbitrarily large squared-ratio excess. This does not extend the active construction to arbitrary \(R\) under clipping: when \(R>1+\epsilon _{\rm clip}\), the positive-advantage term is locally constant, while the negative term remains active. Their clipped gradients are then \(0\) and \(-R(1/2,-1/2)^{\mathsf T}\), with nonzero mean \((-R/4,R/4)^{\mathsf T}\).

The example does not contradict Theorem 4.4.3: that theorem upper-bounds batch-gradient energy by a ratio moment. It supplies no lower bound on the gradient and no converse from zero batch-gradient energy to ratios equal to one. The arbitrary-ratio version concerns only the separate raw surrogate.

Let two consecutive proposed reused updates have the same monitored energy, \(g_{t-1}=g_t=1\). With trailing-window increment mean \(\mu _t=0\), any positive scale \(\sigma _t+\varepsilon \), and any positive threshold, DGG's score is \[ z_t=\frac {(g_t-g_{t-1})-\mu _t}{\sigma _t+\varepsilon }=0, \] so the gate accepts the proposal. Now choose a fixed evaluation interface for which the artifact before the accepted update has performance \(J_{\rm ev}=1\) and the artifact after it has performance \(J_{\rm ev}=0\).

This is consistent because the gate rule of Definition [ftip-005A] imposes no mathematical relation between its scalar monitor and \(J_{\rm ev}\). A safety claim would need an additional assumption connecting proposed updates to the independent evaluation law; the Z-score calculation alone cannot provide it.

Let \(\Delta W\in \mathbb R^{m\times n}\) be nonzero, set \(q=\min (m,n)\), and order its singular values as \(\sigma _1\geq \cdots \geq \sigma _q\geq 0\). Its leading squared singular-energy share is \[ \rho _1(\Delta W) =\frac {\sigma _1(\Delta W)^2}{\sum _{j=1}^q\sigma _j(\Delta W)^2} =\frac {\sigma _1(\Delta W)^2}{\|\Delta W\|_F^2}. \] The nonzero hypothesis prevents an undefined \(0/0\); the value lies in \([1/q,1]\).

NExt instead reports \(E_1=\sigma _1/\sum _j\sigma _j\) in Section 3.2 Low-rank optimization trajectories modeling for LLM RLVR acceleration[chen2026lowrank]. That nuclear-share statistic and \(\rho _1\) answer different questions. The squared share measures contribution to squared Frobenius norm.

For \(\Delta W\in \mathbb R^{m\times n}\) with \(q=\min (m,n)\geq 2\), its leading spectral gap is \[ \operatorname {gap}_1(\Delta W) =\sigma _1(\Delta W)-\sigma _2(\Delta W). \] A positive gap makes the leading left and right one-dimensional singular subspaces unique, although each chosen singular vector still has an arbitrary sign. When the gap is zero, a single leading vector is not intrinsic.

The quantity is undefined when \(q=1\), since there is no second singular value. Projector comparisons require a positive gap at every compared checkpoint.

Let \(\Delta W_t\) and \(\Delta W_s\) be nonzero matrices of the same shape, each with positive leading spectral gap. If \(u_1(t)\) and \(u_1(s)\) are unit leading left singular vectors, define their leading-subspace projector drift by \[ d_{\rm proj}(t,s) =\|\Pi _t^{\rm svd}-\Pi _s^{\rm svd}\|_F, \qquad \Pi _t^{\rm svd}=u_1(t)u_1(t)^{\mathsf T}, \quad \Pi _s^{\rm svd}=u_1(s)u_1(s)^{\mathsf T}. \] The definition is independent of both singular-vector signs and takes values in \([0,\sqrt {2}]\).

This is an additional proposed trajectory diagnostic; NExt does not define it.

A large value of \(\rho _1\) from Definition 4.4.8 says that one singular mode dominates a particular difference matrix. It does not say that the dominant subspace remains fixed across checkpoints; \(d_{\rm proj}\) measures that separate question.

Fix the evaluation interface of Convention 1.1. Let \(P_{t+s}^{\rm act}\) be the protocol that returns the artifact reached after \(s\geq 1\) additional realized training steps from checkpoint \(t\). Let \(\mathcal K_{\leq t}\) be the saved checkpoint history available through \(t\), and let \(\widehat P_{t+s}(\mathcal K_{\leq t})\) instead return the artifact forecast from that history. Their signed path-emulation regret and absolute path-emulation regret are \[ \begin {aligned} \operatorname {Reg}^{\rm path}_{\rm ev}(t,s) &=J_{\rm ev}(P_{t+s}^{\rm act}) -J_{\rm ev}(\widehat P_{t+s}(\mathcal K_{\leq t})),\\ \operatorname {AReg}^{\rm path}_{\rm ev}(t,s) &=\left |\operatorname {Reg}^{\rm path}_{\rm ev}(t,s)\right |. \end {aligned} \]

The sign distinguishes an optimistic forecast from a pessimistic one; the absolute value measures discrepancy without cancellation across checkpoints. Both use the same task law, inference budget, and evaluator. Parameter error alone is not substituted for evaluation error. The NExt extrapolation and recovery schedule described in Sections 4.1--4.3 and 5.1 motivates this comparison Low-rank optimization trajectories modeling for LLM RLVR acceleration[chen2026lowrank], but the paper does not state this regret definition.

Let \(e_1=(1,0)^{\mathsf T}\) and \(e_2=(0,1)^{\mathsf T}\) be the standard basis vectors of \(\mathbb R^2\). In a two-dimensional matrix chart, set \(P=e_1e_1^{\mathsf T}\) and \(Q=e_2e_2^{\mathsf T}\). Two paths share the entire observed history \[ W_0=0,\qquad W_1=P,\qquad W_2=2P. \] Every observed local difference is the same rank-one matrix \(P\). The paths then fork: path A takes \(W_3^A=3P\), whereas path B takes \(W_3^B=2P+Q\). Each next local difference is again rank one.

A deterministic history-only forecaster must return the same matrix \(\widehat W_3\) in both worlds. Since \(\|W_3^A-W_3^B\|_F=\|P-Q\|_F=\sqrt {2}\), the triangle inequality forces its Frobenius error to be at least \(1/\sqrt {2}\) on one continuation. Low-rank early motion therefore does not identify the next subspace or the next checkpoint.

NExt extracts global, local, and target differences from LoRA checkpoint matrices, keeps leading singular factors, learns a nonlinear predictor, jumps in that parameter chart, and resumes RLVR [chen2026lowrank, Sections 3.2, 4.1--4.3, and 5.1]. Its reported results are empirical; the paper states no theorem that a leading subspace remains stable or that a forecast preserves a policy.

The quantities in Definition 4.4.8--Definition 4.4.11 separate four tests. Squared singular-energy share concerns one matrix, spectral gap controls whether its leading subspace is identifiable, projector drift compares those subspaces over time, and path-emulation regret returns to fixed independent evaluation. None, by itself, is evidence that new feedback was acquired or that a new capability was learned. Such a claim must use a declared post-training gain such as Definition 1.3 and must charge forecast training, the parameter jump, and recovery updates.

Theorems 1 and 2 of the v1 preprint [paes2026theoretical] concern an exactly optimal KL-regularized policy and a fixed query. On a finite response set, their expectations and normalizers are ordinary finite sums.

These exact identities do not establish a comparison among best-of-\(N\), PPO, and GRPO, or a general guarantee for proxy rewards and reward ensembles. Those questions require assumptions beyond exact optimization of one reward.

The source writes \(\lambda >0\) for the KL penalty. Here we write \(\beta >0\), matching the DPO notation in Notation [ftip-003M]--Definition [ftip-003O]. This avoids collision with the generalized-advantage parameter in Definition [ftip-003E] and the protocol law in Definition 4.3.1.

The source's gain \(\Delta (r,r')\) is written \(G_p(r;s)\): \(r\) is the reward used to tilt the reference law \(p\), while \(s\) is the reward used to evaluate the tilted law. This avoids collision with both the simplex notation \(\Delta (X)\) of Notation [ftip-000A] and the paired DPO margin of Notation [ftip-003M]. For a finite set \(\mathcal Y\), a law \(p\in \Delta (\mathcal Y)\), and functions \(f,g:\mathcal Y\to \mathbb R\), set

A finite KL-alignment instance consists of a nonempty finite response set \(\mathcal Y\), a reference law \(p\in \Delta (\mathcal Y)\), a penalty \(\beta >0\), and finite real functions \(r,s:\mathcal Y\to \mathbb R\). The reference support is

A prompt is fixed and suppressed. The function \(r\) determines the alignment update; \(s\) measures its result. They may coincide, but they need not. Finiteness makes every exponential weight and expectation below finite.

These objects specialize the fixed-query setting in [paes2026theoretical, Section 2, equations (2.1)--(2.2)]; they are not a definition of alignment in general.

For a finite KL-alignment instance, define the exponential weight and partition function by

Every \(a_r(y)\) is finite and strictly positive. Since \(p\) is a probability law on a nonempty finite set, \(0<Z_r<\infty \).

This weight and normalizer are the fixed-query form of [paes2026theoretical, Equation (2.2)].

The aligned law induced by \(r\) is \(q_r\in \Delta (\mathcal Y)\) given by

Normalization follows from the definition of \(Z_r\). Moreover, \(q_r(y)>0\) exactly when \(p(y)>0\). This is the finite notation for the same exponential-tilt optimizer stated in Definition [ftip-003O]; support preservation was proved in Theorem 4.1.6.

This formula is [paes2026theoretical, Equation (2.2)]. It assumes the exact optimizer, not an iterate returned by a particular training algorithm.

The cross-reward gain from aligning with \(r\) and evaluating with \(s\) is

The semicolon records two roles. Its left argument changes the response law; its right argument scores both laws. Thus \(G_p(r;s)\) is not an independent evaluation unless \(s\) has separately been declared to serve that role.

This is the finite notation for \(\Delta (r,r')\) in Theorem 2, equation (3.4), of Theoretical limits of language model alignment[paes2026theoretical].

For probability laws \(p\) and \(q\) for which both terms are finite, their Jeffreys divergence is

The KL divergence and its argument order are defined in Definition [ftip-006X]. Unlike either directed term, \(J(p,q)=J(q,p)\). In the finite tilt of Definition 5.1.5, \(p\) and \(q_r\) have the same support, so both terms are finite.

See [paes2026theoretical, Theorem 1, equation (3.2)].

For every \(y\in S_p\), the aligned law satisfies

This is equation (B.1) followed by the logarithmic step in [paes2026theoretical, Appendix B.1, equations (B.1)--(B.2)].

For a finite KL-alignment instance,

This is the finite form of [paes2026theoretical, Theorem 1, equation (3.2), with proof in Appendix B.1].

The identity Theorem 5.1.9 is an equality inside one declared model. It says that exact gain in the optimizing reward equals a symmetric divergence from the reference law. It does not say that a larger divergence improves a different utility, nor that a training algorithm reaches the exact tilt.

The identity accounts for neither rollout and update work nor the cost of obtaining \(r\). Consequently it is not, by itself, a bound on the costed post-training potential of Definition 1.10.

For a finite KL-alignment instance,

This is the finite form of Theorem 2, equation (3.4), in Theoretical limits of language model alignment[paes2026theoretical]. Its proof is in Appendix B.1, equations (B.5)--(B.7).

The identity Theorem 5.1.11 expresses exact tilted-law gain using expectations under \(p\). This makes the population quantity accessible from the reference law in principle. A finite-sample estimator still needs its own sampling law, moment assumptions, and error analysis.

The formula also remains indexed by both \(r\) and \(s\). It cannot turn a proxy reward into an independent utility. If \(s=r\), it measures improvement in the same quantity that defined the exact optimizer; if \(s\ne r\), its sign is determined by the displayed covariance.

Take \(\mathcal Y=\{a,b\}\), \(p=(1/2,1/2)\), \(\beta =1\), and \(r(a)=0\), \(r(b)=\log 3\). Exponential weighting changes the reference law as follows.

The reward gain is

Direct calculation gives

whose sum is \(\frac 14\log 3\). Thus the example checks Theorem 5.1.9 exactly; it is not an empirical alignment result.

Let \(\mathcal X\) be finite with prompt law \(\mu \). For each \(x\in \mathcal X\), let \(p_x\), \(r_x\), and \(q_{r_x}\) be a finite KL-alignment instance with the same penalty \(\beta >0\). Then

The source theorem is stated for each fixed query. This corollary performs only finite averaging; it does not introduce a shared neural parameterization across prompts.

For constants \(c,d\in \mathbb R\),

This finite lemma records the same prompt-dependent additive non-identifiability that appears in the DPO reparameterization of Definition [ftip-003P].

For fixed \(\beta \), the exponential tilt solves the penalized problem

under the conventions of Definition 5.1.3. This is not the same specification as choosing a number \(\kappa \) and solving

A Lagrange multiplier can relate the two problems when the relevant duality and activity conditions hold. The coefficient \(\beta \) alone does not declare a hard budget, and the identities Theorem 5.1.9--Theorem 5.1.11 do not supply those conditions.

The source setup writes rewards as \(r(\mathbf x,\mathbf y)\), while the display of Theorem 1 reverses the two arguments in places. This subsection uses the setup order and then suppresses the fixed prompt. The source also moves between a dataset-level penalized objective and fixed-query identities. The finite-averaging result Corollary 5.1.14 makes that step explicit.

The identities specialize Theoretical limits of language model alignment[paes2026theoretical] to a finite response set. They require no extension to the full sequence space.

Finally, exact exponential tilting is a distributional optimizer. It does not account for rollout, gradient, optimizer, or systems cost, and it does not show that PPO, GRPO, DPO, or any frontier training run attains the displayed law. Applying the identities to a training run therefore requires a separate argument that its output law is the exact optimizer.

The routing model and preference limits below are based on the v1 preprint The limits of preference data for post-training[zhao2025limits]. It models a pretrained system as a finite collection of response circuits plus learned maps that route queries to those circuits. Post-training changes the routing maps while retaining the circuit collection.

The noiseless lower bound is proved here under explicit pointwise response separation and a deterministic preference-only learner, whose returned router may be stochastic. Its finite partition estimate and cardinal-utility construction give a restricted reconstruction of the source rate; the proof does not establish a randomized-learner extension.

The routing model is a source assumption, not an architectural theorem about language models. In particular, its lower bounds do not prove that real post-training creates no circuits, that a neural network decomposes into the displayed objects, or that every preference-learning algorithm obeys the same bound outside this model.

The finite preference model uses the following symbols, corresponding to the notation of The limits of preference data for post-training[zhao2025limits]:

Thus \(H\) denotes the source's finite internal-representation set, not a public interaction history. The family \(\mathfrak E\) contains admissible query encoders and is unrelated to the costed potential \(\Phi \) of Definition 1.10.

Following the formal model in Section 2 of The limits of preference data for post-training[zhao2025limits], let \(Q\) be a finite nonempty query set and \(Y\) a response set. A response circuit is a function \(c:Q\to Y\). The finite nonempty set \(C_0\) contains the circuits retained from the pretrained model.

The word ``circuit'' is source terminology for this abstract function. No claim is made here that an element of \(C_0\) is localized in a neural network or corresponds to one human-named capability.

Let \(H\) be a finite nonempty representation set and \(\mathfrak E\subseteq H^Q\) a finite nonempty family of admissible encoders. A query encoder is \(e\in \mathfrak E\). A circuit router is a map

For a query \(\xi \in Q\), the composition \(h(e(\xi ))\) is a law on the retained circuits. Sampling \(c\sim h(e(\xi ))\) and returning \(c(\xi )\) gives the response law. These are the two routing components in the source model [zhao2025limits, Section 2, ``Formal model''].

A routing model is a triple \(\mathfrak m=(e,h,C_0)\) with the types fixed in Definition 5.2.3--Definition 5.2.4. The pretrained model is \(\mathfrak m_0=(e_0,h_0,C_0)\). In the source intervention, a post-trained model may replace \(e_0\) and \(h_0\), but it retains exactly \(C_0\).

Write \(\mathfrak m(\xi )\) for the random response obtained by sampling \(c\sim h(e(\xi ))\) and returning \(c(\xi )\). The source sometimes writes this as though the router selected a circuit rather than a distribution; the stochastic reading follows its declared codomain \(\Delta (C_0)\).

The source fixes the uniform probability law \(\mu _Q\) on \(Q\) and a utility function

For a routing model \(\mathfrak m\), its expected utility is

This is the outcome objective in [zhao2025limits, Section 2, ``Outcome-based optimization with preference data'']. It is an assumed cardinal utility, not something ordinal comparisons directly reveal.

Assume that distinct retained circuits never tie at a query. For each \(\xi \in Q\), utility then induces a strict comparison by

The complete noiseless ordinal profile is \(\succ _u=(\succ _{\xi ,u})_{\xi \in Q}\). The source gives the learner unlimited, unbiased access to every such comparison [zhao2025limits, Section 2, final paragraph]. Utilities with ties require a separately declared deterministic tie-break; the lower bounds below use the source's tie-free constructions.

Within this finite model, a table containing every comparison in \(\succ _u\) and a noiseless online oracle that answers every possible query expose the same ordinal information. The source deliberately grants this ideal access so that its obstruction is not caused by finite sampling or an offline split [zhao2025limits, Section 2].

This equivalence does not price the number of queries, labeler time, or adaptivity. It therefore cannot be transferred to a finite-feedback protocol without a separate query-complexity statement.

A preference-only post-training algorithm \(\mathcal A\) maps the pretrained routing model and the complete profile to

The deterministic comparator class displayed in the source theorems is

The source's formal model initially permits stochastic routers \(h:H\to \Delta (C_0)\), but its theorem comparator uses deterministic \(h:H\to C_0\). The comparator class is therefore the displayed deterministic class.

Let \(\mathcal B\) be any post-training algorithm with output \(\mathfrak m_{\mathcal B}(u)\) under the feedback law induced by \(u\). When the denominator is positive, define its multiplicative post-training distortion by

If the numerator is positive and the denominator is zero, set the ratio to \(+\infty \). The source lower bounds construct bounded nonnegative utilities for which the comparison is well defined [zhao2025limits, Equation (2) and Appendix A]. Distortion measures loss relative to the declared comparator class; it is not an absolute capability score.

Let \(m=|C_0|\), and suppose each \(\succ _{\xi ,u}\) is a strict total order on \(C_0\). If \(\operatorname {rank}_{\succ _{\xi ,u}}(c)\) places the most preferred circuit at rank one, its Borda score is

The Borda-count choice is any circuit in

This is [zhao2025limits, Definition 3.1]. The source cites a prior equivalence between a standard RLHF model and Borda count; the definition here does not assert that every RLHF implementation is a Borda rule.

Example 3.2 of The limits of preference data for post-training[zhao2025limits] prints only \(a\geq 0\) and \(2a<b\), but those conditions do not force its stated rankings or Borda totals. Sufficient conditions are \(0<a<1/2\) and \(2a<b<1\). Take three queries, three circuits, and a singleton representation set, so every query must use one common circuit. Set

These inequalities give the rankings \(c_A\succ c_C\succ c_B\) for \(\xi _1,\xi _2\) and \(c_B\succ c_C\succ c_A\) for \(\xi _3\). Hence the Borda totals are \((4,2,3)\), so the ordinal rule selects \(c_A\). The utility totals are \((2,1,2-2a+b)\). Since \(2a<b\), circuit \(c_C\) has strictly greater total utility than \(c_A\).

This finite example isolates information lost by ranking. The compromise circuit is never first for an individual query, yet its aggregate utility is largest. It does not show that Borda is always suboptimal or that a neural model must collapse all queries to one representation.

Let \(Q,H\) be nonempty finite sets, let \(\varnothing \neq \mathfrak E\subseteq H^Q\) be finite, and let \(C_0=\{c_1,\ldots ,c_m\}\) be a nonempty finite set of deterministic maps \(Q\to Y\). Write \(N=|Q|\), \(r=|H|\), \(p=|\mathfrak E|\) and \(m=|C_0|\). The query law is uniform on \(Q\). Call \(C_0\) pointwise response-separating when

Assume this separation. Let \(\mathcal A\) be a deterministic preference-only learner: from a pretrained model \(\mathfrak m_0\) and the complete strict ordinal profile, it returns \((e,h,C_0)\) with \(e\in \mathfrak E\) and a possibly stochastic router \(h:H\to \Delta (C_0)\). The comparator class is exactly \(\mathcal M_{\rm det}(C_0)\) of Definition 5.2.9, including every map \(H\to C_0\) for each \(e\in \mathfrak E\).

For every such pretrained model and learner there exists \(u:Q\times Y\to [0,1]\), inducing a strict profile, such that

Logarithms are natural. Utility averages over the uniform query and the learner's router draw as in Definition 5.2.6. The comparator maximum exists because its class is finite and nonempty. It is positive for the normalized utilities above: some constant-circuit route has mean at least \(1/m\). A zero learner utility therefore gives distortion \(+\infty \), with no \(0/0\) case.

Appendix Theorem A.1 of The limits of preference data for post-training[zhao2025limits] states the corresponding rate for its routing model. Its displayed balancing parameter retains \(\log N\) factors that are absent from its final rate; the second-moment partition bound in Lemma 5.2.15 supplies a separate argument for the stated restricted model. Its index-based perturbation on queries outside the selected group need not preserve their prescribed favorite; the query-dependent rank weights used here preserve it. When circuits collide at a query, a circuitwise prescription need not define a response utility; pointwise separation excludes that issue.

The utility is a worst-case existential choice for each fixed learner. The proof permits stochastic inference, but it does not cover internal learner randomness: a utility chosen after a realized random output need not be one utility valid before the learner's random draw. No claim about response-colliding circuits or the separate noisy-preference bound follows.

Under all hypotheses of Theorem 5.2.13, including a deterministic preference-only learner and pointwise response separation, assume

For every pretrained model and every such learner, the utility supplied by that theorem satisfies

Theorem 3.3 of The limits of preference data for post-training[zhao2025limits] gives a square-root rate in its large-query regime. The explicit condition above gives a finite regime for the reconstructed theorem with its stated learner and response assumptions.

Let \(Q,H\) be nonempty finite sets with \(N=|Q|\) and \(r=|H|\), and let \(\mathfrak E\subseteq H^Q\) be nonempty and finite with \(p=|\mathfrak E|\). For every integer \(k\geq 1\), there is a map \(f:Q\to [k]\) such that, simultaneously for every \(e\in \mathfrak E\),

where \([k]=\{1,\ldots ,k\}\), logarithms are natural, and

The bound uses a second-moment estimate in place of the occupancy estimate in source Lemma A.2. The chosen labeling depends only on \(Q,H,\mathfrak E,k\); its existence uses proof randomness and does not assume randomness in the learning algorithm.

Fix a query \(\xi \) and nonnegative circuit scores \(a_{\xi ,c}\geq 0\) such that \(a_{\xi ,c_i}+a_{\xi ,c_j}>0\) for every compared pair. The Bradley--Terry comparison law is

Only score ratios are identified: multiplying every score for a fixed query by the same positive constant leaves all comparison probabilities unchanged. The linear and exponential score links discussed in [zhao2025limits, Section 3.2] therefore impose different assumptions on the relation between rewards and comparisons.

The exponential-score link sets

For this link, exact comparison probabilities reveal pairwise utility differences through

This algebra explains why a known link and infinite noiseless frequency information can expose more than an ordinal order. It does not apply when the link is unknown or misspecified.

The linear-score link sets

Write \(p^u_{\xi ,ij}\) for the displayed probability and define the complete pairwise-probability profile

This requires nonnegative utilities and a positive denominator for each queried pair. A linear-score algorithm \(\mathcal A_{\rm lin}\) takes \((\mathfrak m_0,P_u^{\rm lin})\) as input and returns \(\mathfrak m_{\mathcal A_{\rm lin}}(u)\). The following lower bound concerns this specific link; it is not a lower bound for every stochastic preference model.

Assume pointwise response separation as defined in Theorem 5.2.13. For every pretrained routing model and linear-score algorithm \(\mathcal A_{\rm lin}\), there exists a utility

Post-training from even the complete comparison-probability profile of the linear-score link then satisfies

where \(R_{Q,\mathfrak E,H}\) is defined in Theorem 5.2.13. The theorem is again existential in \(u\) and uses the source's deterministic comparator. The same circuitwise-utility issue recorded in Theorem 5.2.13 prevents us from asserting the source's unrestricted pretrained-model quantifier here. Its noise is informative because probabilities depend on cardinal scores; the obstruction survives under the particular linear link.

This lower bound is the form of Appendix Theorem A.5 in The limits of preference data for post-training[zhao2025limits] with the additional pointwise response-separation assumption stated above.

The main-text Theorem 3.4 of The limits of preference data for post-training[zhao2025limits] displays \(\Omega (|C_0|)\) in its stated large-query regime. Appendix Theorem A.5 displays the general bound with \(\widetilde \Omega \) and the minimum in Theorem 5.2.19. The tilde can hide logarithmic factors, so these statements are not textually identical.

The general bound in Theorem 5.2.19 has the appendix's rate with a possible logarithmic loss. Eliminating that loss to obtain the main-text rate requires an additional argument.

In the social-choice source, agents rank alternatives and also have nonnegative cardinal values. A value query takes an agent \(i\) and an alternative \(j\) and returns \(v_{ij}\) [amanatidis2021peeking, Definition 1].

Under the routing analogy, a query \(\xi \in Q\) plays the role of an agent and a circuit \(c\in C_0\) plays the role of an alternative. A cardinal query therefore reveals one value \(u(\xi ,c(\xi ))\). Query counts inherited from the social-choice theorem are per query/agent, not totals across \(Q\).

Let \(m\) be the number of alternatives and let \(k\in \{1,\ldots ,m\}\). Theorem 4 of Peeking behind the ordinal curtain: Improving distortion via cardinal queries[amanatidis2021peeking] states that \(k\)-Acceptable Range Voting uses

value queries per agent and has distortion

This is a social-choice mechanism theorem. It transfers directly to the singleton-representation specialization \(|H|=1\), where every query must use one common circuit. Under that restriction, queries are agents and circuits are alternatives as in Definition 5.2.21. The theorem does not by itself cover a routing comparator that may choose different circuits for different representations, and it is not an implementation of neural post-training.

This statement is Theorem 4 of Peeking behind the ordinal curtain: Improving distortion via cardinal queries[amanatidis2021peeking].

Taking \(k\) proportional to \(\log m\) in Theorem 5.2.22 yields constant distortion with

value queries per agent. This is Corollary 2 of Peeking behind the ordinal curtain: Improving distortion via cardinal queries[amanatidis2021peeking], inherited as Theorem 3.5 by The limits of preference data for post-training[zhao2025limits]. Under the routing analogy, the count is per query, so a full table over \(|Q|\) queries can require \(O(|Q|\log ^2m)\) values. The statement is an upper bound for the named mechanism in the common-circuit specialization of Theorem 5.2.22. It is not a result for the full routing comparator, nor a lower bound saying that this many values are necessary.

The two source families assume different information models; the diagram compares them without imposing an information order.

The KL identities characterize the exact optimizer for a declared scalar reward. The preference lower bounds are worst-case statements inside a fixed- circuit routing model. Bradley--Terry conclusions depend on the chosen score link. In the common-circuit specialization, cardinal queries change the information modality and admit a positive mechanism result. Outside that specialization, the preference and cardinal-query results are not directly comparable without further assumptions.

None of these statements derives the finite transcript theorem Theorem 4.3.5, and none proves that benchmark gain is capability acquisition. Applying either result to a training system requires that system to satisfy the corresponding information, optimizer, and comparator assumptions.

[clay2026demystifying, Sections 4.1--4.3 and Figure 1] organize the reported experiments along three declared axes: a starting model distribution, a training-prompt distribution, and a reward function. Each reported measurement therefore belongs to a cell of a matrix rather than to an unqualified ``RL post-training'' condition. The source holds its broad training recipe fixed while varying selected axes; it does not study variation among RL algorithms.

Each experimental comparison is conditional on its model, task, prompt law, executable reward rule, training budget, and evaluation procedure. The reward rule is especially consequential here because the main text and Appendix C do not state identical sparse-reward semantics. Section 4.2 describes target containment with a length penalty. Appendix C's prose says an exact target receives unit reward, but its displayed equation requires the response to be strictly longer than the target for any positive reward. These two descriptions do not identify a single sparse-reward implementation.

Fix an evaluation instance \(x\), a response space \(\mathcal Y_x\), and a declared target predicate \(h_x^\star :\mathcal Y_x\to \{0,1\}\). For a sampled response \(Y\), the target-response event is

The predicate is part of the evaluation specification. It may test exact string equality after a declared normalization, containment of a target substring, equality of a parsed answer, or another explicitly typed condition. These predicates are not interchangeable. The movie-quote and AIME experiments in [clay2026demystifying, Sections 4.1--4.2] use different response objects and therefore require different target predicates.

Let \(y_x^\star \) be a designated string and let \(N_x\) be a declared normalizer. Exact-string success is the specialization \(h_x^\star (y)=\mathbf 1\{N_x(y)=N_x(y_x^\star )\}\) of Definition 6.1.2. A task utility may instead accept semantically equivalent answers, assign partial credit, invoke a verifier, or depend on evaluator randomness as in Convention 1.1.

Consequently, a change in target-string match rate is not automatically a change in task utility. [clay2026demystifying, Sections 4.1--4.2] deliberately use narrow target behaviours to study controlled coverage. Transferring their measurements to broader capability requires a separately justified evaluator.

For one declared model family and target predicate, a starting-policy triple is

Here \(\pi ^{\mathrm {base}}\) is the unmodified starting policy, \(\pi ^+\) is obtained by a declared supervised update intended to increase target-response frequency, and \(\pi ^-\) is obtained by a declared update intended to decrease it. The superscripts name producing interventions, not mathematical order relations between the resulting policies.

[clay2026demystifying, Section 4.1] instantiates the triple with the source's Base, SFT-positive, and SFT-negative checkpoints. For the movie quote, its SFT mixture places twenty percent weight on the target quote and eighty percent on other quotes from the Cornell Movie-Dialogs corpus; the negative intervention maximizes cross-entropy loss on the target.

The policies \(\pi ^+\) and \(\pi ^-\) of Definition 6.1.4 are produced by optimization. Their weights, output probabilities, representations, and off-target behaviours can all change together. Calling one intervention ``positive'' and the other ``negative'' describes its intended effect on the declared target response; it does not assert that only one probability mass was edited.

The comparisons in [clay2026demystifying, Sections 4.1 and 5.1] therefore test dependence on three constructed starting checkpoints. They do not identify a causal effect of initial target probability alone without an additional intervention model.

A controlled post-training experiment cell is a record

where \(M\) identifies the model architecture and checkpoint lineage, \(\pi ^{\mathrm {init}}\) is its starting policy, \(\mathsf T\) is the task, \(\mathcal D_{\mathrm {tr}}\) is the training-prompt law, \(r^{\mathrm {id}}\) identifies an executable reward rule and version, \(\mathcal A\) is the update procedure, \(b_{\mathrm {tr}}\) is the training budget, and \(\mathsf E\) is the evaluation interface. Random seeds and hyperparameters belong to the relevant record fields even when suppressed from the notation.

[clay2026demystifying, Figure 1 and Sections 4.1--4.3] motivates the coordinates. The source varies starting distribution, prompt distribution, and reward while using a common broad RL setup. Because its main-text and Appendix-C sparse rewards differ, the reward field is an identifier for an executable rule, not merely the word ``sparse.''

Let \(J\) index the coordinates of an experiment cell \(\mathfrak c\) from Definition 6.1.6. For \(S\subseteq J\), two cells \(\mathfrak c\) and \(\mathfrak c'\) are matched outside \(S\) when

The coordinates in \(J\setminus S\) are the held-fixed coordinates; those in \(S\) are the declared intervention coordinates. Equality means equality at the resolution recorded by the experiment, including evaluation sample size when that size affects the reported statistic.

A matched comparison licenses a contrast between the recorded cells. It does not show that a named coordinate is internally atomic. In particular, matching outside the starting-policy coordinate leaves the composite-checkpoint qualification of Remark 6.1.5 in force.

A source comparison may place the three starting policies of Definition 6.1.4 into cells matched outside the starting-policy coordinate. The shared boxes below mean equality of the declared grid fields, not that the three checkpoints differ in only one scalar probability.

This is the comparison pattern used in [clay2026demystifying, Sections 4.1 and 5.1]. An actual cell still needs its model, task, reward implementation, and numerical record; the diagram is not a claim that all experiments in the paper instantiate one identical grid.

Within a declared model, task, training grid, and evaluation procedure, the three-cell comparison can establish that the measured post-training outcome differs across the source's constructed starting checkpoints. The movie-quote and AIME entries in [clay2026demystifying, Section 5.1, Tables 1--2] are empirical records of that form.

The comparison does not isolate initial target probability from every other SFT-induced weight change, prove that an observed zero has zero support, or establish acquisition of a broader capability. The reported model sets differ across the paper: [clay2026demystifying, Section 4.1] names OLMo 3, Qwen2-7B, Qwen3-1.7B, and Qwen2.5-7B-Instruct, while Table 1 and Appendix G also report Qwen3-8B and Qwen2-1.5B. Each measurement is specific to the model and configuration in its table row.

Fix a starting policy, an evaluation sampling law, and the target predicate of Definition 6.1.2. Before post-training, draw evaluation records \((X_j,Y_j)_{j=1}^m\). Define the hit indicators and the initial evaluation hit count by

The sampling law determines whether the instances \(X_j\) are fixed, resampled, or stratified. The adjective ``initial'' locates the measurement before the post-training intervention; it does not assert that the samples are independent or that the starting policy is a foundation checkpoint.

For a nonempty initial evaluation sample of size \(m\), the empirical initial success rate is

using the empirical-average convention of Convention [ftip-006Z]. This is a statistic of the declared sample. It is not, without a sampling model and an uncertainty statement, the exact successful-support probability \(p_P(x)\) of Definition 3.1.

The equality \(K_m=0\) says that no target response occurred in one finite sample. Every success probability \(0\leq p<1\) assigns positive probability \((1-p)^m\) to this observation under independent Bernoulli sampling. Thus \(\widehat p_m=0\) does not imply \(p=0\), absence from mathematical support, or impossibility of discovery under a larger budget.

The entries printed as 0.00 percent in [clay2026demystifying, Section 5.1, Tables 1--2] are empirical rates at the stated evaluation sizes. They must not be rewritten as exact support claims.

Let \(m\in \mathbb N_{\geq 1}\). Assume the hit indicators from Definition 6.2.1 are independent Bernoulli variables with a common success probability \(p\). Then \(K_m\) has the binomial law and

This finite statement follows from the displayed hypotheses. Applying it to a source table requires the additional iid and stationary-success assumptions stated above.

Assume Theorem 6.2.4 with \(m\in \mathbb N_{\geq 1}\) and fix \(0<\delta <1\). If \(K_m=0\), inversion of the exact zero-count likelihood gives the one-sided upper endpoint

Indeed, \(\Pr _{u_{m,\delta }}(K_m=0)=\delta \), and for every \(p>u_{m,\delta }\) one has \(\Pr _p(K_m=0)<\delta \). Thus the observed zero count excludes probabilities above \(u_{m,\delta }\) at exact level \(\delta \) under the declared iid model.

This is a likelihood inversion derived from the displayed finite setup, not a source theorem and not a support certificate.

Take \(m=128\) and \(\delta =0.05\) in Corollary 6.2.5. After zero hits, the exact one-sided upper endpoint is

Under the iid Bernoulli model, probabilities above approximately 2.313 percent make a zero count less than five percent likely. The calculation does not turn a recorded \(0/128\) into proof that the true probability is zero, and it does not apply if the 128 evaluations have a different joint sampling law.

Fix an executable sparse-reward rule and let \(r_j^{\mathrm {sp}}\) be the reward returned on rollout \(j\). The first positive sparse-reward time is the extended-valued index

This definition concerns the reward implementation named by an experiment cell. The event \(r_j^{\mathrm {sp}}>0\) coincides with a target-response event only when that equality is part of the declared reward semantics. In particular, a length penalty or partial-credit rule can separate positive reward from exact target match.

Suppose the events \(\{r_j^{\mathrm {sp}}>0\}\) are independent and have a common probability \(q\). For every positive rollout budget \(B\),

This is a specialization proved from the displayed hypotheses. It does not supply independence, stationarity, or the value of \(q\) for a training run, and it does not equate positive proxy reward with task utility.

[clay2026demystifying, Section 5.1, Table 2] reports the following empirical target-answer rates for AIME Problem 4 with Qwen2.5-7B-Instruct and evaluation size \(m=128\). The task asks for the number of integer pairs \((x,y)\in [-100,100]^2\) satisfying \(12x^2-xy-6y^2=0\); the target answer is \(117\).

Every entry in the displayed array is a percentage.

These are source-reported empirical percentages, not exact policy probabilities. ``No Reward'' is retained as the source's row label; the table alone does not define it as a post-training protocol. The three zero entries for SFT-negative are finite observations and remain subject to Remark 6.2.3. Appendix Figure 5's caption prints a different set of Qwen2.5-7B-Instruct values that duplicates the neighbouring movie-quote caption and is incompatible with Table 2. The displayed percentages are those of Table 2; the caption does not provide a consistent alternative measurement.

[clay2026demystifying, Section 5.2 and Figure 2] report that the Qwen3-1.7B Base movie-quote cell, whose initial empirical match rate is about 0.5 percent, does not optimize under the source's sparse-reward run, whereas its dense-reward run rises toward fifty percent. [clay2026demystifying, Appendix Figure 8] gives final match rates 10.0 percent for sparse reward and 48.8 percent for dense reward. Thus ``failed'' here means failure of the reported sparse run to optimize as intended, not zero final matches.

A plateau is indexed by the model, target, reward implementation, optimizer, sampling process, and budget in its experiment cell. One observed transition therefore does not identify a universal initial-probability threshold for RL learning, support, or capability acquisition.

Let \(\mathcal Y\) be a finite response set and fix a target response \(\tau \in \mathcal Y\). The sparse exact reward is the map \(r_{\rm exact}:\mathcal Y\to \{0,1\}\) defined by

This idealized reward exposes one bit: whether the whole response equals the target. It does not expose a prefix, edit location, or partial milestone. The movie-quote source uses related but textually inconsistent substring and length-penalized rewards; the distinct formulas are compared in Remark 6.3.2.

Section 4.2 of Demystifying Reinforcement Learning Post-Training of Language Models[clay2026demystifying] displays a substring indicator \(\mathbf 1\{\tau \subseteq y\}\) and says that a length penalty is applied. In Appendix C, \(n\) is the maximum generation length and \(s\) is the number of tokens beyond the target. The appendix first defines the excess-length penalty

and then gives Equation (3):

As printed, Equation (3) assigns zero when \(y=\tau \), because its first branch requires strict excess length. This conflicts with the immediately preceding Appendix-C prose, which assigns base reward one when the target is generated, and it is not the exact-match reward of Definition 6.3.1. The two printed descriptions therefore specify different reward rules.

Let \(y=y_1\cdots y_m\) and a nonempty target \(\tau =\tau _1\cdots \tau _n\) be finite strings. Their Levenshtein distance is determined by \(D(0,j)=j\), \(D(i,0)=i\), and, for \(i,j>0\),

Set \(L(y,\tau )=\max \{|y|,|\tau |\}\). The edit-distance reward is

The recurrence is Equation (1) in the main text and Equation (4) in Appendix C of Demystifying Reinforcement Learning Post-Training of Language Models[clay2026demystifying]; the normalized reward is its Equation (5). The nonempty-target assumption makes the displayed denominator positive. This reward does not establish that edit similarity is the correct utility for a different task.

The edit reward of Definition 6.3.3 is not obtained from the binary value in Definition 6.3.1 by a numerical smoothing operation. It also receives the target string, a character-level edit model, and the ordering of symbols. Two non-target responses that both receive sparse reward zero can therefore receive different edit rewards.

That extra resolution can improve credit assignment, but it changes the feedback channel and its assumptions. The source calls edit distance a dense proxy and reports controlled movie-quote experiments [clay2026demystifying, Sections 4.2 and 5.2]. Neither the formula nor those observations establish that edit proximity is an independent measure of general response quality.

For the source's fixed AIME problem, let

record whether a completed response contains five declared milestones: a valid algebraic setup, correct root relations, correct integer bounds, a correct count for at least one branch, and correct subtraction of the overlap at the origin. With

the milestone process reward is

Appendix D.1 of Demystifying Reinforcement Learning Post-Training of Language Models[clay2026demystifying] gives Equation (6) and the milestone list above. In the experiment, a 32-billion-parameter instruction-tuned model classifies the completed trajectory. The displayed map is therefore richer than a deterministic final-answer verifier, even though it returns one scalar after the rollout.

Appendix D.2 of Demystifying Reinforcement Learning Post-Training of Language Models[clay2026demystifying] supplements \(r_{\rm proc}\) with an extracted-answer override and two penalties. Translating its notation, let \(\tau =117\), \(\tau _{\rm near}=118\), and define

Let \(p_{\rm loop}(y)=-0.3\) when the judge flags a loop without the exact answer, and zero otherwise. Let \(p_{\rm format}(y)=-0.5\) when the response lacks the required boxed delimiter, and zero otherwise. Equations (7)--(8) then give

These clauses assume a reliable milestone judge, parser, near-miss choice, loop flag, and formatting rule. They are part of the feedback definition, not consequences of reinforcement learning. The main text describes ``exponentially increasing'' dense rewards, and Appendix D calls the weights ``exponential scaling.'' The printed vector \((0.05,0.05,0.10,0.15,0.25)\) is neither strictly increasing at every milestone nor a geometric progression. We preserve the exact weights and do not infer an exponential law from that wording.

Let \(\mathcal Y\) be a finite response set and \(r:\mathcal Y\to \mathcal R\) any reward map. Two responses are observationally equivalent under \(r\), written \(y\sim _r y'\), when

Equality makes \(\sim _r\) an equivalence relation. Its quotient \(\mathcal Y/{\sim _r}\) is the finite set of response classes distinguished by the reward alone. This proposed equivalence relation ignores any information in the response that is not returned by \(r\); equal rewards need not imply equal latent utility.

Let \(\mathcal Y\) be finite and let \(r_1:\mathcal Y\to \mathcal R_1\) and \(r_2:\mathcal Y\to \mathcal R_2\). Say that \(r_2\) refines \(r_1\) when

If \(r_2\) refines \(r_1\), then

This finite lemma compares observational partitions only. It does not say that the refined reward is cheaper, more accurate, or better aligned with utility.

Take the target string \(\tau =\texttt {abc}\) and two responses \(y=\texttt {abx}\) and \(y'=\texttt {xyz}\). Both fail exact matching, while their edit distances and normalized edit rewards are

Thus \(y\sim _{r_{\rm exact}}y'\) but \(y\not \sim _{r_{\rm edit}}y'\). The example witnesses a strict separation inside one sparse-reward class. It does not claim that every dense proxy refines every sparse verifier.

One finite response set can be observed through three different maps. The arrows below share a domain; they do not assert that the three codomains form a refinement chain.

The exact reward forgets every difference among failures. Edit reward can retain character-level proximity, while the source process reward retains a declared milestone vector only after judge, override, and penalty choices. The partition lemma Lemma 6.3.8 applies to a pair only after its refinement hypothesis has been checked.

A denser reward can distinguish more responses and supply more frequent update signal. It can also require information absent from a sparse verifier. In the controlled source, edit feedback assumes the full target and computes a string metric; process feedback uses a 32-billion-parameter judge, five problem-specific milestones, answer extraction, an enumerated near miss, and loop and format penalties [clay2026demystifying, Appendices C--D].

The richer feedback channel incurs specification, computation, and validation costs. Replacing sparse feedback by dense feedback can alter both the information available to training and the objective being optimized. The resulting comparison is an intervention on feedback resolution, not evidence that one fixed reward was smoothed at zero cost.

Let \(\mathcal X\) be a task-instance set. Write \(\mu _{\rm tr}=\mathcal D_{\rm tr}\) for the training-prompt coordinate of an experiment cell in Definition 6.1.6. A training prompt law is this probability law on \(\mathcal X\), used to draw instances during a post-training protocol. Let \(\mu _{\rm ev}\) be an independently declared evaluation law on the same set.

An evaluated prompt slice is a measurable set \(C\subseteq \mathcal X\) with \(\mu _{\rm ev}(C)>0\). Its conditional evaluation law is

The training and evaluation laws need not agree. A slice records where an effect is measured; it does not assert that prompts inside the slice are equally difficult or represented equally in pretraining. We specialize the task-law convention of Definition [ftip-001Q].

A prompt configuration is the record

containing the starting artifact, training prompt law, finite prompt-pool size, reward, update algorithm, and training budget. We call two particular records \(\mathsf c_{\rm nar}\) and \(\mathsf c_{\rm brd}\) only when a cited experiment names them narrow and broad.

Section 4.3 and Appendix F of Demystifying Reinforcement Learning Post-Training of Language Models[clay2026demystifying] instantiate these labels in two model-family experiments. The Qwen comparison uses 100 DeepScaleR prompts versus 10,000 WildChat prompts. The OLMo comparison uses 100 math-only prompts versus 10,000 prompts split evenly among mathematics, instruction following, and code. These labels identify the two reported configurations; they do not define a general measure or ordering of prompt breadth.

The records in Definition 6.4.2 change several coordinates together. The prompt-pool size, domain mixture, and repetition frequency differ. Across the paper's Qwen and OLMo studies, the starting artifact and evaluated tasks also differ.

We therefore use ``narrow'' and ``broad'' as names for the source cells. They do not define a total order on prompt laws, and their comparison does not identify a causal effect of breadth alone. A breadth theorem would need a declared statistic and matched configurations that differ only in that statistic.

Fix an evaluated slice \(C\) from Definition 6.4.1, and let \(P_0\) and \(P_1\) denote the pre- and post-training protocols whose successful-support probabilities are defined in Definition 3.1. The slice-conditioned success change is

The evaluation interface, inference budget, and success predicate are held fixed across the two terms. The quantity is an average change on one declared slice. It neither locates the change inside the model nor identifies which training coordinate caused it.

Fix pairwise disjoint evaluated slices \(C_1,\ldots ,C_J\) and a declared degradation threshold \(\kappa >0\). Define the set of materially degraded slices

Degradation is localized relative to this slice family and threshold when \(|\mathcal H_\kappa |=1\). It is distributed when \(|\mathcal H_\kappa |\geq 2\).

These terms depend on the chosen partition, threshold, and evaluation budget. They do not turn a finite benchmark vector into a statement about all capabilities.

Section 5.3 and Appendix F of Demystifying Reinforcement Learning Post-Training of Language Models[clay2026demystifying] supply two OLMo paths; Figure 4 reports their outcomes. Both replace the usual verifiable reward with a random scalar drawn from \(\operatorname {Unif}[0,1]\), but their prompt configurations differ.

In the narrow SFT-start path, GSM8K accuracy falls from 86 to about 32 by step 400, while the reported MMLU and IFEval changes are much smaller. In the broad path, the source reports an entropy spike near step 400 together with collapse on GSM8K, MMLU, and IFEval. The diagram records that interpretation; it does not isolate prompt breadth from pool size, mixture, or repetition.

Section 5.3 and Figures 3--4 of Demystifying Reinforcement Learning Post-Training of Language Models[clay2026demystifying] report that the Qwen narrow configuration retains higher MATH and AMC Acc@1 than its broad configuration under random reward. The OLMo experiment reports the different evaluation patterns summarized in Example 6.4.6.

These are finite empirical observations. The random scalar reward removes task alignment from one feedback channel, but it does not hold all other training coordinates fixed. The reported entropy is a named token statistic, not a direct measure of capability or acquisition.

Let two evaluation slices \(C_1,C_2\) have equal mass. A feedback channel returns an independent Bernoulli reward with mean \(1/2\) under either of two post-training protocols. Suppose their fixed-interface success changes are

The reward law and its mean are identical, but the slice-level evaluation vectors differ. Hence marginal training reward does not determine whether damage is localized or distributed. This is a finite counterexample, not a claim about the mechanism in the source experiment.

The source cells motivate tests of starting-policy coverage, feedback resolution, and prompt-conditioned damage. They do not establish that dense reward creates mathematical support, that zero sampled hits imply zero probability, or that random reward has one model-independent effect.

Separating the effects of coverage, feedback resolution, and prompt breadth requires stronger controls. One comparison fixes prompt laws and varies only the feedback sigma-algebra. Another fixes feedback and evaluation while varying one starting-policy coordinate. Confidence regions for the complete vector of slice-conditioned changes would quantify effects that an aggregate score can hide.

The reported observations remain conditional on their experimental configurations. They do not establish general results about capability acquisition, elicitation, or the optimal allocation of post-training compute.

The conclusions depend on their declared probability laws, finite index sets, positivity conditions, and execution inputs. Changing one of these assumptions can change the result even when the observed score is unchanged.

Finite constructions can separate average success from coverage, proxy accuracy from evaluation quality, and observed gradients from the hypotheses needed to bound them. Each separation identifies an assumption that a stronger conclusion would require.

Prompt and response spaces, probability laws, policies, rewards, evaluation scores, and execution traces are distinct mathematical objects.