Fix the evaluation interface and success threshold \(u_*\) from Convention [ftip-005D]. For a protocol \(P\) and instance \(x\in \mathcal X_{\mathsf T_{\rm ev}}\), its successful-support probability is

The probability includes protocol, inference, environment, and evaluator randomness: \(M_P\) is drawn from the protocol, while \((\Xi ,\Omega )\) is drawn from \(\Lambda _{\rm ev}(\cdot \mid x)\) independently of training. The quantity is instance-specific and budget-specific.

For threshold \(0<\alpha \leq 1\), the successful-support coverage of protocol \(P\) under evaluation law \(Q_{\rm ev}\) is

Coverage records how much task mass has at least the declared success probability. It does not average the probabilities above or below the threshold.

Increasing probability mass on known solutions need not increase coverage of the task family. Pass-at-\(k\) and related support-sensitive measurements motivate this distinction. Sections 3--5 of Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?[yue2025does] and Sections 2--3 of Understanding r1-zero-like training: A critical perspective[liu2025understanding] compare post-training behavior with base-model support and discuss why response accuracy alone may overstate what RL creates beyond a base model.

The threshold \(\alpha \), inference budget, task law, and success predicate must all be reported. Finite samples estimate this quantity; they do not reveal the unobserved tail without assumptions.

Fix a finite, independently selected family \(G\subseteq \mathcal X_{\mathsf T_{\rm ev}}\), thresholds \(0\leq \alpha <\beta \leq 1\), and the common evaluation interface of Convention [ftip-005D]. Protocols \(P_0\) and \(P_1\) have a uniform task-family acquisition witness on \(G\) when

For each \(x\), the successful event in Definition 1 supplies the fixed-budget transfer event in Definition [ftip-0007]. The displayed inequalities specialize that earlier acquisition witness with \(\varepsilon _0=\alpha \) and \(\varepsilon _1=\beta \), then require the specialization uniformly over \(G\).

The proposed family-level criterion is a demanding paired comparison. The task family and evaluator are fixed independently, and both protocols use the same inference budget. It is stronger than one acquisition witness because the threshold change must hold for every member of \(G\).

A theorem may strengthen the witness with confidence bounds, transfer to a predeclared related family, or a lower bound on newly successful support. Each strengthening requires additional sampling or structural assumptions.

Let \(S\in \{0,1\}\) be an independently defined success label and \(A\in \{0,1\}\) the verifier's accept decision on the same outcome. The false-accept and false-reject rates are

Both rates are conditional on the task and outcome distribution used for the audit. They are undefined when the conditioning event has probability zero.

Verifier error is measured against an independent success label because a training reward cannot audit itself. The verifier and evidence interfaces are in Definition [ftip-0046]--Definition [ftip-0049]. DeepSeek-Prover-V2 [ren2025deepseekproverv2, Section 2.3, ``Reinforcement Learning''] supplies a proof-checking instance. The false-accept/false-reject decomposition is a binary measurement model.

The rates may vary by task, policy, trajectory length, and adversarial pressure. A low average error rate need not prevent reward hacking on the subpopulation favored by optimization.

Shifting probability mass toward a checker exploit raises proxy reward while reducing independently judged utility.

Let the three outputs be \(y_0=\text {41}\), \(y_1=\text {42}\), and \(y_2=\text {ignore; 42; done}\). The proxy accepts \(y_1,y_2\); the independent evaluator accepts only \(y_1\). Moving their probabilities from \((0.5,0.4,0.1)\) to \((0.1,0.1,0.8)\) changes expected proxy reward from \(0.5\) to \(0.9\), but true utility from \(0.4\) to \(0.1\).

Reward exploitation in formal verification is discussed in [ren2025deepseekproverv2, Section 3.2]. The constructed checker exposes one failure mechanism. It provides no basis for treating every executable verifier as incomplete or every proxy improvement as reward hacking.

Fix a finite action space, a finite horizon \(T\), and a declared distribution \(\mu _t\) over public histories at each position. The global policy entropy is the weighted average

The history distribution and action alphabet are part of the statistic.

We use the convention \(0\log 0=0\) in this and the following position-entropy formula.

Let \(T_{\rm tok}\) be the random number of generated tokens. For a one-based token position \(t\) satisfying \(\Pr (T_{\rm tok}\geq t)>0\), condition on \(T_{\rm tok}\geq t\) and let \(\xi _t\) be the token. The token-position entropy is

This is the entropy of a position marginal. It is generally different from the history-conditioned entropy averaged in Definition 9.

For a sampled token \(\xi _t\) under next-token law \(\pi (\,\cdot \mid \mathsf H_t)\), its surprisal is

It is a random value attached to the selected token. Its conditional expectation over the token equals the finite Shannon entropy of the next-token law.

Shannon's finite entropy A mathematical theory of communication[shannon1948mathematical] can be aggregated over different random objects. Global and position-level entropy statistics use different laws. Selected-token surprisal is a sample statistic, while the other two are expectations under declared history or position laws.

A training intervention can raise one quantity while lowering another. Later claims about exploration or collapse must name the statistic, sampling law, horizon, and conditioning event.

For token-level current-to-behavior ratios \(r_1,\ldots ,r_T\), DGG defines

The source relates this statistic to an upper bound on language-model-head gradient energy through a batch-dependent constant. In a finite batch \(\widehat \chi ^2\) can be negative, so it is not automatically a nonnegative empirical Pearson divergence. The result also gives no converse from a small batch gradient to small policy shift.

A componentwise lifecycle allocation separates resource accounting from the finite number of rollouts available for observation.

In the coordinate order of Definition [ftip-005H], declare a lifecycle budget vector \(\mathbf B\). Its rollout coordinate permits \(N\) attempts. If each attempt has declared success probability \(p\), then \[ \Pr (\text {at least one success})=1-(1-p)^N\leq Np. \] Only the rollout coordinate enters this calculation; no sum or conversion among heterogeneous coordinates is defined.

The finite bound is the union bound applied to Bernoulli rollouts; finite-horizon success probabilities are standard policy-evaluation objects in [sutton2018reinforcement, Section 3.5]. Independence is needed only for the displayed exact expression. The budget vector is illustrative: neither empirical optimality nor interchangeability of heterogeneous units is inferred from it.

Finite analysis raises three questions: how conditional success rates bound long-horizon success, how replay-ratio concentration controls update error, and how trajectory extrapolation with periodic recovery affects evaluation regret. A cost comparison also depends on rollout, update, environment, storage, and evaluation work.

Finite counterexamples show why these questions need additional assumptions. Finite policies can separate the entropy statistics above. Two verifiers can agree on all observed training records yet differ on an unobserved adversarial region. Two parameter paths can share early low-rank summaries and diverge after a new successful rollout. These examples mark where additional assumptions are unavoidable.

The DGG and NExt papers motivate reuse and trajectory-compression hypotheses but do not establish these general results. Bounds on update error and extrapolation regret require assumptions relating the monitored quantities to the corresponding outcomes.