Let \(\mathcal E\) be an evidence space. Write \(\mathsf V\) for a declared verifier, \(e_{\mathrm {ver}}\in \mathcal E\) for its evidence, \(d_{\mathrm {ver}}\in \{0,1\}\) for its decision, and \(R_{\mathrm {ver}}\in \mathbb R\) for the reward derived from that result.

For a prompt \(x\) and group size \(g\geq 2\), write \(\mathbf Y_x=(Y^{(1)},\ldots ,Y^{(g)})\) for a rollout group sampled from the behavior policy. Let \(R_i\) be the reward of member \(i\), \(\bar R\) the group mean, \(S_R\) the group standard deviation, and \(\widehat A_i\) its group-relative advantage.

Reinforcement learning with verifiable rewards is a post-training method that samples responses from a language-model policy, computes rewards with a deterministic check of the response, and applies a reinforcement-learning update to increase expected checked reward, optionally with regularization to a reference policy.

In the cited construction, a correct response receives reward \(\alpha \) and an incorrect response receives \(0\). Equation (8) gives this two-valued form, and the paragraph immediately after it sets \(\alpha =10\). The checked predicate, reward scale, prompt law, rollout procedure, and policy optimizer remain separate parts of the method.

Concrete RLVR methods include the PPO-based construction in [lambert2024tulu3, Section 6], group relative policy optimization (GRPO) in [shao2024deepseekmath, Section 4.1], R1-Zero in [guo2025deepseek, Section 2.2], and Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) in [yu2025dapo, Sections 2--3].

A proposed common description groups methods that acquire policy samples, evaluate a declared verifiable signal, assign credit, and update the policy. A specific claim must still name its verifier, group sampling, estimator, clipping, normalization, reference-policy treatment, and length rule.

For prompts and responses from Notation [ftip-002A], a deterministic verifier correctness indicator is a declared map

Here \(v(x,y)=1\) means that the response satisfies the checked predicate. The map includes the parsing and execution rules needed to reproduce the decision. A stochastic judge or learned reward model is a different feedback mechanism unless its randomness and decision procedure are separately exposed.

A verifier evidence record is a tuple

where \(\iota _{\mathrm {ver}}\) identifies the verifier and its version, \(o\) is the reproducible output used by the check, and \(d_{\mathrm {ver}}\in \{0,1\}\) is the resulting decision. Examples of \(o\) include a parsed exact answer, a test log, or a proof-checker result.

The evidence record preserves how the decision was obtained. It does not assert that the checked predicate is complete for the task's independent success criterion.

The same correctness decision can arise from different parsers, tests, proof kernels, or failure modes. Tülu 3 uses a scaled two-valued reward \(\alpha \) or \(0\), with \(\alpha =10\), in [lambert2024tulu3, Section 6, equation (8), and the immediately following paragraph]; it does not identify that reward with a complete record of the checking procedure.

Evidence enables later audits of false positives, false negatives, version changes, and shortcut exploitation. Reproducibility of the check still does not establish that the checked predicate equals the intended capability.

Given a deterministic correctness indicator \(v\) and reward scale \(\alpha >0\), the verifiable reward in the cited construction is

The paragraph after Equation (8) sets \(\alpha =10\). Another method may choose a different scale or combine several checked components, but that transformation is part of its reward design. A reward is verifiable relative to the implemented check, not relative to every property an independent evaluator may care about.

Rule evidence compares a parsed answer or format with a declared target, as in [guo2025deepseek, Section 2.2]. Test evidence executes a candidate in a sandbox and records test outcomes. Proof evidence submits a term to a specified checker and records acceptance or an error.

These routes expose different predicates and costs. A rule can ignore the derivation, a test suite can omit behavior, and a proof checker establishes only the proposition and trusted-kernel boundary it was given. Their rewards must not be merged without retaining which evidence source fired.

Fix a prompt \(x\), group size \(g\geq 2\), and behavior policy \(\pi _b\). A rollout group is the indexed family

DeepSeekMath samples several outputs for the same prompt so their rewards can define a group-relative baseline. The producing policy, decoder, group size, and conditional sampling dependence are part of the rollout law.

For rollout-group rewards \(R_1,\ldots ,R_g\), define the group mean and group standard deviation by

Both statistics are prompt-local and depend on every member of the sampled group. The source leaves the standard-deviation divisor symbolic and does not add a numerical stabilizer. An implementation must state those conventions and its treatment of zero-variance groups.

When \(S_R>0\), the group-relative advantage of rollout member \(i\) is

The estimator compares responses sampled for the same prompt and does not require a learned value function. Its members are statistically coupled through \(\bar R\) and \(S_R\); it is not an independent estimate of the population advantage of each response.

Outcome-supervision GRPO samples a group of responses for one prompt, assigns each response an outcome reward, standardizes those rewards within the group, and uses the resulting group-relative advantage across the response's token log-likelihood terms.

The construction removes a separately learned critic but retains a behavior policy, likelihood ratios, a reference-policy term in the source formulation, and explicit normalization choices.

Process-supervision GRPO supplies rewards at declared reasoning steps and forms advantages from the subsequent step rewards assigned to each token. The policy objective therefore receives credit that can vary within one response rather than repeating one terminal outcome advantage at every token.

This construction requires a step segmentation and a process reward model. Its estimator is not obtained merely by relabeling an outcome-supervision rollout group.

GRPO names a family of group-relative policy updates rather than one fully determined estimator. Outcome and process supervision assign different reward structures. Implementations may also differ in token versus response normalization, KL terms, clipping intervals, importance weights, and treatment of groups with zero reward variance.

A result about one variant must retain its producing policy and all of those conventions. Sharing the group-relative baseline does not make the resulting gradients identical.

Clip-higher replaces the symmetric upper and lower PPO clipping widths by separate constants. It permits a larger upward likelihood-ratio movement for tokens with positive advantage while retaining a smaller lower-side interval.

The change alters which sampled terms are saturated by the surrogate. It does not impose a policy-wide trust region or guarantee preservation of low-probability actions.

DAPO dynamic sampling filters prompt groups whose sampled responses all receive the same reward and continues sampling until the training batch contains the declared number of groups with nonzero reward variation.

The intervention changes the distribution of prompts and rollouts entering the update. It can prevent zero-advantage groups from consuming an optimizer batch, but the retained batch is no longer an unconditioned sample from the original prompt law.

The token-level loss aggregates active token terms across the complete minibatch and divides by the number of active tokens. A longer response therefore contributes in proportion to its active token count rather than receiving the same total weight as every shorter response.

This normalization changes the empirical gradient even when the sampled responses, rewards, advantages, and likelihood ratios are unchanged. It must be distinguished from averaging a separately normalized loss over responses.

Overlong reward shaping introduces a soft penalty near the maximum response length before the hard truncation boundary. The penalty increases over a declared buffer region rather than assigning the same abrupt terminal penalty to every response that reaches the limit.

The construction changes both the reward value and which length-related behavior receives gradient. The maximum length, buffer width, and penalty schedule are part of the reward design.

Clip-higher changes the saturated region of the policy surrogate, dynamic sampling conditions which prompt groups enter a batch, token-level aggregation changes response-length weighting, and overlong shaping changes the reward near a truncation boundary. These are four different interventions.

An observed training improvement cannot be assigned to a generic ``algorithm'' without an ablation that holds the other sampling, estimator, reward, and compute choices fixed. The interventions also change which rollouts and gradients are observed, so their effects need not add linearly.

Three sampled responses pass through exact verification, group credit, and a proposed update in one finite RLVR round.

The exact-answer checker gives mean reward \(\bar r=2/3\). Using the declared unnormalized centered credit \(\widehat A_i=r_i-\bar r\) yields \((1/3,-2/3,1/3)\). An optimizer may use these numbers to propose a parameter change, but the tuple itself is not yet an accepted update.

The RLVR protocol is Definition 2. Its group-credit objects are Definition 9, Definition 10, and Definition 11. Together they specialize to this finite order of operations. Centered, unscaled credit and an exact-answer checker are local choices rather than requirements on other RLVR algorithms.

A rollout group is normalized to relative advantages and then passed to a clipped surrogate objective.

For a rollout group with rewards \(r_i\), form a group mean and scale and set \(\widehat A_i=(r_i-\bar r)/s\). For current-to-behaviour ratios \(\rho _i\) and clip interval \([1-\epsilon ,1+\epsilon ]\), define each pointwise term by

The mean of the resulting terms is the group contribution to the update; its value depends on the sampled rewards, ratios, and clip width.

Group-relative normalization is grounded in [shao2024deepseekmath, Section 4.1], while clipping is grounded in [schulman2017proximal, Section 3]. The normalization convention and ratios must be declared by the training protocol; a positive surrogate mean alone gives no guarantee of improved held-out performance after an optimizer step.