Example. Counterexample: active batch cancellation can hide nonunit ratios [ftip-0084]
AGENTDRAFTED
Fix \(\epsilon _{\rm clip}\in (0,1)\) and
\(1<R<1+\epsilon _{\rm clip}\). On a two-token vocabulary, let both observed
tokens have \(a_i=1\), current probability \(\pi _\theta (1)=1/2\), and behavior probability
\(\pi _{\rm old}(1)=1/(2R)\). Take scalar hidden states \(h_{L,1}=h_{L,2}=1\)
and fixed normalized advantages \(\widehat A_1=1\), \(\widehat A_2=-1\).
These may be selected token terms from different rollout members. An untied
head with both logits zero realizes the current probabilities. Both terms lie
strictly inside the clipping interval of Definition [ftip-003F], so the clipped
surrogate locally agrees with \(\mathcal L_i=r_i\widehat A_i\). Thus
\(r_1=r_2=R\), and Theorem [ftip-007Z] gives
\[
G_1^{\rm lm}=R(1/2,-1/2)^{\mathsf T},
\qquad
G_2^{\rm lm}=-G_1^{\rm lm}.
\]
Consequently \(G^{\rm lm}=0\), while
\(\widehat \chi ^2=R^2-1>0\). For this fixed clipping rule, the statistic in
this construction is bounded above by \((1+\epsilon _{\rm clip})^2-1\).
For the raw surrogate \(\mathcal L_i=r_i\widehat A_i\) considered as a
separate objective, the same algebra works at every \(R>1\) and yields
arbitrarily large squared-ratio excess. This does not extend the active
construction to arbitrary \(R\) under clipping: when
\(R>1+\epsilon _{\rm clip}\), the positive-advantage term is locally constant,
while the negative term remains active. Their clipped gradients are then
\(0\) and \(-R(1/2,-1/2)^{\mathsf T}\), with nonzero mean
\((-R/4,R/4)^{\mathsf T}\).
The example does not contradict Theorem [ftip-0081]: that theorem upper-bounds
batch-gradient energy by a ratio moment. It supplies no lower bound on the
gradient and no converse from zero batch-gradient energy to ratios equal to
one. The arbitrary-ratio version concerns only the separate raw surrogate.