Notation. Reward, return, policy roles, value, advantage, and likelihood ratios [ftip-002X]
AGENTDRAFTED
The ratio and policy-objective formulas in this section specialize the
measurable policy interface of Definition [ftip-0023] to a declared finite or
countable discrete action space. Thus \(\pi (a\mid h)\) denotes probability
mass, not an unspecified density. Whenever \(\log \pi _\theta (A_t\mid H_t)\) is
displayed, its sampled mass is assumed positive.
For decision times \(t=0,\ldots ,N-1\), write
\(\mathsf H_t:=H_t\) for the public history defined in Definition [ftip-0021],
\(A_t\) for its action, and \(R_{t+1}\in \mathbb R\) for
the next reward. Let \(\gamma \in [0,1]\) be the discount factor and \(G_t\) the
return from time \(t\).
Write \(\pi _b\) for the behavior policy that produced a stored action,
\(\pi _\theta \) for the current trainable policy, and \(\pi _{\mathrm {ref}}\) for
a fixed reference policy. Their value, action-value, and advantage functions
are denoted \(V^\pi \), \(Q^\pi \), and \(A^\pi \); an estimated advantage is
\(\widehat A_t\). The current-to-behavior likelihood ratio is denoted
\(\rho _t(\theta )\).
In a response-level episode, \(\pi _\theta (y\mid x)\) abbreviates the
completed-continuation probability
\(P_\theta (y\mid s_{\mathrm {pr}}(x))\) of the response-task interface, using the
declared prompt serializer; the same convention applies to \(\pi _b\) and
\(\pi _{\mathrm {ref}}\). In an interactive episode these symbols instead denote
the action laws on public histories defined in Definition [ftip-0023].