Let \(\mathcal X_{\mathrm {pr}}\) be the prompt space. For each prompt \(x\in \mathcal X_{\mathrm {pr}}\), let \(\mathcal Y(x)\subseteq \mathcal V^*\) be the set of finite token responses permitted by the training format. We write \(y=(y_1,\ldots ,y_{|y|})\in \mathcal Y(x)\) and \(y_{<t}=(y_1,\ldots ,y_{t-1})\).

A demonstration is denoted \(z=(x,y,\omega )\), where \(\omega \) is its provenance record. A finite multiset of demonstrations is denoted \(D_{\mathrm {sft}}\). For each response position, a loss mask \(m_t(z)\in \{0,1\}\) records whether the token contributes to the supervised objective, and a nonnegative weight \(w_t(z)\) records its declared normalization. Prompt tokens are conditioning context and lie outside the response-position domains of \(m_t(z)\) and \(w_t(z)\).

Using the prompt space of Notation 1, a prompt serializer is a declared map \[ s_{\mathrm {pr}}:\mathcal X_{\mathrm {pr}}\longrightarrow \mathcal V^*. \] For token strings \(u,v\in \mathcal V^*\), write \(u\mathbin {\Vert }v\) for their concatenation. A response token \(y_t\) is therefore predicted from the token prefix \(s_{\mathrm {pr}}(x)\mathbin {\Vert }y_{<t}\).

The serializer fixes system text, role markers, separators, and any other prompt-side control tokens. Changing it changes the conditional examples seen by the language model even when the abstract prompts and responses are unchanged.

The next-token law in Definition [ftip-000H] accepts token prefixes, whereas an instruction dataset may store structured prompts. The proposed serializer in Definition 2 maps between those types. Ouyang et al. describe SFT on labeler demonstrations in [ouyang2022training, Section 3.5 and Appendix C.1]; their training account supplies a concrete pipeline rather than a universal serializer.

A reproducible objective must retain the serializer revision with the tokenizer and loss mask. Otherwise two runs can share an abstract dataset name while optimizing different token sequences.

Instruction tuning fine-tunes a pretrained language model on a mixture of tasks whose examples are expressed through natural-language instructions and corresponding target outputs. The task description and any input are placed in the prompt; the target output supplies the continuation to be learned.

The construction changes the model parameters by supervised learning. It does not require a learned reward model, preference comparisons, or interaction with an environment.

Using the notation of Notation 1, a demonstration record is a triple

where \(\omega \) identifies the declared producer of \(y\), the collection or generation procedure, and any selection rule applied before the record entered \(D_{\mathrm {sft}}\). The response is a training target; its presence in the data does not assert that it is uniquely correct or preferred to every alternative.

The provenance field \(\omega \) distinguishes a demonstration from the process that supplied it. Ouyang et al. collect labeler-written demonstrations in [ouyang2022training, §§3.2, 3.4]. Wei et al. permit a broader task mixture in [wei2022finetuned, §2; Fig. 2].

Human writers, stronger teacher models, filtered self-generations, and programmatically produced targets give different information to the learner. Claims about sample efficiency or capability acquisition must therefore state which producer and selection rule were available.

In the InstructGPT data-collection procedure, a labeler receives a prompt and writes a desired response. The resulting prompt--response pair enters the demonstration data used by the supervised fine-tuning stage. Prompts come from the declared API and labeler-written sources, and the paper records filtering and labeler-selection procedures.

This construction specifies a human demonstration channel. It is distinct from the later comparison channel, in which a labeler ranks model-generated responses rather than writing the target response.

For the language-model policy \(\pi _\theta \) and the notation of Notation 1, a masked supervised fine-tuning (SFT) objective has the form

where the nonnegative weights satisfy \(\sum _{z\in D_{\mathrm {sft}}}\sum _{t=1}^{|y|}w_t(z)m_t(z)=1\). The response index \(t\) does not range over serialized prompt tokens. Minimizing this objective increases conditional likelihood on the selected target tokens. The weights and mask are local conventions and must be declared for each implementation.

For a finite demonstration multiset, define the number of selected response tokens by

When \(N_{\mathrm {resp}}>0\), response-token normalization sets \(w_t(z)=N_{\mathrm {resp}}^{-1}\) at every selected position. Each selected token then has equal weight in \(L_{\mathrm {sft}}\). Equal weighting of records or prompts is a different convention because response lengths vary.

The training account alone does not determine this normalization. Any comparison of losses or gradients must retain the mask and normalization that produced them.

Two instruction positions provide conditioning context but lie outside the response-loss index set. The loss is taken on the two generated tokens.

Declare \(\pi (r_1\mid u_1,u_2)=1/2\) and \(\pi (r_2\mid u_1,u_2,r_1)=1/4\). The instruction tokens are not indexed by the response mask. With mask one on each displayed response token, the sum and mean are \[ L_{\rm sum}=-\log \tfrac 12-\log \tfrac 14=\log 8, \qquad L_{\rm mean}=\tfrac 12\log 8\approx 1.040. \]

The supervised objective in Definition 8 and the causal factorization in Definition [ftip-000I] specialize to the displayed token strip. Its mask is a local choice on response positions; other instruction-tuning implementations may supervise a different subset of response tokens or declare a larger loss domain explicitly.

Instruction tuning changes conditional likelihood under a declared demonstration distribution. It can teach response format, task interpretation, and behavior represented by the targets, but the training loss alone does not identify which of those effects caused an independent evaluation gain.

Ouyang et al. report supervised fine-tuning and its optimization settings in [ouyang2022training, Section 3.5 and Appendix C.1]. They do not define the exact mask and weighting convention displayed in Definition 8; those choices remain part of the declared objective.

The stage also leaves several questions open. A demonstration does not compare its target with alternatives, a teacher may transmit errors or hidden information, and low loss on the collected prompts need not imply transfer to a new task law. Preference acquisition introduces a different observation: which response a judge selected from a displayed pair.