Let \(\mathcal D_{\rm doc}\) be a measurable document space and \(\mathcal S\) a finite set of data sources. Source \(s\in \mathcal S\) has a document law \(Q_s\) on \(\mathcal D_{\rm doc}\), and mixture weights \(\alpha \in \Delta (\mathcal S)\) use the simplex notation of Notation [ftip-000A]. Let \(T\) be the context length. A token window is \(x_{1:T}\in \mathcal V^T\); a prediction mask is \(m_{1:T}\in \{0,1\}^T\), where \(m_t=1\) means that the token at position \(t\) contributes to the loss.

Let \(\Theta \) be the parameter set used by the next-token law in Definition [ftip-000H]. The model parameters at optimizer update \(k\in \{0,1,\ldots ,K\}\) are \(\theta _k\in \Theta \); \(\theta _0\) is the initialization and \(\theta _K\) the stopped parameter value. A minibatch at update \(k\) is \(\mathcal B_k\). Random document draws, window offsets, masks, batch order, and optimizer noise are part of the training run even when the notation suppresses them.

Using Notation 1, a pretraining source mixture is the pair \(((Q_s)_{s\in \mathcal S},\alpha )\). It induces the document law \[ Q_{\alpha }(A)=\sum _{s\in \mathcal S}\alpha _s Q_s(A) \] for every document event \(A\): first sample a source \(s\) with probability \(\alpha _s\), then sample a document from \(Q_s\).

The mixture weights specify sampling frequency. They need not equal the fraction of bytes stored in each source. Resampling, epoch boundaries, or temperature-based weights must therefore be recorded as part of \(\alpha \) or its schedule.

Let \(\dagger \) denote rejection. A document filter is a declared map \[ F:\mathcal D_{\rm doc}\longrightarrow \mathcal D_{\rm doc}\cup \{\dagger \}. \] It may retain a document, return a transformed document, or reject it. The lift \(F_{\#}\) applies \(F\) to a finite corpus and removes every \(\dagger \).

A filter specification includes the features it reads, thresholds, transformation rules, and implementation revision. An acceptance rate alone does not determine the retained document law.

Let \(\mathcal M_{\rm doc}\) be the set of finite multisets of documents. A deduplication operator is a map \[ D:\mathcal M_{\rm doc}\longrightarrow \mathcal M_{\rm doc} \] that selects retained representatives according to a declared duplicate relation. The relation may compare complete documents or spans.

The declaration includes normalization, similarity threshold, cluster construction, and representative tie-breaking. Different choices can remove different training events even when they report the same duplicate rate.

For a protected evaluation collection \(\mathcal E\), an evaluation decontamination operator is a map \[ C_{\mathcal E}:\mathcal M_{\rm doc}\longrightarrow \mathcal M_{\rm doc} \] that removes documents or spans matching \(\mathcal E\) under a declared matching rule.

The protected collection, normalization, match granularity, threshold, and removal policy are part of the operator. Decontamination against one released benchmark does not establish independence from every related task.

Let \(\mathcal C_{\mathrm {raw}}\in \mathcal M_{\rm doc}\) be a finite corpus drawn from the source mixture in Definition 2. Using the declared document filter, deduplication operator, and evaluation decontamination operator, an ordered pretraining data pipeline returns \[ \mathcal C_{\mathrm {clean}} =C_{\mathcal E}\left (D\left (F_{\#}(\mathcal C_{\mathrm {raw}})\right )\right ). \] Changing the order can change the retained corpus. The pipeline therefore records the order and exact revisions in addition to the three operators.

The source mixture and cleaning pipeline determine different parts of the data law. The mixture in Definition 2 describes how raw documents are proposed. The construction in Definition 6 determines which proposals survive and how they are transformed. Together with all random seeds and thresholds, they induce a cleaned document law \(Q_{\mathrm {clean}}\).

Kaplan et al. describe the concrete WebText2 dataset in [kaplan2020scaling, sec. 2.3]. Hoffmann et al. report the MassiveText sources and mixture in [hoffmann2022training, Appendix A, Table A1]. A finite released corpus is one realization of such choices, and an undocumented change in them is not identified by a loss--compute curve.

Fix the context length \(T\) from Notation 1. A context-window sampler draws cleaned text from the law described in Remark 7 and tokenizes it with Definition [ftip-000E]. It then chooses a declared offset and returns a length-\(T\) token sequence \(x_{1:T}\) together with metadata that identifies document boundaries.

If fewer than \(T\) tokens remain, the sampler must specify whether it pads, drops the suffix, or joins another document. These choices change both the distribution of contexts and the prediction mask.

A packed training record is a tuple \(\zeta =(x,m,c,M^{\mathrm {doc}})\). It places tokenized segments \(x^{(1)},\ldots ,x^{(r)}\) into the length-\(T\) token array \(x\), records a prediction mask \(m\), and records a segment identifier \(c_t\in \{0,1,\ldots ,r\}\) at each position. The value \(c_t=0\) denotes padding. Its document mask is \[ M^{\mathrm {doc}}_{ts}= \begin {cases} 0,&c_t=c_s\ne 0\text { and }s\leq t,\\ -\infty ,&\text {otherwise}. \end {cases} \] Replacing the causal mask in Definition [ftip-000S] by this mask prevents one packed document from attending to another. We write \(x_t(\zeta )\), \(m_t(\zeta )\), and \(M^{\mathrm {doc}}(\zeta )\) for the corresponding fields.

The prediction mask \(m_t(\zeta )\) from Notation 1 is one only at positions whose target token is retained for training. Padding, a segment's first token when no predecessor is present, and any deliberately excluded control token receive mask value zero.

[kaplan2020scaling], Section 2.2, uses fixed-length sequence batches.

[phuong2022formal], Section 7, gives the causal language-model objective.

The proposed data interface additionally specifies the sampler, segment identifiers, document mask, and prediction mask. Those boundary choices are not determined by the high-level training accounts.

These objects describe one admissible data interface. A concrete training run must replace them with its offset, padding, joining, masking, and packing rules.

Let \(\zeta =(x,m,c,M^{\mathrm {doc}})\) be a packed training record from Definition 9. Write \(\pi _{\theta ,M^{\mathrm {doc}}(\zeta )}(\,\cdot \mid x_{<t}(\zeta ))\) for the next-token law of Definition [ftip-000H] when every decoder attention layer uses the declared document mask. For a predicted position \(t\geq 2\), its masked token negative log-likelihood is \[ \ell _t(\theta ;\zeta ) =-m_t(\zeta )\log \pi _{\theta ,M^{\mathrm {doc}}(\zeta )} \left (x_t(\zeta )\mid x_{<t}(\zeta )\right ), \] A masked position has zero contribution; an unmasked position penalizes the log probability computed under the same segment boundaries that defined the packed example.

Let \(Q_T\) be the law of packed examples induced by Remark 7--Definition 9, and assume that its expected number of predicted positions is positive. The population pretraining objective is the expected total token loss divided by the expected number of predicted tokens: \[ L_{\mathrm {pop}}(\theta ) =\frac { \mathbb E_{\zeta \sim Q_T} \left [\sum _{t=2}^{T}\ell _t(\theta ;\zeta )\right ] }{ \mathbb E_{\zeta \sim Q_T} \left [\sum _{t=2}^{T}m_t(\zeta )\right ] }, \] using expectation as fixed in Notation [ftip-000A].

Packing documents into one row requires a block-causal mask so that each token attends only within its own document and to earlier positions there.

For a packed row, set \(M_{ij}=0\) only when tokens \(i,j\) belong to the same document and \(j\leq i\); otherwise set \(M_{ij}=-\infty \). Thus a token in document \(B\) cannot inspect any token in document \(A\), even when \(A\) precedes \(B\) in storage.

The within-document triangle instantiates the causal mask in Definition [ftip-000S]. The block boundary instantiates the packing convention in Definition 9. Document-isolated attention is the declared convention in this packed row, not a universal description of pretraining systems.

For a finite training sample of packed records \(\mathcal D_N=(\zeta ^{(i)})_{i=1}^{N}\) with at least one unmasked target token, the empirical pretraining objective is \[ L_N(\theta ) =\frac { \sum _{i=1}^{N}\sum _{t=2}^{T} \ell _t(\theta ;\zeta ^{(i)}) }{ \sum _{i=1}^{N}\sum _{t=2}^{T}m_t(\zeta ^{(i)}) }. \] This is a per-predicted-token average. Reusing or resampling examples changes the stochastic optimization path even when the displayed finite-sample function is unchanged.

The autoregressive token loss follows [phuong2022formal, Section 7]. Kaplan et al. report language-model loss with parameter and compute variables in [kaplan2020scaling, sec. 2.1]. The population ratio-of-expectations in Definition 12 and finite per-token ratio in Definition 14 are proposed normalization conventions with explicit prediction masks. Neither cited source defines those exact ratios.

A theorem or experiment using another record weighting, length weighting, or expectation order must state that change rather than reuse these symbols.

At update \(k\), let \(\mathcal B_k\) be the minibatch named in Notation 1, and let \(N_k=\sum _{\zeta \in \mathcal B_k}\sum _{t=2}^{T}m_t(\zeta )\) be its number of predicted tokens. For \(N_k>0\), the minibatch gradient estimator is \[ g_k=\frac 1{N_k} \sum _{\zeta \in \mathcal B_k}\sum _{t=2}^{T} \nabla _\theta \ell _t(\theta _k;\zeta ), \] where \(\ell _t\) is defined in Definition 11. Whether \(g_k\) is unbiased for the gradient of Definition 12 depends on the window sampler and minibatch weighting. Other sampling schemes require their own bias statement.

Two equal-length records make a full empirical loss directly comparable with a minibatch estimate, while exposing the assumptions behind unbiasedness.

Let \(\ell _{ij}\) denote the token loss at position \(j\) of record \(i\). The full token-average loss is the mean of all \(\ell _{ij}\), while a one-record minibatch averages only the losses from its sampled record. Uniform sampling of equal-length records makes the latter unbiased for the former.

The preceding loss and estimator definitions apply to this equal-length record setup. Equal record lengths make the estimator unbiased here; unequal lengths, padding, dependence, and optimizer noise remain outside its scope.

An optimizer state \(s_k\) is the collection of persistent variables, other than the model parameters \(\theta _k\), that an update rule uses after step \(k\). For Adam, \(s_k\) contains the step index and exponential moving averages of the gradient and squared gradient.

The state is initialized by a declared value \(s_0\). Two runs with the same checkpoint \(\theta _k\) but different optimizer states need not have the same next update.

A parameter update rule is a specified map \(U_k\) that takes the current parameters \(\theta _k\), optimizer state \(s_k\), gradient estimate \(g_k\), and scheduled hyperparameters \(\eta _k\), and returns \[ (\theta _{k+1},s_{k+1}) =U_k(\theta _k,s_k,g_k,\eta _k). \] The learning rate, momentum coefficients, numerical stabilizers, clipping, and weight decay belong to \(\eta _k\) or to the declared form of \(U_k\); they are not determined by the loss alone.

A pretraining run is not determined by minimization of the empirical objective \(L_N\) in Definition 14. It includes the initialization, data order, window offsets, dropout and other model randomness, numerical precision, distributed reduction order, optimizer state, every hyperparameter schedule, and a stopping rule. These objects determine a distribution over final parameters even when the source mixture and nominal objective agree.

The training configuration of [vaswani2017attention, sec. 5.3] and the scaling experiments of [kaplan2020scaling, sec. 2.2] report concrete instances of these choices. An optimization result depends on which randomness it averages over and whether the stopping time is fixed or data-dependent.

A base-model artifact is an executable tuple \[ M_0=(\tau ,\delta ,\mathsf {cfg},\theta _K,\mathsf {num}), \] where \(\tau ,\delta \) are the tokenizer and detokenizer of Definition [ftip-000E]; \(\mathsf {cfg}\) fixes the architecture of Definition [ftip-0012]; \(\theta _K\) is the stopped pretraining checkpoint; and \(\mathsf {num}\) fixes the numerical and inference conventions required to evaluate its next-token law.

The tokenizer model is part of the executable system by Definition [ftip-000E]. A weight file without its vocabulary, architecture configuration, or output convention does not determine the law \(\pi _{\theta _K}\) in Definition [ftip-000H]. Training-data and optimizer provenance may accompany the artifact, but they are not runtime inputs.

Section 6 of [phuong2022formal] describes decoder architecture. Sections 3.1 and 3.5 of [kudo2018sentencepiece] treat the tokenizer as a separate model. Public model releases additionally distribute weights, configuration, and numerical conventions. The proposed artifact model in Definition 21 includes these runtime ingredients because none alone determines an executable next-token law.

Training provenance accompanies the artifact but is not itself a runtime argument. Later causal claims must distinguish the executable endpoint from the process that produced it.

An executable-artifact manifest separates runtime components from evidence about how the corresponding weights were trained.

The pinned DeepSeek-V4-Flash-0731 configuration [⧉] declares architecture and numeric-format fields. Sections 2.4 and 4 of the V4 family report [⧉] describe a mixed Muon/AdamW optimizer allocation and pretraining data construction. The exact 0731 data mixture, optimizer schedule, and post-preview training delta remain undisclosed by those sources.