Remark. Window sampling, document masks, and packing [ftip-006K]
Remark. Window sampling, document masks, and packing [ftip-006K]
[kaplan2020scaling], Section 2.2, uses fixed-length sequence batches.
[phuong2022formal], Section 7, gives the causal language-model objective.
The proposed data interface additionally specifies the sampler, segment identifiers, document mask, and prediction mask. Those boundary choices are not determined by the high-level training accounts.
These objects describe one admissible data interface. A concrete training run must replace them with its offset, padding, joining, masking, and packing rules.