Example. Packing two documents without cross-document attention [ftip-001D]
Example. Packing two documents without cross-document attention [ftip-001D]
Packing documents into one row requires a block-causal mask so that each token attends only within its own document and to earlier positions there.
For a packed row, set \(M_{ij}=0\) only when tokens \(i,j\) belong to the same document and \(j\leq i\); otherwise set \(M_{ij}=-\infty \). Thus a token in document \(B\) cannot inspect any token in document \(A\), even when \(A\) precedes \(B\) in storage.
The within-document triangle instantiates the causal mask in Definition [ftip-000S]. The block boundary instantiates the packing convention in Definition [ftip-001A]. Document-isolated attention is the declared convention in this packed row, not a universal description of pretraining systems.