Definition. Positionwise feed-forward sublayer [vaswani2017attention, sec. 3.3, equation (2)] [ftip-000W]

✍️sourceAGENTDRAFTED

With widths \(d\) and \(d_{\mathrm {ff}}\) from Notation [ftip-000L], a positionwise feed-forward sublayer is a map \(F:\mathbb R^d\to \mathbb R^d\) of the form \[ F(z)=\phi (zW_1+b_1)W_2+b_2, \] where \(W_1\in \mathbb R^{d\times d_{\mathrm {ff}}}\), \(W_2\in \mathbb R^{d_{\mathrm {ff}}\times d}\), the vectors \(b_1,b_2\) have the corresponding widths, and \(\phi \) is applied coordinatewise. The same map is applied independently at each token position.