Definition. Query, key, and value projections [vaswani2017attention, sec. 3.2.1] [ftip-000P]

✍️sourceAGENTDRAFTED

Let \(X\in \mathbb R^{T\times d}\) use the notation of Notation [ftip-000L]. For attention head \(h\), choose learned matrices \[ W_h^Q,W_h^K\in \mathbb R^{d\times d_h}, \qquad W_h^V\in \mathbb R^{d\times d_v}, \] where \(d_v\) is the value width of one head. The query, key, and value matrices are \[ Q_h=XW_h^Q,\qquad K_h=XW_h^K,\qquad V_h=XW_h^V. \] Their rows associate a query, key, and value vector with each token position.