Definition. Sparse mixture-of-experts routing [ftip-000X]
Definition. Sparse mixture-of-experts routing [ftip-000X]
Let \(J\geq 1\) and \(1\leq k\leq J\). Let \(F_1,\ldots ,F_J:\mathbb R^d\to \mathbb R^d\) be feed-forward experts and let \(g:\mathbb R^d\to \mathbb R^J\) be a router. For a hidden vector \(z\), let \(S_k(z)\) be the indices of the \(k\) largest coordinates of \(g(z)\), breaking equal scores by a fixed total order on expert indices. A sparse mixture-of-experts layer returns \[ F_{\mathrm {MoE}}(z) =\sum _{j\in S_k(z)}\alpha _j(z)F_j(z), \qquad \alpha _j(z)= \frac {\exp g_j(z)}{\sum _{i\in S_k(z)}\exp g_i(z)}. \] Only the selected experts are evaluated for that token.