Remark. Dense and routed feed-forward computation [ftip-000Y]
Remark. Dense and routed feed-forward computation [ftip-000Y]
The dense sublayer in Definition [ftip-000W] evaluates one learned map for every token. The sparsely gated construction of Shazeer et al., Section 2, selects among multiple experts and introduces an additional routing and load-balancing problem [⧉]. The top-\(k\) selection rule is described in Definition [ftip-000X].
Parameter count, active parameter count, and executed operations therefore need not agree. A post-training comparison involving routed models must state which quantity is held fixed and whether the router itself is updated.