Abstract
Large and sparse feed-forward layers (S-FFN) such as Mixture-of-Experts (MoE) have proven effective in scaling up Transformers model size for pretraining large language models. By only activating part of the FFN parameters conditioning on input, S-FFN improves generalization performance while keeping training and inference costs (in FLOPs) fixed. In this work, we analyzed two major design choices of S-FFN: the memory block (a.k.a. expert) size and the memory block selection method under a general conceptual framework of sparse neural memory. Using this unified framework, we compare several S-FFN architectures for language modeling and provide insights into their relative efficacy and efficiency. We found a simpler selection method — Avg-K that selects blocks through their mean aggregated hidden states, achieving lower perplexity in language model pretraining compared to existing MoE architectures including Switch Transformer (Fedus et al., 2021) and HashLayer (Roller et al., 2021).- Anthology ID:
- 2023.emnlp-main.930
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15038–15061
- Language:
- URL:
- https://preview.aclanthology.org/remove-affiliations/2023.emnlp-main.930/
- DOI:
- 10.18653/v1/2023.emnlp-main.930
- Cite (ACL):
- Zeyu Liu, Tim Dettmers, Xi Lin, Veselin Stoyanov, and Xian Li. 2023. Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15038–15061, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model (Liu et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/remove-affiliations/2023.emnlp-main.930.pdf