Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models.

Samira Abnar Harshay Shah Dan Busbridge Alaaeldin Mohamed Elnouby Ali Joshua M. Susskind Vimal Thilak Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models. 2025 January abs/2501.12370 CoRR https://doi.org/10.48550/arXiv.2501.12370 db/journals/corr/corr2501.html#abs-2501-12370 streams/journals/corr