Skip to main content
ai2-llm
Projects
olmoe
Reports
Plot: MoE vs Dense

Plot: MoE vs Dense

Niklas Muennighoff
Created on August 5|Last edited on August 29

5k10k15k20k25k30kStep2.533.54
olmoe-8x1b-newhp-newds-cx5-fine1 Run set
olmoe-8x1b-newhp-newds-cx5-fine1 Run set
olmo-1b-newhp-newds-cx5 Run set
olmo-1b-newhp-newds-cx5 Run set
5k10k15k20k25k30kStep23456
olmoe-8x1b-newhp-newds-cx5-fine1 Run set
olmoe-8x1b-newhp-newds-cx5-fine1 Run set
olmo-1b-newhp-newds-cx5 Run set
olmo-1b-newhp-newds-cx5 Run set
Run set
4
Run set 2


Created with ❤️ on Weights & Biases.

https://wandb.ai/ai2-llm/olmoe/reports/Plot-MoE-vs-Dense--Vmlldzo4OTM0Mjkx
Made with Weights & Biases. Sign up or log in to create reports like this one.