Skip to main content
ai2-llm
Projects
olmoe
Reports
Why Nikla's MoE w new tokenizer spikey val?

Why Nikla's MoE w new tokenizer spikey val?

Kyle Lo
Created on August 1|Last edited on August 1

Section 1


5k10k15k20k25k30kStep20406080100
olmoe-8x1b-newhp-newds-cx5-fine1-newtok Run: olmoe-8x1b-newhp-newds-cx5-fine1-newtok
olmoe-8x1b-newhp-newds-cx5-fine1-datafix Run set 2
5k10k15k20k25k30kStep123
olmoe-8x1b-newhp-newds-cx5-fine1-datafix Run set 2
olmoe-8x1b-newhp-newds-cx5-fine1-newtok Run: olmoe-8x1b-newhp-newds-cx5-fine1-newtok
5k10k15k20k25k30kStep46810
olmoe-8x1b-newhp-newds-cx5-fine1-newtok Run: olmoe-8x1b-newhp-newds-cx5-fine1-newtok
olmoe-8x1b-newhp-newds-cx5-fine1-datafix Run set 2
5k10k15k20k25k30kStep22.533.54
olmoe-8x1b-newhp-newds-cx5-fine1-newtok Run: olmoe-8x1b-newhp-newds-cx5-fine1-newtok
olmoe-8x1b-newhp-newds-cx5-fine1-datafix Run set 2
5k10k15k20k25k30kStep20406080
olmoe-8x1b-newhp-newds-cx5-fine1-datafix Run set 2
olmoe-8x1b-newhp-newds-cx5-fine1-newtok Run: olmoe-8x1b-newhp-newds-cx5-fine1-newtok
Run: olmoe-8x1b-newhp-newds-cx5-fine1-newtok
1
Run set 2
1
Run set 3
Run set 4


Created with ❤️ on Weights & Biases.

https://wandb.ai/ai2-llm/olmoe/reports/Why-Nikla-s-MoE-w-new-tokenizer-spikey-val---Vmlldzo4ODkzNzM2
Made with Weights & Biases. Sign up or log in to create reports like this one.