Skip to main content
marin-community
Projects
marin
Reports
Olmo 1b replication report v2

Olmo 1b replication report v2

David Leo Wright Hall, Ivan Zhou
Created on July 15|Last edited on May 12

Various Loss


1k2k3k4k5k6k7k8k9k10k20k30k40k50k60k70k80k90k100k200k300kStep3456
1B Dolma 1.7 with Shuffle Buffer + Weight Decay Masking 0610
1B Dolma 1.7 with Shuffle Buffer 0604
1B Dolma 1.7 w/ Fixed Mixture Sharding
1B Dolma 1.7 fixed Mixture Key Sharding
1B Dolma 1.7 with First Exhausted
1B Dolma 1.7 pre-mixed
1B strange config, Dolma 1.7 Premix
Run set
8



Run set
7


Created with ❤️ on Weights & Biases.

https://wandb.ai/marin-community/marin/reports/Olmo-1b-replication-report-v2--Vmlldzo4NjgyOTE0
Made with Weights & Biases. Sign up or log in to create reports like this one.