Skip to main content
marin-community
Projects
marin
Reports
Olmo 1b replication report v2
Log in
Sign up
Share
Comment
Star
Olmo 1b replication report v2
David Leo Wright Hall
,
Ivan Zhou
Created on July 15
|
Last edited on May 12
Comment
Various Loss
log/log eval/loss
log/log eval/loss
1k
2k
3k
4k
5k
6k
7k
8k
9k
10k
20k
30k
40k
50k
60k
70k
80k
90k
100k
200k
300k
Step
3
4
5
6
1B Dolma 1.7 with Shuffle Buffer + Weight Decay Masking 0610
1B Dolma 1.7 with Shuffle Buffer 0604
1B Dolma 1.7 w/ Fixed Mixture Sharding
1B Dolma 1.7 fixed Mixture Key Sharding
1B Dolma 1.7 with First Exhausted
1B Dolma 1.7 pre-mixed
1B strange config, Dolma 1.7 Premix
Run set
8
Run set
7
Add a comment