Skip to main content
marin-community
Projects
marin
Reports
DCLM Debugging

DCLM Debugging

David Leo Wright Hall
Created on September 25|Last edited on May 12

Section 1


100k200k300k400kStep2000004000006000008000001000000120000014000001600000
dclm_70b_0919_2
dclm_22b0903
dclm_7b_llama3_0828
dclm_7b0828
dclm 7b replication LR=4e-4/beta2=0.95
1k2k3k4k5k6k7k8k9k10k20k30k40k50k60k70k80k90k100k200k300k400kStep345
dclm_70b_0919_2
dclm_22b0903
dclm_7b_llama3_0828
dclm_7b0828
dclm 7b replication LR=4e-4/beta2=0.95
4G5G6G7G8G9G10G20G30G40G50G60G70G80G90G100G200G300G400G500G600G700G800G900G1T2Tthroughput/total_tokens34
dclm_70b_0919_2
dclm_22b0903
dclm_7b_llama3_0828
dclm_7b0828
dclm 7b replication LR=4e-4/beta2=0.95
500G1T1.5T2Tthroughput/total_tokens2345678910
dclm_70b_0919_2
dclm_22b0903
dclm_7b_llama3_0828
dclm_7b0828
dclm 7b replication LR=4e-4/beta2=0.95
110010kStep2345678910
dclm_70b_0919_2
dclm_22b0903
dclm_7b_llama3_0828
dclm_7b0828
dclm 7b replication LR=4e-4/beta2=0.95
Run set
6


Created with ❤️ on Weights & Biases.

https://wandb.ai/marin-community/marin/reports/DCLM-Debugging--Vmlldzo5NDk4MDI1
Made with Weights & Biases. Sign up or log in to create reports like this one.