data efficiency hyperparams

Created on June 22|Last edited on September 8
Comment
﻿
grouped runsoverparametrized wdoverparametrized wd (ensembles)1.4b ensembles (based on rough optimal hps)1.4b, 200m tokens, 16x epoched, ensembles1.4b, 200m tokens, 32x and 16x, 6.4 wd
﻿
﻿
grouped runs﻿
eval/dclm/loss
eval/dclm/loss
0.050.060.070.080.090.10.20.30.40.50.60.70.80.9run_progress45
1_4b4k-209Mx8-dclm-cos-lr0.0010-wd3.20-bs64   1.4b, 209M
1_4b4k-209Mx8-dclm-cos-lr0.0010-wd1.60-bs64   1.4b, 209M
1_4b4k-209Mx8-dclm-cos-lr0.0010-wd0.80-bs64   1.4b, 209M
train/loss
train/loss
0.050.060.070.080.090.10.20.30.40.50.60.70.80.9run_progress345678
1_4b4k-209Mx8-dclm-cos-lr0.0010-wd3.20-bs64   1.4b, 209M
1_4b4k-209Mx8-dclm-cos-lr0.0010-wd1.60-bs64   1.4b, 209M
1_4b4k-209Mx8-dclm-cos-lr0.0010-wd0.80-bs64   1.4b, 209M
 
150m opt hp0
 
300m opt hp0
 
600m opt hp4
 
1.4b opt hp7
 
1.4b, 419M3
1.4b, 209M3
Run set 70
﻿
overparametrized wd﻿
 
1.4b, 209M, lr0.0033
1.4b, 209M, lr0.00110
 
Run set 310
﻿
﻿
overparametrized wd (ensembles)﻿
wd6.40532
wd3.20532
wd1.6532
wd0.8532
﻿
﻿
1.4b ensembles (based on rough optimal hps)﻿
1.4b, 1.7B seed tokens1779
1.4b, 838M seed tokens1779
1.4b, 419M seed tokens1779
1.4b, 209M seed tokens1779
﻿
﻿
﻿
1.4b, 1.7B seed tokens532
1.4b, 838m532
1.4b, 419m532
1.4b, 209m532
﻿
﻿
1.4b, 200m tokens, 16x epoched, ensembles﻿
Run set1779
﻿
﻿
﻿
Run set532
﻿
﻿
1.4b, 200m tokens, 32x and 16x, 6.4 wd﻿
x321779
x16-wd6.41779
﻿
﻿
Add a comment