Skip to main content
stanford-mercury
Projects
suhas-data-efficiency
Reports
Data efficiency scaling laws

Data efficiency scaling laws

Suhas Kotha
Created on June 11|Last edited on June 22

Showing first 10 runs
00.20.40.60.8run_progress00.00020.00040.00060.00080.001
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed0
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed1
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed2
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed4
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed4
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed0
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed2
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed1
Showing first 50 runs
0.050.060.070.080.090.10.20.30.40.50.60.70.80.9run_progress34
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed0
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed1
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed2
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed4
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed4
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed0
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed2
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed1
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.40-seed0
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.40-seed4
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.40-seed1
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.40-seed3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.40-seed2
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd3.20-seed0
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd3.20-seed1
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd3.20-seed4
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd3.20-seed3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd3.20-seed2
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd1.60-seed3
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd1.60-seed0
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd1.60-seed4
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd1.60-seed1
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd1.60-seed2
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.80-seed3
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.80-seed4
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.80-seed0
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.80-seed1
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.80-seed2
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd3.20-seed4
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd3.20-seed0
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd3.20-seed1
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd3.20-seed3
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd3.20-seed2
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.40-seed0
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.40-seed1
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.40-seed3
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.40-seed4
1_4b4k-838Mx8-dclm-cos-lr0.0010-wd0.40-seed2
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed3
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed9
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed5
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed6
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed0
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed4
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed7
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed1
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed2
300m4k-1.7Bx32-dclm-cos-lr0.0030-wd0.80-seed8
Showing first 10 runs
0.050.060.070.080.090.10.20.30.40.50.60.70.80.9run_progress3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed0
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed1
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed2
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd0.80-seed4
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed4
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed0
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed2
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed3
1_4b4k-1.7Bx8-dclm-cos-lr0.0010-wd1.60-seed1
initial debugging
0
epoch scaling bs 64
10
weight decay tuning
32
full scaling laws
744
ensemble members
214
600m hp
0

Main lessons so far
  • Batch size is super important
  • lr 3e-3 is good for 64, smaller lr for larger batch size maybe?

Created with ❤️ on Weights & Biases.

https://wandb.ai/stanford-mercury/suhas-data-efficiency/reports/Data-efficiency-scaling-laws--VmlldzoxMzE3MjUzNQ
Made with Weights & Biases. Sign up or log in to create reports like this one.