An Exploration of whether models with the same/similar loss can be harder to SFT

We present a rigorous study across 10 optimizers on optimization speed in language modeling speedup and reveal that the realistic speedup is significantly lower than claimed. We analyze why this is the case and show new observations about optimization.

See https://github.com/stanford-crfm/marin/issues/600 for narrative

See https://github.com/stanford-crfm/marin/issues/600 for narrative

See https://github.com/stanford-crfm/marin/issues/600 for narrative

See https://github.com/stanford-crfm/marin/issues/600 for narrative

See https://github.com/stanford-crfm/marin/issues/600 for narrative

See https://github.com/stanford-crfm/marin/issues/600 for narrative

See https://github.com/stanford-crfm/marin/issues/600 for narrative

See https://github.com/stanford-crfm/marin/issues/600 for narrative

https://github.com/stanford-crfm/marin/issues/654

Fineweb-EDU 1.4b Tokenizers: * llama2 (~32k) * llama3 (~128k) * neox (~50k)

## 150m Fast run winner: bf4172 or f2c1ce (2-ish hours) Best run: fcd416 ## 330m Fast run winner: 1631b7 (2.5 hours) Best run: e3faf7

* V1 dclm_7b0820: Llama 2 architecture 7b with DCLM's optimizer hyper parameters. Very spiky ,so we reduce the LR in V2 * v2 dclm_7b0820-2: reduced LR * v3 dclm_7b0821: kept reduced LR, added shuffle buffer (100k) and reduced beta2 to match dclm paper * v4 dclm_7b0821-3: v3 but old beta2 (0.999) * v5 dclm_7b0822-1: v3 but with dclm LR (so, beta2=0.95) Other runs Conclusions: * beta2=0.95 important! * higher lr might not matter?

Our attempt at replicating olmo 7b. We used the olmo tokenizer and dolma 1.7. Our architecture was almost identical except we use RMSNorm rather than LayerNorm (both versions do not learn either a bias term or a gain term on the layer norm)

Data browser link: https://marlin-subtle-barnacle.ngrok-free.app/experiment?path=gs%3A//marin-us-central2/experiments/exp636_stackexchange_vs_hqwebpages-a374bc.json

Reproduced Olmo 7b SFT with the following hyperparameters: Batch