Tootsie 8B phoenix cooldown ("starling")

See https://github.com/stanford-crfm/marin/issues/600 for narrative

Created on April 24|Last edited on May 12

Comment

﻿
﻿Main Tootsie Report﻿
﻿GH Issue #600 (with Narrative for tootsie overall.)﻿
﻿GH Issue #977 (This run in particular)
﻿Google Sheet with all Named Checkpoints﻿
Big Idea:From monumental-jellyfish
Core tootsie DCLM mix to 3.7 T tokens
Cooldown on Dolmino HQ data (without synth math or Flan) to 4.8T tokens
rewarm over 2000 steps to peak LR, training on 50/50 DCLM Mix and (nemotron+starcoder)
train for a while
start cooling down on our best mix from #847 We also added zloss and increased BS to 16.8M tokens
NB that the final run (lime green, phase 3) starts from a slightly earlier checkpoint from the red run since I messed the red one up.
﻿
Lineage Runs﻿
This set of panels contains runs from a private project, which cannot be shown in this report
﻿
﻿
﻿
Run set6662
﻿
﻿

Add a comment