Skip to main content
marin-community
Projects
marin
Reports
654 Scaling Law
Log in
Sign up
Share
Comment
Star
654 Scaling Law
https://github.com/stanford-crfm/marin/issues/654
David Leo Wright Hall
Created on December 11
|
Last edited on May 12
Comment
Section 1
c4en/bpb vs log(gflops)
c4en/bpb vs log(gflops)
70G
80G
90G
100G
200G
300G
400G
500G
600G
700G
800G
900G
1T
2T
log(gflops)
0.9
0.95
1
bpb
tootsie-scaling-1024-b45766
tootsie-scaling-768-d17a90
tootsie-scaling-2048-1ed392
tootsie-scaling-512-81c36c
tootsie-scaling-1536-350a3a
tootsie-scaling-1024-f4e4be
c4en/bpb vs log(tokens)
c4en/bpb vs log(tokens)
50G
60G
70G
80G
90G
100G
200G
throughput/total_tokens
0.9
0.95
1
bpb
tootsie-scaling-1024-b45766
tootsie-scaling-768-d17a90
tootsie-scaling-2048-1ed392
tootsie-scaling-512-81c36c
tootsie-scaling-1536-350a3a
tootsie-scaling-1024-f4e4be
Run set
6
Run set
6
Run set
6
Add a comment