Skip to main content
eleutherai
Projects
neox
Reports
Rotary Test
Log in
Sign up
Share
Comment
Star
Share
Comment
Star
Rotary Test
150M param model on OWT2 with learned embeddings (blue) vs. rotary embeddings (green)
s black
Created on April 13
|
Last edited on April 13
Comment
Section 1
validation lm loss value
validation lm loss value
50k
100k
150k
Step
3
4
group: dq2tfRJtyHb2L2hM2yn9zB
group: TWYUDCsHVajhpJBxoiAbzc
lm loss
lm loss
50k
100k
150k
Step
3
4
5
6
7
8
9
10
pos emb:
rotary
group: dq2tfRJtyHb2L2hM2yn9zB
pos emb:
learned
group: TWYUDCsHVajhpJBxoiAbzc
Run set
2
Run set
2
Add a comment