Skip to main content
eleutherai
Projects
neox
Reports
Rotary Test 2
Log in
Sign up
Share
Comment
Star
Share
Comment
Star
Rotary Test 2
150M param model on OWT2 with learned embeddings (blue) vs. rotary embeddings (green) vs. rpe (brown)
s black
Created on April 15
|
Last edited on April 15
Comment
Section 1
validation lm loss value
validation lm loss value
50k
100k
150k
Step
3
4
group: EaSTz5R9RzXz9QjfVHBWcc
group: dq2tfRJtyHb2L2hM2yn9zB
group: TWYUDCsHVajhpJBxoiAbzc
lm loss
lm loss
50k
100k
150k
Step
3
4
5
6
7
8
9
10
pos emb:
rpe
group: EaSTz5R9RzXz9QjfVHBWcc
pos emb:
rotary
group: dq2tfRJtyHb2L2hM2yn9zB
pos emb:
learned
group: TWYUDCsHVajhpJBxoiAbzc
Run set
3
Run set
3
Add a comment