Rotary Test 3

150M param model on OWT2 with learned embeddings (blue) vs. rotary embeddings (green) vs. rpe (brown) vs. rpe with caching (peach)

s black

Created on April 15|Last edited on April 15

Comment

﻿
Section 1﻿
validation lm loss value
validation lm loss value
50k100k150kStep34
group: c8h5RaxCLDDWmEdewhGjTs
group: EaSTz5R9RzXz9QjfVHBWcc
group: dq2tfRJtyHb2L2hM2yn9zB
group: TWYUDCsHVajhpJBxoiAbzc
lm loss
lm loss
50k100k150kStep345678910
pos emb:  rpe    group: c8h5RaxCLDDWmEdewhGjTs
pos emb:  rpe    group: EaSTz5R9RzXz9QjfVHBWcc
pos emb:  rotary    group: dq2tfRJtyHb2L2hM2yn9zB
pos emb:  learned    group: TWYUDCsHVajhpJBxoiAbzc
Run set4
﻿
﻿
﻿
Run set4
﻿
﻿

Add a comment