Skip to main content
ai2-llm
Projects
olmoe
Reports
Plot: QKNorm revisited

Plot: QKNorm revisited

olmoe17-8x1b-final-eps-noqk uses no QK-Norm but RMSNorm with weights olmoe17-8x1b-final-eps uses non-parametric QK-Norm & RMSNorm with weights
Niklas Muennighoff
Created on July 6|Last edited on August 29

6k8k10k12k14kStep2.752.82.852.9
olmoe17-8x1b-final-eps-noqk Run set
olmoe17-8x1b-final-eps Run set
olmoe17-8x1b-final-eps Run set
1k2k3k4k5kStep46810
olmoe17-8x1b-final-eps-noqk Run set
olmoe17-8x1b-final-eps Run set
olmoe17-8x1b-final-eps Run set
Run set
3
Run set 2


Created with ❤️ on Weights & Biases.

https://wandb.ai/ai2-llm/olmoe/reports/Plot-QKNorm-revisited--Vmlldzo4NTc2NTIz
Made with Weights & Biases. Sign up or log in to create reports like this one.