Skip to main content
eleutherai
Projects
neox
Reports
TopK Attention

TopK Attention

Experiments using topk operator within self attention block
s black
Created on March 25|Last edited on March 25

Normal (red) vs. TopK HARD (orange) vs. TopK Soft (TBD)


2k4k6k8kStep405060708090
group: KXtakAjjB3Q58vacVKpX8A
group: SJvwbBjUVfknhhoiCQUmmu
Run set
12



Run set
12


Created with ❤️ on Weights & Biases.

https://wandb.ai/eleutherai/neox/reports/TopK-Attention--Vmlldzo1NTg2MTM
Made with Weights & Biases. Sign up or log in to create reports like this one.