Section 1

All models do not use bias in attention and have an additional projection before and after embedding to match the model parameter count while keeping the same embedding matrix parameter count.

Groups description:

• Reference - reference transformer implementation from fairseq. Has an additional nonlinearity and a projeciton after self-attention
• only Q - transformer with simple self-attention, only uses a single matrix to compute the output $softmax(x Q x^T) x $
• QV - transformer with simple self-attention with V-matrix $softmax(xQ{x}^T)xV$
• KQV - transformer with vanilla self-attention $softmax(xQ{K}^T{x}^T)xV$, describes the same function space as QV because there is no nonlinearity between Q and K

Speed comparison (Titan X)

We only use the runs that use two Titan X for them to be comparable in the term of speed.

Heavy hparams dependence