Our fast.ai community submission to the Reproducibility challenge 2020: "Reformer: The Efficient Transformer" by Nikita Kitaev, Łukasz Kaiser and Anselm Levskaya, accepted to ICLR 2020.

A fast.ai community submission to the Reproducibility challenge 2020: "Reformer: The Efficient Transformer" by Nikita Kitaev, Łukasz Kaiser and Anselm Levskaya, accepted to ICLR 2020.

Submission to Reproducibility challenge 2020: "Reformer: The Efficient Transformer" by Nikita Kitaev, Łukasz Kaiser and Anselm Levskaya, accepted to ICLR 2020.

Comparing shared/separate qk transformer lm for the synthetic task

Brief explanation of Rev Layers operation

LSH attention performance as a function of hashing rounds number

We demonstrate how the memory allocation of the various reformer variants compare to the transformer during training

Results from the synthetic task of the Reformer paper: https://arxiv.org/abs/2001.04451

We demonstrate how the memory allocation of the various reformer variants compare to the transformer during training

We demonstrate how LSH evaluation speed changes with increased sequence lenghts.

We'll give a brief explanation of LSH (locality sensing hashing) attention. For a step-by-step walk through of LSH-attention we refer to our project documentation: https://arampacha.github.io/reformer_fastai/exploration.LSH.html