We evaluate standard and homebrew techniques for parameter sharing in GPT-like language models on OpenWebText dataset; conducted partly by Tim Dettmers, Aleksandr Borzunov, Michael Diskin and Max Ryabinin