ML Reproducibility Challenge Spring 2021
All information and guidelines for the ML Reproducibility Challenge (MLRC) 2021 Spring Edition
Created on April 26|Last edited on July 15
Comment
Welcome to the 2021 Spring Edition of the ML Reproducibility Challenge!
Supporting the Reproducibility Challenge participants
At W&B, we strongly believe that research should be reproducible and accessible, so we’re excited to support people participating in the PaperswithCode Reproducibility Challenge. We believe this initiative is very important, and we’re happy to do what we can to support participants.
Specifically, we're looking to support efforts to make sure that every NeurIPS, ACL, CVPR, ECCV, ICLR, ICML, and EMNLP paper is reproducible and that the claims made are verifiable by tracking the model performance and predictions in W&B. If you'd like to help us achieve that, please join the #ml-reproducibility channel in our Slack community, where we coordinate these efforts.
We realise that some of these papers are computationally expensive to reproduce, and we're happy to offer participants $500 per paper reproduced, as long as it meets these guidelines.
How to participate
- Pick a paper from the list of accepted papers of the supported conferences and check this list to confirm that the paper hasn't been claimed yet.
- Use existing code released by the authors, or create a minimal implementation, to verify the main claims and results made by the paper.
- Write a short summary of your findings and ablation studies as a WandB Report – was the paper reproducible? Did you uncover any key insights?
- Submit your findings:
- Our W&B editors will later contact you to go through an editing cycle before featuring the report on our Gallery. This is referred to as the "Post Submission Editing".
- Authors will also be invited to share their work at the W&B Deep Learning Salon and the Weekly AMAs hosted in the W&B community.
Compensation
We understand that the current landscape of deep learning and machine learning research has matured in terms of compute requirements, and that this can prevent many researchers and students from participating in the reproducibility challenge. We will award every participant who meets the above listed guidelines with $500 towards compute costs. Exact details regarding the form and method of compensation will be communicated with eligible participants.
W&B Reproducibility Grant
To facilitate maximum experimentation and thorough validation, for the Spring edition of MLRC, we are introducing the W&B Reproducibility Grant wherein we will provide $500 upfront to each selected participant for reproducing their selected papers while adhering to all the guidelines stated above.
To submit an application for the W&B Reproducibility Grant, please fill and submit this Google Form.
FAQ:
- The grants are provided on a rolling basis. However, the deadline for submitting an application for the grant is July 14th, 2021.
- The grant winners shall be selected on the basis of their aim & motivation of reproducing the particular paper of choice and their prior works in the same space.
- The grant of $500 is designated to cover compute costs for the reproducibility of one (1) paper of choice. If a participant is reproducing multiple papers, the prize money/ reimbursement of $500 for the remaining papers shall be provided post submission of the W&B reports for those remaining papers.
- An individual participating in the W&B MLRC 2021 Spring is limited to reproducing only 2 papers of choice at maximum.
- A total of 10 grants shall be awarded for the W&B MLRC Spring 2021.
Timeline
- Challenge Starts: April 1st, 2021
- Final submission deadline: July 20th, 2021
- You can submit your report at any time before the deadline.
- We understand that authors may require more time to polish and add all their findings to their reports. Therefore, we will accept reports up to two weeks after the challenge deadline (August 5th, 2021)
- Post Submission Editing completion: September 15th, 2021
- Authors notified for compensation: September 21st, 2021
Resources
Join the #ml-reproducibility channel in our Slack community, where you can collaborate on paper reproductions and ask questions about reproducing models or using W&B.
Also, please see the helpful list of resources that the folks at the Reproducibility Challenge have put together.
Here are some Reproducibility Challenge Reports from the Winter 2020 edition:
We will soon release the full list of accepted reports from the Winter 2020 edition.
Integrating Weights & Biases ?
W&B makes it easier to reproduce papers by allowing you to log all your runs in one centralised dashboard and see the training progress live. W&B also provides you with features that allow you to:
- condense your project into a report with all your charts and panels exported from your dashboard using WandB Report,
and much more.
To learn more, check out the following two YouTube playlists:
Get started
If you're excited to start, join the #ml-reproducibility channel in our Slack community where we coordinate our reproducibility efforts. You'll also find code and analysis guidelines for ensuring that the papers are reproducible.
People
- Diganta Misra
- Lavanya Shukla
- Morgan McGuire
- Angelica Pan
- Justin Tenuto
Questions
Add a comment