ML Reproducibility Challenge Spring 2021
All information and guidelines for the ML Reproducibility Challenge (MLRC) 2021 Spring Edition. Made by Angelica Pan using Weights & Biases
Welcome to the 2021 Spring Edition of the ML Reproducibility Challenge!
Supporting the Reproducibility Challenge participants
At W&B, we strongly believe that research should be reproducible and accessible, so we’re excited to support people participating in the PaperswithCode Reproducibility Challenge
. We believe this initiative is very important, and we’re happy to do what we can to support participants.
Specifically, we're looking to support efforts to make sure that every NeurIPS, ACL, CVPR, ECCV, ICLR, ICML, and EMNLP paper is reproducible and that the claims made are verifiable by tracking the model performance and predictions in W&B. If you'd like to help us achieve that, please join the #ml-reproducibility channel in our Slack community
, where we coordinate these efforts.
We realise that some of these papers are computationally expensive to reproduce, and we're happy to offer participants $500 per paper reproduced, as long as it meets these guidelines
How to participate
We understand that the current landscape of deep learning and machine learning research has matured in terms of compute requirements, and that this can prevent many researchers and students from participating in the reproducibility challenge. We will award every participant who meets the above listed guidelines with $500 towards compute costs. Exact details regarding the form and method of compensation will be communicated with eligible participants.
W&B Reproducibility Grant
To facilitate maximum experimentation and thorough validation, for the Spring edition of MLRC, we are introducing the W&B Reproducibility Grant wherein we will provide $500 upfront to each selected participant for reproducing their selected papers while adhering to all the guidelines stated above.
To submit an application for the W&B Reproducibility Grant, please fill and submit this Google Form
The grants are provided on a rolling basis. However, the deadline for submitting an application for the grant is July 14th, 2021.
The grant winners shall be selected on the basis of their aim & motivation of reproducing the particular paper of choice and their prior works in the same space.
The grant of $500 is designated to cover compute costs for the reproducibility of one (1) paper of choice. If a participant is reproducing multiple papers, the prize money/ reimbursement of $500 for the remaining papers shall be provided post submission of the W&B reports for those remaining papers.
An individual participating in the W&B MLRC 2021 Spring is limited to reproducing only 2 papers of choice at maximum.
A total of 10 grants shall be awarded for the W&B MLRC Spring 2021.
- Challenge Starts: April 1st, 2021
Final submission deadline: July 20th, 2021
You can submit your report at any time before the deadline.
We understand that authors may require more time to polish and add all their findings to their reports. Therefore, we will accept reports up to two weeks after the challenge deadline (August 5th, 2021)
Post Submission Editing completion: September 15th, 2021
Authors notified for compensation: September 21st, 2021
Join the #ml-reproducibility channel in our Slack community
, where you can collaborate on paper reproductions and ask questions about reproducing models or using W&B.
Also, please see the helpful list of resources
that the folks at the Reproducibility Challenge have put together.
Here are some Reproducibility Challenge Reports from the Winter 2020 edition:
We will soon release the full list of accepted reports from the Winter 2020 edition.
Integrating Weights & Biases ?
W&B makes it easier to reproduce papers by allowing you to log all your runs in one centralised dashboard and see the training progress live. W&B also provides you with features that allow you to:
perform hyper-parameter optimisation using Sweeps
perform dataset/ model versioning using Artifacts
perform dataset evaluation and inspection using DSViz
condense your project into a report with all your charts and panels exported from your dashboard using WandB Report
and much more.
To learn more, check out the following two YouTube playlists:
You can also view our docs
for extensive code-based documentation, along with detailed examples.
If you're excited to start, join the #ml-reproducibility channel in our Slack community
where we coordinate our reproducibility efforts. You'll also find code and analysis guidelines for ensuring that the papers are reproducible.