Neurips 2019 was once again an overwhelming festival of all flavors and patterns of machine learning. It’s impossible to see or capture in its entirety, and the late-night conversations and coffee break encounters are more magical than the talks anyway.
As a field, I think we’re shifting from breakneck growth and hype towards a starting semblance of maturity, or at least self-awareness, with the rising themes of ethics, robustness, and interpretability. Big data, big compute, and big ad-driven attention-usurping tech are out, and active learning, application-specific hardware and urgent real-world applications like social good and climate change are in.
The niches are proliferating— I noticed way more biotech (protein sequences as a language), experimental cognitive psychology and neuroscience, and of course graphs. Classic math and computational theory are still relevant, but you can be effective without them—fewer equations and more straightforward step-by-step instructions make your work more reproducible, quoth Edward Raff, the hero who implemented two hundred and fifty-five machine learning papers without looking at any supporting code .
I particularly liked this table from his poster. It’s surprising that there is no relation between reproduction and the year a paper was published, and that papers with detailed pseudo code and no pseudo code were equally reproducible.
This year’s posters included a breath of fresh air— large TL;DR takeaways visible from a distance.
The reproducibility exercise demonstrates the necessity of the update step in the Earth-scale metalearning program: the more insights, workflows, and code libraries we can reuse, the faster the machine learning community can converge on better practices and better models. W&B strives to support this with tools for automatic logging, easy visualization, and flexible mind-mapping of your workflow—from data to model to training loop to hyperparameter sweeps and so on up the meta-ladder, all organized in one place.
The Weights & Biases team enjoyed the conference, chatting with hundreds of users about what we’re doing well and what to fix next. Internet allowing, we ran a fun interactive demo raffle at our booth where users could configure one guess at—or randomly roll—the best hyperparameters for a small convnet (a toy version of a serious benchmark on drought prediction from satellite). We successfully nerd-sniped, improved val_acc, and delighted folks with real-time training plots.
We’re excited to keep simplifying and enhancing collaboration in deep learning— please message Carey (firstname.lastname@example.org) if you have ideas of how to make W&B more useful for you! Stay tuned for deeper dives into Neurips, more benchmarks on meaningful problems, and perhaps more speculative forays into consciousness and AGI.
 Celeste Kidd, How to Know. [link] This talk yields the most generalizable gradient updates of the conference, imo.
 Rao, R. et al. Evaluating Protein Transfer Learning with TAPE. [link]
 Raff, E. A Step Toward Quantifying Independently Reproducible Machine Learning Research [link]
 Yoshua Bengio, Andrew Ng, Carla Gomes, Lester Mackey, Jeff Dean: Panel - Climate Change: A Grand Challenge for ML [link]