ECMWF Accelerates Machine Learning Research in Weather Forecasting With W&B
"W&B has been helpful for us to try loads of different approaches and understand which ideas and what changes led to improvements"
Matthew Chantry
Machine Learning Coordinator
Machine Learning at ECMWF
Weather is a complex puzzle with many evolving pieces—it can be pretty tricky just to keep an eye on it. And when it comes to making a reliable forecast, well, that’s another beast of its own.
Considered to have the most accurate global weather model, the European Centre for Medium-Range Weather Forecasts (ECMWF) has been exploring machine learning in weather forecasting for the past few years. A key milestone was achieved earlier this year when ECMWF made operational use of modern ML, using neural networks to monitor observations.
As ECMWF continues its research of applying ML to improve weather prediction, there’s a need for constant experimentation and iteration. With multiple experiments running concurrently, it becomes essential to use tools that can effectively track, compare, and analyze the results of different runs. Weights & Biases emerged as the platform of choice with its robust experiment tracking capabilities and rich, interactive visualizations, helping the team facilitate data-driven decision-making and model optimization.
A Distributed ML Team
“We’re very aware that what we’re undertaking here is more than a step up in complexity and would require a much larger size of collaborators,” said Matthew Chantry, Machine Learning Coordinator at ECMWF. “This isn’t just one or two people working on a project together, but more like ten or even more collaborating on a code base to train models, compare results, and decide what to do next.
On top of managing a growing ML team, ECMWF also has to work with the fact that its ML engineers are distributed across the organization. Each practitioner is embedded into different groups to leverage the expertise and knowledge of various disciplines. Past that, the team is also spread across multiple countries and timezones. The challenge then becomes facilitating alignment and ensuring everyone is on the same page and working cohesively.
Designed for seamless collaboration, Weights & Biases serves as the hub for consolidating the entire team’s ML efforts and project insights. Everything from the latest git commit, hyperparameters, model weights, and metrics are stored in one shared workspace. This gives visibility into everyone’s work and makes it easy to iterate on past experiments, leading to better model performance, higher accuracy, and more reliable results. Leveraging W&B as the single source of truth, the ML team at ECMWF can work collectively and autonomously with confidence.
AIFS
Building on several notable companies making rapid progress in ML-based weather forecasts, ECMWF launched its first forecasting system based entirely on ML, with live forecasts available on their website. Developed as a companion to the ECMWF’s existing Integrated Forecasting System (IFS), the Artificial Intelligence/Integrated Forecasting System (AIFS) aims to expand the organization’s applications of ML to Earth system modeling. While the model is still in its alpha phase, it already has a resolution of approximately one degree (111 km) and can make predictions for wind, temperature, humidity, geopotential, and more.
“W&B has played a crucial role in supporting the work around AIFS,” said Chantry. “We’re able to log various media types and create plots for a range of forecasts. This makes it easy to compare and contrast results, ensuring there’s no spurious patterns and identifying the best models.”
Example maps of truth earth humidity and AIFS prediction, logged to W&B during training to evaluate spatial consistency of the model.
The development of the AIFS requires iterative work – which means experiments, a lot of them. To keep track of their model development efforts, the team relies on W&B. Using the W&B dashboard, ECMWF can organize and visualize experiments in real time, storing the data and results in one convenient place. With a record of what worked and what didn’t, the team can expedite their process of putting the best models into production.
“W&B has been helpful for us to try loads of different approaches and understand which ideas and what changes led to improvements,” said Chantry.
The Future of Earth System Prediction
Today, ECMWF’s forecasting system is recognized as one of the best in the world. Adopting machine learning into their work represents a significant step forward on ECMWF’s journey to provide more skillful forecasts of the weather phenomena that affect us most.
As the team continues to explore potential areas of ML in weather forecasting, having tools that support rapid iteration and experimentation is critical. By harnessing W&B’s inherent collaboration capabilities, customizable visualizations, and comprehensive model training records, ECMWF accelerates the pace of innovation substantially.
“W&B encourages us to experiment more, and we can go from idea to testing much more quickly,” said Chantry.