Exploiting AI to enhance extreme weather and climate information

The Destination Earth (DestinE) Initiative of the European Commission is accelerating the development of machine learning techniques that can dramatically improve the accuracy of extreme weather simulations and climate change projections.

“Machine learning is revolutionising our industry,” says Florian Pappenberger, deputy director-general at the European Centre for Medium-Range Weather Forecasts (ECMWF). “I’ve never seen anything else revolutionise the industry like this,” he adds.  Whereas, weather forecasting used to improve 10% every 10 years, Pappenberger says machine learning-based forecasts have improved 90% over the past four years and are now competitive with traditional weather forecasting models in several aspects.

A subset of artificial intelligence (AI), effective machine learning requires training on high quality training data —the kind of data that traditional weather and climate models and observing systems, such as the ERA5 reanalysis produced by ECMWF within the framework of the Copernicus Climate Change Service, provide in abundance.

Advances in computing power are also enabling machine learning to be employed effectively for weather and climate forecasting. The European High Performance Computing Joint Undertaking (EuroHPC JU) provides several of the fastest supercomputers on the planet to DestinE.“It’s only when we had the right computing infrastructure combined with the right data and top-level scientists that everything got such a big boost,” notes Pappenberger.

ECMWF is implementing the DestinE initiative together with the European Space Agency (ESA) and the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) under the leadership of the European Commission’s Directorate-General for Communications Networks, Content and Technology (CNECT).  Overall, the initiative involves 116 organisations from more than 25 countries across Europe.

Helping to deliver the Green Deal

Financed by the European Union in the context of the Green Deal, the DestinE initiative is creating  “digital twins” of the Earth.  The first two digital twins, developed by ECMWF together with its many partners create a window on extreme events on a time scale of 2-4 days ahead, and a window on the climate on a timescale of a few decades ahead. Running on the EuroHPC JU’s supercomputers, they create a high-definition view of the earth system and enable national and European decision-makers to better respond and adapt to the environmental challenges of our changing climate.  

"The DestinE initiative's substantial investment in machine learning, along with the higher resolution weather and climate datasets produced by the DestinE Digital Twins, will significantly enhance the use of machine learning in meteorology and climate sciences,” says Pappenberger. “This effort, combined with the collaborative endeavours on machine learning at ECMWF and in our Member and Cooperating States, positions us to secure European leadership in machine learning applications for the Earth system. It also aids the creation of a machine learning-based digital twin of the Earth."

AI in DestinE encompasses a wide range of activities including helping to understand uncertainty and making the digital twins more accessible and interactive. Pappenberger notes the entire team working on the DestinE initiative is helping with these activities either through generating high resolution data for training or working directly on AI activities.

Improving knowledge of uncertainty through AI

Mariana Clare, a mathematician at ECMWF, is using machine learning techniques to quantify the “uncertainties” that have always been a feature of weather forecasting. “Weather is chaotic and this makes it inherently uncertain,” she says. “There are also additional uncertainties that come from the forecasting itself, for example, uncertainty in the starting point of the weather forecasts and from the model when you run a simulation. As you forecast further out into the future, this uncertainty and randomness become larger and therefore it is important that you capture it. If you are a policymaker, it’s important to both be aware of the likely scenarios, some more probable than others, and to be aware of potential worst-case scenarios and know how probable those are, too.”  

In traditional models, insights into uncertainty are obtained by running multiple simulations in parallel, slightly changing their starting point and parts of the model and seeing how the weather forecast reacts to this change. A weather forecaster’s job is then to take these scenarios and make a weather forecast of what is most likely to happen, whilst assessing the risk of other scenarios occurring. But this is a very computationally-intensive approach because each different scenario is essentially an independent forecast. Using machine learning, algorithms can take a single forecast and then introduce “randomness and noise” after-the-fact, using a fraction of the computing power needed for the traditional approach, according to Clare. 

Interactivity and accessibility

At the same time, the arrival of AI-based tools, such as chatbots, can help make the weather and climate information produced for example by DestinE’s digital twins more easily accessible for users. For example, DestinE will build on existing developments to create a chatbot for weather and climate applications, changing the way that DestinE’s users and stakeholders can interact with data.

“We’re going to develop machine learning-based elements that will allow us to send the model to users instead of the forecast itself. This so-called forecast-in-a-box will streamline data pipelines and make forecasts more accessible,” Pappenberger, says with a note of pride.

Such tools, in turn, could help stimulate private initiatives that build on the foundation of the public investment in the DestinE initiative. The initiative already has more than 60 private sector partners and Pappenberger foresees tenders that will make its work more accessible to even more partners.