PROSE-PDE: A Foundation Model for Solving and Extrapolating Partial Differential Equations

Have you ever wondered how complex phenomena like fluid flows, heat transfer, or even the formation of patterns in nature can be described mathematically? The answer lies in partial differential equations (PDEs), which are powerful tools used to model and understand intricate spatio-temporal processes across various scientific domains. However, solving these equations analytically can be a daunting task, often requiring computational methods or simulations. This is where machine learning comes into play, offering a novel approach to tackle PDE problems by learning to approximate the solutions directly from data.

Traditionally, solving PDEs involved numerical methods that could be computationally expensive, especially for complex systems or high-dimensional problems. Recently, researchers have been exploring using neural networks to learn the mappings between input conditions and output solutions of PDEs. However, most existing approaches are limited to specific equations or struggle to generalize to unseen systems without fine-tuning.

In a remarkable step forward, a team of researchers has developed PROSE-PDE (Figure 3), a multimodal neural network model designed to be a foundation for solving a wide range of time-dependent PDEs, including nonlinear diffusive, dispersive, conservation laws, and wave equations. The key innovation lies in PROSE-PDE’s ability to learn multiple operators simultaneously and extrapolate physical phenomena across different governing systems. But how does it work?

At the core of PROSE-PDE is a novel technique called Multi-Operator Learning (MOL). Unlike traditional approaches that learn a single operator for a specific PDE, MOL trains a unified model to approximate multiple operators simultaneously. This is achieved through symbolic encoding (shown in Figure 2), where equations are represented as trainable tokens in a Polish notation format. The model can then learn to associate these symbolic representations with the corresponding data solutions.

The PROSE-PDE architecture comprises five main components: Data Encoder, Symbol Encoder, Feature Fusion, Data Decoder, and Symbol Decoder. The Data Encoder processes the input data sequence, while the Symbol Encoder handles the symbolic equation guesses. These encoded features are then fused together, allowing information exchange between the data and symbolic representations. The Data Decoder synthesizes the fused features to predict the output solutions, and the Symbol Decoder refines and generates the corresponding symbolic expressions.

However, what sets PROSE-PDE apart is its ability to extrapolate physical features across different systems. Through extensive experiments, the researchers demonstrated that PROSE-PDE could generalize to unseen model parameters, predict variables at future time points, and even handle entirely new physical systems not encountered during training. This remarkable capability is attributed to the model’s ability to abstract and transfer underlying physical laws from the training data.

The evaluation results are promising, with PROSE-PDE achieving low relative prediction errors (< 3.1%) and high R^2 scores on a diverse set of 20 PDEs. Moreover, the model successfully recovered unknown equations with an error of only 0.549%. These findings pave the way for a general-purpose foundation model for scientific applications capable of efficiently solving complex PDE problems and extrapolating physical insights across different systems.

While the current work focuses on one-dimensional time-dependent PDEs, the researchers envision extending PROSE-PDE to multi-dimensional and non-time-dependent equations. As data becomes increasingly abundant in scientific domains, the potential for such foundation models to revolutionize our understanding and modeling of complex physical phenomena is truly exciting. 

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 40k+ ML SubReddit