PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations

Abstract

TL;DR: We propose Physics-Informed Gaussians, an adaptive mesh representation where Gaussian parameters are trained to dynamically adjust their positions and shapes.

The approximation of Partial Differential Equations (PDEs) using neural networks has seen significant advancements through Physics-Informed Neural Networks (PINNs). Despite their straightforward optimization framework and flexibility in implementing various PDEs, PINNs often suffer from limited accuracy due to the spectral bias of Multi-Layer Perceptrons (MLPs), which struggle to effectively learn high-frequency and non-linear components. Recently, parametric mesh representations in combination with neural networks have been investigated as a promising approach to eliminate the inductive biases of neural networks. However, they usually require very high-resolution grids and a large number of collocation points to achieve high accuracy while avoiding overfitting issues. In addition, the fixed positions of the mesh parameters restrict their flexibility, making it challenging to accurately approximate complex PDEs. To overcome these limitations, we propose Physics-Informed Gaussians (PIGs), which combine feature embeddings using Gaussian functions with a lightweight neural network. Our approach uses trainable parameters for the mean and variance of each Gaussian, allowing for dynamic adjustment of their positions and shapes during training. This adaptability enables our model to optimally approximate PDE solutions, unlike models with fixed parameter positions. Furthermore, the proposed approach maintains the same optimization framework used in PINNs, allowing us to benefit from their excellent properties. Experimental results show the competitive performance of our model across various PDEs, demonstrating its potential as a robust tool for solving complex PDEs.

Architecture

(a) PINN directly takes input coordinates (four collocation points) as inputs and produces outputs. (b) Parametric grids first map input coordinates to output feature vectors. Each vertex in the grids holds learnable parameters, and output features are extracted through interpolation schemes. (c) The proposed PIG consists of numerous Gaussians moving around within the input domain, and their shapes change dynamically during training. Each Gaussian has learnable parameters, and a feature vector for an input coordinate is the weighted sum of the learnable parameters based on the distance to the Gaussians.

Method

1. Learnable Gaussian Feature Embedding $\texttt{FE}_\phi$

Let $\phi = \{(\mu_i, \Sigma_i, f_i): i=1, \dots, N\}$ be the set of Gaussian model parameters, where $\mu_i \in \mathbb{R}^{d}$ is a position of a Gaussian, $\Sigma_i \in \mathbb{S}^{d}_{++}$ is a covariance matrix, and each Gaussian has a learnable feature embedding $f_i \in \mathbb{R}^{k}$. Given an input coordinate $x \in \mathbb{R}^d$, the learnable Gaussian feature embedding $\texttt{FE}_\phi:\mathbb{R}^d \rightarrow \mathbb{R}^{k}$ is extracted as follows:

\[ \texttt{FE}_\phi(x) = \sum_{i=1}^N f_i G_i(x), \quad G_i(x) = e^{-\frac{1}{2}(x - \mu_i)^\top \Sigma_i^{-1} (x - \mu_i)}. \]

where $k$ is the input dimension of MLP, $N$ is the number of Gaussians and $G_i$ represents the $i$-th Gaussian function. $\texttt{FE}_\phi$ maps an input coordinate to a feature embedding by a weighted sum of the individual features $f_i$ of each Gaussian. To enhance the expressive capability, we can use different Gaussians for each feature dimension. Further details are provided in Appendix A.1. All Gaussian parameters $\phi$ are learnable and iteratively updated throughout the training process. This dynamic adjustment, akin to adaptive mesh-based numerical methods, optimizes the structure of the underlying Gaussian functions to accurately approximate the solution functions.

2. Solution Approximation with Gaussians followed by a Lightweight Neural Network $\texttt{NN}_\theta$

Once the features are extracted, a neural network processes the feature to produce the solution outputs.

\[ u_{\phi,\theta}(x) = \texttt{NN}_\theta(\texttt{FE}_\phi(x)). \]

where $\texttt{NN}_\theta$ is a lightweight MLP with the parameter $\theta$. We employed a single hidden layer MLP with a limited number of hidden units, resulting in negligible additional computational costs.

Visualized Results

Klein-Gordon Equation

$$\large{\frac{\partial^2 u}{\partial t^2}-\left(\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial y^2}\right)+u^2=f}$$

Flow-Mixing Equation

$$\large{\frac{\partial u}{\partial t}+a\frac{\partial u}{\partial x}+b\frac{\partial u}{\partial y}=0}$$

Helmholtz Equation

$$\large{\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial y^2}+k^2u=q}$$

2D Helmholtz equation with a low wavenumber $(a_1, a_2) = (1, 4)$. PIG achieved a relative $L^2$ error of $2.22 \times 10^{-5}$, while the parametric fixed grid method PIXEL reached a relative $L^2$ error of $8.63\times 10^{-4}$.

2D Helmholtz equation with a high wavenumber $(a_1, a_2) = (10, 10)$. PIG achieved a relative $L^2$ error of $7.09\times 10^{-3}$, while the parametric fixed grid method PIXEL reached a relative $L^2$ error of $7.47\times 10^{-2}$. PINN failed to converge.

Lid-Driven Cavity Equation

$$\nabla \cdot \mathbf{u} = 0$$ $$\rho (\mathbf{u} \cdot \nabla)\mathbf{u} = -\nabla p + \mu \nabla^2 \mathbf{u}$$

Lid-driven cavity flow problem. PIG achieved $4.04 \times 10^{-4}$ relative $L^2$ error whereas the baseline parametric grid method PGCAN resulted in $1.22\times 10^{-3}$.

BibTeX

@misc{kang2024pigphysicsinformedgaussiansadaptive,
      title={PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations}, 
      author={Namgyu Kang and Jaemin Oh and Youngjoon Hong and Eunbyung Park},
      year={2024},
      eprint={2412.05994},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2412.05994}, 
}