Meta-Auto-Decoder: a Meta-Learning-Based Reduced Order Model for Solving Parametric Partial Differential Equations

doi:10.1007/s42967-023-00293-7

Abstract

Abstract: Many important problems in science and engineering require solving the so-called parametric partial differential equations (PDEs), i.e., PDEs with different physical parameters, boundary conditions, shapes of computational domains, etc. Typical reduced order modeling techniques accelerate the solution of the parametric PDEs by projecting them onto a linear trial manifold constructed in the offline stage. These methods often need a predefined mesh as well as a series of precomputed solution snapshots, and may struggle to balance between the efficiency and accuracy due to the limitation of the linear ansatz. Utilizing the nonlinear representation of neural networks (NNs), we propose the Meta-Auto-Decoder (MAD) to construct a nonlinear trial manifold, whose best possible performance is measured theoretically by the decoder width. Based on the meta-learning concept, the trial manifold can be learned in a mesh-free and unsupervised way during the pre-training stage. Fast adaptation to new (possibly heterogeneous) PDE parameters is enabled by searching on this trial manifold, and optionally fine-tuning the trial manifold at the same time. Extensive numerical experiments show that the MAD method exhibits a faster convergence speed without losing the accuracy than other deep learning-based methods.

Key words: Parametric partial differential equations(PDEs), Meta-learning, Reduced order modeling, Neural networks (NNs), Auto-decoder

Zhanhong Ye, Xiang Huang, Hongsheng Liu, Bin Dong. Meta-Auto-Decoder: a Meta-Learning-Based Reduced Order Model for Solving Parametric Partial Differential Equations[J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 1096-1130.

TrendMD

References

[1] Antoniou, A., Edwards, H., Storkey, A.: How to train your MAML. In: Seventh International Conference on Learning Representations (2019)
[2] Baydin, A.G., Pearlmutter, B.A., Radul, A.A., Siskind, J.M.: Automatic differentiation in machine learning: a survey. J. Mach. Learn. Res. 18, 1-43 (2018)
[3] Benner, P., Ohlberger, M., Cohen, A., Willcox, K.: Model Reduction and Approximation. Society for Industrial and Applied Mathematics, Philadelphia, PA (2017)
[4] Bhattacharya, K., Hosseini, B., Kovachki, N.B., Stuart, A.M.: Model reduction and neural networks for parametric PDEs. arXiv:2005.03180 (2020)
[5] Cai, S., Wang, Z., Wang, S., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks for heat transfer problems. J. Heat Transfer 143(6), 060801 (2021)
[6] Chan, E.R., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-GAN: periodic implicit generative adversarial networks for 3D-aware image synthesis. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021). https://doi.org/10.1109/cvpr46437.2021.00574
[7] Chen, Y., Dong, B., Xu, J.: Meta-MgNet: meta multigrid networks for solving parameterized partial differential equations. J. Comput. Phys. 455, 110996 (2022)
[8] Cohen, A., DeVore, R.: Approximation of high-dimensional parametric PDEs. Acta Numerica 24, 1-159 (2015)
[9] Cohen, A., DeVore, R., Petrova, G., Wojtaszczyk, P.: Optimal stable nonlinear approximation. Foundat. Comput. Math. 22, 607-648 (2022)
[10] Cohen, A., DeVore, R., Schwab, C.: Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE’s. Anal. Appl. 9(1), 11-47 (2011). https://doi.org/10.1142/S0219530511001728
[11] DeVore, R.A., Howard, R., Micchelli, C.A.: Optimal nonlinear approximation. Manuscripta Mathematica 63, 469-478 (1989)
[12] Dupont, E., Kim, H., Eslami, S.M.A., Rezende, D.J., Rosenbaum, D.: From data to functa: your data point is a function and you can treat it like one. In: Proceedings of the 39th International Conference on Machine Learning (2022)
[13] E, W., Yu, B.: The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat. 6(1), 1-12 (2018)
[14] Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning. PMLR, pp. 1126-1135 (2017)
[15] Franco, N.R., Manzoni, A., Zunino, P.: A deep learning approach to reduced order modelling of parameter dependent partial differential equations. Math. Comput. 92(340), 483-524 (2023)
[16] Fresca, S., Dede’, L., Manzoni, A.: A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs. J. Sci. Comput. 87(2), 61 (2021). https://doi.org/10.1007/s10915-021-01462-7
[17] Fresca, S., Gobat, G., Fedeli, P., Frangi, A., Manzoni, A.: Deep learning-based reduced order models for the real-time simulation of the nonlinear dynamics of microstructures. Int. J. Numer. Methods Eng. 123(20), 4749-4777 (2022). https://doi.org/10.1002/nme.7054
[18] Gedney, S.D.: Introduction to the finite-difference time-domain (FDTD) method for electromagnetics. Syn. Lectures Comput. Electromag. 6(1), 1-250 (2011)
[19] Greif, C., Urban, K.: Decay of the Kolmogorov N-width for wave problems. Appl. Math. Lett. 96, 216-222 (2019). https://doi.org/10.1016/j.aml.2019.05.013
[20] Huang, X., Liu, H., Shi, B., Wang, Z., Yang, K., Li, Y., Weng, B., Wang, M., Chu, H., Zhou, J., Fan, Y., Hua, B., Chen, L., Dong, B.: Solving partial differential equations with point source based on physics-informed neural networks. arXiv:2111.01394 (2021)
[21] Huang, X., Ye, Z., Liu, H., Shi, B., Wang, Z., Yang, K., Li, Y., Weng, B., Wang, M., Chu, H., Zhou, J., Fan, Y., Hua, B., Chen, L., Dong, B.: Meta-auto-decoder for solving parametric partial differential equations. Adv. Neural Infor. Process. Syst. 35, 23426-23438 (2022)
[22] Jiang, C.M., Esmaeilzadeh, S., Azizzadenesheli, K., Kashinath, K., Mustafa, M., Tchelepi, H.A., Marcus, P., Prabhat, M., Anandkumar, A.: MeshfreeFlownet: a physics-constrained deep continuous space-time super-resolution framework. In: International Conference for High Performance Computing, Networking, Storage, and Analysis (2020)
[23] Khoo, Y., Lu, J., Ying, L.: Solving parametric PDE problems with artificial neural networks. Eur. J. Appl. Math. 32(3), 421-435 (2021)
[24] Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)
[25] Kochkov, D., Smith, J., Alieva, A., Wang, Q., Brenner, M., Hoyer, S.: Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences of the United States of America. 118(21), 1 (2021)
[26] Lee, K., Carlberg, K.T.: Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. J. Comput. Phys. 404, 108973 (2020). https://doi.org/10.1016/j.jcp.2019.108973
[27] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., Anandkumar, A.: Fourier neural operator for parametric partial differential equations. arXiv:2010.08895 (2020)
[28] Liszka, T., Orkisz, J.: The finite difference method at arbitrary irregular grids and its application in applied mechanics. Comput. Struct. 11(1/2), 83-95 (1980)
[29] Liu, X., Zhang, X., Peng, W., Zhou, W., Yao, W.: A novel meta-learning initialization method for physics-informed neural networks. arXiv:2107.10991 (2021)
[30] Liu, Y., Li, J., Sun, S., Yu, B.: Advances in Gaussian random field generation: a review. Comput. Geosci. 23(5), 1011-1047 (2019)
[31] Long, Z., Lu, Y., Dong, B.: PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network. J. Comput. Phys. 399, 108925 (2019)
[32] Long, Z., Lu, Y., Ma, X., Dong, B.: PDE-Net: learning PDEs from data. In: International Conference on Machine Learning, PMLR, pp. 3208-3216 (2018)
[33] Lu, Y., Chen, H., Lu, J., Ying, L., Blanchet, J.: Machine learning for elliptic PDEs: fast rate generalization bound, neural scaling law and minimax optimality. In: International Conference on Learning Representations (2022)
[34] Lu, L., Jin, P., Karniadakis, G.E.: DeepONet: learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv:1910.03193 (2019)
[35] Lu, L., Meng, X., Cai, S., Mao, Z., Goswami, S., Zhang, Z., Karniadakis, G.E.: A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data. Comput. Methods Appl. Mech. Eng. 393, 114778 (2022)
[36] Lu, L., Pestourie, R., Yao, W., Wang, Z., Verdugo, F., Johnson, S.G.: Physics-informed neural networks with hard constraints for inverse design. SIAM J. Sci. Comput. 43, B1105-B1132 (2021)
[37] Mehta, I., Gharbi, M., Barnes, C., Shechtman, E., Ramamoorthi, R., Chandraker, M.: Modulated periodic activations for generalizable local functional representations. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021). https://doi.org/10.1109/iccv48922.2021.01395
[38] Nichol, A., Schulman, J.: Reptile: a scalable metalearning algorithm. arXiv:1803.02999 (2018)
[39] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165-174 (2019)
[40] Psaros, A.F., Kawaguchi, K., Karniadakis, G.E.: Meta-learning PINN loss functions. J. Comput. Phys. 458, 111121 (2022). https://doi.org/10.1016/j.jcp.2022.111121
[41] Quarteroni, A., Manzoni, A., Negri, F.: Reduced Basis Methods for Partial Differential Equations: an Introduction. Springer, Switzerland (2015)
[42] Raissi, M.: Deep hidden physics models: deep learning of nonlinear partial differential equations. J. Mach. Learn. Res. 19(1), 932-955 (2018)
[43] Schneider, J.B.: Understanding the Finite-Difference Time-Domain Method. School of Electrical Engineering and Computer Science. Washington State University 28 (2010)
[44] Sirignano, J., Spiliopoulos, K.: DGM: a deep learning algorithm for solving partial differential equations. J. Comput. Phys. 375, 1339-1364 (2018)
[45] Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. Adv. Neural Informat. Process. Syst. 33, 7462-7473 (2020)
[46] Stanziola, A., Arridge, S.R., Cox, B.T., Treeby, B.E.: A Helmholtz equation solver using unsupervised learning: application to transcranial ultrasound. J. Comput. Phys. 441, 110430 (2021). https://doi.org/10.1016/j.jcp.2021.110430
[47] Tran, H., Webster, C.G., Zhang, G.: Analysis of quasi-optimal polynomial approximations for parameterized PDEs with deterministic and stochastic coefficients. Numerische Mathematik 137(2), 451-493 (2017). https://doi.org/10.1007/s00211-017-0878-6
[48] Treeby, B.E., Cox, B.T.: k-Wave: Matlab toolbox for the simulation and reconstruction of photoacoustic wave fields. J. Biomed. Optics 15(2), 021314 (2010)
[49] Wang, C., Li, S., He, D., Wang, L.: Is

\begin{document}$ L^2 $\end{document}

physics informed loss always suitable for training physics informed neural network? In: Advances in Neural Information Processing Systems (2022)
[50] Wang, S., Wang, H., Perdikaris, P.: Learning the solution operator of parametric partial differential equations with physics-informed deeponets. arXiv:2103.10974 (2021)
[51] Yoon, J., Kim, T., Dia, O., Kim, S., Bengio, Y., Ahn, S.: Bayesian model-agnostic meta-learning. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7343-7353 (2018)
[52] Zang, Y., Bao, G., Ye, X., Zhou, H.: Weak adversarial networks for high-dimensional partial differential equations. J. Comput. Phys. 411, 109409 (2020)
[53] Zienkiewicz, O.C., Taylor, R.L., Nithiarasu, P., Zhu, J.: The Finite Element Method, vol. 3. McGraw-hill, London (1977)