On Loss Functionals for Physics-Informed Neural Networks for Steady-State Convection-Dominated Convection-Diffusion Problems

doi:10.1007/s42967-024-00433-7

Abstract

Abstract: Solutions of convection-dominated convection-diffusion problems usually possess layers, which are regions where the solution has a steep gradient. It is well known that many classical numerical discretization techniques face difficulties when approximating the solution to these problems. In recent years, physics-informed neural networks (PINNs) for approximating the solution to (initial-)boundary value problems ((I)BVPs) received a lot of interest. This paper studies various loss functionals for PINNs that are especially designed for convection-dominated convection-diffusion problems and that are novel in the context of PINNs. They are numerically compared to the vanilla and an hp-variational loss functional from the literature based on two steady-state benchmark problems whose solutions possess different types of layers. We observe that the best novel loss functionals reduce the L²(Ω) error by 17.3% for the first and 5.5% for the second problem compared to the methods from the literature.

Key words: Steady-state convection-diffusion problems, Convection-dominated regime, Physics-informed neural networks (PINNs), Loss functionals

CLC Number:

Derk Frerichs-Mihov, Linus Henning, Volker John. On Loss Functionals for Physics-Informed Neural Networks for Steady-State Convection-Dominated Convection-Diffusion Problems[J]. Communications on Applied Mathematics and Computation, 2026, 8(1): 287-308.

TrendMD

References

[1] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: large-scale machine learning on heterogeneous systems. Software available from https://www.tensorflow.org/ (2015)
[2] Arzani, A., Cassel, K.W., D’Souza, R.M.: Theory-guided physics-informed neural networks for boundary layer problems with singular perturbation. J. Comput. Phys. 473, 111768 (2023). https://doi.org/10.1016/j.jcp.2022.111768
[3] Augustin, M., Caiazzo, A., Fiebach, A., Fuhrmann, J., John, V., Linke, A., Umla, R.: An assessment of discretizations for convection-dominated convection-diffusion equations. Comput. Methods Appl. Mech. Eng. 200(47/48), 3395-3409 (2011). https://doi.org/10.1016/j.cma.2011.08.012
[4] Barrenechea, G.R., John, V., Knobloch, P.: Finite element methods respecting the discrete maximum principle for convection-diffusion equations. SIAM Rev. 66, 3-88 (2024). https://doi.org/10.1137/22M1488934
[5] Beck, A., Flad, D., Munz, C.-D.: Deep neural networks for data-driven LES closure models. J. Comput. Phys. 398, 108910 (2019). https://doi.org/10.1016/j.jcp.2019.108910
[6] Bengio, Y.: Practical recommendations for gradient-based training of deep architectures. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade: Second Edition. Lecture Notes in Computer Science, vol. 7700, pp. 437-478. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-35289-8_26
[7] Cai, S., Mao, Z., Wang, Z., Yin, M., Karniadakis, G.E.: Physics-informed neural networks (PINNs) for fluid mechanics: a review. Acta. Mech. Sin. 37(12), 1727-1738 (2021). https://doi.org/10.1007/s10409-021-01148-1
[8] Cuomo, S., Di Cola, V.S., Giampaolo, F., Rozza, G., Raissi, M., Piccialli, F.: Scientific machine learning through physics-informed neural networks: where we are and what’s next. J. Sci. Comput. 92(3), 88 (2022). https://doi.org/10.1007/s10915-022-01939-z
[9] Daw, A., Bu, J., Wang, S., Perdikaris, P., Karpatne, A.: Mitigating propagation failures in physics-informed neural networks using retain-resample-release (r3) sampling. In: Proceedings of the 40th International Conference on Machine Learning. ICML’23, vol. 202, pp. 7264-7302. JMLR.org, Honolulu, Hawaii, USA (2023)
[10] De Ryck, T., Mishra, S., Molinaro, R.: wPINNs: weak physics informed neural networks for approximating entropy solutions of hyperbolic conservation laws. SIAM J. Numer. Anal. 62(2), 811-841 (2024). https://doi.org/10.1137/22M1522504
[11] Dissanayake, G., Phan-Thien, N.: Neural-network-based approximations for solving partial differential equations. Commun. Numer. Methods Eng. 10(3), 195-201 (1994). https://doi.org/10.1002/cnm.1640100303
[12] Doumèche, N., Biau, G., Boyer, C.: Convergence and error analysis of PINNs (2023). https://doi.org/10.48550/arXiv.2305.01240
[13] Farrell, P., Rotundo, N., Doan, D.H., Kantner, M., Fuhrmann, J., Koprucki, T.: Drift-diffusion models. In: Piprek, J. (ed.) Handbook of Optoelectronic Device Modeling and Simulation: Lasers, Modulators, Photodetectors, Solar Cells, and Numerical Methods, vol. 2, 1st edn., pp. 733-771. CRC Press, Boca Raton (2017). https://doi.org/10.4324/9781315152318
[14] Frerichs, D., John, V.: On reducing spurious oscillations in discontinuous Galerkin (DG) methods for steady-state convection-diffusion equations. J. Comput. Appl. Math. 393, 113487 (2021). https://doi.org/10.1016/j.cam.2021.113487
[15] Frerichs-Mihov, D.: On slope limiting and deep learning techniques for the numerical solution to convection-dominated convection-diffusion problems. Ph.D. Thesis. Free University Berlin, Berlin (2023)
[16] Frerichs-Mihov, D., Henning, L., John, V.: Data and code from the paper “On loss functionals for physics-informed neural networks for convection-dominated convection-diffusion problems” (2023). https://doi.org/10.20347/wias.data.7. https://archive.wias-berlin.de/receive/wias_mods_00007477
[17] Frerichs-Mihov, D., Henning, L., John, V.: Using deep neural networks for detecting spurious oscillations in discontinuous Galerkin solutions of convection-dominated convection-diffusion equations. J. Sci. Comput. 97(2), 36 (2023). https://doi.org/10.1007/s10915-023-02335-x
[18] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 9, pp. 249-256. PMLR, Chia Laguna Resort, Sardinia, Italy (2010)
[19] Gomes, A.T.A., Silva, L.M., Valentin, F.: Physics-aware neural networks for boundary layer linear problems (2022). https://doi.org/10.48550/arXiv.2208.12559
[20] Gomes, A.T.A., Silva, L.M., Valentin, F.: Improving boundary layer predictions using parametric physics-aware neural networks. In: Navaux, P., Barrios, H.C.J., Osthoff, C., Guerrero, G. (eds.) High Performance Computing. Communications in Computer and Information Science, pp. 90-102. Springer, Porto Alegre (2022). https://doi.org/10.1007/978-3-031-23821-5_7
[21] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). https://www.deeplearningbook.org/. Accessed 2023-07-19
[22] He, Q., Tartakovsky, A.M.: Physics-informed neural network method for forward and backward advection-dispersion equations. Water Resour. Res. 57(7), 2020-029479 (2021). https://doi.org/10.1029/2020WR029479
[23] Hemker, P.W.: A singularly perturbed model problem for numerical computation. J. Comput. Appl. Math. 76(1/2), 277-285 (1996). https://doi.org/10.1016/S0377-0427(96)00113-6
[24] Higham, C.F., Higham, D.J.: Deep learning: an introduction for applied mathematicians. SIAM Rev. 61(4), 860-891 (2019). https://doi.org/10.1137/18M1165748
[25] Hou, Q., Sun, Z., He, L., Karemat, A.: Orthogonal grid physics-informed neural networks: a neural network-based simulation tool for advection-diffusion-reaction problems. Phys. Fluids 34(7), 077108 (2022). https://doi.org/10.1063/5.0095536
[26] John, V., Knobloch, P.: On spurious oscillations at layers diminishing (SOLD) methods for convection-diffusion equations: part I—a review. Comput. Methods Appl. Mech. Eng. 196(17/18/19/20), 2197-2215 (2007). https://doi.org/10.1016/j.cma.2006.11.013
[27] John, V., Knobloch, P.: Adaptive computation of parameters in stabilized methods for convection-diffusion problems. In: Cangiani, A., Davidchack, R.L., Georgoulis, E., Gorban, A.N., Levesley, J., Tretyakov, M.V. (eds.) Numerical Mathematics and Advanced Applications 2011—Proceedings of ENUMATH 2011, vol. 1, pp. 275-283. Springer, Berlin (2013). https://doi.org/10.1007/978-3-642-33134-3_30
[28] John, V., Knobloch, P., Novo, J.: Finite elements for scalar convection-dominated equations and incompressible flow problems: a never ending story? Comput. Vis. Sci. 19(5/6), 47-63 (2018). https://doi.org/10.1007/s00791-018-0290-5
[29] John, V., Knobloch, P., Savescu, S.B.: A posteriori optimization of parameters in stabilized methods for convection-diffusion problems—part I. Comput. Methods Appl. Mech. Eng. 200(41/42/43/44), 2916-2929 (2011). https://doi.org/10.1016/j.cma.2011.04.016
[30] John, V., Knobloch, P., Wilbrandt, U.: A posteriori optimization of parameters in stabilized methods for convection-diffusion problems—part II. J. Comput. Appl. Math. 428, 115167-17 (2023). https://doi.org/10.1016/j.cam.2023.115167
[31] John, V., Maubach, J.M., Tobiska, L.: Nonconforming streamline-diffusion-finite-element-methods for convection-diffusion problems. Numer. Math. 78(2), 165-188 (1997). https://doi.org/10.1007/s002110050309
[32] Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L.: Physics-informed machine learning. Nat. Rev. Phys. 3(6), 422-440 (2021). https://doi.org/10.1038/s42254-021-00314-5
[33] Kharazmi, E., Zhang, Z., Karniadakis, G.E.: VPINNs: variational physics-informed neural networks for solving partial differential equations (2019). https://doi.org/10.48550/arXiv.1912.00873
[34] Kharazmi, E., Zhang, Z., Karniadakis, G.E.M.:

\begin{document}$ hp $\end{document}

-VPINNs: variational physics-informed neural networks with domain decomposition. Comput. Methods Appl. Mech. Eng. 374, 113547 (2021). https://doi.org/10.1016/j.cma.2020.113547
[35] Khodayi-Mehr, R., Zavlanos, M.: VarNet: variational neural networks for the solution of partial differential equations. In: Proceedings of the 2nd Conference on Learning for Dynamics and Control, pp. 298-307. PMLR, Virtual, Online (2020)
[36] Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: ICLR 2015, p. 13. San Diego, California, USA (2014). https://doi.org/10.48550/ARXIV.1412.6980
[37] Knobloch, P., Lukáš, P., Solin, P.: On error indicators for optimizing parameters in stabilized methods. Adv. Comput. Math. 45(4), 1853-1862 (2019). https://doi.org/10.1007/s10444-019-09662-4
[38] Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., Mahoney, M.W.: Characterizing possible failure modes in physics-informed neural networks. In: Advances in Neural Information Processing Systems, vol. 34, pp. 26548-26560. Curran Associates, Inc., Virtual, Online (2021)
[39] Laghi, L., Schiassi, E., De Florio, M., Furfaro, R., Mostacci, D.: Physics-informed neural networks for 1-D steady-state diffusion-advection-reaction equations. Nucl. Sci. Eng. 197(9), 1-31 (2023). https://doi.org/10.1080/00295639.2022.2160604
[40] Lu, L., Pestourie, R., Yao, W., Wang, Z., Verdugo, F., Johnson, S.G.: Physics-informed neural networks with hard constraints for inverse design. SIAM J. Sci. Comput. 43(6), 1105-1132 (2021). https://doi.org/10.1137/21M1397908
[41] Margenberg, N., Lessig, C., Richter, T.: Structure preservation for the deep neural network multigrid solver. ETNA Electron. Trans. Numer. Anal. 56, 86-101 (2022). https://doi.org/10.1553/etna_vol56s86
[42] Mishra, S., Molinaro, R.: Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDEs. IMA J. Numer. Anal. 42(2), 981-1022 (2022). https://doi.org/10.1093/imanum/drab032
[43] Mishra, S., Molinaro, R.: Estimates on the generalization error of physics-informed neural networks for approximating PDEs. IMA J. Numer. Anal. 43(1), 1-43 (2023). https://doi.org/10.1093/imanum/drab093
[44] Misra, D.: Mish: a self regularized non-monotonic activation function. In: British Machine Vision Conference (2020). https://api.semanticscholar.org/CorpusID:221113156
[45] Mojgani, R., Balajewicz, M., Hassanzadeh, P.: Lagrangian PINNs: a causality-conforming solution to failure modes of physics-informed neural networks. Comput. Methods Appl. Mech. Eng. 404, 115810 (2023). https://doi.org/10.1016/j.cma.2022.115810. arXiv:2205.02902
[46] Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numer. 8, 143-195 (1999). https://doi.org/10.1017/S0962492900002919
[47] Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686-707 (2019). https://doi.org/10.1016/j.jcp.2018.10.045
[48] Ray, D., Hesthaven, J.S.: Detecting troubled-cells on two-dimensional unstructured grids using a neural network. J. Comput. Phys. 397, 108845 (2019). https://doi.org/10.1016/j.jcp.2019.07.043
[49] Roos, H.-G., Stynes, M., Tobiska, L.: Robust Numerical Methods for Singularly Perturbed Differential Equations: Convection-Diffusion-Reaction and Flow Problems. Springer Series in Computational Mathematics, vol. 24, 2nd edn. Springer, Berlin (2008). https://doi.org/10.1007/978-3-540-34467-4
[50] Saadat, M.H., Gjorgiev, B., Das, L., Sansavini, G.: Neural tangent kernel analysis or PINN for advection-diffusion equation (2022). https://doi.org/10.48550/arXiv.2211.11716
[51] TensorFlow Developers: TensorFlow (v2.13.0). Zenodo (2023). https://doi.org/10.5281/zenodo.8117732
[52] von Wahl, H., Richter, T.: Using a deep neural network to predict the motion of underresolved triangular rigid bodies in an incompressible flow. Int. J. Numer. Methods Fluids 93(12), 3364-3383 (2021). https://doi.org/10.1002/fld.5037
[53] Wang, Y., Xu, C., Yang, M., Zhang, J.: Less emphasis on hard regions: curriculum learning of PINNs for singularly perturbed convection-diffusion-reaction problems. East Asian J. Appl. Math. 14(1), 104-123 (2024). https://doi.org/10.4208/eajam.2023-062.170523
[54] Wolff, T., Carrillo, H., Martí, L., Sanchez-Pi, N.: Towards optimally weighted physics-informed neural networks in ocean modelling (2021). https://doi.org/10.48550/arXiv.2106.08747
[55] Yang, L., Shami, A.: On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing 415, 295-316 (2020). https://doi.org/10.1016/j.neucom.2020.07.061
[56] Zang, Y., Bao, G., Ye, X., Zhou, H.: Weak adversarial networks for high-dimensional partial differential equations. J. Comput. Phys. 411, 109409 (2020). https://doi.org/10.1016/j.jcp.2020.109409
[57] Zong, Y., He, Q., Tartakovsky, A.M.: Improved training of physics-informed neural networks for parabolic differential equations with sharply perturbed initial conditions. Comput. Methods Appl. Mech. Eng. 414, 116125 (2023). https://doi.org/10.1016/j.cma.2023.116125