Overview Frequency Principle/Spectral Bias in Deep Learning

doi:10.1007/s42967-024-00398-7

Abstract

Abstract: Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds light on this magical “black box” by showing a Frequency principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs)—DNNs often fit functions from low to highfrequencies during the training. The F-Principle is first demonstrated by one-dimensional (1D) synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural networks in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of the F-Principle and propose some open problems for future research.

Key words: Neural network, Frequency principle (F-Principle), Deep learning, Generalization, Training, Optimization

CLC Number:

Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo. Overview Frequency Principle/Spectral Bias in Deep Learning[J]. Communications on Applied Mathematics and Computation, 2025, 7(3): 827-864.

TrendMD

References

1. Agarwal, R., Frosst, N., Zhang, X., Caruana, R., Hinton, G.E.:Neural additive models:interpretable machine learning with neural nets. arXiv:2004.13912 (2020)
2. Arora, S., Du, S., Hu, W., Li, Z., Wang, R.:Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In:International Conference on Machine Learning, pp. 322-332 (2019)
3. Arpit, D., Jastrze?bski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M.S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., Lacoste-Julien, S.:A closer look at memorization in deep networks. In: International Conference on Machine Learning, pp. 233-242 (2017)
4. Aubin, B., Maillard, A., Barbier, J., Krzakala, F., Macris, N., Zdeborová, L.:The committee machine: computational to statistical gaps in learning a two-layers neural network. Adv. Neural Inf. Process. Syst. 31, 3223-3234 (2018)
5. Baratin, A., George, T., Laurent, C., Hjelm, R.D., Lajoie, G., Vincent, P., Lacoste-Julien, S.:Implicit regularization via neural feature alignment. arXiv:2008.00938 (2020)
6. Basri, R., Galun, M., Geifman, A., Jacobs, D., Kasten, Y., Kritchman, S.:Frequency bias in neural networks for input of non-uniform density. In:International Conference on Machine Learning, pp. 685- 694 (2020)
7. Basri, R., Jacobs, D., Kasten, Y., Kritchman, S.:The convergence rate of neural networks for learned functions of different frequencies. Adv. Neural Inf. Process. Syst. 32, 4761-4771 (2019)
8. Bi, S., Xu, Z., Srinivasan, P., Mildenhall, B., Sunkavalli, K., Hašan, M., Hold-Geoffroy, Y., Kriegman, D., Ramamoorthi, R.:Neural reflectance fields for appearance acquisition. arXiv:2008.03824 (2020)
9. Biland, S., Azevedo, V.C., Kim, B., Solenthaler, B.:Frequency-aware reconstruction of fluid simulations with generative networks. arXiv:1912.08776 (2019)
10. Bordelon, B., Canatar, A., Pehlevan, C.:Spectrum dependent learning curves in kernel regression and wide neural networks. In:International Conference on Machine Learning, pp. 1024-1034 (2020)
11. Breiman, L.:Reflections after refereeing papers for nips. In:The Mathematics of Generalization, pp. 11-15 (1995)
12. Brenner, S.C., Scott, L.R.:The Mathematical Theory of Finite Element Methods. Springer, New York (2008)
13. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.:Language models are few-shot learners. arXiv:2005.14165 (2020)
14. Cai, W., Li, X., Liu, L.:A phase shift deep neural network for high frequency approximation and wave problems. SIAM J. Sci. Comput. 42(5), 3285-3312 (2020)
15. Cai, W., Xu, Z.-Q.J.:Multi-scale deep neural networks for solving high dimensional PDEs. arXiv:1910.11710 (2019)
16. Campo, M., Chen, Z., Kung, L., Virochsiri, K., Wang, J.:Band-limited soft actor critic model. arXiv:2006.11431 (2020)
17. Camuto, A., Willetts, M., Sim sekli, U., Roberts, S., Holmes, C.:Explicit regularisation in Gaussian noise injections. arXiv:2007.07368 (2020)
18. Cao, Y., Fang, Z., Wu, Y., Zhou, D.-X., Gu, Q.:Towards understanding the spectral bias of deep learning. In:Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2205-2211 (2021)
19. Chakrabarty, P.:The spectral bias of the deep image prior. In:Bayesian Deep Learning Workshop and Advances in Neural Information Processing Systems (NeurIPS) (2019)
20. Chen, G.-Y., Gan, M., Chen, C.P., Zhu, H.-T., Chen, L.:Frequency principle in broad learning system. IEEE Trans. Neural Netw. Learn. Syst. 33, 6983 (2021)
21. Chen, H., Lin, M., Sun, X., Qi, Q., Li, H., Jin, R.:MuffNet:multi-layer feature federation for mobile deep learning. In:Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)
22. Chen, Y., Li, G., Jin, C., Liu, S., Li, T.:SSD-GAN:measuring the realness in the spatial and spectral domains. Proc. AAAI Conf. Artif. Intell. 35, 1105-1112 (2021)
23. Chizat, L., Bach, F.:On the global convergence of gradient descent for over-parameterized models using optimal transport. Adv. Neural Inf. Process. Syst. 31, 3036-3046 (2018)
24. Choromanska, A., Henaff, M., Mathieu, M., Arous, G.B., LeCun, Y.:The loss surfaces of multilayer networks. In:Artificial Intelligence and Statistics, pp. 192-204 (2015)
25. Deng, X., Zhang, Z.M.:Is the meta-learning idea able to improve the generalization of deep neural networks on the standard supervised learning? In:2020 25th International Conference on Pattern Recognition (ICPR), pp. 150-157 (2021)
26. Dissanayake, M., Phan-Thien, N.:Neural-network-based approximations for solving partial differential equations. Commun. Numer. Methods Eng. 10(3), 195-201 (1994)
27. Dong, B., Hou, J., Lu, Y., Zhang, Z.:Distillation ≈ early stopping? Harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network. arXiv:1910.01255 (2019)
28. Dyson, F.:A meeting with Enrico Fermi. Nature 427(6972), 297 (2004)
29. E, W., Han, J., Jentzen, A.:Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Commun. Math. Stat. 5(4), 349-380 (2017)
30. E, W., Han, J., Jentzen, A.:Algorithms for solving high dimensional PDEs:from nonlinear Monte Carlo to machine learning. Nonlinearity 35(1), 278 (2021)
31. E, W., Ma, C., Wang, J.:Model reduction with memory and the machine learning of dynamical systems. Commun. Comput. Phys. 25(4), 947-962 (2018)
32. E, W., Ma, C., Wu, L.:A priori estimates of the population risk for two-layer neural networks. Commun. Math. Sci. 17(5), 1407-1425 (2019)
33. E, W., Ma, C., Wu, L.:Machine learning from a continuous viewpoint, I. Sci. China Math. 63, 2233-2266 (2020)
34. E, W., Yu, B.:The deep Ritz method:a deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat. 6(1), 1-12 (2018)
35. Engel, A., Broeck, C.V.d.:Statistical Mechanics of Learning. Cambridge University Press, Cambridge (2001)
36. Evans, L.C.:Partial Differential Equations. American Mathematical Society, Providence, Rhode Island (2010)
37. Fan, Y., Lin, L., Ying, L., Zepeda-Núnez, L.:A multiscale neural network based on hierarchical matrices. Multiscale Model. Simul. 17(4), 1189-1213 (2019)
38. Frankle, J., Carbin, M.:The lottery ticket hypothesis:finding sparse, trainable neural networks. arXiv:1803.03635 (2018)
39. Fu, Y., Guo, H., Li, M., Yang, X., Ding, Y., Chandra, V., Lin, Y.:CPT:efficient deep neural network training via cyclic precision. arXiv:2101.09868 (2021)
40. Fu, Y., You, H., Zhao, Y., Wang, Y., Li, C., Gopalakrishnan, K., Wang, Z., Lin, Y.:Fractrain:fractionally squeezing bit savings both temporally and spatially for efficient DNN training. In:Advances in Neural Information Processing Systems 33:Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual (2020)
41. Giryes, R., Bruna, J.:How can we use tools from signal processing to understand better neural networks? Inside Signal Processing Newsletter (2020)
42. Goldt, S., Mézard, M., Krzakala, F., Zdeborová, L.:Modeling the influence of data structure on learning in neural networks:the hidden manifold model. Phys. Rev. X 10(4), 041044 (2020)
43. Guo, M., Fathi, A., Wu, J., Funkhouser, T.:Object-centric neural scene rendering. arXiv:2012.08503 (2020)
44. Han, J., Ma, C., Ma, Z., Weinan, E.:Uniformly accurate machine learning-based hydrodynamic models for kinetic equations. Proc. Natl. Acad. Sci. 116(44), 21983-21991 (2019)
45. Han, J., Zhang, L., Car, R., E, W.:Deep potential:a general representation of a many-body potential energy surface. Commun. Comput. Phys. 23, 3 (2018)
46. Häni, N., Engin, S., Chao, J.-J., Isler, V.:Continuous object representation networks:novel view synthesis without target view supervision. arXiv:2007.15627 (2020)
47. He, J., Li, L., Xu, J., Zheng, C.:ReLU deep neural networks and linear finite elements. arXiv:1807.03973 (2018)
48. He, J., Xu, J.:MgNet:A unified framework of multigrid and convolutional neural network. Sci. China Math. 62(7), 1331-1354 (2019)
49. He, S., Wang, X., Shi, S., Lyu, M.R., Tu, Z.:Assessing the bilingual knowledge learned by neural machine translation models. arXiv:2004.13270 (2020)
50. Hennigh, O., Narasimhan, S., Nabian, M.A., Subramaniam, A., Tangsali, K., Fang, Z., Rietmann, M., Byeon, W., Choudhry, S.:NVIDIA SimNet:an AI-accelerated multi-physics simulation framework. In:Paszynski, M., Kranzlmüller, D., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science-ICCS 2021. Lecture Notes in Computer Science, vol. 12746, pp. 447-461. Springer, Cham (2021)
51. Hu,W., Xiao, L., Adlam, B., Pennington, J.:The surprising simplicity of the early-time learning dynamics of neural networks. arXiv:2006.14599 (2020)
52. Huang, J., Wang, H., Yang, H.:Int-Deep:a deep learning initialized iterative method for nonlinear problems. J. Comput. Phys. 419, 109675 (2020)
53. Huang, X., Liu, H., Shi, B., Wang, Z., Yang, K., Li, Y., Weng, B., Wang, M., Chu, H., Zhou, J., Yu, F., Hua, B., Chen, L., Dong, B.:Solving partial differential equations with point source based on physicsinformed neural networks. arXiv:2111.01394 (2021)
54. Jacot, A., Gabriel, F., Hongler, C.:Neural tangent kernel:convergence and generalization in neural networks. In:Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8580-8589 (2018)
55. Jagtap, A.D., Kawaguchi, K., Karniadakis, G.E.:Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. J. Comput. Phys. 404, 109136 (2020)
56. Jiang, L., Dai, B., Wu, W., Loy, C.C.:Focal frequency loss for generative models. arXiv:2012.12821 (2020)
57. Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., Bengio, S.:Fantastic generalization measures and where to find them. In:International Conference on Learning Representations (2019)
58. Jin, H., Montúfar, G.:Implicit bias of gradient descent for mean squared error regression with wide neural networks. arXiv:2006.07356 (2020)
59. Jin, P., Lu, L., Tang, Y., Karniadakis, G.E.:Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness. Neural Netw. 130, 85-99 (2020)
60. Kalimeris, D., Kaplun, G., Nakkiran, P., Edelman, B., Yang, T., Barak, B., Zhang, H.:SGD on neural networks learns functions of increasing complexity. Adv. Neural Inf. Process. Syst. 32, 3496-3506 (2019)
61. Khoo, Y., Ying, L.:SwitchNet:a neural network model for forward and inverse scattering problems. SIAM J. Sci. Comput. 41(5), 3182-3201 (2019)
62. Kingma, D.P., Ba, J.:Adam:a method for stochastic optimization. arXiv:1412.6980 (2014)
63. Kopitkov, D., Indelman, V.:Neural spectrum alignment:empirical study. In:International Conference on Artificial Neural Networks, pp. 168-179. Springer (2020)
64. Lampinen, A.K., Ganguli, S.:An analytic theory of generalization dynamics and transfer learning in deep linear networks. In:The International Conference on Learning Representations (2019)
65. Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., Pennington, J.:Wide neural networks of any depth evolve as linear models under gradient descent. Adv. Neural Inf. Process. Syst. 32, 8572-8583 (2019)
66. Li, M., Soltanolkotabi, M., Oymak, S.:Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In:International Conference on Artificial Intelligence and Statistics, pp. 4313-4324 (2020)
67. Li, X.-A., Xu, Z.-Q.J., Zhang, L.:A multi-scale DNN algorithm for nonlinear elliptic equations with multiple scales. Commun. Comput. Phys. 28(5), 1886-1906 (2020). https://doi.org/10.4208/cicp.OA- 2020-0187
68. Li, X.-A., Xu, Z.-Q.J., Zhang, L.:Subspace decomposition based DNN algorithm for elliptic-type multiscale PDEs. J. Comput. Phys. 488, 112242 (2023). https://doi.org/10.2139/ssrn.4020731
69. Li, Y., Peng, W., Tang, K., Fang, M.:Spatio-frequency decoupled weak-supervision for face reconstruction. Comput. Intell. Neurosci. 2022, 1-12 (2022)
70. Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., Anandkumar, A.:Fourier neural operator for parametric partial differential equations. arXiv:2010.08895 (2020)
71. Liang, S., Lyu, L., Wang, C., Yang, H.:Reproducing activation function for deep learning. arXiv:2101.04844 (2021)
72. Lin, J., Camoriano, R., Rosasco, L.:Generalization properties and implicit regularization for multiple passes SGM. In:International Conference on Machine Learning, pp. 2340-2348 (2016)
73. Liu, Z., Cai, W., Xu, Z.-Q.J.:Multi-scale deep neural network (MscaleDNN) for solving PoissonBoltzmann equation in complex domains. Commun. Comput. Phys. 28(5), 1970-2001 (2020)
74. Lu, L., Jin, P., Pang, G., Zhang, Z., Karniadakis, G.E.:Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nat. Mach. Intell. 3(3), 218-229 (2021)
75. Lu, L., Meng, X., Mao, Z., Karniadakis, G.E.:DeepXDE:a deep learning library for solving differential equations. SIAM Rev. 63(1), 208-228 (2021)
76. Luo, T., Ma, Z., Wang, Z., Xu, Z.-Q.J., Zhang, Y.:Fourier-domain variational formulation and its wellposedness for supervised learning. arXiv:2012.03238 (2020)
77. Luo, T., Ma, Z., Xu, Z.-Q.J., Zhang, Y.:On the exact computation of linear frequency principle dynamics and its generalization. arXiv:2010.08153 (2020)
78. Luo, T., Ma, Z., Xu, Z.-Q.J., Zhang, Y.:Theory of the frequency principle for general deep neural networks. CSIAM Trans. Appl. Math. 2(3), 484-507 (2021). https://doi.org/10.4208/csiam-am.SO- 2020-0005
79. Luo, T., Xu, Z.-Q.J., Ma, Z., Zhang, Y.:Phase diagram for two-layer ReLU neural networks at infinitewidth limit. J. Mach. Learn. Res. 22, 1-47 (2021)
80. Ma, C., Wu, L., E., W.:The slow deterioration of the generalization error of the random feature model. In:Mathematical and Scientific Machine Learning, pp. 373-389 (2020)
81. Ma, Y., Xu, Z.-Q.J., Zhang, J.:Frequency principle in deep learning beyond gradient-descent-based training. arXiv:2101.00747 (2021)
82. Mei, S., Montanari, A., Nguyen, P.-M.:A mean field view of the landscape of two-layer neural networks. Proc. Natl. Acad. Sci. 115(33), 7665-7671 (2018)
83. Michoski, C., Milosavljevic, M., Oliver, T., Hatch, D.:Solving irregular and data-enriched differential equations using deep neural networks. arXiv:1905.04351 (2019)
84. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.:NeRF:representing scenes as neural radiance fields for view synthesis. In:European Conference on Computer Vision, pp. 405-421. Springer (2020)
85. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.:NeRF:representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99-106 (2021)
86. Mingard, C., Skalse, J., Valle-Pérez, G., Martínez-Rubio, D., Mikulik, V., Louis, A.A.:Neural networks are a priori biased towards boolean functions with low entropy. arXiv:1909.11522 (2019)
87. Nye, M., Saxe, A.:Are efficient deep representations learnable? arXiv:1807.06399 (2018)
88. Peng, S., Zhang, Y., Xu, Y., Wang, Q., Shuai, Q., Bao, H., Zhou, X.:Neural body:implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9054-9063 (2021)
89. Peng, W., Zhou, W., Zhang, J., Yao, W.:Accelerating physics-informed neural network training with prior dictionaries. arXiv:2004.08151 (2020)
90. Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.:D-NeRF:neural radiance fields for dynamic scenes. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327 (2021)
91. Rabinowitz, N.C.:Meta-learners' learning dynamics are unlike learners'. arXiv:1905.01320 (2019)
92. Rahaman, N., Arpit, D., Baratin, A., Draxler, F., Lin, M., Hamprecht, F.A., Bengio, Y., Courville, A.: On the spectral bias of deep neural networks. In:International Conference on Machine Learning (2019)
93. Raissi, M., Perdikaris, P., Karniadakis, G.E.:Physics-informed neural networks:a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686-707 (2019)
94. Rotskoff, G.M., Vanden-Eijnden, E.:Parameters as interacting particles:long time convergence and asymptotic error scaling of neural networks. In:Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7146-7155 (2018)
95. Saxe, A.M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B.D., Cox, D.D.:On the information bottleneck theory of deep learning. J. Stat. Mech. Theory Exp. 2019(12), 124020 (2019). https://doi.org/10.1088/1742-5468/ab3985
96. Saxe, A.M., McClelland, J.L., Ganguli, S.:Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In:The International Conference on Learning Representations (2014)
97. Schwarz, K., Liao, Y., Geiger, A.:On the frequency bias of generative models. Adv. Neural Inf. Process. Syst. 34, 18126 (2021)
98. Shalev-Shwartz, S., Shamir, O., Shammah, S.:Failures of gradient-based deep learning. In:International Conference on Machine Learning, pp. 3067-3075 (2017)
99. Sharma, R., Ross, A.:D-NetPAD:an explainable and interpretable iris presentation attack detector. In: 2020 IEEE International Joint Conference on Biometrics (IJCB), pp. 1-10 (2020)
100. Shen, J., Tang, T., Wang, L.L.:Spectral Methods:Algorithm, Analysis and Applications. Springer, Berlin (2011)
101. Shwartz-Ziv, R., Tishby, N.:Opening the black box of deep neural networks via information. arXiv:1703.00810 (2017)
102. Simonyan, K., Zisserman, A.:Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)
103. Sirignano, J., Spiliopoulos, K.:Mean field analysis of neural networks:a central limit theorem. Stoch. Process. Appl. 130(3), 1820-1852 (2020)
104. Strofer, C.M., Wu, J.-L., Xiao, H., Paterson, E.:Data-driven, physics-based feature extraction from fluid flow fields using convolutional neural networks. Commun. Comput. Phys. 25(3), 625-650 (2019)
105. Tancik, M., Mildenhall, B., Wang, T., Schmidt, D., Srinivasan, P.P., Barron, J.T., Ng, R.:Learned initializations for optimizing coordinate-based neural representations. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2846-2855 (2021)
106. Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., Ng, R.:Fourier features let networks learn high frequency functions in low dimensional domains. Adv. Neural Inf. Process. Syst. 33, 7537-7547 (2020)
107. Wang, B., Zhang, W., Cai, W.:Multi-scale deep neural network (MscaleDNN) methods for oscillatory stokes flows in complex domains. Commun. Comput. Phys. 28(5), 2139-2157 (2020)
108. Wang, J., Xu, Z.-Q.J., Zhang, J., Zhang, Y.:Implicit bias with Ritz-Galerkin method in understanding deep learning for solving PDEs. arXiv:2002.07989 (2020)
109. Wang, S., Wang, H., Perdikaris, P.:On the eigenvector bias of Fourier feature networks:from regression to solving multi-scale PDEs with physics-informed neural networks. Comput. Methods Appl. Mech. Eng. 384, 113938 (2021)
110. Xi, Y., Jia, W., Zheng, J., Fan, X., Xie, Y., Ren, J., He, X.:DRL-GAN:dual-stream representation learning GAN for low-resolution image classification in UAV applications. IEEE J. Select. Top. Appl. Earth Observ. Remote Sens. 14, 1705-1716 (2020)
111. Xie, B., Liang, Y., Song, L.:Diverse neural network learns true target functions. Int. Conf. Artif. Intell. Stat. 54, 1216-1224 (2017)
112. Xu, R., Wang, X., Chen, K., Zhou, B., Loy, C.C.:Positional encoding as spatial inductive bias in GANs. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13569- 13578 (2021)
113. Xu, Z.-Q.J.:Understanding training and generalization in deep learning by Fourier analysis. arXiv:1808.04295 (2018)
114. Xu, Z.-Q.J., Zhang, Y., Luo, T., Xiao, Y., Ma, Z.:Frequency principle:Fourier analysis sheds light on deep neural networks. Commun. Comput. Phys. 28(5), 1746-1767 (2020)
115. Xu, Z.-Q.J., Zhang, Y., Xiao, Y.:Training behavior of deep neural network in frequency domain. In: International Conference on Neural Information Processing, pp. 264-274. Springer (2019)
116. Xu, Z.-Q.J., Zhou, H.:Deep frequency principle towards understanding why deeper learning is faster. Proc. AAAI Conf. Artif. Intell. 35, 10541 (2021)
117. Yang, G., Salman, H.:A fine-grained spectral perspective on neural networks. arXiv:1907.10599 (2019)
118. Yang, M., Wang, Z., Chi, Z., Zhang, Y.:FreGAN:exploiting frequency components for training GANs under limited data. arXiv:2210.05461 (2022)
119. You, H., Li, C., Xu, P., Fu, Y., Wang, Y., Chen, X., Lin, Y., Wang, Z., Baraniuk, R.G.:Drawing earlybird tickets:towards more efficient training of deep networks. In:International Conference on Learning Representations (2020)
120. Zang, Y., Bao, G., Ye, X., Zhou, H.:Weak adversarial networks for high-dimensional partial differential equations. J. Comput. Phys. 411, 109409 (2020)
121. Zdeborová, L.:Understanding deep learning is also a job for physicists. Nat. Phys. 16, 1-3 (2020). https://doi.org/10.1038/s41567-020-0929-2
122. Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.:Understanding deep learning requires rethinking generalization. In:5th International Conference on Learning Representations (2017)
123. Zhang, L., Luo, T., Zhang, Y., Xu, Z.-Q.J., Ma, Z.:MOD-NET:a machine learning approach via modeloperator-data network for solving PDEs. arXiv:2107.03673 (2021)
124. Zhang, Y., Li, Y., Zhang, Z., Luo, T., Xu, Z.-Q.J.:Embedding principle:a hierarchical structure of loss landscape of deep neural networks. arXiv:2111.15527 (2021)
125. Zhang, Y., Luo, T., Ma, Z., Xu, Z.-Q.J.:A linear frequency principle model to understand the absence of overfitting in neural networks. Chin. Phys. Lett. 38(3), 038701 (2021)
126. Zhang, Y., Xu, Z.-Q.J., Luo, T., Ma, Z.:Explicitizing an implicit bias of the frequency principle in two-layer neural networks. arXiv:1905.10264 (2019)
127. Zhang, Y., Xu, Z.-Q.J., Luo, T., Ma, Z.:A type of generalization error induced by initialization in deep neural networks. In:Mathematical and Scientific Machine Learning, pp. 144-164 (2020)
128. Zhang, Y., Zhang, Z., Luo, T., Xu, Z.-Q.J.:Embedding principle of loss landscape of deep neural networks. NeurIPS (2021)
129. Zheng, Q., Babaei, V., Wetzstein, G., Seidel, H.-P., Zwicker, M., Singh, G.: Neural light field 3D printing. ACM Trans. Graph. (TOG) 39(6), 1–12 (2020)
130. Zhu, H., Qiao, Y., Xu, G., Deng, L., Yu, Y.-F.: DSPNet: a lightweight dilated convolution neural networks for spectral deconvolution with self-paced learning. IEEE Trans. Ind. Inf. 16, 7392 (2019)

[1]	Yujian Cao, Jianguo Huang, Haoqin Wang. A Hybrid Iterative Method for Elliptic Variational Inequalities of the Second Kind [J]. Communications on Applied Mathematics and Computation, 2025, 7(3): 910-928.
[2]	Xiaotao Liang, Shuo Yin, Xing Zhao, XuYing Zhao. Immunity-Based Orthogonal Weights Modification Algorithm [J]. Communications on Applied Mathematics and Computation, 2025, 7(2): 718-732.
[3]	Liqun Qi. Motion, Dual Quaternion Optimization and Motion Optimization [J]. Communications on Applied Mathematics and Computation, 2025, 7(1): 228-238.
[4]	Jarrod Mau, Jia Zhao. Discovery of Governing Equations with Recursive Deep Neural Networks [J]. Communications on Applied Mathematics and Computation, 2025, 7(1): 239-263.
[5]	Jim Magiera, Christian Rohde. A Multiscale Method for Two-Component, Two-Phase Flow with a Neural Network Surrogate [J]. Communications on Applied Mathematics and Computation, 2024, 6(4): 2265-2294.
[6]	Vitaliy Gyrya, Evan Lieberman, Mark Kenamond, Mikhail Shashkov. Optimization of Artificial Viscosity in Production Codes Based on Gaussian Regression Surrogate Models [J]. Communications on Applied Mathematics and Computation, 2024, 6(3): 1521-1550.
[7]	Zhiwei Gao, Tao Tang, Liang Yan, Tao Zhou. Failure-Informed Adaptive Sampling for PINNs, Part II: Combining with Re-sampling and Subset Simulation [J]. Communications on Applied Mathematics and Computation, 2024, 6(3): 1720-1741.
[8]	Vitaly Gyrya, Mikhail Shashkov, Alexei Skurikhin, Svetlana Tokareva. Machine Learning Approaches for the Solution of the Riemann Problem in Fluid Dynamics: a Case Study [J]. Communications on Applied Mathematics and Computation, 2024, 6(3): 1832-1859.
[9]	Sergio Torregrosa, Victor Champaney, Amine Ammar, Vincent Herbert, Francisco Chinesta. Physics-Based Active Learning for Design Space Exploration and Surrogate Construction for Multiparametric Optimization [J]. Communications on Applied Mathematics and Computation, 2024, 6(3): 1899-1923.
[10]	Yongqiang Cai, Qianxiao Li, Zuowei Shen. Optimization in Machine Learning: a Distribution-Space Approach [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 1217-1240.
[11]	Wes Whiting, Bao Wang, Jack Xin. Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 1175-1188.
[12]	Liangchen Liu, Louis Ly, Colin B. Macdonald, Richard Tsai. Nearest Neighbor Sampling of Point Sets Using Rays [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 1131-1174.
[13]	Zhanhong Ye, Xiang Huang, Hongsheng Liu, Bin Dong. Meta-Auto-Decoder: a Meta-Learning-Based Reduced Order Model for Solving Parametric Partial Differential Equations [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 1096-1130.
[14]	Jane Wu, Michael Bao, Xinwei Yao, Ronald Fedkiw. Deep Energies for Estimating Three-Dimensional Facial Pose and Expression [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 837-861.
[15]	Howard Heaton, Samy Wu Fung, Stanley Osher. Global Solutions to Nonconvex Problems by Evolution of Hamilton-Jacobi PDEs [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 790-810.