Communications on Applied Mathematics and Computation >

2024 , Vol. 6 >Issue 2: 757 - 789

DOI: https://doi.org/10.1007/s42967-022-00234-w

ORIGINAL PAPERS

A Dynamical System-Based Framework for Dimension Reduction

Expand
  • Department of Mathematics, University of Utah, Salt Lake City, UT, USA

Received date: 2022-04-23

  Revised date: 2022-10-11

  Accepted date: 2022-11-21

  Online published: 2023-02-03

Fold

Abstract

We propose a novel framework for learning a low-dimensional representation of data based on nonlinear dynamical systems, which we call the dynamical dimension reduction (DDR). In the DDR model, each point is evolved via a nonlinear flow towards a lower-dimensional subspace; the projection onto the subspace gives the low-dimensional embedding. Training the model involves identifying the nonlinear flow and the subspace. Following the equation discovery method, we represent the vector field that defines the flow using a linear combination of dictionary elements, where each element is a pre-specified linear/nonlinear candidate function. A regularization term for the average total kinetic energy is also introduced and motivated by the optimal transport theory. We prove that the resulting optimization problem is well-posed and establish several properties of the DDR method. We also show how the DDR method can be trained using a gradient-based optimization method, where the gradients are computed using the adjoint method from the optimal control theory. The DDR method is implemented and compared on synthetic and example data sets to other dimension reduction methods, including the PCA, t-SNE, and Umap.

Key words： Dimension reduction; Equation discovery; Dynamical systems; Adjoint method; Optimal transportation

Cite this article

Ryeongkyung Yoon, Braxton Osting . A Dynamical System-Based Framework for Dimension Reduction[J]. Communications on Applied Mathematics and Computation, 2024 , 6(2) : 757 -789 . DOI: 10.1007/s42967-022-00234-w

References

[1] Baldi, P.: Autoencoders, unsupervised learning, and deep architectures. In: Guyon, G., Dror, G., Lemaire, V., Taylor, G., Silver, D. (eds.) Proceedings of ICML Workshop on Unsupervised and Transfer Learning. Proceedings of Machine Learning Research, pp. 37-49. PMLR, Bellevue (2012)
[2] Benamou, J.-D., Brenier, Y.: A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem. Numer. Math. 84(3), 375-393 (2000). https://doi.org/10.1007/s002110050002
[3] Borwein, J.M., Lewis, A.S.: Convex Analysis and Nonlinear Optimization. Springer, New York (2000). https://doi.org/10.1007/978-1-4757-9859-3
[4] Brunton, S.L., Proctor, J.L., Kutz, J.N.: Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. 113(15), 3932-3937 (2016). https://doi.org/10.1073/pnas.1517384113
[5] Calvetti, D., Morigi, S., Reichel, L., Sgallari, F.: Tikhonov regularization and the L-curve for large discrete ill-posed problems. J. Comput. Appl. Math. 123(1/2), 423-446 (2000). https://doi.org/10.1016/s0377-0427(00)00414-3
[6] Chalvidal, M., Ricci, M., VanRullen, R., Serre, T.: Go with the flow: adaptive control for neural ODEs. arXiv: 2006.09545 (2020). https://doi.org/10.48550/arXiv.2006.09545
[7] Chang, B., Chen, M., Haber, E., Chi, E.H.: Antisymmetric RNN: a dynamical system view on recurrent neural networks. arXiv: 1902.09689 (2019). https://doi.org/10.48550/arXiv.1902.09689
[8] Chen, R.T., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. Adv. Neural. Inf. Process. Syst. 31, 07366 (2018). https://doi.org/10.48550/arXiv.1806
[9] Finlay, C., Jacobsen, J.-H., Nurbekyan, L., Oberman, A.: How to train your neural ODE: the world of Jacobian and kinetic regularization. In: International Conference on Machine Learning, pp. 3154-3164. PMLR, Cham (2020)
[10] Garsdal, M., Søgaard, V., Sørensen, S.: Generative time series models using neural ODE in variational autoencoders. arXiv: 2201.04630 (2022). https://doi.org/10.48550/arXiv.2201.04630
[11] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, London (2016). https://doi.org/10.1007/s10710-017-9314-z
[12] Grathwohl, W., Chen, R.T., Bettencourt, J., Sutskever, I., Duvenaud, D.: FFJORD: free-form continuous dynamics for scalable reversible generative models. arXiv: 1810.01367 (2018). https://doi.org/10.48550/arXiv.1810.01367
[13] Greydanus, S., Dzamba, M., Yosinski, J.: Hamiltonian neural networks. In: NIPS'19: Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 15379-15389. ACM (2019). https://doi.org/10.48550/arXiv.1906.01563
[14] Haber, E., Ruthotto, L.: Stable architectures for deep neural networks. Inverse Prob. 34, 014004 (2017). https://doi.org/10.1088/1361-6420/aa9a90
[15] Heinonen, M., Yildiz, C., Mannerström, H., Intosalmi, J., Lähdesmäki, H.: Learning unknown ODE models with Gaussian processes. In: International Conference on Machine Learning, pp. 1959-1968. PMLR, Cham (2018). https://doi.org/10.1109/cdc45484.2021.9683426
[16] Hotelling, H.: Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 24(6), 417 (1933). https://doi.org/10.1037/h0071325
[17] Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv: 1312.6114 (2013). https://doi.org/10.48550/arXiv.1312.6114
[18] Long, Z., Lu, Y., Dong, B.: PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network. J. Comput. Phys. 399, 108925 (2019). https://doi.org/10.1016/j.jcp.2019.108925
[19] McInnes, L., Healy, J., Melville, J.: Umap: uniform manifold approximation and projection for dimension reduction. arXiv: 1802.03426 (2018). https://doi.org/10.48550/arXiv.1802.03426
[20] Santambrogio, F.: Optimal transport for applied mathematicians: calculus of variations, PDEs, and modeling. In: Progress in Nonlinear Differential Equations and Their Applications. Birkäuser, New York (2015). https://doi.org/10.1007/978-3-319-20828-2
[21] Sideris, T.C.: Ordinary Differential Equations and Dynamical Systems. Springer, Cham (2013). https://doi.org/10.2991/978-94-6239-021-8
[22] Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(86), 2579-2605 (2008)
[23] Xia, H., Suliafu, V., Ji, H., Nguyen, T., Bertozzi, A., Osher, S., Wang, B.: Heavy ball neural ordinary differential equations. In: Thirty-Fifth Conference on Neural Information Processing Systems (NeurIPS 2021), arXiv: 2110.04840 (2021). https://doi.org/10.48550/arXiv.2110.04840
[24] Yoon, R., Bhat, H.S., Osting, B.: A nonautonomous equation discovery method for time signal classification. SIAM J. Appl. Dyn. Syst. 21(1), 33-59 (2022). https://doi.org/10.1137/21m1405216
[25] Zhang, L., Schaeffer, H.: On the convergence of the SINDy algorithm. Multiscale Model. Simul. 17(3), 948-972 (2019). https://doi.org/10.1137/18m1189828
[26] Zhong, Y.D., Dey, B., Chakraborty, A.: Symplectic ODE-net: learning Hamiltonian dynamics with control. arXiv: 1909.12077 (2019). https://doi.org/10.48550/arXiv.1909.12077
Options
Outlines

/

〈 〉

Copyright © Editorial By Communications on Applied Mathematics and Computation
沪ICP备09014157