Joint Spectral Regression Methods for Large-Scale Discriminant Analysis

doi:10.1007/s42967-024-00402-0

Communications on Applied Mathematics and Computation ›› 2025, Vol. 7 ›› Issue (5): 1791-1814.doi: 10.1007/s42967-024-00402-0

• ORIGINAL PAPERS • Previous Articles

Joint Spectral Regression Methods for Large-Scale Discriminant Analysis

Gang Wu^1,2,3, Wen Yang²

1. Key Laboratory of Data Science and Intelligence Education of Hainan Normal University, Ministry of Education, Haikou, 571158, Hainan, China;
2. School of Mathematics, China University of Mining and Technology, Xuzhou, 221116, Jiangsu, China;
3. School of Big Data, Fuzhou University of International Studies and Trade, Fuzhou, 350202, Fujian, China

Received:2023-11-26 Revised:2024-02-16 Accepted:2024-03-21 Online:2024-06-21 Published:2024-06-21
Contact: Gang Wu,E-mail:gangwu@cumt.edu.cn E-mail:gangwu@cumt.edu.cn

Abstract

Abstract: Spectral regression discriminant analysis (SRDA) is one of the most popular methods for large-scale discriminant analysis. It is a stepwise algorithm composed of two steps. First, the response vectors are obtained from solving an eigenvalue problem. Second, the projection vectors are computed by solving a least-squares problem. However, the independent two steps can not guarantee the optimality of the two terms. In this paper, we propose a unified framework to compute both the response matrix and the projection matrix in SRDA, so that one can extract the discriminant information of classification tasks more effectively. The convergence of the proposed method is discussed. Moreover, we shed light on how to choose the joint parameter adaptively, and propose a parameter-free joint spectral regression discriminant analysis (JointSRDA-PF) method. Numerical experiments are made on some real-world databases, which show the numerical behavior of the proposed methods and the effectiveness of our strategies.

Key words: Dimension reduction, Spectral regression discriminant analysis (SRDA), Joint principal component and discriminant analysis (JPCDA) algorithm, Joint spectral regression discriminant analysis (JointSRDA)

Gang Wu, Wen Yang. Joint Spectral Regression Methods for Large-Scale Discriminant Analysis[J]. Communications on Applied Mathematics and Computation, 2025, 7(5): 1791-1814.

TrendMD

References

[1] Belhumeour, P., Hespanha, J., Kriegman, D.: Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19, 711-720 (1997)
[2] Cai, D., He, X., Han, J.: SRDA: an efficient algorithm for large-scale discriminant analysis. IEEE Trans. Knowl. Data Eng. 20, 1-12 (2008)
[3] Chung F.: Spectral Graph Theory. AMS, Providence (1997)
[4] Dai, D., Uen, P.: Face recognition by regularized discriminant analysis. IEEE Trans. Syst. Man. Cybern. Part B (Cybernetics) 37, 1080-1085 (2007)
[5] Duda, R., Hart, P.: Pattern Classication and Scene Analysis. Wiley, New York (1973)
[6] Fisher, R.: The use of multiple measurements in taxonomic problems. Ann. Eugen. 7, 179-188 (1936)
[7] Friedman, J.: Regularized discriminant analysis. J. Am. Stat. Assoc. 84, 165-175 (1989)
[8] Gao, W., Cao, B., Shan, S., Chen, X., Zhou, D., Zhang, X., Zhao, D.: The CAS-PEAL large-scale Chinese face database and baseline evaluations. IEEE Trans. Syst. Man. Cybern. Part A 38, 149-161 (2008)
[9] Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. Johns Hopkins University Press, Baltimore (2013)
[10] Halko, N., Martinsson, P., Tropp, J.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53, 217-288 (2011)
[11] He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.: Face recognition using Laplacian faces. IEEE Trans. Pattern Anal. Mach. Intell. 27, 328-340 (2005)
[12] Hu, L., Zhang, W.: Orthogonal neighborhood preserving discriminant analysis with patch embedding for face recognition. Pattern Recognit. 106, 107450 (2020)
[13] Huang, J., Nie, F., Huang H.: Spectral rotation versus K-means in spectral clustering. In: Proceedings of the 27th AAAI Conference on Artificial Intelligence, pp. 431-437 (2013)
[14] Krzanowski, W., Jonathan, P., Mcarthy, W., Thomash, M.: Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. Appl. Stat. 44, 101-115 (1995)
[15] Lu, Y., Wu, G.: Fast and incremental algorithms for exponential semi-supervised discriminant embedding. Pattern Recognit. 108, 107530 (2020)
[16] Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of Machine Learning. The MIT Press, Cambridge (2012)
[17] Moulin, C., Largeron, C., Ducottet, C., Gery, M., Barat, C.: Fisher linear discriminant analysis for text-image combination in multimedia information retrieval. Pattern Recognit. 47, 260-269 (2014)
[18] Nie, F., Zhang, R., Li, X.: A generalized power iteration method for solving quadratic problem on the Stiefel manifold. Sci. China Inf. Sci. 60, 112101 (2017)
[19] Paige, C., Saunders, M.: LSQR: an algorithm for sparse linear equations and sparse least squares. ACM Tran. Math. Soft. 8, 43-71 (1982)
[20] Pang, Y., Xie, J., Nie, F., Li, X.: Spectral clustering by joint spectral embedding and spectral rotation. IEEE Trans. Cybern. 50, 247-258 (2020)
[21] Park, C., Park, H.: A comparison of generalized linear discriminant analysis algorithms. Pattern Recognit. 41, 1083-1097 (2008)
[22] Rao, R.: The utilization of multiple measurements in problems of biological classification. J. R. Stat. Soc. 10, 159-203 (1948)
[23] Sasithradevi, A., Mohamed Mansoor, R.S.: Video classification and retrieval through spatio-temporal Radon features. Pattern Recognit. 99, 107099 (2020)
[24] Shi, W., Wu, G.: Perturbation analysis on PCA plus graph embedding methods and PCA plus exponential graph embedding methods. J. Comput. Appl. Math. 444, 115788 (2024)
[25] Wang, F., Wang, Q., Nie, F., Li, Z., Yu, W., Wang, R.: Unsupervised linear discriminant analysis for jointly clustering and subspace learning. IEEE Trans. Knowl. Data Eng. 33, 1276-1290 (2021)
[26] Wold, S., Esbensen, K., Geladi, P.: Principal component analysis. Chemom. Intell. Lab. Syst. 2, 37-52 (1987)
[27] Wolf, L., Hassner, T., Maoz, I.: Face recognition in unconstrained videos with matched background similarity. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 529-534 (2011)
[28] Wu, G., Feng, T., Zhang, L., Yang, M.: Inexact implementation using Krylov subspace methods for large scale exponential discriminant analysis with applications to high dimensionality reduction problems. Pattern Recognit. 66, 328-341 (2017)
[29] Yu, S., Shi, J.: Multiclass spectral clustering. In: Proceedings Ninth IEEE International Conference on Computer Vision. pp. 313-319 (2003)
[30] Zhang, R., Nie, F., Li, X.: Self-weighted spectral clustering with parameter-free constraint. Neurocomputing 241, 164-170 (2017)
[31] Zhang, X., Cheng, L., Chu, D., Liao, L., NG, M., Tan, R.: Incremental regularized least squares for dimensionality reduction of large-scale data. SIAM J. Sci. Comput. 38, B414-B439 (2016)
[32] Zhang, X., Wang, L., Xiang, S., Liu, C.: Retargeted least squares regression algorithm. IEEE Trans. Neur. Netw. Learn. Syst. 26, 2206-2213 (2014)
[33] Zhao, X., Guo, J., Nie, F., Chen, L., Li, Z., Zhang, H.: Joint principal component and discriminant analysis for dimensionality reduction. IEEE Trans. Neur. Netw. Learn. Sys. 31, 433-444 (2020)
[34] Zhou, Z.: Machine Learning and Its Applications. Tsinghua University Press, Beijing (2006)
[35] Zhu, L., Huang, D.: A Rayleigh-Ritz style method for large-scale discriminant analysis. Pattern Recogn. 47, 1698-1708 (2014)

Joint Spectral Regression Methods for Large-Scale Discriminant Analysis

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 1

Recommended Articles

Metrics

Comments