Immunity-Based Orthogonal Weights Modification Algorithm

doi:10.1007/s42967-024-00377-y

Abstract

Abstract: Recently, the catastrophic forgetting problem of neural networks in the process of continual learning (CL) has attracted more and more attention with the development of deep learning. The orthogonal weight modification (OWM) algorithm to some extent overcomes the catastrophic forgetting problem in CL. It is well-known that the mapping rule learned by the network is usually not accurate in the early stage of neural network training. Our main idea is to establish an immune mechanism in CL, which rejects unreliable mapping rules at the beginning of the training until those are reliable enough. Our algorithm showed a very good competitive advantage in the permuted and disjoint MNIST tasks and disjoint CIFAR-10 tasks. As for the more challenging task of Chinese handwriting character recognition, our algorithm showed a notable improvement compared with the OWM algorithm. In view of the context-dependent processing (CDP) module in [37], we revealed that the module may result in a loss of information and we proposed a modified CDP module to overcome this weakness. The performance of the system with the modified CDP module outperforms the original one in the CelebFaces attributed recognition task. Besides continual multi-task, we also considered a single task, where the immunity-based OWM (IOWM) algorithm was designed as an optimization solver of neural networks for low-dose computed tomography (CT) denoising task.

Key words: Image processing, Neural network, Continual learning (CL), Computed tomography (CT)

CLC Number:

68T07
68U10

Xiaotao Liang, Shuo Yin, Xing Zhao, XuYing Zhao. Immunity-Based Orthogonal Weights Modification Algorithm[J]. Communications on Applied Mathematics and Computation, 2025, 7(2): 718-732.

TrendMD

References

1. Abraham, W.C., Robins, A.: Memory retention–the synaptic stability versus plasticity dilemma. Trends Neurosci. 28(2), 73–78 (2005)
2. Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P., Torr, P., Ranzato, M.: Continual learning with tiny episodic memories. In: Workshop on Multi-Task and Lifelong Reinforcement Learning. arXiv: 1902. 10486 (2019)
3. Chen, H., Zhang, Y., Kalra, M.K., Lin, F., Chen, Y., Liao, P., Zhou, J., Wang, G.: Low-dose CT with a residual encoder-decoder convolutional neural network. IEEE Trans. Med. Imaging 36(12), 2524–2535 (2017)
4. Chen, H., Zhang, Y., Zhang, W., Liao, P., Wang, G.: Low-dose CT via convolutional neural network. Biomed. Opt. Express 8(2), 679 (2017)
5. Coop, R., Mishtal, A., Arel, I.: Ensemble learning in fixed expansion layer networks for mitigating catastrophic forgetting. IEEE Trans. Neural Netw. Learn. Syst. 24(10), 1623–1634 (2013)
6. Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A.A., Pritzel, A., Wierstra, D.: PathNet: evolution channels gradient descent in super neural networks. arXiv: 1701. 08734 (2017)
7. French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)
8. Gepperth, A., Karaoguz, C.: A bio-inspired incremental learning architecture for applied perceptual problems. Cogn. Comput. 8(5), 924–934 (2016)
9. Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. In: The Second International Conference on Learning Representations (2014)
10. He, X., Jaeger, H.: Overcoming catastrophic interference using conceptor-aided backpropagation. In: The Sixth International Conference on Learning Representations (2018)
11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. IEEE (2016)
12. Hetherington, P.A., Seidenberg, M.S.: Is there “catastrophic interference” in connectionist networks? In: Proceedings of the 11th Annual Conference Cognitive Science Society, pp. 26–33. Psychology Press (2014)
13. Hu, W., Lin, Z., Liu, B., Tao, C., Tao, Z., Ma, J., Zhao, D., Yan, R.: Overcoming catastrophic forgetting for continual learning via model adaptation. In: The Seventh International Conference on Learning Representations (2019)
14. Karlik, B., Olgac, A.V.: Performance analysis of various activation functions in generalized MLP architectures of neural networks. Int. J. Artif. Intell. Expert Syst. 1(4), 111–122 (2011)
15. Kemker, R., Kanan, C.: FearNet: brain-inspired model for incremental learning. In: The Sixth International Conference on Learning Representations. arXiv: 1711. 10563 (2018)
16. Kemker, R., McClure, M., Abitino, A., Hayes, T., Kanan, C.: Measuring catastrophic forgetting in neural networks. Proc. AAAI Conf. Artif. Intell. 32(1) (2018)
17. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: The Third International Conference on Learning Representations. arXiv: 1412. 6980 (2015)
18. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)
19. Kumaran, D., Hassabis, D., McClelland, J.L.: What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends Cogn. Sci. 20(7), 512–534 (2016)
20. Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., Zhang, B.-T.: Overcoming catastrophic forgetting by incremental moment matching. Adv. Neural Inf. Process. Syst. 30, 4652–4662 (2017)
21. Liang, X.: A study of immunity-based continual learning algorithm (in Chinese). Master’s thesis, Capital Normal University, China (2021)
22. Liu, C.-L., Yin, F., Wang, D.-H., Wang, Q.-F.: Chinese handwriting recognition contest 2010. In: 2010 Chinese Conference on Pattern Recognition, pp. 1–5. IEEE (2010)
23. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision. IEEE (2015)
24. Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Dec 4–9, 2017, Long Beach, CA, USA, pp. 6467–6476 (2017)
25. Masse, N.Y., Grant, G.D., Freedman, D.J.: Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proc. Natl. Acad. Sci. 115(44), 10467–10475 (2018)
26. McClelland, J.L., McNaughton, B.L., O’Reilly, R.C.: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 102(3), 419 (1995)
27. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Academic Press (1989)
28. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning, pp. 807–814 (2010)
29. Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019)
30. Ratcliff, R.: Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol. Rev. 97(2), 285 (1990)
31. Řehůřek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pp. 45–50. ELRA, Valletta, Malta (2010)
32. Robins, A.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci. 7(2), 123–146 (1995)
33. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323(6088), 533–536 (1986)
34. Saha, G., Garg, I., Roy, K.: Gradient projection memory for continual learning. In: The Ninth International Conference on Learning Representations (2021)
35. Sharkey, N.E., Sharkey, A.J.: An analysis of catastrophic interference. Connect. Sci. 7(3/4), 301–330 (1995)
36. Yang, Q., Yan, P., Zhang, Y., Yu, H., Shi, Y., Mou, X., Kalra, M.K., Zhang, Y., Sun, L., Wang, G.: Low-dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss. IEEE Trans. Med. Imaging 37(6), 1348–1357 (2018)
37. Zeng, G., Chen, Y., Cui, B., Yu, S.: Continual learning of context-dependent processing in neural networks. Nat. Mach. Intell. 1(8), 364–372 (2019)
38. Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 3987–3995 (2017)
39. Zhu, Y., Zhao, M., Zhao, Y., Li, H., Zhang, P.: Noise reduction with low dose CT data based on a modified ROF model. Opt. Express 20(16), 17987–18004 (2012)

[1]	Jarrod Mau, Jia Zhao. Discovery of Governing Equations with Recursive Deep Neural Networks [J]. Communications on Applied Mathematics and Computation, 2025, 7(1): 239-263.
[2]	Jim Magiera, Christian Rohde. A Multiscale Method for Two-Component, Two-Phase Flow with a Neural Network Surrogate [J]. Communications on Applied Mathematics and Computation, 2024, 6(4): 2265-2294.
[3]	Vitaly Gyrya, Mikhail Shashkov, Alexei Skurikhin, Svetlana Tokareva. Machine Learning Approaches for the Solution of the Riemann Problem in Fluid Dynamics: a Case Study [J]. Communications on Applied Mathematics and Computation, 2024, 6(3): 1832-1859.
[4]	Zhiwei Gao, Tao Tang, Liang Yan, Tao Zhou. Failure-Informed Adaptive Sampling for PINNs, Part II: Combining with Re-sampling and Subset Simulation [J]. Communications on Applied Mathematics and Computation, 2024, 6(3): 1720-1741.
[5]	Wes Whiting, Bao Wang, Jack Xin. Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 1175-1188.
[6]	Zhanhong Ye, Xiang Huang, Hongsheng Liu, Bin Dong. Meta-Auto-Decoder: a Meta-Learning-Based Reduced Order Model for Solving Parametric Partial Differential Equations [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 1096-1130.
[7]	Jane Wu, Michael Bao, Xinwei Yao, Ronald Fedkiw. Deep Energies for Estimating Three-Dimensional Facial Pose and Expression [J]. Communications on Applied Mathematics and Computation, 2024, 6(2): 837-861.
[8]	Wei-Hung Su, Ching-Shan Chou, Dongbin Xiu. Data-Driven Modeling of Partially Observed Biological Systems [J]. Communications on Applied Mathematics and Computation, 2024, 6(1): 739-754.
[9]	Rémi Abgrall, Maria Han Veiga. Neural Network-Based Limiter with Transfer Learning [J]. Communications on Applied Mathematics and Computation, 2023, 5(2): 532-572.
[10]	Jia Zhao. Discovering Phase Field Models from Image Data with the Pseudo-Spectral Physics Informed Neural Networks [J]. Communications on Applied Mathematics and Computation, 2021, 3(2): 357-369.
[11]	Jun Hou, Tong Qin, Kailiang Wu, Dongbin Xiu. A Non-intrusive Correction Algorithm for Classifcation Problems with Corrupted Data [J]. Communications on Applied Mathematics and Computation, 2021, 3(2): 337-356.