Communications on Applied Mathematics and Computation >

2025 , Vol. 7 >Issue 2: 718 - 732

DOI: https://doi.org/10.1007/s42967-024-00377-y

Immunity-Based Orthogonal Weights Modification Algorithm

Expand
  • 1 School of Mathematical Sciences, Capital Normal University, Beijing 100048, China;
    2 Beijing Advanced Innovation Center for Imaging Technology, Capital Normal University, Beijing 100048, China

Received date: 2023-10-22

  Revised date: 2024-01-24

  Accepted date: 2024-01-26

  Online published: 2025-04-21

Supported by

The research is supported by the Beijing Natural Science Foundation, China (Grant/ Award Number: Z210003), the National Nature Science Foundation of China (NSFC) (Grant/Award Numbers: 12071313, 61827809), the key research project of the Academy for Multidisciplinary Studies, Capital Normal University, China, the National Key Research and Development Program of China (Grant/ Award Number: 2020YFA0712200), and the Major Technologies R & D Program of Shenzhen, China (JSGGZD20220822095600001).

Fold

Abstract

Recently, the catastrophic forgetting problem of neural networks in the process of continual learning (CL) has attracted more and more attention with the development of deep learning. The orthogonal weight modification (OWM) algorithm to some extent overcomes the catastrophic forgetting problem in CL. It is well-known that the mapping rule learned by the network is usually not accurate in the early stage of neural network training. Our main idea is to establish an immune mechanism in CL, which rejects unreliable mapping rules at the beginning of the training until those are reliable enough. Our algorithm showed a very good competitive advantage in the permuted and disjoint MNIST tasks and disjoint CIFAR-10 tasks. As for the more challenging task of Chinese handwriting character recognition, our algorithm showed a notable improvement compared with the OWM algorithm. In view of the context-dependent processing (CDP) module in [37], we revealed that the module may result in a loss of information and we proposed a modified CDP module to overcome this weakness. The performance of the system with the modified CDP module outperforms the original one in the CelebFaces attributed recognition task. Besides continual multi-task, we also considered a single task, where the immunity-based OWM (IOWM) algorithm was designed as an optimization solver of neural networks for low-dose computed tomography (CT) denoising task.

Key words： Image processing; Neural network; Continual learning (CL); Computed tomography (CT)

Cite this article

Xiaotao Liang, Shuo Yin, Xing Zhao, XuYing Zhao . Immunity-Based Orthogonal Weights Modification Algorithm[J]. Communications on Applied Mathematics and Computation, 2025 , 7(2) : 718 -732 . DOI: 10.1007/s42967-024-00377-y

References

1. Abraham, W.C., Robins, A.: Memory retention–the synaptic stability versus plasticity dilemma. Trends Neurosci. 28(2), 73–78 (2005)
2. Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P., Torr, P., Ranzato, M.: Continual learning with tiny episodic memories. In: Workshop on Multi-Task and Lifelong Reinforcement Learning. arXiv: 1902. 10486 (2019)
3. Chen, H., Zhang, Y., Kalra, M.K., Lin, F., Chen, Y., Liao, P., Zhou, J., Wang, G.: Low-dose CT with a residual encoder-decoder convolutional neural network. IEEE Trans. Med. Imaging 36(12), 2524–2535 (2017)
4. Chen, H., Zhang, Y., Zhang, W., Liao, P., Wang, G.: Low-dose CT via convolutional neural network. Biomed. Opt. Express 8(2), 679 (2017)
5. Coop, R., Mishtal, A., Arel, I.: Ensemble learning in fixed expansion layer networks for mitigating catastrophic forgetting. IEEE Trans. Neural Netw. Learn. Syst. 24(10), 1623–1634 (2013)
6. Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A.A., Pritzel, A., Wierstra, D.: PathNet: evolution channels gradient descent in super neural networks. arXiv: 1701. 08734 (2017)
7. French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)
8. Gepperth, A., Karaoguz, C.: A bio-inspired incremental learning architecture for applied perceptual problems. Cogn. Comput. 8(5), 924–934 (2016)
9. Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. In: The Second International Conference on Learning Representations (2014)
10. He, X., Jaeger, H.: Overcoming catastrophic interference using conceptor-aided backpropagation. In: The Sixth International Conference on Learning Representations (2018)
11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. IEEE (2016)
12. Hetherington, P.A., Seidenberg, M.S.: Is there “catastrophic interference” in connectionist networks? In: Proceedings of the 11th Annual Conference Cognitive Science Society, pp. 26–33. Psychology Press (2014)
13. Hu, W., Lin, Z., Liu, B., Tao, C., Tao, Z., Ma, J., Zhao, D., Yan, R.: Overcoming catastrophic forgetting for continual learning via model adaptation. In: The Seventh International Conference on Learning Representations (2019)
14. Karlik, B., Olgac, A.V.: Performance analysis of various activation functions in generalized MLP architectures of neural networks. Int. J. Artif. Intell. Expert Syst. 1(4), 111–122 (2011)
15. Kemker, R., Kanan, C.: FearNet: brain-inspired model for incremental learning. In: The Sixth International Conference on Learning Representations. arXiv: 1711. 10563 (2018)
16. Kemker, R., McClure, M., Abitino, A., Hayes, T., Kanan, C.: Measuring catastrophic forgetting in neural networks. Proc. AAAI Conf. Artif. Intell. 32(1) (2018)
17. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: The Third International Conference on Learning Representations. arXiv: 1412. 6980 (2015)
18. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)
19. Kumaran, D., Hassabis, D., McClelland, J.L.: What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends Cogn. Sci. 20(7), 512–534 (2016)
20. Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., Zhang, B.-T.: Overcoming catastrophic forgetting by incremental moment matching. Adv. Neural Inf. Process. Syst. 30, 4652–4662 (2017)
21. Liang, X.: A study of immunity-based continual learning algorithm (in Chinese). Master’s thesis, Capital Normal University, China (2021)
22. Liu, C.-L., Yin, F., Wang, D.-H., Wang, Q.-F.: Chinese handwriting recognition contest 2010. In: 2010 Chinese Conference on Pattern Recognition, pp. 1–5. IEEE (2010)
23. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision. IEEE (2015)
24. Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Dec 4–9, 2017, Long Beach, CA, USA, pp. 6467–6476 (2017)
25. Masse, N.Y., Grant, G.D., Freedman, D.J.: Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proc. Natl. Acad. Sci. 115(44), 10467–10475 (2018)
26. McClelland, J.L., McNaughton, B.L., O’Reilly, R.C.: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 102(3), 419 (1995)
27. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Academic Press (1989)
28. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning, pp. 807–814 (2010)
29. Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019)
30. Ratcliff, R.: Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol. Rev. 97(2), 285 (1990)
31. Řehůřek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pp. 45–50. ELRA, Valletta, Malta (2010)
32. Robins, A.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci. 7(2), 123–146 (1995)
33. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323(6088), 533–536 (1986)
34. Saha, G., Garg, I., Roy, K.: Gradient projection memory for continual learning. In: The Ninth International Conference on Learning Representations (2021)
35. Sharkey, N.E., Sharkey, A.J.: An analysis of catastrophic interference. Connect. Sci. 7(3/4), 301–330 (1995)
36. Yang, Q., Yan, P., Zhang, Y., Yu, H., Shi, Y., Mou, X., Kalra, M.K., Zhang, Y., Sun, L., Wang, G.: Low-dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss. IEEE Trans. Med. Imaging 37(6), 1348–1357 (2018)
37. Zeng, G., Chen, Y., Cui, B., Yu, S.: Continual learning of context-dependent processing in neural networks. Nat. Mach. Intell. 1(8), 364–372 (2019)
38. Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 3987–3995 (2017)
39. Zhu, Y., Zhao, M., Zhao, Y., Li, H., Zhang, P.: Noise reduction with low dose CT data based on a modified ROF model. Opt. Express 20(16), 17987–18004 (2012)
Options
Outlines

/

〈 〉

Copyright © Editorial By Communications on Applied Mathematics and Computation
沪ICP备09014157