Communications on Applied Mathematics and Computation >

2021 , Vol. 3 >Issue 2: 337 - 356

DOI: https://doi.org/10.1007/s42967-020-00084-4

ORIGINAL PAPER

A Non-intrusive Correction Algorithm for Classifcation Problems with Corrupted Data

Expand
  • Department of Mathematics, The Ohio State University, Columbus, OH 43210, USA

Received date: 2020-02-10

  Revised date: 2020-06-02

  Online published: 2021-05-26

Fold

Abstract

A novel correction algorithm is proposed for multi-class classifcation problems with corrupted training data. The algorithm is non-intrusive, in the sense that it post-processes a trained classifcation model by adding a correction procedure to the model prediction. The correction procedure can be coupled with any approximators, such as logistic regression, neural networks of various architectures, etc. When the training dataset is sufciently large, we theoretically prove (in the limiting case) and numerically show that the corrected models deliver correct classifcation results as if there is no corruption in the training data. For datasets of fnite size, the corrected models produce signifcantly better recovery results, compared to the models without the correction algorithm. All of the theoretical fndings in the paper are verifed by our numerical examples.

Key words： Data corruption; Deep neural network; Cross-entropy; Label corruption; Robust loss

Cite this article

Jun Hou, Tong Qin, Kailiang Wu, Dongbin Xiu . A Non-intrusive Correction Algorithm for Classifcation Problems with Corrupted Data[J]. Communications on Applied Mathematics and Computation, 2021 , 3(2) : 337 -356 . DOI: 10.1007/s42967-020-00084-4

References

1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.:TensorFlow:large-scale machine learning on heterogeneous systems (2015). http://tensorfow.org/
2. Brooks, J.P.:Support vector machines with the ramp loss and the hard margin loss. Oper. Res. 59, 467-479 (2011)
3. Chang, L.-B.:Partial order relations for classifcation comparisons. The Canadian Journal of Statistics. https://doi.org/10.1002/cjs.11524 (2019)
4. Chollet, F., et al.:Keras. https://keras.io(2015)
5. Frénay, B., Verleysen, M.:Classifcation in the presence of label noise:a survey. IEEE Trans. Neural Netw. Learn. Syst. 25, 845-869 (2014)
6. Ghosh, A., Kumar, H., Sastry, P.:Robust loss functions under label noise for deep neural networks. In:Thirty-First AAAI Conference on Artifcial Intelligence, arXiv:1712.09482v1 (2017)
7. Ghosh, A., Manwani, N., Sastry, P.:Making risk minimization tolerant to label noise. Neurocomputing 160, 93-107 (2015)
8. Graves, A., Mohamed, A.-r., Hinton, G.:Speech recognition with deep recurrent neural networks. In:IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645-6649 (2013)
9. Haber, E., Ruthotto, L.:Stable architectures for deep neural networks. Inverse Probl. 34, 014004 (2017)
10. He, K., Zhang, X., Ren, S., Sun, J.:Deep residual learning for image recognition, In:IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778 (2016)
11. Hendrycks, D., Mazeika, M., Wilson, D., Gimpel, K.:Using trusted data to train deep networks on labels corrupted by severe noise. In:Advances in Neural Information Processing Systems 31, pp. 10477-10486 (2018)
12. Jiang, L., Zhou, Z., Leung, T., Li, L.-J., Fei-Fei, L.:Mentornet:regularizing very deep neural networks on corrupted labels. arXiv:1712.05055 (2017)
13. Khetan, A., Lipton, Z.C., Anandkumar, A.:Learning from noisy singly-labeled data. arXiv:1712.04577 (2017)
14. Kingma, D.P., Ba, J.:Adam:a method for stochastic optimization. arXiv:1412.6980 (2014)
15. Larsen, J., Nonboe, L., Hintz-Madsen, M., Hansen, L.K.:Design of robust neural network classifers. In:Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP'98 (Cat. No. 98CH36181), vol. 2, IEEE, pp. 1205-1208 (1998)
16. LeCun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.:Handwritten digit recognition with a back-propagation network. In:Advances in Neural Information Processing Systems 2 (NIPS 1990), pp. 396-404 (1990)
17. Li, B., Wang, Y., Singh, A., Vorobeychik, Y.:Data poisoning attacks on factorization-based collaborative fltering. In:Advances in Neural Information Processing Systems 29 (NIPS 2016), pp. 1885-1893 (2016)
18. Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., Li, L.-J.:Learning from noisy labels with distillation. In:Proceedings of the IEEE International Conference on Computer Vision (2017), pp. 1910-1918 (2017)
19. Liu, T., Tao, D.:Classifcation with noisy labels by importance reweighting. IEEE Trans. Pattern. Anal. Mach. Intell. 38, 447-461 (2016)
20. Long, P.M., Servedio, R.A.:Random classifcation noise defeats all convex potential boosters. Mach. Learn. 78, 287-304 (2010)
21. Manwani, N., Sastry, P.:Noise tolerance under risk minimization. IEEE Trans. Cybern. 43, 1146-1151 (2013)
22. Masnadi-Shirazi, H., Vasconcelos, N.:On the design of loss functions for classifcation:theory, robustness to outliers, and savageboost. In:Advances in Neural Information Processing Systems 21 (NIPS 2008), pp. 1049-1056 (2008)
23. Menon, A., Van Rooyen, B., Ong, C.S., Williamson, B.:Learning from corrupted binary labels via class-probability estimation. In:International Conference on Machine Learning, pp. 125-134 (2015)
24. Mnih, V., Hinton, G.E.:Learning to label aerial images from noisy data. In:Proceedings of the 29th International Conference on Machine Learning (ICML-12), pp. 567-574 (2012)
25. Nair, V., Hinton, G.E.:Rectifed linear units improve restricted boltzmann machines. In:Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807-814 (2010)
26. Natarajan, N., Dhillon, I.S., Ravikumar, P.K., Tewari, A.:Learning with noisy labels. In:Advances in Neural Information Processing Systems 26, pp. 1196-1204 (2013)
27. Nettleton, D.F., Orriols-Puig, A., Fornells, A.:A study of the efect of diferent types of noise on the precision of supervised learning techniques. Artif. Intell. Rev. 33, 275-306 (2010)
28. Northcutt, C.G., Wu, T., Chuang, I.L.:Learning with confdent examples:rank pruning for robust classifcation with noisy labels. arXiv:1705.01936 (2017)
29. Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., Qu, L.:Making deep neural networks robust to label noise:a loss correction approach. In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952 (2017)
30. Ren, M., Zeng, W., Yang, B., Urtasun, R.:Learning to reweight examples for robust deep learning. arXiv:1803.09050 (2018)
31. Steinhardt, J., Koh, P.W.W., Liang, P.S.:Certifed defenses for data poisoning attacks. In:Advances in Neural Information Processing Systems NIPS, pp. 3520-3532 (2017)
32. Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L., Fergus, R.:Training convolutional networks with noisy labels. arXiv:1406.2080 (2014)
33. Van Rooyen, B., Menon, A., Williamson, R.C.:Learning with symmetric label noise:the importance of being unhinged. In:Advances in Neural Information Processing Systems, pp. 10-18 (2015)
34. Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., Belongie, S.:Learning from noisy large-scale datasets with minimal supervision. In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 839-847 (2017)
35. Xiao, H., Rasul, K., Vollgraf, R.:Fashion-mnist:a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747 (2017)
36. Xiao, T., Xia, T., Yang, Y., Huang, C., Wang, X.:Learning from massive noisy labeled data for image classifcation. In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2691-2699 (2015)
37. Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.:Understanding deep learning requires rethinking generalization. arXiv:1611.03530 (2016)
38. Zhang, J., Yang, Y.:Robustness of regularized linear classifcation methods in text categorization. In:Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval, ACM, pp. 190-197 (2003)
39. Zhang, Z., Sabuncu, M.:Generalized cross entropy loss for training deep neural networks with noisy labels. In:The 32nd Conference on Neural Information Processing Systems, pp. 8792-8802 (2018)
40. Zhu, X., Wu, X.:Class noise vs. attribute noise:a quantitative study. Artif. Intell. Rev. 22, 177-210 (2004)
Options
Outlines

/

〈 〉

Copyright © Editorial By Communications on Applied Mathematics and Computation
沪ICP备09014157