An Online Gradient Method with Smoothing L_0 Regularization for Pi-Sigma Network
DOI:
https://doi.org/10.14738/tmlai.66.5838Keywords:
Convergence, Online gradient method, Pi-Sigma networks, Smoothing L_0 regularization.Abstract
The description of this study is to make possibility analysis solution of online gradient method with smoothing regularization for pi-sigma network training. Due to the effectiveness computational and theoretical analysis are a very important issues to improve the generalization performance of networks and the gradient descent algorithm with regularization is widely used method. However, regularization is reefed to NP-hard nature problems, which has not differentiable objective functional-penalty term. In this paper to avoid this trick, we use a smoothing function to recover the origin regularization into smoothing regularization. Under this condition, the resulting obtained as a good decreases solution when compared with others. The monotonically of the error function, weak and strong convergence theorems are proved.
References
(1) Shin, Y., et al., The pi-sigma network: An efficient higher-order neural network for pattern classification and function approximation. In Neural Networks, 1991., IJCNN-91-Seattle International Joint Conference on IEEE, 1991. 1: p. 13-18.
(2) Akram, U., et al., A comprehensive Survey on Pi-Sigma Neural Network for Time Series Prediction. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), 2017. 9(3-3): p.57-62.
(3) Shin, Y., et al., Computationally efficient invariant pattern recognition with higher order Pi-Sigma Networks. The University of Texas at Austin, 1992.
(4) Setiono, R., A penalty-function approach for pruning feedforward neural networks. Neural computation, 1997. 9(1): p.185-204.
(5) Reed, R., Pruning algorithms-a survey, IEEE Trans Neural Network, 1993. 4 (5): p.740–7.
(6) Wu, W., et al., Deterministic convergence of an online gradient method for BP neural networks. IEEE Transactions on Neural Networks, 2005. 16(3): p.533-540.
(7) Hagan, M.T., et al., Training feedforward networks with the Marquardt algorithm. IEEE transactions on Neural Networks, 1994. 5(6): p.989-993.
(8) Akaike, H., A new look at the statistical model identification. IEEE transactions on automatic control, 1974. 19(6): p.716-723.
(9) Donoho, D. L., Neighborly Polytopes and the Sparse Solution of Underdetermined Systems of Linear Equations. To appear. IEEE Trans. Info. Thry (2005).
(10) Tibshirani, R., Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 1996. P. 267-288.
(11) Reed, R., Pruning algorithms-a survey, IEEE Trans Neural Network, 1993. 4 (5): p.740–7.
(12) Hinton, G.E., Connectionist learning procedures. Artificial intelligence, 1989. 40(1-3): 185-234.
(13) Zhang, H., et al., Online gradient method with smoothing ‘0regularization for feedforward neural networks. Neurocomputing, 2017. 224: p.1-8.
(14) Fan, Q., et al., Convergence of online gradient method for feedforward neural networks with smoothing L1/2regularization penalty. Neurocomputing, 2014. 131: p.208-216.
(15) Xu, Z.B., et al., When does online BP training converge?. IEEE Transactions on Neural Networks, 2009. 20(10): p.1529-1539.
