Improved Three-term Conjugate Gradient Algorithm For Training Neural Network
DOI:
https://doi.org/10.31642/JoKMC/2018/020309Keywords:
Feed Forward Neural Network, Training Algorithms, Optimization, Three Terms Conjugate Gradient algorithms Neural Network.Abstract
A new three-term conjugate gradient algorithm for training feed-forward neural networks is developed. It is a vector based training algorithm derived from DFP quasi-Newton and has only O(n) memory. The global convergence to the proposed algorithm has been established for convex function under Wolfe condition. The results of numerical experiments are included and compared with other well known training algorithms in this field.Downloads
References
Haykin S(1994). 'Neural Network A comprehensive function: MacmillanCollege Publishing Company New york".
Rumelhart D., Hinton G., and Williams R (1986).'Learning internal representation by error propagation' . In Rumelhart D and Mcllandeditors, parallel Distributed processing explorations in theMicrostructure of conjugate Cambridge.
Livieris I. and Pintelas P.(2009).' Performance evaluation ofdescent CG methods for neural network training.In E.A. Lipitakis, editor, 9th Hellenic European Research on Computer Mathematic & its Applications Conference (HERCMA’09).
Kalil A. and Hind M. (2014). 'Conjugate gradient algorithm based on Aiten'sProcessfor training neural networks'. Raf. J. of Comp & Math. Vol.(11), No.(1). DOI: https://doi.org/10.33899/csmj.2014.163730
Battiti R. (1992). '1st and 2nd order method for learning betweensteepest descent and Newton method'. Neural Comp. 4(2). DOI: https://doi.org/10.1162/neco.1992.4.2.141
Fletcher R. and Reeves C. (1964).' function minimization byconjugate gradients'. Computer Journal. (7). DOI: https://doi.org/10.1093/comjnl/7.2.149
Noccedal J.(1996).' conjugate gradient methods and nonlinearoptimization. In [Adams.J.L.Nazareth (EDS.) linear and nonlinear conjugate gradient related methods. SIAM.
Parker D.(1987).'Optimal algorithms for adaptive Networks:second order back- propagation, second order Direct propagation,and second order Hebian learning.' Proceedings of the 1987 IEEE International Conference on Neural Networks, vol (2).
Andreas A. and Wu S. (2007).'Practical Optimization Algorithms and Engineering Applications', Springer Science and Business Media, LLC New York.
Livieris I., and pintelas P.(2011).'An advanced conjugate gradientraining algorithm based on a modified secant equation'. TechnicalReport No. TR11-03'. University of patras. Dep.of Math. GR-265 04 patras , Greece.
Andrei N.(2013). 'A simple three-term conjugate gradient f unconstrained optimization'. J. of Computation and AppliedMathematics (241). DOI: https://doi.org/10.1016/j.cam.2012.10.002
Zoutendijk G. (1970). 'Non-linear programming computational Methods', in J. Abadi (Ed). Integer and Non-linear Programming,North Holland.
Nguyen D. and Widrow B. (1990).' Improving the learning speedof 2-layer neural network by choosing initial values of adaptiveweights. Biological Cybernetics, 59. DOI: https://doi.org/10.1109/IJCNN.1990.137819
Shanno D. and Phua K.(1976).' Minimization of unconstrained multivariate functions'. ACM Transactions on Mathematical Software, 2. DOI: https://doi.org/10.1145/355666.355673
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Abbas H. Taqi
This work is licensed under a Creative Commons Attribution 4.0 International License.
which allows users to copy, create extracts, abstracts, and new works from the Article, alter and revise the Article, and make commercial use of the Article (including reuse and/or resale of the Article by commercial entities), provided the user gives appropriate credit (with a link to the formal publication through the relevant DOI), provides a link to the license, indicates if changes were made and the licensor is not represented as endorsing the use made of the work.
