Nonlinear Time Series Prediction Performance Using Constrained Motion Particle Swarm Optimization
DOI:
https://doi.org/10.14738/tmlai.55.3566Keywords:
Support Vector Regression, Constrained Motion Particle Swarm Optimization (CMPSO), Particle Swarm Optimization, Nonlinear Time Series Prediction.Abstract
Constrained Motion Particle Swarm Optimization (CMPSO) is a general framework for optimizing Support Vector Regression (SVR) free parameters for nonlinear time series regression and prediction. CMPSO uses Particle Swarm Optimization (PSO) to determine the SVR free parameters. However, CMPSO attempts to fuse the PSO and SVR algorithms by constraining the SVR Lagrange multipliers such that every PSO epoch yields a candidate solution that meets the SVR constraint criteria. The fusion of these two algorithms provides a numerical efficiency advantage since an SVR Quadratic Program (QP) solver is not necessary for every particle at every epoch. This reduces the search space of the overall optimization problem. It has been demonstrated that CMPSO provides similar (and in some cases superior) performance to other contemporary time series prediction algorithms for nonlinear time series benchmarks such as Mackey-Glass data. This paper details the CMPSO algorithm framework and tests its performance against other SVR time series prediction algorithms and data including the European Network on Intelligent Technologies for Smart Adaptive Systems (EUNITE) competition data and the Competition on Artificial Time Series (CATS) competition data.
References
(1) N. Sapankevych and R. Sankar, “Time Series Prediction Using Support Vector Machines: A Survey”, IEEE Computational Intelligence, vol. 4, no. 2, pp. 24-38, May 1999.
(2) N. Sapankevych and R. Sankar, “Constrained Motion Particle Swarm Optimization and Support Vector Regression for Non-Linear Time Series Regression and Prediction Applications”, 12th International Conference on
Machine Learning and Applications (ICMLA 2013), December 2013.
(3) V. N. Vapnik, “The Nature of Statistical Learning Theory”, Springer, 1995.
(4) V. N. Vapnik, “Statistical Learning Theory”, John Wiley and Sons, 1998.
(5) A. J. Smola and B. Schölkopf, “A Tutorial on Support Vector Regression”, NeuroCOLT Technical Report, Royal Holloway College, London, UK, 1998.
(6) N. Cristianini and J. Shawe-Taylor, “An Introduction to Support Vector Machines and Other Kernel-based Learning Methods”, Cambridge University
Press, 2000.
(7) John C. Platt, "Sequential Minimization Optimization: A Fast Algorithm for Training Support Vector Machines", Technical Report MSR-TR-98-14, Microsoft Research, 1998.
(8) J. Kennedy and R. Eberhart, "Particle Swarm Optimization", Proceedings of the IEEE International Conference on Neural Networks, vol. 4, pp. 1942–1948, 1995.
(9) R. Poli, J. Kennedy, and T. Blackwell, "Particle Swarm Optimization: An Overview", Swarm Intell, vol. 1, pp. 33-57, 2007.
(10) J. Robinson and Y. Rahmat-Samii, "Particle Swarm Optimization in Electromagnetics", IEEE Transactions on Antennas and Propagation, vol. 52, no. 2, February 2004.
(11) R. Poli, "An Analysis of Publications on Particle Swarm Optimization Applications", University of Essex, UK, Department of Computer Science, Technical Report CSM-469, May 2007, Revised November 2007.
(12) M. Reyes-Sierra and C. A. C. Coello, “Multi-Objective Particle Swarm Optimizers: A Survey of the State-of-the-Art,” International Journal of Computational Intelligence Research, vol. 2, no. 3, pp. 287-308, 2006
(13) J. Fieldsend, "Multi–Objective Particle Swarm Optimization Methods", Technical Report No. 419, Department of Computer Science, University of Exeter, 2004.
(14) W.-C. Hong, "Chaotic Particle Swarm Optimization Algorithm in a Support Vector Regression Electric Load Forecasting Model", Energy Conversion and Management, vol. 50, no. 1, pp. 105-117, 2009.
(15) X. C. Guo, J. H. Yang, C. G. Wu, C. Y. Wang, and Y. C. Liang, "A Novel LS-SVMs Hyper-Parameter Selection Based on Particle Swarm Optimization", Neurocomputing, vol. 71, no. 16-18, pp. 3211-3215, 2008.
(16) H. Yuan, Y. Zhang, D. Zhang, and G. Yang, "A Modified Particle Swarm
Optimization Algorithm for Support Vector Machine Training", Proceedings of the 6th World Congress on Intelligent Control and Automation, June
-23, 2006.
(17) U. Paquest and A. P. Engelbrecht, "A New Particle Swarm Optimizer for Linearly Constrained Optimization", Proceedings IEEE Congress on Evolutionary Computation, pp. 227-233, 2003.
(18) U. Paquest and A. P. Engelbrecht, "Training Support Vector Machines with Particle Swarms", Proceedings International Joint Conference on Neural Networks, pp. 1593-1598, 2003.
(19) X. Wang, J. Lu and J. Liu, “Wavelet Transform and PSO Support Vector Machine Based approach for Time Series Forecasting”, International Conference on Artificial Intelligence and Computational Intelligence, vol. 1, pp. 46-50, Nov 2009.
(20) J. Hu, P. Gao, Y. Yao, and X. Xie, “Traffic Flow Forecasting with Particle Swarm Optimization and Support Vector Regression”, IEEE 17th International Conference on Intelligent Transportation Systems, pp. 2267-2268, October 2014.
(21) Y. Wen and Y. Chen, “Modified Parallel Cat Swarm Optimization in SVM Modeling for Short-term Cooling Load Forecasting”, Journal of Software, vol. 9, no. 8, August 2014.
(22) X. Qiu, L. Zhang, Y. Ren, P. N. Suganthan, and G. Amaratunga, “Ensemble Deep Learning for Regression and Time Series Forecasting”, IEEE Symposium on Computational Intelligence in Ensemble Learning, pp. 1-6, December 2014.
(23) C. Rajeswari, B. Sathiyabhama, S. Devendiran, and K. Manivannan, “Bearing Fault Diagnosis Using Wavelet Packet Transform, Hybrid PSO, and Support Vector Machine”, 12th Global Congress on Manufacturing and Management – Procedia Engineering, vol. 97, 2014.
(24) P. Shinde, T. Parvat, “Analysis on: Intrusions Detection Based on Support Vector Machine Optimized with Swarm Intelligence”, International Journal of Computer Science and Mobile Computing, vol. 3, no. 12, pp. 559-566, December 2014.
(25) W. Wenhai, D. Jiandong, “Short-term Wind Power Forecasting Based on Maximum Correntropy Criterion”, International Conference on Power System Technology, pp. 2800-2805, October 2014.
(26) Z. Hu, Q. Liu, Y. Tian, Y. Liao, “A Short-term Wind Speed Forecasting Model Based on Improved QPSO Optimizing LSSVM”, International Conference on Power System Technology, pp. 2806-2811, October 2014.
(27) H. Dong, X. Zhu, Y. Liu, F. He, and G. Huo, “Iris Recognition Based on CPSO Algorithm for Optimizing Multichannel Gabor Parameters”, Journal of Computational Information Systems, vol. 11, pp. 333-340, 2015.
(28) J. F. L. de Oliveira, T. B. Ludermir, “A Distributed PSO-ARIMA-SVR Hybrid System for Time Series Forecasting”, IEEE International Conferences on Systems, Man, and Cybernetics, pp. 3867-3872, October 2014.
(29) http://www.eunite.org (January 2015)
(30) http://neuron-ai.tuke.sk/competition/ (January 2015)
(31) M.-W. Chang, B.-J. Chen, and C.-J. Lin, “EUNITE Network Competition: Electricity Load Forecasting”, November 2001. (see [19]).
(32) B.-J. Chen, M.-W. Chang, and C.-J. Lin, "Load Forecasting Using Support Vector Machines: A Study on EUNITE Competition 2001", IEEE Transactions on Power Systems, vol. 19, no. 4, pp. 1821-1830, November 2004.
(33) B. Bican, Y. Yaslan, “A Hybrid Method for Time Series Prediction Using EMD and SVR”, 6th International Conference on Communications, Control, and Signal Processing, pp. 566-569, May 2014.
(34) X. Yang, “Comparison of the LS-SVM Based Load Forecasting Models”, International Conference on Electronic & Mechanical Engineering and Information Technology, pp. 2942-2945, Aug 2011.
(35) I. Fernandez, C. E. Borges, and Y. K. Penya, “Efficient Building Load Forecasting”, IEEE 16th Conference on Emerging Technologies & Factory Automation, pp. 1-8, September 2011.
(36) B. E. Turkay and D. Demren, “Electrical Load Forecasting Using Support Vector Machines”, 7th International Conference on Electrical and Electronics Engineering, vol. I, pp. 49-53, December 2011.
(37) T. T. Chen, S. J. Lee, “A Weighted LS-SVM Based Learning System for Time Series Forecasting”, Information Sciences, pp. 99-116, 2015.
(38) Lendasse, E. Oja, O. Simula, and M. Verleysen, “Time Series Prediction Competition: The CATS Benchmark”, Proc. IEEE International Joint Conference on Neural Networks, pp. 1615-1620, 25-29 July 2004.
(39) S. Sarkka, A. Vehtari, and J. Lampinen, “Time Series Prediction by Kalman Smoother with Cross Validated Noise Density”, Proc. IEEE
International Joint Conference on Neural Networks, vol. 2, pp. 1653-1657,
-29 July 2004.
(40) X. Cai, N. Zhang, G. Venayagamoorthy and D. Wunsch, “Time Series Prediction with Recurrent Neural Networks Using a Hybrid PSO-EA Algorithm”, International Joint Conference on Neural Networks, vol. 2, pp. 1647-1652, 2004.
(41) S. Kurogi, T. Ueno and M. Sawa, “Batch Learning Competitive Associative Net and Its Application to Time Series Prediction”, International Joint Conference on Neural Networks, vol. 2, pp. 1591-1596, 2004.
(42) X. Hu and D. Wunsch, “IJCNN 2004 Challenge Problem: Time Series Prediction with a Weighted Bidirectional Multi-stream Extended Kalman Filter”, International Joint Conference on Neural Networks, vol. 2, pp. 1641-1645, 2004.
(43) F. Palacios-Gonzalez, “A SVCA Model for The Competition on Artificial Time Series”, International Joint Conference on Neural Networks, vol. 4, pp. 2777-2782, 2004.
(44) L. J. Herrera Maldonado, H. Pomares, I. Rojas, J. Gonzalez and M. Awad, “MultiGrid-Based Fuzzy Systems for Time Series Forecasting: CATS Benchmark IJCNN Competition”, International Joint Conference on Neural Networks, vol. 2, pp. 1603-1608, 2004.
(45) G. Simon, J. A. Lee, M. Verleysen and M. Cottrell, “Double Quantization Forecasting Method for Filling Missing Data in the CATS Time Series”, International Joint Conference on Neural Networks, vol. 2, pp. 1635-1640, 2004.
(46) P. F. Verdes, P. M. Granitto, M. I. Szeliga, A. Rebola and H. A. Ceccatto, “Prediction of the CATS benchmark exploiting time-reversal symmetry”, International Joint Conference on Neural Networks, vol. 2, pp. 1631-1634, 2004.
(47) H.-W. Chan, W.-C. Leung, K.-C. Chiu and L. Xu, “BYY Harmony Learning Based Mixture of Experts Model for Non-stationary Time Series Prediction”, International Joint Conference on Neural Networks, 2004.
(48) J. Wichard and M. Ogorzalek, “Time Series Prediction with Ensemble Models”, International Joint Conference on Neural Networks, vol. 2, pp. 1625-1630, 2004.
(49) D. Samek and D. Manas, “Comparison of Artificial Neural Networks Using Prediction Benchmarking”, Proceedings of the 13th International Conference on Automatic Control, Modelling & Simulation, pp. 152-157, 2011.
(50) Y. Dong, J. Zhang, and J. M. Garibaldi, “Neural Networks and AdaBoost Algorithm Based Ensemble Models for Enhance Forecasting of Nonlinear Time Series”, International Joint Conference on Neural Networks, vol. 1, pp. 149-156, July 2014.
(51) T. Kuremoto, S. Kimura, K. Kobayashi, and M. Obayashi, “Time Series Forecasting Using a Deep Belief Network with Restricted Boltzmann Machines”, Neurocomputing, pp. 47-56, 2014.
(52) R. Ormandi, “Applications of Support Vector-Based Learning”, Research Group on Artificial Intelligence of the University of Szeged and the Hungarian Academy of Sciences, Ph. D. Dissertation, 2013.
(53) P. Mirowski, “Time Series Modeling with Hidden Variables and Gradient-Based Algorithms”, Department of Computer Science, Courant Institute of Mathematical Sciences, New York University, Ph. D. Dissertation, 2011.
