基于输入K-近邻的正则化路径上SVR贝叶斯组合

doi:10.6040/j.issn.1672-3961.1.2016.294

Abstract

Abstract: A model combination method of ε-insensitive support vector regression(ε-SVR)based on regularization path with K-Nearest Neighbor(KNN)of input was proposed. The model set was constructed with ε-SVR regularization path, which was trained by using the same original training set. The initial model set was obtained according to the piecewise linearity of SVR regularization path. The average of Bayesian Information Criterion(BIC)was applied to exclude models with poor performance and prune the initial model set. In the testing or predicting phase, the combination model set was determined with the KNN method, and Bayesian combination was performed. The pruning policy improves not only the computational efficiency of model combination but also the generalization performance. The L_ε-risk consistency for model combination of ε-SVR was defined and proved, which gave the mathematical foundation of the proposed method. Experimental results demonstrated the effectiveness and efficiency of the Bayesian combination of ε-SVR on regularization path.

Key words: regularization path, model combination, support vector regression, KNN, consistency

CLC Number:

TP181

WANG Mei, ZENG Zhaohu, SUN Yingqi, YANG Erlong, SONG Kaoping. Bayesian combination of SVR on regularization path based on KNN of input[J].JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(6): 8-14.

References

[1] BURGES C J. A tutorial on support vector machines for pattern recognition[J]. Data mining and knowledge discovery, 1998, 2(2):121-167.
[2] 邓乃扬, 田英杰. 数据挖掘中的新方法: 支持向量机[M]. 北京: 科学出版社, 2004.
[3] ANTHONY M, HOLDEN S B. Cross-validation for binary classification by real-valued functions: theoretical analysis[C] //Proceedings of the 11th Annual Conference on Computational Learning Theory. Berlin, Germany: Springer, 1998: 218-229.
[4] CHAPELLE O, VAPNIK V. Model selection for support vector machines[C] //Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, 1999.
[5] VAPNIK V, CHAPELLE O. Bounds on error expectation for support vector machines[J]. Neural Computation, 2000, 12(9):2013-2036.
[6] GOLD C, SOLLICH P. Model selection for support vector machine classification [J]. Neurocomputing, 2003, 55(1):221-249.
[7] KEERTHI S S. Efficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms[J]. IEEE Transactions on Neural Networks, 2002, 13(5):1225-1229.
[8] 刘向东, 骆斌, 陈兆乾. 支持向量机最优模型选择的研究[J]. 计算机研究与发展, 2005, 42(4):576-581. LIU Xiangdong, LUO Bin, CHEN Zhaoqian. Optimal model selection for support vector machine[J]. Journal of Computer Research and Development, 2005, 42(4):576-581.
[9] 汪廷华. 支持向量机模型选择研究 [D]. 北京:北京交通大学, 2010. WANG Tinghua. Reseach on model selection for support vector machine[D].Beijing: Beijing Jiaotong University, 2010.
[10] 丁立中, 廖士中. 基于正则化路径的支持向量机近似模型选择[J]. 计算机研究与发展, 2012, 49(6):1248-1255. DING Lizhong, LIAO Shizhong. Approximate model selection on regularization path for support vector machines[J]. Journal of Computer Research and Development, 2012, 49(6):1248-1255.
[11] BREIMAN L. Bagging predictors[J]. Machine Learning, 1996, 24(2):123-140.
[12] VALENTINI G, MUSELLI M, RUFFINO F. Bagged ensembles of support vector machines for gene expression data analysis[C] //Proceedings of the Int Joint Conference on Neural Networks. Piscataway, USA: IEEE Computer Society, 2003: 1844-1849.
[13] SUN B Y, HUANG D S. Least squares support vector machine ensemble [C] //Proceedings of the Int Joint Confence on Neural Networks. Piscataway, USA: IEEE Computer Society, 2004:2013-2016.
[14] KIM H C, PANG S, JE H M. Constructing support vector machine ensemble[J]. Pattern Recognition, 2003, 36(12):2757-2767.
[15] KIM H C, PANG S, JE H M. Pattern classification using support vector machine ensemble[C] //Proceedings of the 16th Int Confence on Pattern Recognition. Los Alamitos, CA: IEEE Computer Society, 2002:160-163.
[16] LI X, WANG L, SUNG E. AdaBoost with SVM-based component classifiers[J]. Engineering Applications of Artificial Intelligence, 2008, 21(5):785-795.
[17] 王梅,廖士中. 正则化路径上三步式SVM贝叶斯组合[J]. 计算机研究与发展, 2013, 50(9):1855-1864. WANG Mei, LIAO Shizhong. Three-step Bayesian combination of SVM on regularization path[J]. Journal of Computer Research and Development, 2013, 50(9): 1855-1864.
[18] GUNTER Lacey, ZHU Ji. Efficient computation and model selection for the support vector regression[J]. Neural Computation, 2007, 19(6):1633-1655.
[19] 廖士中,王梅,赵志辉. 正定矩阵支持向量机正则化路径算法[J]. 计算机研究与发展, 2013, 50(11): 2253-2261. LIAO Shizhong, WANG Mei, ZHAO Zhihui. Regularization path algorithm of SVM via positive definite matrix[J]. Journal of Computer Research and Development, 2013, 50(11): 2253-2261.
[20] WANG Mei, LIAO Shizhong. Model combination for support vector regression via regularization path[C] //Proceedings of 12th Pacific Rim International Conference on Artificial Intelligence(PRICAI 2012). Beijing, China:Science Press, 2012:649-660.
[21] STEINWART I. On the influence of the kernel on the consistency of support vector machines[J]. The Journal of Machine Learning Research, 2002(2):67-93.
[22] CHRISTMANN Andreas, STEINWART Ingo. Consistency and robustness of kernel-based regression in convex risk minimization[J]. Bernoulli, 2007, 13(3):799-819.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 10

[1]	LI Ke,LIU Chang-chun,LI Tong-lei . Medical registration approach using improved maximization of mutual information[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 107 -110 .
[2]	YUE Yuan-Zheng. Relaxation in glasses far from equilibrium[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 1 -20 .
[3]	CHENG Daizhan, LI Zhiqiang. A survey on linearization of nonlinear systems[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 26 -36 .
[4]	WANG Yong, XIE Yudong. Gas control technology of largeflow pipe[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 70 -74 .
[5]	LIU Xin ¹, SONG Sili ¹, WANG Xinhong ². [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 98 -100 .
[6]	. [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 104 -107 .
[7]	CHEN Huaxin, CHEN Shuanfa, WANG Binggang. The aging behavior and mechanism of base asphalts[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 125 -130 .
[8]	. [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 131 -136 .
[9]	LI Shijin, WANG Shengte, HUANG Leping. Change detection with remote sensing images based on forward-backward heterogenicity[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(3): 1 -9 .
[10]	ZHAO Ke-Jun, WANG Xin-Jun, LIU Xiang, CHOU Yi-Hong. Algorithms of continuous top-k join query over structured overlay networks[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 32 -37 .

Bayesian combination of SVR on regularization path based on KNN of input

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 4

Metrics

Comments

Recommended 10

[1]	LI Sun, WANG Chao, ZHANG Guilin, XU Zhigen, CHENG Tao, WANG Yiyuan, WANG Ruiqi. Short-term power load forecasting based on support vector regression [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(6): 52-56.
[2]	ZHANG Jian-cheng, SU Lian-ta. Properties and applications of the roots of theories in propositional logic systems [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2013, 43(4): 87-92.
[3]	XU Long-qin1, LIU Shuang-yin1,2,3,4*. Water quality prediction model based on APSO-WLSSVR [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2012, 42(5): 80-86.
[4]	ZHAO Yanyan, FAN Liya. The application of a multi-output support vector regression machine in time-dependent variational inequalities [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2011, 41(3): 23-30.