您的位置:山东大学 -> 科技期刊社 -> 《山东大学学报(工学版)》

山东大学学报 (工学版) ›› 2024, Vol. 54 ›› Issue (4): 13-20.doi: 10.6040/j.issn.1672-3961.0.2023.163

• 机器学习与数据挖掘 • 上一篇    下一篇

基于神经正切核草图的多核学习方法

王梅1,许传海2*,王伟东1,韩非3   

  1. 1.东北石油大学计算机与信息技术学院, 黑龙江 大庆 163318;2.新疆理工学院信息工程学院, 新疆 阿克苏 843100;3.东北石油大学人工智能能源研究院, 黑龙江 大庆 163318
  • 发布日期:2024-08-20
  • 作者简介:王梅(1975— ),女,河北安国人,教授,硕士生导师,博士,主要研究方向为机器学习、核方法、模型选择等. E-mail: wangmei@nepu.edu.cn. *通信作者简介:许传海(1998— ),男,黑龙江鸡西人,助教,硕士,主要研究方向为机器学习、深度核学习. E-mail: 2023266@xjit.edu.cn
  • 基金资助:
    国家自然科学基金资助项目(51774090,62073070);黑龙江省博士后科研启动金资助项目(LBH-Q20080);黑龙江省研究生精品课程建设项目(15141220103)

Multi-kernel learning method based on neural tangent kernel sketch

WANG Mei1, XU Chuanhai2*, WANG Weidong1, HAN Fei3   

  1. 1. School of Computer and Information Technology, Northeast Petroleum University, Daqing 163318, Heilongjiang, China;
    2. College of Information Engineering, Xinjiang Institute of Technology, Aksu 843100, Xinjiang, China;
    3. Artificial Intelligence Energy Research Institute, Northeast Petroleum University, Daqing 163318, Heilongjiang, China
  • Published:2024-08-20

摘要: 为提高多核学习对大规模及分布不均衡问题的处理能力,提出一种基于神经正切核草图的多核学习方法(neural tangent kernel sketch multiple kernel learning, NS-MKL )。应用神经正切核代替单层核函数作为多核学习基核函数,提高多核学习方法表示能力;使用神经正切核草图算法对神经正切核进行近似,减少神经正切核的特征数量和特征维度,提高多核学习方法计算效率;使用核目标对齐计算每个近似神经正切核的基核权重,根据权重进行多核线性组合,得到多核决策函数。在3个UCI数据集上对神经正线核(neural tangent kernel, NTK)核支持向量机(support vector machine, SVM)与传统核SVM进行比较分析,NTK核SVM比传统核SVM预测准确率最低提高1.9%,精度最低提高2.0%,召回率最低提高2.0%。在3个UCI数据集上对NS-MKL与传统核MKL进行比较分析,NS-MKL比应用传统核MKL预测准确率最低提高2.0%,运行时间最低减少9 s。NS-MKL能提高预测准确率,降低计算速度。

关键词: 多核学习, 神经正切核, 核目标对齐, 反余弦核, 草图算法

中图分类号: 

  • TP391
[1] ZHANG T. An introduction to support vector machines and other kernel-based learning methods[J]. AI Magazine, 2001, 22(2): 103-104.
[2] SOLICH P. Bayesian methods for support vector machines: Evidence and predictive class probabilities [J]. Machine Learning, 2002, 46(1): 21-52.
[3] KLOFT M, BLANCHARD G. On the convergence rate of lp-norm multiple kernel learning [J]. Journal of Machine Learning Research, 2012(1): 2465-2502.
[4] CRISTIANINI N, SHAWE-TAYLOR J, ELISSEEFF A, et al. On kernel-target alignment[C] // Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic. Vancouver, Canada: MIT Press, 2001: 367-373.
[5] WILLIAMS C. Computing with infinite networks[C] // Proceedings of the 9th International Conference on Neural Information Processing Systems. Denver, USA: MIT Press, 1996: 295-301.
[6] LEE J, BAHRI Y, NOVAK R, et al. Deep neural networks as gaussian processes[C] // Proceedings of the 6th International Conference on Learning Representations. Vancouver, Canada: MIT Press, 2018: 1-17.
[7] JACOT A, GABRIEL F, HONGLER C. Neural tangent kernel: convergence and generalization in neural networks[C] //Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montreal, Canada: MIT Press, 2018: 8580-8589.
[8] LEE J, XIAO L, SCHOENHOLZ S, et al. Wide neural networks of any depth evolve as linear models under gradient descent [EB/OL].(2019-08-18)[2019-12-08]. https://arxiv.org/abs/1902.06720.
[9] 王梅,许传海,刘勇. 基于神经正切核的多核学习方法[J].计算机应用,2021,41(12): 3462-3467. WANG Mei, XU Chuanhai, LIU Yong. Multi-kernel learning method based on neural tangent kernel[J]. Journal of Computer Applications, 2021, 41(12): 3462-3467.
[10] 王梅, 宋晓晖, 刘勇, 等.神经正切核K-Means聚类[J].计算机应用, 2022, 42(11): 3330-3336. WAND Mei, SONG Xiaohui, LIU Yong, et al. Neural tangent kernel K-Means clustering[J]. Journal of Computer Applications, 2022, 42(11): 3330-3336.
[11] ARORA S, DU S S, HU W, et al. On exact computation with an infinitely wide neural net [EB/OL].(2019-04-26)[2019-11-04]. https://arxiv. org/abs/1904.11955.
[12] CHEN L, XU S. Deep neural tangent kernel and laplace kernel have the same RKHS[EB/OL].(2020-09-22)[2021-03-18]. https://doi.org/10.48550/ arXiv.2009.10683.
[13] 张琳, 汪廷华, 周慧颖.基于群智能算法的SVR参数优化研究进展[J].计算机工程与应用, 2021, 57(16): 50-64. ZHANG Lin, WANG Tinghua, ZHOU Huiying. Research progress on parameter optimization of SVR based on swarm intelligence algorithm[J]. Computer Engineering and Applications, 2021, 57(16): 50-64.
[14] 祁祥洲, 邢红杰. 基于中心核对齐的多核单类支持向量机[J]. 计算机应用, 2022, 42(2): 349-356. QI Xiangzhou, XING Hongjie. Centered kernel alignment based multiple kernel one-class support vector machine[J]. Journal of Computer Applications, 2022, 42(2): 349-356.
[15] LANCKRIET G, CRISTIANINI N, BARTLETT P L, et al. Learning the kernel matrix with semidefinite programming[J]. Journal of Machine Learning Research, 2002, 5(1): 27-72.
[16] 侯能干. 基于特征融合和多核学习的行人检测方法研究[D]. 合肥:合肥工业大学, 2014. HOU Nenggan. Research on pedestrian detection methods based on feature fusion and multi-core learning[D]. Hefei: Hefei University of Technology, 2014.
[17] GONEN M, ALPAYDIN E. Localized multiple kernel learning[C] //Proceedings of the 25th International conference on Machine learning, Helsinki, Finland: MIT Press, 2008: 352-359.
[18] 梁俊. 基于多核学习支持向量机的货币识别[D]. 长沙:中南大学, 2014. LIANG Jun. Currency Recognition Based on Multikernel Learning Support Vector Machines[D]. Changsha: Central South University, 2014.
[19] HE Q, ZHANG Q, WANG H. Kernel-target alignment based multiple kernel one-class support vector machine[C] //Proceedings of the 2019 IEEE International Conference on Systems, Man and Cybernetics. Bari, Italy: IEEE, 2019: 2083-2088.
[20] 邵朝, 李强. 基于特征加权的多核支持向量机[J]. 西安邮电大学学报, 2017, 22(2): 84-88. SHAO Chao, LI Qiang. Multi-kernel support vector machines based on feature weighting[J]. Journal of Xi'an University of Posts and Telecommunications, 2017, 22(2): 84-88.
[21] 贾涵, 连晓峰, 潘兵. 基于模糊松弛约束的外观缺陷多核学习技术[J]. 测控技术, 2019, 38(8): 43-47. JIA Han, LIAN Xiaofeng, PAN Bing. Appearance defects multiple kernel learning technology based on fuzzy relaxation constraints [J]. Measurement and Control Technology, 2019, 38(8): 43-47.
[22] 王梅, 薛成龙, 张强. 基于秩空间差异的多核组合方法[J]. 山东大学学报(工学版), 2021, 51(1): 108-113. WANG Mei, XUE Chenglong, ZHANG Qiang. Multi-kernel combination method based on rank spatial difference[J]. Journal of Shandong University(Engineering Science), 2021, 51(1): 108-113.
[23] 李湘眷, 孙显, 王宏琦. 基于多核学习的高分辨率遥感图像目标检测方法[J]. 测绘科学, 2013, 38(5): 84-87. LI Xiangjuan, SUN Xian, WANG Hongqi. Target detection method for high-resolution remote sensing images based on multi-core learning [J]. Science of Surveying and Mapping, 2013, 38(5): 84-87.
[24] NOVAK R, XIAO L, HRON J, et al. Neural tangents: fast and easy infinite neural networks in Python[EB/OL].(2019-12-05)[2019-12-05]. https://doi.org/10.48550/arXiv.1912.02803.
[25] LIU X, LEI W, ZHU X, et al. Absent multiple kernel learning algorithms [J]. IEEE Transactions on Pattern Analysis and Machine Intelligencev, 2019, 42(6):1303-1316.
[26] MITCHEL A P, OVENEKE M C, H SAHLI. SVRG-MKL: a fast and scalable multiple kernel learning solution for features combination in multi-class classification problems[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(5): 1710-1723.
[27] WANG X, WANG S, DU Y, et al. Minimum class variance multiple kernel learning [J]. Knowledge-Based Systems, 2020, 208(5): 106469.
[28] RAHIMI A, RECHT B. Random features for large-scale Kernel machines[EB/OL].(2019-05-28)[2019-05-28]. https://doi.org/10.48550/arXiv.2209.01958.
[29] HAN I, AVRON H, SHIN J. Polynomial Tensor Sketch for Element-wise Function of Low-Rank Matrix[C] // Proceedings of the 37th International Conference on Machine Learning. Vienna, Austria: MIT Press, 2020: 3942-3951.
[30] CHO Y, SAUL L. Kernel methods for deep learning[C] // In Advances in Neural Information Processing Systems 22. Vancouver, Canada: Red Hook: 2009:342-350.
[31] BIETTI A, MAIRAL J. On the inductive bias of neural tangent kernels[EB/OL].(2019-05-29)[2019-10-31]. ttp://arxiv.org/abs/1905.12173.
[32] ZANDIEH A, HAN I, AVORN H, et al. Scaling neural tangent kernels via sketching and random features [J]. Advances in Neural Information Processing Systems, 2021, 34: 1062-1073.
[33] PHAN N, PAGH R. Fast and scalable polynomial kernels via explicit feature maps[C] //Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Chicago, USA: ACM, 2013: 239-247.
[34] AIOLLI F, DONINI M. EasyMKL: a scalable multiple kernel learning algorithm[J]. Neurocomputing, 2015, 169: 215-224.
[35] TANABE H, HO T B, Nguyen C H, et al. Simple but effective methods for combining kernels in computational biology[C] //Proceedings of the 2008 IEEE International Conference on Research, Innovation and Vision for the Future in Computing and Communication Technologies. Ho Chi Minh City, Vietnam: IEEE, 2008: 71-78.
[1] 王梅,薛成龙,张强. 基于秩空间差异的多核组合方法[J]. 山东大学学报 (工学版), 2021, 51(1): 108-113.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] 张宏博 黄茂松 宋修广. 基于应变软化与剪胀性特征的粉砂土双硬化弹塑性本构模型[J]. 山东大学学报(工学版), 2008, 38(6): 55 -60 .
[2] 任玉玲, 路文, 徐红强, 何立火. 一种基于Shearlet变换的图像质量客观评价方法[J]. 山东大学学报(工学版), 2015, 45(3): 15 -21 .
[3] 黄斌. 基于离散趋近律的变结构系统控制[J]. 山东大学学报(工学版), 2011, 41(1): 45 -48 .
[4] 焦建民1 ,孙小军1 ,吴保卫2 . 带有非线性扰动的不确定奇异时滞系统的保性能控制[J]. 山东大学学报(工学版), 2009, 39(2): 64 -69 .
[5] 梅清琳,张化祥. 基于全局距离和类别信息的邻域保持嵌入算法[J]. 山东大学学报(工学版), 2016, 46(1): 10 -14 .
[6] 张建明,刘泉声,唐志成,占婷,蒋亚龙. 考虑剪切变形历史影响的节理峰值剪切强度准则[J]. 山东大学学报 (工学版), 2015, 45(5): 77 -81 .
[7] 赵文忠 史军. 复功率注入空间的电力系统概率安全性指标研究[J]. 山东大学学报(工学版), 2009, 39(6): 135 -138 .
[8] 孙国富1,2,李术才2,路为2,陈雷3. 复合胶凝材料钢管混凝土拱肋水化过程截面温度场试验及数值模拟分析[J]. 山东大学学报(工学版), 2011, 41(3): 106 -111 .
[9] 曹红根1,袁宝华1,朱辉生2. 结合对比度信息与LBP的分块人脸识别[J]. 山东大学学报(工学版), 2012, 42(4): 29 -34 .
[10] 何山 尤晓春. 复杂条件下巷道围岩破坏机理及支护方案研究[J]. 山东大学学报(工学版), 2009, 39(4): 121 -124 .