融合多特征和多头自注意力机制的高校学业命名实体识别

doi:10.6040/j.issn.1672-3961.0.2024.114

摘要/Abstract

摘要： 为了有效解决高校学业领域实体归属和实体嵌套问题,提出一种基于多特征融合和多头自注意力机制的中文命名实体识别模型Multi-feature BiGRU-MHSA-CRF(MBMC)。该模型从字、词、位置三个方面对文本语义特征进行表示,丰富多维度学业文本语义特征,并将特征向量输入到双向循环神经网络(Bi-directional recurrent neural network, BiGRU)以捕获全局语义特征。为了解决高校学业领域实体边界划分问题,对注意力机制进行改进,引入带有Q、K、V权重矩阵的多头自注意力机制,增加学习参数,提升识别准确率,将所有可能的标签序列输出到条件随机场(conditional random fields, CRF),通过CRF解码生成实体标签序列。试验结果表明,该模型F1值在公开数据集CoNLL2003和高校学业领域数据集分别达到89.57%、86.14%,优于其它传统模型。

关键词: 高校学业, 命名实体识别, 多特征融合, BiGRU, 多头自注意力机制

Abstract: In order to effectively solve the entity attribution and entity nesting in the academic domain of universities, a Chinese named entity recognition model Multi-feature BiGRU-MHSA-CRF(MBMC)was proposed based on multi-feature fusion. The text semantic features of the model were represented from three aspects of character, word and position to enrich the multi-dimensional semantic features of academic text. The feature vectors were fed into BiGRU(Bi-directional Recurrent Neural Network)to capture global semantic features. In order to solve the problem of entity boundary delimitation in the academic domain of higher education, the attention mechanism was improved by introducing a multi-head self-attention mechanism with Q, K, and V weight matrices and increasing the learning parameters to improve the recognition accuracy. All possible label sequences were output to the CRF, and the entity label sequence was generated by CRF decoding. The experimental results showned that the F1 value of the model reached 89.57% and 86.14% in the public dataset CoNLL2003 and the college academic domain dataset, respectively. It was better than other traditional models.

Key words: university academic, named entity recognition, Multi-feature fusion, Bi-directional recurrent neural network, Multi-head self-attention mechanism

中图分类号:

TP391

王禹鸥,苑迎春,何振学,何晨. 融合多特征和多头自注意力机制的高校学业命名实体识别[J]. 山东大学学报 (工学版), 2025, 55(6): 35-44.

WANG Yuou, YUAN Yingchun, HE Zhenxue, HE Chen. University academic named entity recognition based on the fusion of multi-feature and multi-head self-attention mechanism[J]. Journal of Shandong University(Engineering Science), 2025, 55(6): 35-44.

参考文献

[1] ZHAO Y, ZHANG A Y, LI X C, et al. RETRACTED: construction of a performance evaluation system for private higher education institutions in China based on balanced scorecard[J]. International Journal of Electrical Engineering & Education, 2023, 60(Suppl.1): 923-930.
[2] 孟平贵. 普通高等学校学生管理规定与学生工作规范化实务手册[M]. 长春: 吉林音像出版社, 2005.
[3] FIGUEROA A. Automatically generating effective search queries directly from community question-answering questions for finding related questions[J]. Expert Systems with Applications, 2017, 77: 11-19.
[4] YANG Z J, WANG Y, GAN J H, et al. Design and research of intelligent question-answering(Q&A)system based on high school course knowledge graph[J]. Mobile Networks and Applications, 2021, 26(5): 1884-1890.
[5] LI J, SUN A X, HAN J L, et al. A survey on deep learning for named entity recognition[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(1): 50-70.
[6] YADAV V, BETHARD S. A survey on recent advances in named entity recognition from deep learning models[C] //International Conference on Computational Linguistics, Santa Fe: USA, 2018: 20-26.
[7] GOYAL A, GUPTA V, KUMAR M. Recent named entity recognition and classification techniques: a systematic review[J]. Computer Science Review, 2018, 29: 21-43.
[8] 储德平, 万波, 李红, 等. 基于ELMO-CNN-BiLSTM-CRF模型的地质实体识别[J]. 地球科学, 2021, 46: 3039-3048. CHU Deping, WAN Bo, LI Hong, et al. Geological entity recognition based on ELMO-CNN-BiLSTM-CRF model[J]. Editorial Committee of Earth Science-Journal of China University of Geosciences, 2021, 46: 3039-3048.
[9] LUO L, YANG Z H, YANG P, et al. An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition[J]. Bioinformatics, 2018, 34(8): 1381-1388.
[10] YUAN T P, QIN X Z, WEI C J. A Chinese named entity recognition method based on ERNIE-BiLSTM-CRF for food safety domain[J]. Applied Sciences, 2023, 13(5): 2849.
[11] YANG J L, LIU Y N, QIAN M H, et al. Information extraction from electronic medical records using multitask recurrent neural network with contextual word embedding[J]. Applied Sciences, 2019, 9(18): 3658.
[12] CHENG X W. Named entity recognition in the education domain based on BERT-BiLSTM-CRF-using data structures as an example[C] //2023 International Conference on Educational Knowledge and Infor-matization(EKI). 2023, Guangzhou, China: IEEE, 2023: 5-9.
[13] ZHANG J, JING S Y, HU J H. Named entity recognition method based on bert-bilstm-attention-CRF for education field[C] //2023 4th International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering(ICBAIE). Hangzhou, China: IEEE, 2023: 383-388.
[14] KYUNGHYUN C, BARTVAN M, CAGLAR G, et al. Learning phrase representations using RNN Encoder-Decoder for statistical machine translation[J]. Statistics, 2014: 1724-1734.
[15] 张召武, 徐彬, 高克宁, 等. 面向教育领域的基于SVR-BiGRU-CRF中文命名实体识别方法[J]. 中文信息学报, 2022, 36(7): 114-122. ZHANG Zhaowu, XU Bin, GAO Kening, et al. SVR-BiGRU-CRF based Chinese named entity recognition for education domain[J]. Journal of Chinese Information Processing, 2022, 36(7): 114-122.
[16] 谢雪景, 谢忠, 马凯, 等. 结合BERT与BiGRU-Attention-CRF模型的地质命名实体识别[J]. 地质通报, 2023, 42(5): 846-855. XIE Xuejing, XIE Zhong, MA Kai, et al. Geological named entity recognition combined BERT and BiGRU-Attention-CRF model[J]. Geological Bulletin of China, 2023, 42(5): 846-855.
[17] XU C W, WANG F Y, HAN J L, et al. Exploiting multiple embeddings for Chinese named entity recognition[C] //Proceedings of the 28th ACM International Conference on Information and Knowledge Management. Beijing, China: ACM, 2019: 2269-2272.
[18] 李浩, 刘永坚, 解庆, 等. 基于多层次注意力机制的远程监督关系抽取模型[J]. 计算机科学, 2019, 46(10): 252-257. LI Hao, LIU Yongjian, XIE Qing, et al. Distant supervision relation extraction model based on multi-level attention mechanism[J]. Computer Science, 2019, 46(10): 252-257.
[19] YANG X N, XIAO Y L. Named entity recognition based on BERT-MBiGRU-CRF and multi-head self-attention mechanism[C] //2022 4th International Conference on Natural Language Processing(ICNLP). Xi'an, China: IEEE, 2022: 178-183.
[20] ZHANG Q H, WU M, LV P T, et al. Research on named entity recognition of Chinese electronic medical records based on multi-head attention mechanism and character-word information fusion[J]. Journal of Intelligent & Fuzzy Systems, 2022, 42(4): 4105-4116.
[21] YANG Z W, MA J, CHEN H C, et al. Context-aware attentive multilevel feature fusion for named entity recognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(1): 973-984.
[22] MA X D, YU R L, GAO C X, et al. Research on named entity recognition method of marine natural products based on attention mechanism[J]. Frontiers in Chemistry, 2023, 11: 958002.
[23] KIM J, KO Y, SEO J. A bootstrapping approach with CRF and deep learning models for improving the biomedical named entity recognition in multi-domains[J]. IEEE Access, 2019, 7: 70308-70318.
[24] 何晨, 苑迎春, 王克俭, 等. 高校学业文本命名实体识别及数据集构建研究[J]. 计算机工程与应用, 2023, 59(22): 322-328. HE Chen, YUAN Yingchun, WANG Kejian, et al. Research on college academic text named entity recognition and dataset construction[J]. Computer Engineering and Applications, 2023, 59(22): 322-328.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed