改进RoBERTa、多实例学习和双重注意力机制的关系抽取方法

doi:10.6040/j.issn.1672-3961.0.2024.050

摘要/Abstract

摘要： 针对远程监督关系抽取不能充分利用句子上下文高层信息、易带来噪声标注的问题,提出一种基于改进鲁棒优化的双向编码器表征预训练模型(robustly optimized bidirectional encoder representations from Transformers pretraining approach, RoBERTa)、多实例学习(multiple-instance learning, MI)和双重注意力(dual attention, DA)机制的关系抽取方法。在RoBERTa中引入全词动态掩码,获取文本上下文信息,获得词级别语义向量;将特征向量输入双向门控循环单元(bidirectional gated recurrent unit, BiGRU),挖掘文本深层次语义表征;引入多实例学习,通过学习实例级别特征缩小关系抽取类别范围;引入双重注意力机制,结合词语级注意力机制和句子级注意力机制的优势,充分捕捉句子中实体词语特征信息和对有效语句的关注度,增强句子表达能力。试验结果表明,在公开数据集纽约时报(New York Times, NYT)数据集和谷歌IISc远程监督(Google IISc distant supervision, GIDS)数据集中,关系抽取方法的F₁值分别为88.63%、90.13%,均优于主流对比方法,能够有效降低远程监督噪声影响,实现关系抽取,为构建知识图谱提供理论基础。

关键词: 远程监督, 关系抽取, 改进RoBERTa, 多实例学习, 双重注意力机制

Abstract: Aiming at the problem that distant supervision relation extraction could not make full use of the high-level information of sentence context and was easy to bring noise annotations, a relation extraction method based on improved robustly optimized bidirectional encoder representations from Transformers pretraining approach(RoBERTa), multiple-instance learning(MI)and dual attention(DA)mechanism was proposed. The full-word dynamic mask was introduced on the RoBERTa to obtain the text context information and the word-level semantic vector. The feature vectors were input into bidirectional gated recurrent unit(BiGRU)to mine the deep semantic representation of the text. Multiple-instance learning was introduced to narrow the range of relation extraction categories by learning instance-level features. Dual attention mechanism was introduced, which combined the advantages of word-level attention mechanism and sentence-level attention mechanism to fully capture the feature information of entity words in the sentence, improved the model's attention of effective sentences, and enhanced the expression ability of sentences. The experimental results showed that the F₁ value of the method reached 88.63% and 90.13% on the public dataset New York Times(NYT)and Google IISc distant supervision(GIDS), which were better than the mainstream comparison methods. It could effectively reduce the noise influence of distant supervision, realize the relation extraction, and lay a theoretical foundation for the construction of knowledge graph.

Key words: distant supervision, relation extraction, improved RoBERTa, multiple-instance learning, dual attention mechanism

中图分类号:

TP391

王禹鸥,苑迎春,何振学,王克俭. 改进RoBERTa、多实例学习和双重注意力机制的关系抽取方法[J]. 山东大学学报 (工学版), 2025, 55(2): 78-87.

WANG Yuou, YUAN Yingchun, HE Zhenxue, WANG Kejian. A relation extraction method based on improved RoBERTa, multiple-instance learning and dual attention mechanism[J]. Journal of Shandong University(Engineering Science), 2025, 55(2): 78-87.

参考文献

[1] 袁泉, 陈昌平, 陈泽, 等. 基于BERT的两次注意力机制远程监督关系抽取[J]. 计算机应用, 2024, 44(4): 1080-1085. YUAN Quan, CHEN Changping, CHEN Ze, et al. Twice attention mechanism distantly supervised relation extraction based on BERT[J]. Journal of Computer Applications, 2024, 44(4): 1080-1085.
[2] GRISHMAN R. Information extraction: techniques and challenges[C] //Information Extraction A Multidis-ciplinary Approach to an Emerging Information Technology. Berlin, Germany: Springer, 1997: 10-27.
[3] 刘峤, 李杨, 段宏, 等. 知识图谱构建技术综述[J]. 计算机研究与发展, 2016, 53(3): 582-600. LIU Qiao, LI Yang, DUAN Hong, et al. Knowledge graph construction techniques[J]. Journal of Computer Research and Development, 2016, 53(3): 582-600.
[4] 王传栋, 徐娇, 张永. 实体关系抽取综述[J]. 计算机工程与应用, 2020, 56(12): 25-36. WANG Chuandong, XU Jiao, ZHANG Yong. Survey of entity relation extraction[J]. Computer Engineering and Applications, 2020, 56(12): 25-36.
[5] 李枫林, 柯佳. 基于深度学习框架的实体关系抽取研究进展[J]. 情报科学, 2018, 36(3): 169-176. LI Fenglin, KE Jia. Research progress of entity relation extraction base on deep learning framework[J]. Information Science, 2018, 36(3): 169-176.
[6] MINTZ M, BILLS S, SNOW R, et al. Distant supervision for relation extraction without labeled data[C] //Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. Singapore:ACL, 2009: 1003-1011.
[7] BONAN M, RALPH G, LI W. Distant supervision for relation extraction with an incomplete knowledge base[C] //Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Atlanta, USA: ACL, 2013: 777-782.
[8] 郑志蕴, 徐亚媚, 李伦, 等. 融合位置特征注意力与关系增强机制的远程监督关系抽取[J]. 小型微型计算机系统, 2023, 44(12): 2678-2684. ZHENG Zhiyun, XU Yamei, LI Lun, et al. Distantly supervised relation extraction with position feature attention and relation enhancement[J]. Journal of Chinese Computer Systems, 2023, 44(12): 2678-2684.
[9] 张欢, 李卫疆. 基于类型注意力和GCN的远程监督关系抽取[J]. 计算机工程与科学, 2024, 46(2): 316-324. ZHANG Huan, LI Weijiang. Distant supervision relation extraction based on type attention and GCN[J]. Computer Engineering & Science, 2024, 46(2): 316-324.
[10] 崔仕林, 闫蓉. 基于SoftLexicon和注意力机制的中文因果关系抽取[J]. 中文信息学报, 2023, 37(4):81-89. CUI Shilin, YAN Rong. Chinese causality extraction based on SoftLexicon and attention mechanism[J]. Journal of Chinese Information Processing, 2023, 37(4): 81-89.
[11] 李浩, 刘永坚, 解庆, 等. 基于多层次注意力机制的远程监督关系抽取模型[J]. 计算机科学, 2019, 46(10): 252-257. LI Hao, LIU Yongjian, XIE Qing, et al. Distant supervision relation extraction model based on multi-level attention mechanism[J]. Computer Science, 2019, 46(10): 252-257.
[12] WEI Q, JI Z C, SI Y Q, et al. Relation extraction from clinical narratives using pre-trained language models[C] // American Medical Informatics Association Annual Symposium. Washington, USA: AMIA, 2019: 1236-1245.
[13] SU P, VIJAY-SHANKER K. Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction[J]. BMC Bioinformatics, 2022, 23(1): 120.
[14] FENG P, ZHANG X, ZHAO J, et al. Relation extraction based on prompt information and feature reuse[J]. Data Intelligence, 2023, 5(3): 824-840.
[15] FAN C Y. The entity relationship extraction method using improved RoBERTa and multi-task learning[J]. Computers, Materials & Continua, 2023, 77(2): 1719-1738.
[16] YE Q, CAI T T, JI X, et al. Subsequence and distant supervision based active learning for relation extraction of Chinese medical texts[J]. BMC Medical Informatics and Decision Making, 2023, 23(1): 34.
[17] 张鲁, 段友祥, 刘娟, 等. 基于RoBERTa和加权图卷积网络的中文地质实体关系抽取[J]. 计算机科学, 2024, 51(8): 297-303. ZHANG Lu, DUAN Youxiang, LIU Juan, et al. Chinese geological entity relation extraction based on RoBERTa and weighted graph convolutional networks[J]. Computer Science, 2024, 51(8): 297-303.
[18] ZHOU G D, SU J, ZHANG J, et al. Exploring various knowledge in relation extraction[C] //Proceedings of the 43rd Annual Meeting on Association for Compu-tational Linguistics. Ann Arbor, USA: ACL, 2005: 427-434.
[19] JIANG X, WANG Q, LI P, et al. Relation extraction with multi-instance multi-label convolutional neural networks[C] //Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics. Osaka, Japan: COLING, 2016: 1471-1480.
[20] 刘哲. 基于句子级注意力机制的远程监督实体关系抽取[D]. 南昌: 江西财经大学, 2021. LIU Zhe. Distant supervised entity relationship extraction based on sentence-level attention mechanism[D]. Nanchang: Jiangxi University of Finance and Eco-nomics, 2021.
[21] YUAN Y, LIU L, TANG S, et al. Cross-relation cross-bag attention for distantly-supervised relation extraction[C] //Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. Honolulu, USA: AAAI, 2019: 419-426.
[22] 王红, 李晗, 李浩飞. 民航突发事件领域本体关系提取方法的研究[J]. 计算机科学与探索, 2020, 14(2): 285-293. WANG Hong, LI Han, LI Haofei. Research of relation extraction method of civil aviation emergency domain ontology[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(2): 285-293.
[23] ZHANG J, CAO M L. Distant supervision for relation extraction with hierarchical attention-based networks[J]. Expert Systems with Applications, 2023, 220: 119727.
[24] LI R, XIAO Q, YANG J X, et al. Few-shot relation extraction via the entity feature enhancement and attention-based prototypical network[J]. International Journal of Intelligent Systems, 2023, 1: 1186977.
[25] ZHAI Z W, FAN R L, HUANG J, et al. A novel joint extraction model based on cross-attention mechanism and global pointer using context shield window[J]. Computer Speech & Language, 2024, 87: 101643.
[26] HUANG S J, CHEN Y, ZHOU E J, et al. A RoBERTa based model for identifying non-substantive factual elements of the case[C] // 2021 2nd International Conference on Intelligent Computing and Human-Computer Interaction(ICHCI). Shenyang, China: IEEE, 2021: 65-71.
[27] ZHU Z D, SU J D, HONG X B. Improving relation extraction using semantic role and multi-task learning[C] //Proceedings of the Knowledge Graph and Semantic Computing: Knowledge Graph and Cognitive Intelligence. Singapore: Springer, 2021: 93-105.
[28] ZENG D, LIU K, CHEN Y, et al. Distant supervision for relation extraction via piecewise convolutional neural networks[C] //Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: ACL, 2015: 1753-1762.
[29] JAT S,KHANDELWAL S, TALUKDAR P. Improving distantly supervised relation extraction using word and entity based attention[EB/OL].(2018-04-19)[2024-03-05]. https://arxiv.org/abs/1804.06987
[30] BASTOS A, NADGERI A, SINGH K, et al. RECON:relation extraction using knowledge graph context in a graph neural network[C] //Proceedings of the Web Conference 2021. Ljubljana, Slovenia: ACM, 2021: 1673-1685.
[31] 赵晋斌, 王琦, 马黎雨, 等. 基于知识图谱的远程监督关系抽取降噪方法[J]. 火力与指挥控制, 2023, 48(10): 160-169. ZHAO Jinbin, WANG Qi, MA Liyu, et al. A noise reduction method for distant supervision relation extraction based on knowledge graph[J]. Fire Control & Command Control, 2023, 48(10): 160-169.
[32] CAI F Z, HU Q, ZHOU R J, et al. REEGAT: RoBERTa entity embedding and graph attention networks enhanced sentence representation for relation extraction[J]. Electronics, 2023, 12(11): 2429.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed