Journal of Shandong University(Engineering Science) ›› 2025, Vol. 55 ›› Issue (2): 78-87.doi: 10.6040/j.issn.1672-3961.0.2024.050

• Machine Learning & Data Mining • Previous Articles     Next Articles

A relation extraction method based on improved RoBERTa, multiple-instance learning and dual attention mechanism

WANG Yuou1, YUAN Yingchun1,2*, HE Zhenxue1, WANG Kejian1   

  1. WANG Yuou1, YUAN Yingchun1, 2*, HE Zhenxue1, WANG Kejian1(1. College of Information Science and Technology, Hebei Agricultural University, Baoding 071001, Hebei, China;
    2. Hebei Province Key Laboratory of Agricultural Big Data, Hebei Agricultural University, Baoding 071001, Hebei, China
  • Published:2025-04-15

PDF (PC)

72

Abstract: Aiming at the problem that distant supervision relation extraction could not make full use of the high-level information of sentence context and was easy to bring noise annotations, a relation extraction method based on improved robustly optimized bidirectional encoder representations from Transformers pretraining approach(RoBERTa), multiple-instance learning(MI)and dual attention(DA)mechanism was proposed. The full-word dynamic mask was introduced on the RoBERTa to obtain the text context information and the word-level semantic vector. The feature vectors were input into bidirectional gated recurrent unit(BiGRU)to mine the deep semantic representation of the text. Multiple-instance learning was introduced to narrow the range of relation extraction categories by learning instance-level features. Dual attention mechanism was introduced, which combined the advantages of word-level attention mechanism and sentence-level attention mechanism to fully capture the feature information of entity words in the sentence, improved the model's attention of effective sentences, and enhanced the expression ability of sentences. The experimental results showed that the F1 value of the method reached 88.63% and 90.13% on the public dataset New York Times(NYT)and Google IISc distant supervision(GIDS), which were better than the mainstream comparison methods. It could effectively reduce the noise influence of distant supervision, realize the relation extraction, and lay a theoretical foundation for the construction of knowledge graph.

Key words: distant supervision, relation extraction, improved RoBERTa, multiple-instance learning, dual attention mechanism

CLC Number: 

  • TP391
[1] 袁泉, 陈昌平, 陈泽, 等. 基于BERT的两次注意力机制远程监督关系抽取[J]. 计算机应用, 2024, 44(4): 1080-1085. YUAN Quan, CHEN Changping, CHEN Ze, et al. Twice attention mechanism distantly supervised relation extraction based on BERT[J]. Journal of Computer Applications, 2024, 44(4): 1080-1085.
[2] GRISHMAN R. Information extraction: techniques and challenges[C] //Information Extraction A Multidis-ciplinary Approach to an Emerging Information Technology. Berlin, Germany: Springer, 1997: 10-27.
[3] 刘峤, 李杨, 段宏, 等. 知识图谱构建技术综述[J]. 计算机研究与发展, 2016, 53(3): 582-600. LIU Qiao, LI Yang, DUAN Hong, et al. Knowledge graph construction techniques[J]. Journal of Computer Research and Development, 2016, 53(3): 582-600.
[4] 王传栋, 徐娇, 张永. 实体关系抽取综述[J]. 计算机工程与应用, 2020, 56(12): 25-36. WANG Chuandong, XU Jiao, ZHANG Yong. Survey of entity relation extraction[J]. Computer Engineering and Applications, 2020, 56(12): 25-36.
[5] 李枫林, 柯佳. 基于深度学习框架的实体关系抽取研究进展[J]. 情报科学, 2018, 36(3): 169-176. LI Fenglin, KE Jia. Research progress of entity relation extraction base on deep learning framework[J]. Information Science, 2018, 36(3): 169-176.
[6] MINTZ M, BILLS S, SNOW R, et al. Distant supervision for relation extraction without labeled data[C] //Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. Singapore:ACL, 2009: 1003-1011.
[7] BONAN M, RALPH G, LI W. Distant supervision for relation extraction with an incomplete knowledge base[C] //Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Atlanta, USA: ACL, 2013: 777-782.
[8] 郑志蕴, 徐亚媚, 李伦, 等. 融合位置特征注意力与关系增强机制的远程监督关系抽取[J]. 小型微型计算机系统, 2023, 44(12): 2678-2684. ZHENG Zhiyun, XU Yamei, LI Lun, et al. Distantly supervised relation extraction with position feature attention and relation enhancement[J]. Journal of Chinese Computer Systems, 2023, 44(12): 2678-2684.
[9] 张欢, 李卫疆. 基于类型注意力和GCN的远程监督关系抽取[J]. 计算机工程与科学, 2024, 46(2): 316-324. ZHANG Huan, LI Weijiang. Distant supervision relation extraction based on type attention and GCN[J]. Computer Engineering & Science, 2024, 46(2): 316-324.
[10] 崔仕林, 闫蓉. 基于SoftLexicon和注意力机制的中文因果关系抽取[J]. 中文信息学报, 2023, 37(4):81-89. CUI Shilin, YAN Rong. Chinese causality extraction based on SoftLexicon and attention mechanism[J]. Journal of Chinese Information Processing, 2023, 37(4): 81-89.
[11] 李浩, 刘永坚, 解庆, 等. 基于多层次注意力机制的远程监督关系抽取模型[J]. 计算机科学, 2019, 46(10): 252-257. LI Hao, LIU Yongjian, XIE Qing, et al. Distant supervision relation extraction model based on multi-level attention mechanism[J]. Computer Science, 2019, 46(10): 252-257.
[12] WEI Q, JI Z C, SI Y Q, et al. Relation extraction from clinical narratives using pre-trained language models[C] // American Medical Informatics Association Annual Symposium. Washington, USA: AMIA, 2019: 1236-1245.
[13] SU P, VIJAY-SHANKER K. Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction[J]. BMC Bioinformatics, 2022, 23(1): 120.
[14] FENG P, ZHANG X, ZHAO J, et al. Relation extraction based on prompt information and feature reuse[J]. Data Intelligence, 2023, 5(3): 824-840.
[15] FAN C Y. The entity relationship extraction method using improved RoBERTa and multi-task learning[J]. Computers, Materials & Continua, 2023, 77(2): 1719-1738.
[16] YE Q, CAI T T, JI X, et al. Subsequence and distant supervision based active learning for relation extraction of Chinese medical texts[J]. BMC Medical Informatics and Decision Making, 2023, 23(1): 34.
[17] 张鲁, 段友祥, 刘娟, 等. 基于RoBERTa和加权图卷积网络的中文地质实体关系抽取[J]. 计算机科学, 2024, 51(8): 297-303. ZHANG Lu, DUAN Youxiang, LIU Juan, et al. Chinese geological entity relation extraction based on RoBERTa and weighted graph convolutional networks[J]. Computer Science, 2024, 51(8): 297-303.
[18] ZHOU G D, SU J, ZHANG J, et al. Exploring various knowledge in relation extraction[C] //Proceedings of the 43rd Annual Meeting on Association for Compu-tational Linguistics. Ann Arbor, USA: ACL, 2005: 427-434.
[19] JIANG X, WANG Q, LI P, et al. Relation extraction with multi-instance multi-label convolutional neural networks[C] //Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics. Osaka, Japan: COLING, 2016: 1471-1480.
[20] 刘哲. 基于句子级注意力机制的远程监督实体关系抽取[D]. 南昌: 江西财经大学, 2021. LIU Zhe. Distant supervised entity relationship extraction based on sentence-level attention mechanism[D]. Nanchang: Jiangxi University of Finance and Eco-nomics, 2021.
[21] YUAN Y, LIU L, TANG S, et al. Cross-relation cross-bag attention for distantly-supervised relation extraction[C] //Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. Honolulu, USA: AAAI, 2019: 419-426.
[22] 王红, 李晗, 李浩飞. 民航突发事件领域本体关系提取方法的研究[J]. 计算机科学与探索, 2020, 14(2): 285-293. WANG Hong, LI Han, LI Haofei. Research of relation extraction method of civil aviation emergency domain ontology[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(2): 285-293.
[23] ZHANG J, CAO M L. Distant supervision for relation extraction with hierarchical attention-based networks[J]. Expert Systems with Applications, 2023, 220: 119727.
[24] LI R, XIAO Q, YANG J X, et al. Few-shot relation extraction via the entity feature enhancement and attention-based prototypical network[J]. International Journal of Intelligent Systems, 2023, 1: 1186977.
[25] ZHAI Z W, FAN R L, HUANG J, et al. A novel joint extraction model based on cross-attention mechanism and global pointer using context shield window[J]. Computer Speech & Language, 2024, 87: 101643.
[26] HUANG S J, CHEN Y, ZHOU E J, et al. A RoBERTa based model for identifying non-substantive factual elements of the case[C] // 2021 2nd International Conference on Intelligent Computing and Human-Computer Interaction(ICHCI). Shenyang, China: IEEE, 2021: 65-71.
[27] ZHU Z D, SU J D, HONG X B. Improving relation extraction using semantic role and multi-task learning[C] //Proceedings of the Knowledge Graph and Semantic Computing: Knowledge Graph and Cognitive Intelligence. Singapore: Springer, 2021: 93-105.
[28] ZENG D, LIU K, CHEN Y, et al. Distant supervision for relation extraction via piecewise convolutional neural networks[C] //Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: ACL, 2015: 1753-1762.
[29] JAT S,KHANDELWAL S, TALUKDAR P. Improving distantly supervised relation extraction using word and entity based attention[EB/OL].(2018-04-19)[2024-03-05]. https://arxiv.org/abs/1804.06987
[30] BASTOS A, NADGERI A, SINGH K, et al. RECON:relation extraction using knowledge graph context in a graph neural network[C] //Proceedings of the Web Conference 2021. Ljubljana, Slovenia: ACM, 2021: 1673-1685.
[31] 赵晋斌, 王琦, 马黎雨, 等. 基于知识图谱的远程监督关系抽取降噪方法[J]. 火力与指挥控制, 2023, 48(10): 160-169. ZHAO Jinbin, WANG Qi, MA Liyu, et al. A noise reduction method for distant supervision relation extraction based on knowledge graph[J]. Fire Control & Command Control, 2023, 48(10): 160-169.
[32] CAI F Z, HU Q, ZHOU R J, et al. REEGAT: RoBERTa entity embedding and graph attention networks enhanced sentence representation for relation extraction[J]. Electronics, 2023, 12(11): 2429.
[1] CHEN Zehua, SHANG Xiaohui, CHAI Jing. Neighborhood related multiple-instance classifiers based on integrated Hausdorff distance [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(6): 15-22.
[2] XU Qing, DUAN Liguo, LI Aiping, YIN Guimei. Chinese entity relation extraction based on entity semantic similarity [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2015, 45(6): 7-15.
[3] SHAO Fa, HUANG Yinge, ZHOU Lanjiang, GUO Jianyi, YU Zhengtao, ZHANG Jinpeng. Chinese entity relation extraction based on entity disambiguation [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2014, 44(6): 32-37.
[4] LEI Chun-ya1, GUO Jian-yi1,2, YU Zheng-tao1,2, MAO Cun-li1,2, ZHANG Shao-min1, HUANG Pu1. Domain of automatic entity relation extraction based on seed self-expansion and maximum entropy machine learning [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2010, 40(5): 141-145.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] XIA Bin,ZHANG Lian-jun . Energy comparison-based TOA estimation algorithm for the DS-CDMA UWB system[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(1): 70 -73 .
[2] YUAN Dong-ling,DENG Jian-xin,DING Ze-liang,DUAN Zhen-xing . Finite element analysis for the thermal residual stress of gradient CWS ceramic nozzles[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(2): 18 -22 .
[3] DENG Bin,WANG Jiang . Estimating parameters of a neuron model based on chaos synchronization and adaptive control[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(5): 19 -23 .
[4] WANG Jin-ye,YAO Rui-ying,ZHANG Ji-liang,WANG Qi-jun . System stability control of fuzzy hyperbolic model[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(2): 63 -66 .
[5] YU Jia yuan1, TIAN Jin ting1, ZHU Qiang zhong2. Computational intelligence and its application in psychology[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 1 -5 .
[6] GAO Hou-Lei, TIAN Jia, DU Jiang, WU Zhi-Gang, LIU Chu-Min. Distributed generation—new technology in energy development[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 106 -110 .
[7] YANG Yongshun1, WANG Lin2, GAO Xuechi1, JIA Haiqing3, WEI Jincheng2. Analysis of perpetual pavement strain distribution and fatigue damage[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 118 -124 .
[8] XU Ming. Reaction kinetics of ［BPy］Br and ［BMIm］Br under microwave-assisted synthesis[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2010, 40(4): 113 -116 .
[9] ZHOU Ting-ting1, DU Ai-ling1*, GAO Yu1, YANG Yan-ling2. Influence of pH value in a phosphating bath on  zinc phosphate coatings on the AZ61 magnesium alloy[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2011, 41(1): 114 -119 .
[10] ZHANG Qi-cong1, YANG Gong-ping2*. Study on Agent based simulation of banking queuing system[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2011, 41(4): 68 -72 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd