基于大语言模型辅助知识和对比学习策略的图文多模态情感分析

doi:10.6040/j.issn.1672-3961.0.2025.098

山东大学学报 (工学版) ›› 2026, Vol. 56 ›› Issue (5): 71-78.doi: 10.6040/j.issn.1672-3961.0.2025.098

• 机器学习与数据挖掘 • 上一篇

基于大语言模型辅助知识和对比学习策略的图文多模态情感分析

汤凯钰,张恒源,艾孜尔古丽·玉素甫^*

新疆师范大学计算机科学技术学院, 新疆乌鲁木齐 830054

发布日期:2026-10-19
作者简介:汤凯钰(1999— ),男,湖南新邵人,硕士研究生,主要研究方向为自然语言处理. E-mail:tky8966653@qq.com. *通信作者简介:艾孜尔古丽·玉素甫(1987— ),女,新疆乌鲁木齐人,副教授,硕士生导师,博士,主要研究方向为自然语言处理. E-mail:Azragul2010@126.com
基金资助:
国家自然科学基金资助项目(61662081);国家社会科学基金资助项目(14AZD11);新疆维吾尔自治区创新环境(人才、基地)建设专项-自然科学计划(少数民族科技人才特殊培养)资助项目(2022D03001);新疆师范大学青年拔尖人才资助项目(XJNUQB2022-22)

Multimodal sentiment analysis of images and texts based on auxiliary knowledge of large language models and contrastive learning strategy

Tang Kaiyu, Zhang Hengyuan, Aizierguli·Yusufu^*

Tang Kaiyu, Zhang Hengyuan, Aizierguli·Yusufu^*(College of Computer Science and Technology, Xinjiang Normal University, Urumqi 830054, Xinjiang, China

Published:2026-10-19

摘要/Abstract

摘要： 随着大模型在人工智能领域的快速发展,多模态大语言模型(multimodal large language model, MLLM)对文本和视觉输入展现出强大的理解能力,同时还具有相当丰富的常识和推理特性。受此启发,本研究利用多模态大语言模型(large language and vision assistant, LLaVA)生成辅助知识,并在此基础上设计一种基于辅助知识和对比学习策略的多任务框架(based on auxiliary knowledge and contrastive learning strategy, BAK-CLS),通过文本+文本范式在预训练语言模型上实现多模态情感分析任务。该多任务框架包括掩码语言建模任务和对比学习任务,通过掩码语言建模学习数据样本情感知识提升模型表现,同时在模型训练过程中引入对比学习任务,提升模型在图文语义不一致场景下的情感识别能力,增强模型的鲁棒性。试验结果表明,相较于传统的图文特征融合网络,得益于大模型所学习的常识以及强大的推理能力,本研究提出的方法在MVSA-Single和MVSA-Multiple两个公开数据集上得到的情感分类结果在多个指标上有明显提升。

关键词: 多模态情感分析, 大语言模型, 对比学习, 辅助知识, 预训练语言模型

Abstract: With the rapid development of large models in the field of artificial intelligence, multimodal large language models(MLLM)have shown strong understanding capabilities for text and visual inputs, and they also have quite rich common sense and reasoning characteristics.Inspired by this, this paper designed a method for multimodal sentiment analysis by using a multimodal large language model LLaVA(large language and vision assistant)to generate auxiliary knowledge. At the same time, the paper designed a multi-task framework based on auxiliary knowledge and contrastive learning strategy(BAK-CLS)was designed to implement multimodal sentiment analysis tasks on pre-trained language models through the text + text paradigm. Masked language modeling tasks and contrastive learning tasks were included in the multi-task framework. The model performance was improved by learning the sentiment knowledge of data samples through masked language modeling. Meanwhile, a contrastive learning task was incorporated during model training to improve the models sentiment recognition capability in scenarios where textual and visual semantics were inconsistent, thereby enhancing its robustness. Experimental results showed that compared with the traditional image-text feature fusion network, thanks to the common sense learned by the large model and its powerful reasoning ability, the sentiment classification results obtained by the method proposed in this chapter on the two public datasets MVSA-Single and MVSA-Multiple were significantly improved in multiple indicators.

Key words: Multimodal sentiment analysis, large language model, contrastive learning, auxiliary knowledge, pre-trained language model

中图分类号:

TP391

汤凯钰,张恒源,艾孜尔古丽·玉素甫. 基于大语言模型辅助知识和对比学习策略的图文多模态情感分析[J]. 山东大学学报 (工学版), 2026, 56(5): 71-78.

Tang Kaiyu, Zhang Hengyuan, Aizierguli·Yusufu. Multimodal sentiment analysis of images and texts based on auxiliary knowledge of large language models and contrastive learning strategy[J]. Journal of Shandong University(Engineering Science), 2026, 56(5): 71-78.

参考文献

[1] 刘继明, 张培翔, 刘颖, 等.多模态的情感分析技术综述[J]. 计算机科学与探索, 2021, 15(7): 1165-1182. Liu Jiming, Zhang Peixiang, Liu Ying, et al. Summary of multi-modal sentiment analysis technology[J]. Journal of Frontiers of Computer Science & Technology, 2021, 15(7): 1165-1182.
[2] 秦小林, 古徐, 李弟诚, 等. 大语言模型综述与展望[J]. 计算机应用, 2025, 45(3): 685-696. Qin Xiaolin, Gu Xu, Li Dicheng, et al. Survey and prospect of large language models[J]. Journal of Computer Applications, 2025, 45(3): 685-696.
[3] Wu J Y, Gan W S, Chen Z F, et al. Multimodal large language models: a survey[C] //2023 IEEE International Conference on Big Data. Sorrento, Italy:IEEE, 2023: 2247-2256.
[4] Liu H T, Li C Y, Wu Q Y, et al. Visual instruction tuning[C] //Advances in Neural Information Processing Systems. New Orleans, USA: [s.n.] : 34892-34916.
[5] Li A, Xu L W, Ling C, et al. EmoVerse: exploring multimodal large language models for sentiment and emotion understanding[PP/OL]. V3.(2025-03-31)[2025-04-25]. https://doi.org/10.48550/arXiv.2412.08049
[6] Yang Y, Dong X D, Qiang Y P. MSE-adapter: a lightweight plugin endowing LLMs with the capability to perform multimodal sentiment analysis and emotion recognition[PP/OL].(2025-02-18)[2025-04-25]. https://doi.org/10.48550/arXiv.2502.12478
[7] Ando A, Masumura R, Takashima A, et al. On the use of modality-specific large-scale pre-trained encoders for multimodal sentiment analysis[C] //2022 IEEE Spoken Language Technology Workshop(SLT). Doha, Qatar:IEEE, 2023: 739-746.
[8] Li S X, Okada S. Interpretable multimodal sentiment analysis based on textual modality descriptions by using large-scale language models[PP/OL]. V3.(2023-05-12)[2025-04-25]. https://doi.org/10.48550/arXiv.2305.06162
[9] 张乐, 陈岩松, 张雷瀚. 大模型特征增强与多层次交叉融合的多模态情感分析方法[J]. 数据分析与知识发现, 2025, 9(8): 47-58. Zhang Le, Chen Yansong, Zhang Leihan. A multimodal sentiment analysis method based on LLM feature enhancement and multi-level cross-fusion[J]. Data Analysis and Knowledge Discovery, 2025, 9(8): 47-58.
[10] OpenAI, Achiam J, Adler S, et al. GPT-4 technical report[PP/OL]. V6.(2024-03-04)[2025-04-25]. https://doi.org/10.48550/arXiv.2303.08774
[11] Salazar J, Liang D, Nguyen T Q, et al.Masked language model scoring[PP/OL]. V3.(2021-01-01)[2025-04-25]. https://doi.org/10.48550/arXiv.1910.14659
[12] Gao T Y, Yao X C, Chen D Q. SimCSE: simple contrastive learning of sentence embeddings[PP/OL]. V4.(2022-05-18)[2025-04-25]. https://doi.org/10.48550/arXiv.2104.08821
[13] Niu T, Zhu S, Pang L, et al. Sentiment analysis on multi-view social data[C] //22nd International Conference on MultiMedia Modeling, Miami, USA: Springer International Publishing, 2016: 15-27.
[14] Devlin J, Chang M W, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[PP/OL].V2.(2019-05-24)[2025-04-25]. https://doi.org/10.48550/arXiv.1810.04805
[15] Liu Y H, Ott M, Goyal N, et al. RoBERTa: a robustly optimized BERT pretraining approach[PP/OL].(2019-07-26)[2025-04-25]. https://doi.org/10.48550/arXiv.1907.11692
[16] Li Z, Xu B, Zhu C H, et al. CLMLF: a contrastive learning and multi-layer fusion method for multimodal sentiment detection[PP/OL]. V4.(2022-06-14)[2025-04-25]. https://doi.org/10.48550/arXiv.2204.05515
[17] Zhu T, Li L D, Yang J F, et al. Multimodal sentiment analysis with image-text interaction network[J]. IEEE Transactions on Multimedia, 2023, 25: 3375-3385.
[18] Yang X C, Feng S, Wang D L, et al. Multiple contrastive learning for multimodal sentiment analysis[C] //ICASSP 2023-2023 IEEE International Con-ference on Acoustics, Speech and Signal Processing(ICASSP). Rhodes Island, Greece: IEEE, 2023: 1-5.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于大语言模型辅助知识和对比学习策略的图文多模态情感分析

Multimodal sentiment analysis of images and texts based on auxiliary knowledge of large language models and contrastive learning strategy

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 4

多维度评价

本文评价

推荐阅读 0

[1]	陈颖鑫,陈振涵,林晓宇,蔡志铃,陈林峰,王李进. QD-RAG:一种问题分解驱动的迭代检索增强生成方法[J]. 山东大学学报 (工学版), 2026, 56(4): 75-83.
[2]	陈宇,孟广婷,宗臣,袁卫华,王洁宁,王星. 双侧协同过滤多模态推荐对比表示提升算法[J]. 山东大学学报 (工学版), 2026, 56(3): 106-117.
[3]	刁振宇,韩小凡,张承宇,聂慧佳,赵秀阳,牛冬梅. 基于实例判别与特征增强的单图三维模型检索[J]. 山东大学学报 (工学版), 2025, 55(2): 71-77.
[4]	郑顺,王绍卿,刘玉芳,李可可,孙福振. 基于动态掩码和多对对比学习的序列推荐模型[J]. 山东大学学报 (工学版), 2023, 53(6): 47-55.