Abstract: With the rapid development of large models in the field of artificial intelligence, multimodal large language models(MLLM)have shown strong understanding capabilities for text and visual inputs, and they also have quite rich common sense and reasoning characteristics.Inspired by this, this paper designed a method for multimodal sentiment analysis by using a multimodal large language model LLaVA(large language and vision assistant)to generate auxiliary knowledge. At the same time, the paper designed a multi-task framework based on auxiliary knowledge and contrastive learning strategy(BAK-CLS)was designed to implement multimodal sentiment analysis tasks on pre-trained language models through the text + text paradigm. Masked language modeling tasks and contrastive learning tasks were included in the multi-task framework. The model performance was improved by learning the sentiment knowledge of data samples through masked language modeling. Meanwhile, a contrastive learning task was incorporated during model training to improve the models sentiment recognition capability in scenarios where textual and visual semantics were inconsistent, thereby enhancing its robustness. Experimental results showed that compared with the traditional image-text feature fusion network, thanks to the common sense learned by the large model and its powerful reasoning ability, the sentiment classification results obtained by the method proposed in this chapter on the two public datasets MVSA-Single and MVSA-Multiple were significantly improved in multiple indicators.

Key words: Multimodal sentiment analysis, large language model, contrastive learning, auxiliary knowledge, pre-trained language model