基于空间注意力和卷积神经网络的视觉情感分析

doi:10.6040/j.issn.1672-3961.0.2019.422

摘要/Abstract

摘要：

为了解决现有基于深度学习方法的视觉情感分析忽略了图像各局部区域情感呈现的强度差异问题,提出一种结合空间注意力的卷积神经网络spatial attention with CNN, SA-CNN用于提升视觉情感分析效果。设计一个情感区域探测神经网络用于发现图像中诱发情感的局部区域;通过空间注意力机制对情感映射中各个位置赋予注意力权重,恰当抽取各区域的情感特征表示,从而有助于利用局部区域情感信息进行分类;整合局部区域特征和整体图像特征形成情感判别性视觉特征,并用于训练视觉情感的神经网络分类器。该方法在3个真实数据集TwitterⅠ、TwitterⅡ和Flickr上的情感分类准确率分别达到82.56%、80.23%、79.17%,证明利用好图像局部区域情感表达的差异性,能提升视觉情感分类效果。

关键词: 图像处理, 情感分析, 深度学习, 注意力机制, 神经网络

Abstract:

Existing visual sentiment analysis based on deep learning mainly ignored the intensity differences of emotional presentation in different parts of the image. In order to solve this problem, the convolutional neural network based on spatial attention (SA-CNN) was proposed to improve the effect of visual sentiment analysis. The affective region detection neural network was designed to discover the local areas of sentiment induced in images. The spatial attention mechanism was used to assign attention weights to each location in the sentiment map, and the sentiment features of each region were extracted appropriately, which was helpful for sentiment classification by using local information. The discriminant visual features were formed by integrating local region features and global image features, and were used to train the neural network classifier of visual sentiment. Classification accuracy of the method achieved 82.56%, 80.23% and 79.17% on three real datasets Twitter Ⅰ, Twitter Ⅱ and Flickr, which proved that the method could improve the visual emotion classification effect by making good use of the difference of emotion expression in the local area of the image.

Key words: image process, sentiment analysis, deep learning, attention mechanism, neural network

中图分类号:

TP391

蔡国永,贺歆灏,储阳阳. 基于空间注意力和卷积神经网络的视觉情感分析[J]. 山东大学学报 (工学版), 2020, 50(4): 8-13.

Guoyong CAI,Xinhao HE,Yangyang CHU. Visual sentiment analysis based on spatial attention mechanism and convolutional neural network[J]. Journal of Shandong University(Engineering Science), 2020, 50(4): 8-13.

图/表 4

图1

图2

图3

图4

参考文献 20

1	YANG J, SHE D, LAI Y K, et al. Weakly supervised coupled networks for visual sentiment analysis[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City, USA: IEEE Press, 2018: 7584-7592.
2	JIN X, GALLAGHER A, CAO L, et al. The wisdom of social multimedia: using flickr for prediction and forecast[C]//Proceedings of the 2010 International Conference on Multimedea 2010.Firenze, Italy: ACM Press, 2010: 1235-1244.
3	YUAN J, MCDONOUGH S, YOU Q, et al. Sentribute: image sentiment analysis from a mid-level perspective[C]//Proceedings of the 2013 International Workshop on Issues of Sentiment Discovery and Opinion Mining. Chicago, USA: ACM Press, 2013: 1-8.
4	MACHAJDIK J, HANBURY A. Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM international conference on Multimedia. Firenze, Italy: ACM Press, 2010: 83-92.
5	CHEN M, ZHANG L, ALLEBACH J P. Learning deep features for image emotion classification[C]//Procee-dings of the 2015 IEEE International Conference on Image Processing. Piscataway, USA: IEEE Press, 2015: 4491-4495.
6	YOU Q, YANG J, YANG J, et al. Building a large scale dataset for image emotion recognition: the fine print and the benchmark[C]//Proceedings of the 2016 Thirtieth AAAI Conference on Artificial Intelligence. Phoenix, USA: AAAI Press, 2016: 308-314.
7	YANG J , SHE D , SUN M , et al. Visual sentiment prediction based on automatic discovery of affective regions[J]. IEEE Transactions on Multimedia, 2018, 20 (9): 2513- 2525.
8	SIERSDORFER S, MINACK E, DENG F, et al. Analyzing and predicting sentiment of images on the social web[C]//Proceedings of the 18th ACM international conference on Multimedia. Firenze, Italy: ACM Press, 2010: 715-718.
9	BORTH D, JI R, CHEN T, et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM international conference on Multimedia. Barcelona, Spain: ACM Press, 2013(9): 223-232.
10	CHEN T , BORTH D , DARRELL T , et al. DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks[J]. OALIB Journal-Computer Science, 2014, 1- 6.
11	YOU Q, YANG J, YANG J, et al. Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]// Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin, USA: AAAI Press, 2015: 381-388.
12	SUN M, YANG J, WANG K, et al. Discovering affective regions in deep convolutional neural networks for visual sentiment prediction[C]// 2016 IEEE International Conference on Multimedia and Expo (ICME). Seattle, USA: IEEE Press, 2016: 1-6.
13	LI B, XIONG W, HU W, et al. Context-aware affective images classification based on bilayer sparse repres-entation[C]// Proceedings of the 20th ACM international conference on Multimedia. Nara, Japan: ACM Press, 2012: 721-724.
14	ITTI L , KOCH C . Computational modelling of visual attention[J]. Nature reviews neuroscience, 2001, 2 (3): 194- 195.
15	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in neural information processing systems. California, USA: MIT Press, 2017: 5998-6008.
16	LI L, TANG S, DDENG L, et al. Image caption with global-local attention[C]//Thirty-First AAAI Conference on Artificial Intelligence 2017. San Francisco, USA: AAAI Press.
17	MNIN V, HEESS N, GRAVES A. Recurrent models of visual attention[C]//Advances in Neural Information Processing Systems. Montreal, Canada: MIT Press, 2014: 2204-2212.
18	CHEN L C, YANG Y, WANG J, et al. Attention to scale: scale-aware semantic image segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, USA: IEEE Press, 2016: 3640-3649.
19	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA.: IEEE Press, 2016: 770-778.
20	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015-04-10)[2019-02-01]. http://arxiv.org/abs/1410.8586v.

相关文章 15

[1]	彭岩,冯婷婷,王洁. 基于集成学习的O₃的质量浓度预测模型[J]. 山东大学学报 (工学版), 2020, 50(4): 1-7.
[2]	廖南星,周世斌,张国鹏,程德强. 基于类激活映射-注意力机制的图像描述方法[J]. 山东大学学报 (工学版), 2020, 50(4): 28-34.
[3]	刘帅,王磊,丁旭涛. 基于Bi-LSTM的脑电情绪识别[J]. 山东大学学报 (工学版), 2020, 50(4): 35-39.
[4]	李怡霏,郭尊华. 一种Chirplet神经网络自动目标识别算法[J]. 山东大学学报 (工学版), 2020, 50(3): 8-14.
[5]	金保明,卢光毅,王伟,杜伦阅. 基于弹性梯度下降算法的BP神经网络降雨径流预报模型[J]. 山东大学学报 (工学版), 2020, 50(3): 117-124.
[6]	陈宁宁,赵建伟,周正华. 基于校正神经网络的视频追踪算法[J]. 山东大学学报 (工学版), 2020, 50(2): 17-26.
[7]	宋士奇,朴燕,蒋泽新. 基于改进YOLOv3的复杂场景车辆分类与跟踪[J]. 山东大学学报 (工学版), 2020, 50(2): 27-33.
[8]	陈艳平,冯丽,秦永彬,黄瑞章. 一种基于深度神经网络的句法要素识别方法[J]. 山东大学学报 (工学版), 2020, 50(2): 44-49.
[9]	李春阳,李楠,冯涛,王朱贺,马靖凯. 基于深度学习的洗衣机异常音检测[J]. 山东大学学报 (工学版), 2020, 50(2): 108-117.
[10]	蔡国永,林强,任凯琪. 基于域对抗网络和BERT的跨领域文本情感分析[J]. 山东大学学报 (工学版), 2020, 50(1): 1-7,20.
[11]	陈德蕾,王成,陈建伟,吴以茵. 基于门控循环单元与主动学习的协同过滤推荐算法[J]. 山东大学学报 (工学版), 2020, 50(1): 21-27,48.
[12]	曹小洁,李小华,刘辉. 一类非仿射非线性大系统的结构在线扩展[J]. 山东大学学报 (工学版), 2020, 50(1): 35-48.
[13]	杨巨成,韩书杰,毛磊,代翔子,陈亚瑞. 胶囊网络模型综述[J]. 山东大学学报 (工学版), 2019, 49(6): 1-10.
[14]	刘玉田, 孙润稼, 王洪涛, 顾雪平. 人工智能在电力系统恢复中的应用综述[J]. 山东大学学报 (工学版), 2019, 49(5): 1-8.
[15]	梁志祥,刘晓明,牟颖,刘玉田. 基于深度学习的新能源爬坡事件预测方法[J]. 山东大学学报 (工学版), 2019, 49(5): 24-28.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed