Journal of Shandong University(Engineering Science) ›› 2020, Vol. 50 ›› Issue (4): 8-13.doi: 10.6040/j.issn.1672-3961.0.2019.422

• Machine Learning & Data Mining • Previous Articles     Next Articles

Visual sentiment analysis based on spatial attention mechanism and convolutional neural network

Guoyong CAI(),Xinhao HE,Yangyang CHU   

  1. Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, Guilin 541004, Guangxi, China
  • Received:2019-07-23 Online:2020-08-20 Published:2020-08-13

RichHTML

113

PDF (PC)

673

Abstract:

Existing visual sentiment analysis based on deep learning mainly ignored the intensity differences of emotional presentation in different parts of the image. In order to solve this problem, the convolutional neural network based on spatial attention (SA-CNN) was proposed to improve the effect of visual sentiment analysis. The affective region detection neural network was designed to discover the local areas of sentiment induced in images. The spatial attention mechanism was used to assign attention weights to each location in the sentiment map, and the sentiment features of each region were extracted appropriately, which was helpful for sentiment classification by using local information. The discriminant visual features were formed by integrating local region features and global image features, and were used to train the neural network classifier of visual sentiment. Classification accuracy of the method achieved 82.56%, 80.23% and 79.17% on three real datasets Twitter Ⅰ, Twitter Ⅱ and Flickr, which proved that the method could improve the visual emotion classification effect by making good use of the difference of emotion expression in the local area of the image.

Key words: image process, sentiment analysis, deep learning, attention mechanism, neural network

CLC Number: 

  • TP391

Fig.1

The framework of visual sentiment analysis based on SA-CNN"

Fig.2

Schematic diagram of resnet residual unit"

Fig.3

Classification accuracy of different methods on Twitter Ⅰ and Twitter Ⅱ datasets"

Fig.4

Classification accuracy of different methods on Flickr datasets"

1 YANG J, SHE D, LAI Y K, et al. Weakly supervised coupled networks for visual sentiment analysis[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City, USA: IEEE Press, 2018: 7584-7592.
2 JIN X, GALLAGHER A, CAO L, et al. The wisdom of social multimedia: using flickr for prediction and forecast[C]//Proceedings of the 2010 International Conference on Multimedea 2010.Firenze, Italy: ACM Press, 2010: 1235-1244.
3 YUAN J, MCDONOUGH S, YOU Q, et al. Sentribute: image sentiment analysis from a mid-level perspective[C]//Proceedings of the 2013 International Workshop on Issues of Sentiment Discovery and Opinion Mining. Chicago, USA: ACM Press, 2013: 1-8.
4 MACHAJDIK J, HANBURY A. Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM international conference on Multimedia. Firenze, Italy: ACM Press, 2010: 83-92.
5 CHEN M, ZHANG L, ALLEBACH J P. Learning deep features for image emotion classification[C]//Procee-dings of the 2015 IEEE International Conference on Image Processing. Piscataway, USA: IEEE Press, 2015: 4491-4495.
6 YOU Q, YANG J, YANG J, et al. Building a large scale dataset for image emotion recognition: the fine print and the benchmark[C]//Proceedings of the 2016 Thirtieth AAAI Conference on Artificial Intelligence. Phoenix, USA: AAAI Press, 2016: 308-314.
7 YANG J , SHE D , SUN M , et al. Visual sentiment prediction based on automatic discovery of affective regions[J]. IEEE Transactions on Multimedia, 2018, 20 (9): 2513- 2525.
8 SIERSDORFER S, MINACK E, DENG F, et al. Analyzing and predicting sentiment of images on the social web[C]//Proceedings of the 18th ACM international conference on Multimedia. Firenze, Italy: ACM Press, 2010: 715-718.
9 BORTH D, JI R, CHEN T, et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM international conference on Multimedia. Barcelona, Spain: ACM Press, 2013(9): 223-232.
10 CHEN T , BORTH D , DARRELL T , et al. DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks[J]. OALIB Journal-Computer Science, 2014, 1- 6.
11 YOU Q, YANG J, YANG J, et al. Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]// Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin, USA: AAAI Press, 2015: 381-388.
12 SUN M, YANG J, WANG K, et al. Discovering affective regions in deep convolutional neural networks for visual sentiment prediction[C]// 2016 IEEE International Conference on Multimedia and Expo (ICME). Seattle, USA: IEEE Press, 2016: 1-6.
13 LI B, XIONG W, HU W, et al. Context-aware affective images classification based on bilayer sparse repres-entation[C]// Proceedings of the 20th ACM international conference on Multimedia. Nara, Japan: ACM Press, 2012: 721-724.
14 ITTI L , KOCH C . Computational modelling of visual attention[J]. Nature reviews neuroscience, 2001, 2 (3): 194- 195.
15 VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in neural information processing systems. California, USA: MIT Press, 2017: 5998-6008.
16 LI L, TANG S, DDENG L, et al. Image caption with global-local attention[C]//Thirty-First AAAI Conference on Artificial Intelligence 2017. San Francisco, USA: AAAI Press.
17 MNIN V, HEESS N, GRAVES A. Recurrent models of visual attention[C]//Advances in Neural Information Processing Systems. Montreal, Canada: MIT Press, 2014: 2204-2212.
18 CHEN L C, YANG Y, WANG J, et al. Attention to scale: scale-aware semantic image segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, USA: IEEE Press, 2016: 3640-3649.
19 HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA.: IEEE Press, 2016: 770-778.
20 SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015-04-10)[2019-02-01]. http://arxiv.org/abs/1410.8586v.
[1] LIAO Nanxing, ZHOU Shibin, ZHANG Guopeng, CHENG Deqiang. Image caption generation method based on class activation mapping and attention mechanism [J]. Journal of Shandong University(Engineering Science), 2020, 50(4): 28-34.
[2] LIU Shuai, WANG Lei, DING Xutao. Emotional EEG recognition based on Bi-LSTM [J]. Journal of Shandong University(Engineering Science), 2020, 50(4): 35-39.
[3] Yifei LI,Zunhua GUO. A Chirplet neural network for automatic target recognition [J]. Journal of Shandong University(Engineering Science), 2020, 50(3): 8-14.
[4] Baoming JIN,Guangyi LU,Wei WANG,Lunyue DU. Research on BP neural network rainfall runoff forecasting model based on elastic gradient descent algorithm [J]. Journal of Shandong University(Engineering Science), 2020, 50(3): 117-124.
[5] Ningning CHEN,Jianwei ZHAO,Zhenghua ZHOU. Visual tracking algorithm based on verifying networks [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 17-26.
[6] Shiqi SONG,Yan PIAO,Zexin JIANG. Vehicle classification and tracking for complex scenes based on improved YOLOv3 [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 27-33.
[7] Yanping CHEN,Li FENG,Yongbin QIN,Ruizhang HUANG. A syntactic element recognition method based on deep neural network [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 44-49.
[8] Chunyang LI,Nan LI,Tao FENG,Zhuhe WANG,Jingkai MA. Abnormal sound detection of washing machines based on deep learning [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 108-117.
[9] Guoyong CAI,Qiang LIN,Kaiqi REN. Cross-domain text sentiment classification based on domain-adversarialnetwork and BERT [J]. Journal of Shandong University(Engineering Science), 2020, 50(1): 1-7,20.
[10] Delei CHEN,Cheng WANG,Jianwei CHEN,Yiyin WU. GRU-based collaborative filtering recommendation algorithm with active learning [J]. Journal of Shandong University(Engineering Science), 2020, 50(1): 21-27,48.
[11] Xiaojie CAO,Xiaohua LI,Hui LIU. Construction expansion online for a class of nonaffine nonlinear large-scale systems [J]. Journal of Shandong University(Engineering Science), 2020, 50(1): 35-48.
[12] Jucheng YANG,Shujie HAN,Lei MAO,Xiangzi DAI,Yarui CHEN. Review of capsule network [J]. Journal of Shandong University(Engineering Science), 2019, 49(6): 1-10.
[13] Yutian LIU, Runjia SUN, Hongtao WANG, Xueping GU. Review on application of artificial intelligence in power system restoration [J]. Journal of Shandong University(Engineering Science), 2019, 49(5): 1-8.
[14] Zhixiang LIANG,Xiaoming LIU,Ying MU,Yutian LIU. Prediction method of wind power and PV ramp event based on deep learning [J]. Journal of Shandong University(Engineering Science), 2019, 49(5): 24-28.
[15] Ji ZHANG,Cui JIN,Hongyuan WANG,Shoubing CHEN. Pedestrian recognition based on singular value decomposition pedestrian alignment network [J]. Journal of Shandong University(Engineering Science), 2019, 49(5): 91-97.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] ZHANG Yong-hua,WANG An-ling,LIU Fu-ping . The reflected phase angle of low frequent inhomogeneous[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 22 -25 .
[2] SHI Lai-shun,WAN Zhong-yi . Synthesis and performance evaluation of a novel betaine-type asphalt emulsifier[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 112 -115 .
[3] CHEN Rui, LI Hongwei, TIAN Jing. The relationship between the number of magnetic poles and the bearing capacity of radial magnetic bearing[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(2): 81 -85 .
[4] LI Ke,LIU Chang-chun,LI Tong-lei . Medical registration approach using improved maximization of mutual information[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 107 -110 .
[5] QIN Tong, SUN Fengrong*, WANG Limei, WANG Qinghao, LI Xincai. 3D surface reconstruction using the shape based interpolation guided by maximal discs[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2010, 40(3): 1 -5 .
[6] LIU Wen-liang, ZHU Wei-hong, CHEN Di, ZHANG Hong-quan. Detection and tracking of moving targets using the morphology match in radar images[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2010, 40(3): 31 -36 .
[7] ZHANG Ying,LANG Yongmei,ZHAO Yuxiao,ZHANG Jianda,QIAO Peng,LI Shanping . Research on technique of aerobic granular sludge cultivationby seeding EGSB anaerobic granular sludge[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(4): 56 -59 .
[8] WANG Li-ju,HUANG Qi-cheng,WANG Zhao-xu . [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(6): 51 -56 .
[9] SUN Dianzhu, ZHU Changzhi, LI Yanrui. [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 84 -86 .
[10] YUE Yuan-Zheng. Relaxation in glasses far from equilibrium[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 1 -20 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd