Journal of Shandong University(Engineering Science) ›› 2020, Vol. 50 ›› Issue (4): 8-13.doi: 10.6040/j.issn.1672-3961.0.2019.422

• Machine Learning & Data Mining • Previous Articles     Next Articles

Visual sentiment analysis based on spatial attention mechanism and convolutional neural network

Guoyong CAI(),Xinhao HE,Yangyang CHU   

  1. Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, Guilin 541004, Guangxi, China
  • Received:2019-07-23 Online:2020-08-20 Published:2020-08-13

RichHTML

118

PDF (PC)

2002

Abstract:

Existing visual sentiment analysis based on deep learning mainly ignored the intensity differences of emotional presentation in different parts of the image. In order to solve this problem, the convolutional neural network based on spatial attention (SA-CNN) was proposed to improve the effect of visual sentiment analysis. The affective region detection neural network was designed to discover the local areas of sentiment induced in images. The spatial attention mechanism was used to assign attention weights to each location in the sentiment map, and the sentiment features of each region were extracted appropriately, which was helpful for sentiment classification by using local information. The discriminant visual features were formed by integrating local region features and global image features, and were used to train the neural network classifier of visual sentiment. Classification accuracy of the method achieved 82.56%, 80.23% and 79.17% on three real datasets Twitter Ⅰ, Twitter Ⅱ and Flickr, which proved that the method could improve the visual emotion classification effect by making good use of the difference of emotion expression in the local area of the image.

Key words: image process, sentiment analysis, deep learning, attention mechanism, neural network

CLC Number: 

  • TP391

Fig.1

The framework of visual sentiment analysis based on SA-CNN"

Fig.2

Schematic diagram of resnet residual unit"

Fig.3

Classification accuracy of different methods on Twitter Ⅰ and Twitter Ⅱ datasets"

Fig.4

Classification accuracy of different methods on Flickr datasets"

1 YANG J, SHE D, LAI Y K, et al. Weakly supervised coupled networks for visual sentiment analysis[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City, USA: IEEE Press, 2018: 7584-7592.
2 JIN X, GALLAGHER A, CAO L, et al. The wisdom of social multimedia: using flickr for prediction and forecast[C]//Proceedings of the 2010 International Conference on Multimedea 2010.Firenze, Italy: ACM Press, 2010: 1235-1244.
3 YUAN J, MCDONOUGH S, YOU Q, et al. Sentribute: image sentiment analysis from a mid-level perspective[C]//Proceedings of the 2013 International Workshop on Issues of Sentiment Discovery and Opinion Mining. Chicago, USA: ACM Press, 2013: 1-8.
4 MACHAJDIK J, HANBURY A. Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM international conference on Multimedia. Firenze, Italy: ACM Press, 2010: 83-92.
5 CHEN M, ZHANG L, ALLEBACH J P. Learning deep features for image emotion classification[C]//Procee-dings of the 2015 IEEE International Conference on Image Processing. Piscataway, USA: IEEE Press, 2015: 4491-4495.
6 YOU Q, YANG J, YANG J, et al. Building a large scale dataset for image emotion recognition: the fine print and the benchmark[C]//Proceedings of the 2016 Thirtieth AAAI Conference on Artificial Intelligence. Phoenix, USA: AAAI Press, 2016: 308-314.
7 YANG J , SHE D , SUN M , et al. Visual sentiment prediction based on automatic discovery of affective regions[J]. IEEE Transactions on Multimedia, 2018, 20 (9): 2513- 2525.
8 SIERSDORFER S, MINACK E, DENG F, et al. Analyzing and predicting sentiment of images on the social web[C]//Proceedings of the 18th ACM international conference on Multimedia. Firenze, Italy: ACM Press, 2010: 715-718.
9 BORTH D, JI R, CHEN T, et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM international conference on Multimedia. Barcelona, Spain: ACM Press, 2013(9): 223-232.
10 CHEN T , BORTH D , DARRELL T , et al. DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks[J]. OALIB Journal-Computer Science, 2014, 1- 6.
11 YOU Q, YANG J, YANG J, et al. Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]// Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin, USA: AAAI Press, 2015: 381-388.
12 SUN M, YANG J, WANG K, et al. Discovering affective regions in deep convolutional neural networks for visual sentiment prediction[C]// 2016 IEEE International Conference on Multimedia and Expo (ICME). Seattle, USA: IEEE Press, 2016: 1-6.
13 LI B, XIONG W, HU W, et al. Context-aware affective images classification based on bilayer sparse repres-entation[C]// Proceedings of the 20th ACM international conference on Multimedia. Nara, Japan: ACM Press, 2012: 721-724.
14 ITTI L , KOCH C . Computational modelling of visual attention[J]. Nature reviews neuroscience, 2001, 2 (3): 194- 195.
15 VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in neural information processing systems. California, USA: MIT Press, 2017: 5998-6008.
16 LI L, TANG S, DDENG L, et al. Image caption with global-local attention[C]//Thirty-First AAAI Conference on Artificial Intelligence 2017. San Francisco, USA: AAAI Press.
17 MNIN V, HEESS N, GRAVES A. Recurrent models of visual attention[C]//Advances in Neural Information Processing Systems. Montreal, Canada: MIT Press, 2014: 2204-2212.
18 CHEN L C, YANG Y, WANG J, et al. Attention to scale: scale-aware semantic image segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, USA: IEEE Press, 2016: 3640-3649.
19 HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA.: IEEE Press, 2016: 770-778.
20 SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015-04-10)[2019-02-01]. http://arxiv.org/abs/1410.8586v.
[1] HUANG Fang, WANG Xin, GAO Guohai, SHEN Lingzhen, FU Xun, FANG Yu. Mining Top-k frequent patterns for graphs based on subjective and objective metrics [J]. Journal of Shandong University(Engineering Science), 2025, 55(6): 1-12.
[2] WANG Yuou, YUAN Yingchun, HE Zhenxue, HE Chen. University academic named entity recognition based on the fusion of multi-feature and multi-head self-attention mechanism [J]. Journal of Shandong University(Engineering Science), 2025, 55(6): 35-44.
[3] SHAO Mengwei, YUAN Shifei, ZHOU Hongzhi, WANG Naihua. Optimisation of finned tube structure based on BP neural network and genetic algorithm [J]. Journal of Shandong University(Engineering Science), 2025, 55(6): 76-82.
[4] LI Changgang, LI Baoliang, CAO Yongji, WANG Jiaying. Review and prospect on artificial intelligence application in power system power flow calculation [J]. Journal of Shandong University(Engineering Science), 2025, 55(5): 1-17.
[5] DENG Bin, ZHANG Zongbao, ZHAO Wenmeng, LUO Xinhang, WU Qiuwei. Cloud-edge collaborative and graph neural network based load forecasting method for electric vehicle charging stations [J]. Journal of Shandong University(Engineering Science), 2025, 55(5): 62-69.
[6] ZHOU Qunying, SUI Jiacheng, ZHANG Ji, WANG Hongyuan. Industrial product surface defect detection based on self supervised convolution and parameter free attention mechanism [J]. Journal of Shandong University(Engineering Science), 2025, 55(4): 40-47.
[7] XUE Bingbing, WANG Yong, YANG Weihao, WANG Chuan, YU Di, WANG Xu. Real-time expressway traffic data imputation and state prediction based on ETC system data [J]. Journal of Shandong University(Engineering Science), 2025, 55(3): 58-71.
[8] DONG Mingshu, CHEN Liqi, MA Chuanyi, ZHANG Zhuhao, SUN Renjuan, GUAN Yanhua, ZHUANG Peizhi. Deep learning-based intelligent judgment for radar detection of pavement cracks [J]. Journal of Shandong University(Engineering Science), 2025, 55(3): 72-79.
[9] LI Feng, WEN Yimin. Multi-scale visual and textual semantic feature fusion for image captioning [J]. Journal of Shandong University(Engineering Science), 2025, 55(3): 80-87.
[10] JIA Xuan, XU Jikai, REN Yijing, LIU Decai, XU Qiang, ZHANG Li. Calculation method of theoretical line loss in transformer districts based on sample expansion and data-driven [J]. Journal of Shandong University(Engineering Science), 2025, 55(3): 158-164.
[11] ZHU Ming, SHI Chenglong, LÜ Pan, LIU Xianrong, SUN Chi, CHEN Jiancheng, FAN Hongyun. Deformation prediction method and engineering application of deep foundation pit based on optimized LSTM method [J]. Journal of Shandong University(Engineering Science), 2025, 55(3): 141-148.
[12] WANG Yuou, YUAN Yingchun, HE Zhenxue, WANG Kejian. A relation extraction method based on improved RoBERTa, multiple-instance learning and dual attention mechanism [J]. Journal of Shandong University(Engineering Science), 2025, 55(2): 78-87.
[13] Jiachun LI,Bowen LI,Jianbo CHANG. An efficient and lightweight RGB frame-level face anti-spoofing model [J]. Journal of Shandong University(Engineering Science), 2023, 53(6): 1-7.
[14] Hong YU,Juan DU,Lin WEI,Li ZHANG. Data fitting method for electricity consumption of power market users considering behavioral characteristics [J]. Journal of Shandong University(Engineering Science), 2023, 53(4): 113-119.
[15] Haoyuan LI,Jingming YU,Guilin ZHANG,Bin ZHANG. Optimization of manufacturing parameters for optical fiber preform core based on intelligent algorithm [J]. Journal of Shandong University(Engineering Science), 2023, 53(4): 149-156.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] WANG Su-yu,<\sup>,AI Xing<\sup>,ZHAO Jun<\sup>,LI Zuo-li<\sup>,LIU Zeng-wen<\sup> . Milling force prediction model for highspeed end milling 3Cr2Mo steel[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 1 -5 .
[2] ZHANG Yong-hua,WANG An-ling,LIU Fu-ping . The reflected phase angle of low frequent inhomogeneous[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 22 -25 .
[3] LI Kan . Empolder and implement of the embedded weld control system[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 37 -41 .
[4] KONG Xiang-zhen,LIU Yan-jun,WANG Yong,ZHAO Xiu-hua . Compensation and simulation for the deadband of the pneumatic proportional valve[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 99 -102 .
[5] LAI Xiang . The global domain of attraction for a kind of MKdV equations[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 87 -92 .
[6] YU Jia yuan1, TIAN Jin ting1, ZHU Qiang zhong2. Computational intelligence and its application in psychology[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 1 -5 .
[7] CHEN Rui, LI Hongwei, TIAN Jing. The relationship between the number of magnetic poles and the bearing capacity of radial magnetic bearing[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(2): 81 -85 .
[8] LI Ke,LIU Chang-chun,LI Tong-lei . Medical registration approach using improved maximization of mutual information[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 107 -110 .
[9] JI Tao,GAO Xu/sup>,SUN Tong-jing,XUE Yong-duan/sup>,XU Bing-yin/sup> . Characteristic analysis of fault generated traveling waves in 10 Kv automatic blocking and continuous power transmission lines[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 111 -116 .
[10] . [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 27 -32 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd