Journal of Shandong University(Engineering Science) ›› 2020, Vol. 50 ›› Issue (1): 1-7.doi: 10.6040/j.issn.1672-3961.0.2019.293

• Machine Learning & Data Mining •     Next Articles

Cross-domain text sentiment classification based on domain-adversarialnetwork and BERT

Guoyong CAI(),Qiang LIN,Kaiqi REN   

  1. School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, Guangxi, China
  • Received:2019-06-10 Online:2020-02-20 Published:2020-02-14
  • Supported by:
    国家自然科学基金资助项目(61763007);广西自然科学基金重点资助项目(2017JJD160017)

RichHTML

50

PDF (PC)

836

Abstract:

In order to capture more sentence semantic information from the extracted shared sentiment features for cross-domain sentiment analysis, a deep network model based on domain adversarial mechanism and BERT (bidirectional encoder representations from transformers) was proposed. The model firstly used BERT to obtain the semantic representation vectors of sentences, and then extracted the local features of sentences with a convolutional neural network. A domain adversarial neural network was designed to make the representations of features extracted from different domains to be as indistinguishable as possible, that was, the features extracted from source domain and target domain had much more similarities; and a sentiment classifier was trained on the source domain dataset with sentiment labels, and it was expected that the trained sentiment classifier would have good classification performance in the source domain, and in the target domain. The experimental results on Amazon product reviews dataset showed that the proposed method achieved the expectation and was competent for achieving cross-domain text sentiment classification.

Key words: cross-domain, sentiment analysis, convolution neural network, domain adversarial network, shared sentiment features

CLC Number: 

  • TP391

Fig.1

Model BCN on cross-domain text sentiment analysis"

Fig.2

Transformer structure"

Fig.3

Multi-Head Attention"

Fig.4

Convolution for text"

Table 1

Statistical information of datasets  条"

领域 积极评论 消极评论 无标签评论
Book 3 000 3 000 9 750
dvd 3 000 3 000 11 843
electronics 3 000 3 000 17 009
Kitchen 3 000 3 000 13 856
Video 3 000 3 000 30 180

Table 2

Accuracy performance of cross-domain sentiment analysis on Amazon product review data set"

源领域 目标领域 方法
S-only DANN mSDA AMN HATN BD BCN
Books DVD 0.805 7 0.834 2 0.861 2 0.856 2 0.870 7 0.893 8 0.893 3
Kitchen 0.716 3 0.779 0 0.810 5 0.818 8 0.870 3 0.917 0 0.919 0
Electronics 0.736 5 0.762 7 0.790 2 0.805 5 0.857 5 0.913 5 0.914 0
Video 0.814 5 0.832 3 0.849 8 0.872 5 0.878 0 0.896 3 0.901 7
DVD Books 0.764 5 0.807 7 0.851 7 0.845 3 0.877 8 0.905 0 0.910 5
Kitchen 0.734 3 0.781 5 0.826 0 0.816 7 0.874 7 0.915 5 0.922 3
Electronics 0.731 2 0.763 5 0.761 7 0.804 2 0.863 2 0.915 0 0.914 7
Video 0.827 5 0.859 5 0.838 0 0.874 0 0.891 2 0.910 3 0.917 1
Video DVD 0.824 3 0.841 5 0.859 0 0.868 8 0.879 0 0.905 0 0.909 1
Kitchen 0.713 3 0.752 2 0.795 2 0.809 0 0.864 5 0.915 3 0.917 1
Electronics 0.718 7 0.757 2 0.776 7 0.796 8 0.859 8 0.8985 0.900 0
Books 0.770 3 0.800 3 0.830 0 0.835 0 0.871 0 0.909 1 0.907 6
Electronics DVD 0.726 0 0.762 7 0.826 3 0.805 3 0.843 2 0.866 0 0.868 3
Kitchen 0.846 3 0.845 3 0.858 0 0.878 3 0.900 8 0.939 3 0.941 7
Video 0.724 8 0.772 0 0.817 0 0.821 2 0.841 8 0.864 5 0.870 0
Books 0.688 7 0.735 3 0.799 2 0.775 2 0.840 3 0.887 6 0.882 1
Kitchen DVD 0.733 2 0.753 2 0.821 8 0.795 0 0.847 2 0.867 2 0.870 8
Video 0.760 8 0.763 7 0.814 7 0.821 5 0.848 5 0.872 8 0.874 5
Electronics 0.831 5 0.855 3 0.880 0 0.866 8 0.893 3 0.933 8 0.930 8
Books 0.715 3 0.741 7 0.805 5 0.790 5 0.848 8 0.888 8 0.882 3
平均准确率 0.759 2 0.790 0 0.823 6 0.827 9 0.866 1 0.900 7 0.902 3

Fig.5

Accuracy of different convolution kernel size"

Fig.6

Accuracy of different β values"

1 TAN S, CHENG X, WANG Y, et al. Adapting naive bayes to domain adaptation for sentiment analysis[C]//European Conference on Information Retrieval. Berlin, Germany: Springer, 2009: 337-349.
2 PAN S J, NI X, SUN J, et al. Cross-domain sentiment classification via spectral feature alignment[C]//19thInternational Conference on World Wide Web. Raleigh, North Carolina, USA: ACM, 2010: 751-760.
3 GLOROT X, BORDES A, BENGIO Y, et al. Domain adaptation for large-scale sentiment classification: a deep learning approach[C]//28th International Conference on Machine Learning. Bellevue, Washington, USA: Omnipress, 2011: 513-520.
4 CHEN M, XU Z, SHA F, et al. Marginalized Denoising Autoencoders for Domain Adaptation[C]//29th International Conference on Machine Learning. Edinburgh, Scotland, UK: [s.n.], 2012: 1627-1634.
5 GANIN Y , USTINOVA E , AJAKAN H , et al. Domain-adversarial training of neural networks[J]. Journal of Machine Learning Research, 2016, 17 (1): 1- 35.
6 AJAKAN H , GERMAIN P , LAROCHELLE H , et al. Domain-Adversarial Neural Networks[J]. Statistics, 2014, (1050): 1- 8.
7 LI Z, ZHANG Y, WEI Y, et al. End-to-end adversarial memory network for cross-domain sentiment classification[C]// 26th International Joint Conference on Artificial Intelligence. Melbourne, Australia: [s.n.], 2017: 2237-2243.
8 LI Z, WEI Y, ZHANG Y, et al. Hierarchical attention transfer network for cross-domain sentiment classification[C]//32th AAAI Conference on Artificial Intelligence. Hilton New Orleans Riverside, USA: AAAI, 2018: 5852-5859.
9 BLITZER J, DREDZE M, PEREORA F, et al. Biographies, bollywood, boom-boxes and blenders: domain adaptation for sentiment classification[C]//Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics. Prague, Czech Republic: Association for Computational Linguistics, 2007: 440-447.
10 GOODFELLOW I J, POUGETABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems. Montreal, Canada: [s.n.], 2014: 2672-2680.
11 KRIZHEVSKY A, SUTSKEVER I, HINTON G E, et al. ImageNet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems. Lake Tahoe, Nevada, USA: [s.n.], 2012: 1106-1114.
12 KARPATHY A, TODERICI G. Large-scale video classification with convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition. Columbus, Ohio, USA: IEEE Xplore, 2014: 1725-1732.
13 KIM Y. Convolutional neural networks for sentence classification[C]//Conference on empirical methods in natural language processing. Doha, Qatar: [S.n.], 2014: 1746-1751.
14 WEI X , LIN H , YU Y , et al. Low-resource cross-domain product review sentiment classification based on a CNN with an auxiliary large-scale corpus[J]. Algorithms, 2017, 10 (81): 1- 15.
15 WU F, HUANG Y. Sentiment domain adaptation with multiple sources[C]//Proceedings of the Annual Meeting of the Association for Computational Linguistics. Berlin, Germany: Association for Computational Linguistics, 2016: 301-310.
16 DEVLIN J, CHANG M, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv: Computation and Language, 2018, 23(2): 3-19.
17 VASWANI A, SHAZEER N, PARMAR N, et al. Attention is All you Need[C]//Advances in neural information processing systems. Long Beach, USA: [s.n.], 2017: 5998-6008.
18 HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE Xplore, 2016: 770-778.
[1] YANG Xiuyuan, PENG Tao, YANG Liang, LIN Hongfei. Adaptive multi-domain sentiment analysis based on knowledge distillation [J]. Journal of Shandong University(Engineering Science), 2021, 51(3): 15-21.
[2] Guoyong CAI,Xinhao HE,Yangyang CHU. Visual sentiment analysis based on spatial attention mechanism and convolutional neural network [J]. Journal of Shandong University(Engineering Science), 2020, 50(4): 8-13.
[3] Chunyang LI,Nan LI,Tao FENG,Zhuhe WANG,Jingkai MA. Abnormal sound detection of washing machines based on deep learning [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 108-117.
[4] Wenwen QUAN,Mingxing LIN. Algorithm of underwater target recognition based on CNN features with BOF [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 107-113.
[5] Chunlin QIAN,Xingfang ZHANG,Lihua SUN. Advanced collaborative filtering recommendation model based on sentiment analysis of online review [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 47-54.
[6] Rongxiang ZHOU,Xiuyi JIA. Features analysis for Chinese irony detection [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 41-46.
[7] SHEN Ji, MA Zhiqiang, LI Tuya, ZHANG Li. A word extend LDA model for short text sentiment [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(3): 120-126.
[8] ZHOU Zhe, SHANG Lin. A sentiment analysis method based on dynamic lexicon and three-way decision [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2015, 45(1): 19-23.
[9] ZHOU Yongmei1, YANG Aimin1, LIN Jianghao2. A method of building Chinese microblog sentiment lexicon [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2014, 44(3): 36-40.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] LI Liang, LUO Qiming, CHEN Enhong. Graph-based ranking model for object-level search
[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 15 -21 .
[2] Yue Khing Toh1， XIAO Wendong2， XIE Lihua1. Wireless sensor network for distributed target tracking: practices via real test bed development[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 50 -56 .
[3] WANG Shan,LI Tian-ze . A new method for the control of a wound-rotor induction machine[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(3): 86 -89 .
[4] ZHANG Ying,LANG Yongmei,ZHAO Yuxiao,ZHANG Jianda,QIAO Peng,LI Shanping . Research on technique of aerobic granular sludge cultivationby seeding EGSB anaerobic granular sludge[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(4): 56 -59 .
[5] LIU Xin 1, SONG Sili 1, WANG Xinhong 2. [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 98 -100 .
[6] HUANG Le-jian,WANG Jian-ming . Dynamic analysis of the stabilized smoothing nodal integration meshfree method[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(5): 68 -72 .
[7] WU Hao,TIAN Guo-hui,HUANG Bin .

Research on the collaboration strategy of multi-robot for exploring unknown environment

[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 27 -31 .
[8] FANG Ting,YANG Zhong,SHEN Chun-Lin . Multiple targets accurate tracking on UAV formation video sequences[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 22 -26 .
[9] LI Meng-li, WANG Wei-qiang ，XU Shu-gen ， SONG Ming-da. Possibility analysis on chemical explosion of material causing urea  reactor cylinder fracture[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(6): 1 -6 .
[10] NIU Xiu-ming,FU Chun-hua . The effect of carbon on organic wastewater degradation in the process of pulse discharge[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(1): 121 -126 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd