Journal of Shandong University(Engineering Science) ›› 2020, Vol. 50 ›› Issue (1): 1-7,20.doi: 10.6040/j.issn.1672-3961.0.2019.293

• Machine Learning & Data Mining •     Next Articles

Cross-domain text sentiment classification based on domain-adversarialnetwork and BERT

Guoyong CAI(),Qiang LIN,Kaiqi REN   

  1. School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, Guangxi, China
  • Received:2019-06-10 Online:2020-02-20 Published:2020-02-14
  • Supported by:
    国家自然科学基金资助项目(61763007);广西自然科学基金重点资助项目(2017JJD160017)

RichHTML

50

PDF (PC)

809

Abstract:

In order to capture more sentence semantic information from the extracted shared sentiment features for cross-domain sentiment analysis, a deep network model based on domain adversarial mechanism and BERT (bidirectional encoder representations from transformers) was proposed. The model firstly used BERT to obtain the semantic representation vectors of sentences, and then extracted the local features of sentences with a convolutional neural network. A domain adversarial neural network was designed to make the representations of features extracted from different domains to be as indistinguishable as possible, that was, the features extracted from source domain and target domain had much more similarities; and a sentiment classifier was trained on the source domain dataset with sentiment labels, and it was expected that the trained sentiment classifier would have good classification performance in the source domain, and in the target domain. The experimental results on Amazon product reviews dataset showed that the proposed method achieved the expectation and was competent for achieving cross-domain text sentiment classification.

Key words: cross-domain, sentiment analysis, convolution neural network, domain adversarial network, shared sentiment features

CLC Number: 

  • TP391

Fig.1

Model BCN on cross-domain text sentiment analysis"

Fig.2

Transformer structure"

Fig.3

Multi-Head Attention"

Fig.4

Convolution for text"

Table 1

Statistical information of datasets  条"

领域 积极评论 消极评论 无标签评论
Book 3 000 3 000 9 750
dvd 3 000 3 000 11 843
electronics 3 000 3 000 17 009
Kitchen 3 000 3 000 13 856
Video 3 000 3 000 30 180

Table 2

Accuracy performance of cross-domain sentiment analysis on Amazon product review data set"

源领域 目标领域 方法
S-only DANN mSDA AMN HATN BD BCN
Books DVD 0.805 7 0.834 2 0.861 2 0.856 2 0.870 7 0.893 8 0.893 3
Kitchen 0.716 3 0.779 0 0.810 5 0.818 8 0.870 3 0.917 0 0.919 0
Electronics 0.736 5 0.762 7 0.790 2 0.805 5 0.857 5 0.913 5 0.914 0
Video 0.814 5 0.832 3 0.849 8 0.872 5 0.878 0 0.896 3 0.901 7
DVD Books 0.764 5 0.807 7 0.851 7 0.845 3 0.877 8 0.905 0 0.910 5
Kitchen 0.734 3 0.781 5 0.826 0 0.816 7 0.874 7 0.915 5 0.922 3
Electronics 0.731 2 0.763 5 0.761 7 0.804 2 0.863 2 0.915 0 0.914 7
Video 0.827 5 0.859 5 0.838 0 0.874 0 0.891 2 0.910 3 0.917 1
Video DVD 0.824 3 0.841 5 0.859 0 0.868 8 0.879 0 0.905 0 0.909 1
Kitchen 0.713 3 0.752 2 0.795 2 0.809 0 0.864 5 0.915 3 0.917 1
Electronics 0.718 7 0.757 2 0.776 7 0.796 8 0.859 8 0.8985 0.900 0
Books 0.770 3 0.800 3 0.830 0 0.835 0 0.871 0 0.909 1 0.907 6
Electronics DVD 0.726 0 0.762 7 0.826 3 0.805 3 0.843 2 0.866 0 0.868 3
Kitchen 0.846 3 0.845 3 0.858 0 0.878 3 0.900 8 0.939 3 0.941 7
Video 0.724 8 0.772 0 0.817 0 0.821 2 0.841 8 0.864 5 0.870 0
Books 0.688 7 0.735 3 0.799 2 0.775 2 0.840 3 0.887 6 0.882 1
Kitchen DVD 0.733 2 0.753 2 0.821 8 0.795 0 0.847 2 0.867 2 0.870 8
Video 0.760 8 0.763 7 0.814 7 0.821 5 0.848 5 0.872 8 0.874 5
Electronics 0.831 5 0.855 3 0.880 0 0.866 8 0.893 3 0.933 8 0.930 8
Books 0.715 3 0.741 7 0.805 5 0.790 5 0.848 8 0.888 8 0.882 3
平均准确率 0.759 2 0.790 0 0.823 6 0.827 9 0.866 1 0.900 7 0.902 3

Fig.5

Accuracy of different convolution kernel size"

Fig.6

Accuracy of different β values"

1 TAN S, CHENG X, WANG Y, et al. Adapting naive bayes to domain adaptation for sentiment analysis[C]//European Conference on Information Retrieval. Berlin, Germany: Springer, 2009: 337-349.
2 PAN S J, NI X, SUN J, et al. Cross-domain sentiment classification via spectral feature alignment[C]//19thInternational Conference on World Wide Web. Raleigh, North Carolina, USA: ACM, 2010: 751-760.
3 GLOROT X, BORDES A, BENGIO Y, et al. Domain adaptation for large-scale sentiment classification: a deep learning approach[C]//28th International Conference on Machine Learning. Bellevue, Washington, USA: Omnipress, 2011: 513-520.
4 CHEN M, XU Z, SHA F, et al. Marginalized Denoising Autoencoders for Domain Adaptation[C]//29th International Conference on Machine Learning. Edinburgh, Scotland, UK: [s.n.], 2012: 1627-1634.
5 GANIN Y , USTINOVA E , AJAKAN H , et al. Domain-adversarial training of neural networks[J]. Journal of Machine Learning Research, 2016, 17 (1): 1- 35.
6 AJAKAN H , GERMAIN P , LAROCHELLE H , et al. Domain-Adversarial Neural Networks[J]. Statistics, 2014, (1050): 1- 8.
7 LI Z, ZHANG Y, WEI Y, et al. End-to-end adversarial memory network for cross-domain sentiment classification[C]// 26th International Joint Conference on Artificial Intelligence. Melbourne, Australia: [s.n.], 2017: 2237-2243.
8 LI Z, WEI Y, ZHANG Y, et al. Hierarchical attention transfer network for cross-domain sentiment classification[C]//32th AAAI Conference on Artificial Intelligence. Hilton New Orleans Riverside, USA: AAAI, 2018: 5852-5859.
9 BLITZER J, DREDZE M, PEREORA F, et al. Biographies, bollywood, boom-boxes and blenders: domain adaptation for sentiment classification[C]//Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics. Prague, Czech Republic: Association for Computational Linguistics, 2007: 440-447.
10 GOODFELLOW I J, POUGETABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems. Montreal, Canada: [s.n.], 2014: 2672-2680.
11 KRIZHEVSKY A, SUTSKEVER I, HINTON G E, et al. ImageNet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems. Lake Tahoe, Nevada, USA: [s.n.], 2012: 1106-1114.
12 KARPATHY A, TODERICI G. Large-scale video classification with convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition. Columbus, Ohio, USA: IEEE Xplore, 2014: 1725-1732.
13 KIM Y. Convolutional neural networks for sentence classification[C]//Conference on empirical methods in natural language processing. Doha, Qatar: [S.n.], 2014: 1746-1751.
14 WEI X , LIN H , YU Y , et al. Low-resource cross-domain product review sentiment classification based on a CNN with an auxiliary large-scale corpus[J]. Algorithms, 2017, 10 (81): 1- 15.
15 WU F, HUANG Y. Sentiment domain adaptation with multiple sources[C]//Proceedings of the Annual Meeting of the Association for Computational Linguistics. Berlin, Germany: Association for Computational Linguistics, 2016: 301-310.
16 DEVLIN J, CHANG M, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv: Computation and Language, 2018, 23(2): 3-19.
17 VASWANI A, SHAZEER N, PARMAR N, et al. Attention is All you Need[C]//Advances in neural information processing systems. Long Beach, USA: [s.n.], 2017: 5998-6008.
18 HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE Xplore, 2016: 770-778.
[1] Wenwen QUAN,Mingxing LIN. Algorithm of underwater target recognition based on CNN features with BOF [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 107-113.
[2] Chunlin QIAN,Xingfang ZHANG,Lihua SUN. Advanced collaborative filtering recommendation model based on sentiment analysis of online review [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 47-54.
[3] Rongxiang ZHOU,Xiuyi JIA. Features analysis for Chinese irony detection [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 41-46.
[4] SHEN Ji, MA Zhiqiang, LI Tuya, ZHANG Li. A word extend LDA model for short text sentiment [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(3): 120-126.
[5] ZHOU Zhe, SHANG Lin. A sentiment analysis method based on dynamic lexicon and three-way decision [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2015, 45(1): 19-23.
[6] ZHOU Yongmei1, YANG Aimin1, LIN Jianghao2. A method of building Chinese microblog sentiment lexicon [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2014, 44(3): 36-40.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] ZHANG Yong-hua,WANG An-ling,LIU Fu-ping . The reflected phase angle of low frequent inhomogeneous[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 22 -25 .
[2] CHEN Rui, LI Hongwei, TIAN Jing. The relationship between the number of magnetic poles and the bearing capacity of radial magnetic bearing[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(2): 81 -85 .
[3] QIN Tong, SUN Fengrong*, WANG Limei, WANG Qinghao, LI Xincai. 3D surface reconstruction using the shape based interpolation guided by maximal discs[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2010, 40(3): 1 -5 .
[4] LIU Wen-liang, ZHU Wei-hong, CHEN Di, ZHANG Hong-quan. Detection and tracking of moving targets using the morphology match in radar images[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2010, 40(3): 31 -36 .
[5] SUN Dianzhu, ZHU Changzhi, LI Yanrui. [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 84 -86 .
[6] SUN Cong-zheng,GUAN Cong-sheng,QIN Jing-yu,CHENG Chuan . The structure and performances of the electroless Ni-P alloy coating on aluminum alloy[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(5): 108 -112 .
[7] LI Shijin, WANG Shengte, HUANG Leping. Change detection with remote sensing images based on forward-backward heterogenicity[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(3): 1 -9 .
[8] ZHAO Ke-Jun, WANG Xin-Jun, LIU Xiang, CHOU Yi-Hong. Algorithms of continuous top-k join query over structured overlay networks[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 32 -37 .
[9] WANG Wei,MAO Hua-yong,LI Guo-xiang,PAN Shi-yan,GONG Ting-fang,JIN Shi-qiang,HAO Sheng-bing . Numerical simulation of the flow in a fuel burned vehicle heater[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(3): 64 -68 .
[10] XU Xiaodan, DUAN Zhengjie, CHEN Zhongyu. The sentiment mining method based on extended sentiment dictionary and integrated features[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2014, 44(6): 15 -18 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd