Journal of Shandong University(Engineering Science) ›› 2020, Vol. 50 ›› Issue (3): 31-37.doi: 10.6040/j.issn.1672-3961.0.2019.364

• Machine Learning & Data Mining • Previous Articles     Next Articles

A semantictag generation method based on multi-model subspace learning

Feng TIAN(),Xin LI,Fang LIU*(),Chuang LI,Xiaoqiang SUN,Ruishan DU   

  1. School of Computer and Information Technology, Northeast Petroleum University, Daqing 163318, Heilongjiang, China
  • Received:2019-05-14 Online:2020-06-20 Published:2020-06-16
  • Contact: Fang LIU E-mail:tianfeng1980@163.com;lfliufang1983@126.com
  • Supported by:
    国家自然科学基金资助项目(61502094);东北石油大学优秀中青年科研创新团队资助项目(KYCXTD201903);黑龙江省高等教育教学改革研究项目(SJGY20180079);黑龙江省高等教育教学改革研究项目(SJGY20190098);黑龙江省哲学社会科学研究规划项目资助项目(19SHE280);大庆市哲学社会科学规划研究项目(DSGB2019042)

RichHTML

10

PDF (PC)

563

Abstract:

A multi-model subspace learning semantic tag generation method was proposed, whic was based on the visual space and label space tag correlation modeling method separately. This method reconstructed the "image-tag" correlation in a non-linear manner by establishing a visual feature similarity map, thereby unifying the visual modal representation of the image and the text modal representation of the tag into a multi-model subspace, and ensuring space structure preservation before and after conversion. In this space, the text modal information of the label and the modal information of the visual content of the image were complementary to each other. The semantically related images and labels were mapped to similar sample points in the space, and the semantic label generation problem was then transformed into the nearest label-neighbors retrieval problem. The results showed that the performance of the proposed method was 36.88% on FLICKR-25K data set, and 44.17% on NUS-WIDE data set, which indicated that the proposed method could greatly improve the accuracy of label generation.

Key words: image tag generation, multimodal learning, subspace learning, space transformation, structure preservation

CLC Number: 

  • TP391

Fig.1

Pipeline of multimodal subspace learning"

Table 1

Dataset statistic"

数据集 图像/
幅		 标签/
个		 图像/
幅		 图像平均标签/个
Flickr-25K(2008) 17 512 457 8 756 6.71
NUS-WIDE(2009) 55 615 2 892 27 808 7.83
MS-COCO(2014) 123 287 80 82 783 2.95

Fig.2

The F1 value changes"

Fig.3

Average image accuracy on NUS-WIDE dataset"

Fig.4

Average image recall rate on NUS-WIDE dataset"

Fig.5

Average image accuracy on FLICKR dataset"

Fig.6

Average image recall rate on FLICKR dataset"

Table 2

Annotation performance of MIR-FLICKR dataset %"

方法 API ARI APL ARL F1
lres 19.12 16.45 38.00 26.75 31.40
mpmf 16.58 15.83 38.53 26.01 31.06
twtv 21.05 17.03 38.37 26.61 31.43
cmusl 23.07 18.27 55.56 27.60 36.88

Table 3

Annotation performance of NUS-WIDE dataset %"

方法 API ARI APL ARL F1
lres 18.25 14.33 18.56 6.76 9.91
mpmf 17.02 13.41 13.30 8.24 10.18
twtv 19.47 15.48 17.24 7.64 10.59
cmusl 21.14 16.44 22.27 14.36 17.46

Table 4

Annotation performance of MSCOCOdataset %"

方法 API ARI APL ARL F1
lres 33.17 32.45 37.52 32.35 34.74
mpmf 33.21 32.39 38.11 32.24 34.93
twtv 34.34 33.26 40.50 32.01 35.76
cmusl 35.28 34.79 46.87 41.76 44.17
lres 35.83 33.96 47.16 42.53 44.73

Fig.7

Annotation precision of some labels in NUS-WIDE dataset"

Fig.8

Samples of annotated images in the NUS-WIDE dataset"

1 TIAN Feng , WANG Quge , LI Xin , et al. Heterogeneous multimedia cooperative annotation based on multimodal correlation learning[J]. Journal of Visual Communication and Image Representation, 2019, 58 (2): 544- 553.
2 LI Xirong , URICCHIO Tiberio , BALLAN Lamberto , et al. Socializing the semantic gap: a comparative survey on image tag assignment, refinement, and retrieval[J]. ACM Computing Surveys, 2016, 49 (1): 14.
3 田枫, 沈旭昆. 基于标签集相关性学习的大规模网络图像在线标注[J]. 自动化学报, 2014, 40 (8): 1635- 1643.
TIAN Feng , SHEN Xukun . Large scale web image online annotation by learning label set relevance[J]. Acta Automatica Sinica, 2014, 40 (8): 1635- 1643.
4 李瞳, 李彤, 赵宏伟. 基于残差神经网络的视频内容快速检索系统研究[J]. 吉林大学学报(信息科学版), 2018, 36 (4): 112- 116.
LI Tong , LI Tong , ZHAO Hongwei . Video content quick search system based on residual neural network[J]. Journal of Jilin University(Information Science Edition), 2018, 36 (4): 112- 116.
5 LI Xirong . Tag relevance fusion for social image retrieval[J]. Multimedia Systems, 2017, 23 (1): 29- 40.
6 LI Jundong , CHENG Kewei , WANG Suhang , et al. Feature selection: a data perspective[J]. ACM Computing Surveys (CSUR), 2018, 50 (6): 94.
7 MAFARJA Majdi , ALJARAH Ibrahim , HEIDARI Ali Asghar , et al. Binary dragonfly optimization for feature selection using time-varying transfer functions[J]. Knowledge-Based Systems, 2018, 161 (2): 185- 204.
8 XU Xing , HE Li , LU Huimin , et al. Deep adversarial metric learning for cross-modal retrieval[J]. World Wide Web, 2019, 22 (2): 657- 672.
doi: 10.1007/s11280-018-0541-x
9 CHENG Gong , YANG Ceyuan , YAO Xiwen , et al. When deep learning meets metric learning: remote sensing image scene classification via learning discriminative CNNs[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56 (5): 2811- 2821.
doi: 10.1109/TGRS.2017.2783902
10 HOTELLING Harold . Relations between two sets of variates[J]. Biometrika, 1936, 16 (2): 321- 377.
11 HARDOON David R , SHAWE-TAYLOR John . Sparse canonical correlation analysis[J]. Machine Learning, 2011, 83 (3): 331- 353.
12 庄凌, 庄越挺, 吴江琴. 一种基于稀疏典型性相关分析的图像检索方法[J]. 软件学报, 2012, 34 (5): 1295- 1304.
ZHUANG Ling , ZHUANG Yueting , WU Jiangqin , et al. Image retrieval approach based on sparse canonical correlation analysis[J]. Journal of Software, 2012, 34 (5): 1295- 1304.
13 LI Zechao, LIU Jing, ZHU Xiaobin. Image annotation using multi-correlation probabilistic matrix factorization[C]//Proceedings of the International Conference on Multimedia. New York, USA: ACM, 2010: 1187-1190.
14 ZHUANG Jinfeng, STEVEN C H H. A two-view learning approach for image tag ranking[C]// Proceedings of the Fourth ACM International Conference on Web Search and Data Mining. New York, USA: ACM, 2011: 625-634.
15 ZHU Guangyu, YAN Shuicheng, MA Yi. Image tag refinement towards low-rank, content-tag prior and error sparsity[C]// Proceedings of the ACM Multimedia Conference. New York, USA: ACM Press, 2010: 461-470.
16 LIU Yang , WEN Kaiwen , GAO Quanxue , et al. SVM based multi-label learning with missing labels for image annotation[J]. Pattern Recognition, 2018, 78 (2): 307- 317.
17 LI Xirong, LIAO Shuai, LAN Weiyu, et al. Zero-shot image tagging by hierarchical semantic embedding[C]// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM, 2016: 879-882.
18 XIA Hao, WU Pengcheng, STEVEN C H H. Online multi-modal distance learning for scalable multimedia retrieval[C]//Proceedings of the Sixth ACM International Conference on Web Search and Data Mining. New York, USA: ACM, 2016: 455-464.
19 XIE Liang, SHEN Jialie, ZHU Lei. Online cross-modal hashing for web image retrieval[C]//Proceedings of thirtieth AAAI Conference on Artificial Intelligence. New York, USA: AAAI Press, 2016: 294-300.
20 DONG Jianfeng, LI Xirong, LIAO Shuai, et al. Image retrieval by cross-media relevance fusion[C]// Proceedings of the 23rd ACM international conference on Multimedia. New York, USA: ACM, 2016: 173-176.
21 TIAN Feng , SHEN Xukun , LIU Xianmei , et al. Image tagging by semantic neighbor learning using user-contributed social image datasets[J]. Tsinghua Science and Technology, 2017, 22 (6): 551- 563.
doi: 10.23919/TST.2017.8195340
22 TIAN Feng , SHEN Xukun , LIU Xianmei . Multimedia automatic annotation by mining label set correlation[J]. Multimedia Tools and Applications, 2018, 77 (3): 3473- 3491.
doi: 10.1007/s11042-017-5170-3
23 MARK J H, MICHAEL S L. The MIR flickr retrieval evaluation[C]// Proceedings of the ACM Intenational Conference on Multimedia Information Retrieval. New York, USA: ACM, 2008: 39-43.
24 CHUA T S, TANG Jinhui, HONG Richang. NUS-WIDE: a real-world web image database from national university of singapore[C]// Proceedings of the ACM Conference on Image and Video Retrieval. New York, USA: ACM, 2009: 1-9
25 LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the European Conference on Computer Vision (ECCV). Zurich, Switzerland: Springer Cham, 2014: 740-755.
[1] WANG Xi-zhao,BAI Li-jie*,HUA Qiang, LIU Yu-chao. Locally linear discriminant embedding with nonparametric method [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2011, 41(4): 1-6.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] SHI Lai-shun,WAN Zhong-yi . Synthesis and performance evaluation of a novel betaine-type asphalt emulsifier[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 112 -115 .
[2] YU Jia yuan1, TIAN Jin ting1, ZHU Qiang zhong2. Computational intelligence and its application in psychology[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(1): 1 -5 .
[3] HU Tian-liang,LI Peng,ZHANG Cheng-rui,ZUO Yi . Design of a QEP decode counter based on VHDL[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(3): 10 -13 .
[4] JI Tao,GAO Xu/sup>,SUN Tong-jing,XUE Yong-duan/sup>,XU Bing-yin/sup> . Characteristic analysis of fault generated traveling waves in 10 Kv automatic blocking and continuous power transmission lines[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(2): 111 -116 .
[5] SUN Cong-zheng,GUAN Cong-sheng,QIN Jing-yu,CHENG Chuan . The structure and performances of the electroless Ni-P alloy coating on aluminum alloy[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(5): 108 -112 .
[6] XU Li-li,JI Zhong,XIA Ji-mei . The optimum algorithm for the container loading problem with homogeneous cargoes[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(3): 14 -17 .
[7] PAN Duo-tao,LIU Gui-ping,LIU Chang-feng . Screening of microbe producing flocculant and optimizationon its cultural conditions[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(3): 99 -103 .
[8] DIAO Yong, TIAN Si-Meng, CAO Zhe-Meng. Geological work method for the construction of the Yichang Wanzhou Railway tunnel in high risk karst areas[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 91 -95 .
[9] DENG Bin,WANG Jiang . Estimating parameters of a neuron model based on chaos synchronization and adaptive control[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(5): 19 -23 .
[10] GAO Hou-Lei, TIAN Jia, DU Jiang, WU Zhi-Gang, LIU Chu-Min. Distributed generation—new technology in energy development[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 106 -110 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd