Journal of Shandong University(Engineering Science) ›› 2021, Vol. 51 ›› Issue (2): 19-25.doi: 10.6040/j.issn.1672-3961.0.2020.225

• Machine Learning & Data Mining • Previous Articles     Next Articles

Real-time semantic segmentation of high-resolution remote sensing image based on multi-level feature cascade

Chunhong CAO(),Hongxuan DUAN*(),Ling CAO,Lele ZHANG,Kai HU,Fen XIAO   

  1. The MOE Key Laboratory of Intelligent Computing & Information Processing (Xiangtan University), Xiangtan 411100, Hunan, China
  • Received:2020-06-17 Online:2021-04-20 Published:2021-04-16
  • Contact: Hongxuan DUAN E-mail:caoch@xtu.edu.cn;201821562036@smail.xtu.edu.cn

RichHTML

40

PDF (PC)

435

Abstract:

Aiming at the problems of long segmentation time and inaccurate segmentation of small targets in remote sensing image semantic segmentation, a fast semantic segmentation model of high-resolution remote sensing image based on multi-level feature cascade network (MFCNet) was proposed. The model was mainly composed of feature encoding, feature fusion and target refinement. Feature encoding extracted the input images feature of different resolutions and used different backbone networks. Due to the lower resolution of low-resolution images, heavy-weight backbone networks were used to obtain rich semantic information with fewer parameters. For medium and high-resolution images, lightweight backbone network was used to reduce the amount of parameters and obtain global information. While medium and low-resolution encoding used the way of weights and calculation sharing to further reduce model parameters and computational complexity. The feature fusion section fused features from different branches to obtain information at different scales. The target refinement used residual to correction the fused features and the features of the coded part to restore the spatial detail information of the image, making the segmentation more accurate. And the entire model worked efficiently in an end-to-end manner. The experimental verified the validity of the model in semantic segmentation of remote sensing images, and achieved a good balance between model complexity and accuracy.

Key words: remote sensing image, real-time semantic segmentation, multi-level feature fusion, feature extraction, end-to-end

CLC Number: 

  • TP18

Fig.1

Network model based on multi-level feature cascade"

Fig.2

Module of cascade feature fusion"

Fig.3

Scheme of residual correction"

Fig.4

Comparison results of different segmentation methods under the Vaihingen dataset"

Fig.5

Comparison results of different segmentation methods under the Potsdam dataset"

Table 1

Comparison results of MFCNet and other methods under Vaihingen dataset  %"

方法 F1 OA mIOU
不透水的表面 建筑物 低植被 树木 汽车 平均
UNet 85.14 89.24 73.21 82.12 71.84 80.31 83.82 67.88
ICNet 85.84 90.36 72.71 81.99 36.86 73.55 83.68 61.69
PSPNet 85.68 90.13 73.46 83.02 65.40 79.53 84.52 67.20
MFCNet 86.90 91.02 75.26 83.54 74.15 82.18 85.53 70.53

Table 2

Comparison results of MFCNet and other methods under Potsdam dataset  %"

方法 F1 OA mIOU
不透水的表面 建筑物 低植被 树木 汽车 平均
UNet 88.16 89.74 80.68 80.36 88.90 85.57 86.04 75.12
ICNet 88.49 90.10 78.22 77.73 79.75 82.86 85.19 71.20
PSPNet 90.04 93.59 82.08 80.89 88.00 86.92 87.89 77.25
MFCNet 89.84 92.98 81.37 79.76 90.82 86.91 87.47 77.39

Table 3

Comparison of complexity under Vaihingen dataset"

方法 分割图像的平均时间/s 参数量
UNet 26.38 7846822
ICNet 64.85 6743442
PSPNet 31.28 46692053
MFCNet 34.17 7795320

Table 4

Comparison of complexity under Potsdam dataset"

方法 分割图像的平均时间/s 参数量
UNet 194.55 7846822
ICNet 908.18 6743442
PSPNet 220.13 46692053
MFCNet 210.40 7795320

Table 5

Influence of different branch connection positions on experimental results  %"

分支 OA F1 mIOU
分支1 81.70 77.02 63.72
分支2 85.53 82.18 70.53
1 刘航, 汪西莉. 基于注意力机制的遥感图像分割模型[J]. 激光与光电子学进展, 2020, 57 (4): 170- 180.
LIU Hang , WANG Xili . Remote sensing image segmentation model based on attention mechanism[J]. Laser & Optoelectronics Progress, 2020, 57 (4): 170- 180.
2 LIU C , YUEN J , TORRALBA A . Nonparametric scene parsing via label transfer[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33 (12): 2368- 2382.
doi: 10.1109/TPAMI.2011.131
3 LONG J , SHELHAMER E , DARRELL T . Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2014, 39 (4): 640- 651.
4 BADRINARAYANAN V , KENDALL A , CIPOLLA R . Segnet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39 (12): 2481- 2495.
5 ZHAO H, SHI J, QI X, et al. Pyramid scene parsing network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE, 2017: 2881-2890.
6 CHEN L, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[C]// Proceedings of the International Conference on Learning Representations (ICLR). San Diego, USA: IEEE, 2015: 1-14.
7 CHEN L , PAPANDREOU G , KOKKINOS I , et al. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40 (4): 834- 848.
doi: 10.1109/TPAMI.2017.2699184
8 CHEN L, ZHU Y, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European Conference on Computer Vision (ECCV). Munich, Germany: Springer, 2018: 801-818.
9 NOH H, HONG S, HAN B. Learning deconvolution network for semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015: 1520-1528.
10 MEHTA S, RASTEGARI M, SHAPIRO L, et al. Espnetv2: a light-weight, power efficient, and general purpose convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recogintion (CVPR). California, USA: IEEE, 2019: 9190-9200.
11 WANG Y, ZHOU Q, LIU J, et al. Lednet: a lightweight encoder-decoder network for real-time semantic segmentation[C]//Proceedings of the IEEE International Conference on Image Processing (ICIP). Taipei, China: IEEE, 2019: 1860-1864.
12 YU C, WANG J, PENG C, et al. Bisenet: bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of the European Conference on Computer Vision (ECCV). Munich, Germany: Springer, 2018: 334-349.
13 LI H, XIONG P, FAN H, et al. Dfanet: deep feature aggregation for real-time semantic segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recogintion (CVPR). California, USA: IEEE, 2019: 9522-9531.
14 HE T, SHEN C, TIAN Z, et al. Knowledge adaptation for efficient semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). California, USA: IEEE, 2019: 578-587.
15 ZHAO H, QI X, SHEN X, et al. Icnet for real-time semantic segmentation on high-resolution images[C]// Proceedings of the European Conference on Computer Vision (ECCV). Munich, Germany: Springer, 2018: 405-420.
16 ZHUANG J, YANG J, GU L, et al. Shelfnet for fast semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision Workshop (ICCVW). Seoul, Korea: IEEE, 2019: 847-856.
17 MARIUS C, MOHAMED O, SEBASTIAN R, et al. The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recogintion (CVPR). Las Vegas, USA: IEEE, 2016: 3213-3223.
18 LIN T, MICHAEL M, SERGE B, James Hays, et al. Microsoft coco: common objects in context[C]// Proceedings of the European Conference on Computer Vision (ECCV). Zurich, Switzerland: Springer, 2014: 740-755.
19 OLAF R, PHILIPP F, THOMAS B. U-net: convolutional networks for biomedical image segmentation[C]// Proceedings of the Medical Image Computing and Computer-Assisted Intervention (MICCA). Munich, Germany: Springer, 2015: 234-241.
20 HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 770-778.
[1] Donglan LIU,Xin LIU,Jianfei CHEN,Wenting WANG,Hao ZHANG,Lei MA,Dong LI. End-to-end security encryption scheme of NB-IoT for smart grid based on physical unclonable function [J]. Journal of Shandong University(Engineering Science), 2020, 50(1): 63-71.
[2] Jun FAN,Qiaolin YE,Ning YE. Face recognition based on improved prameter-free supervised localitypreserving projections [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 10-16.
[3] Guoxin WANG,Fengdong CHEN,Guodong LIU. Feature extraction method of color pseudo-random coded structured light [J]. Journal of Shandong University(Engineering Science), 2018, 48(5): 55-60.
[4] YE Ziyun, YANG Jinfeng. A finger-vein recognition method based on weighted graph model [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(3): 103-109.
[5] ZHANG Zhenyue, LI Fei, JIANG Mingyan. Unsupervised face image feature extraction based on low-rank representation projection [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2018, 48(1): 15-20.
[6] YU Qingmin, LI Xiaolei, ZHAI Yong. Feature extraction method of rolling bearing inner ring in wind turbine based on improved EMD and feature box [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(3): 89-95.
[7] ZHANG Yuling, YIN Chuanhuan. Android malware detection based on SVM [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(1): 42-47.
[8] GUO Chao, YANG Yan, JIANG Yongquan, SONG Yi. Condition recognition of high-speed train based on multi-view classification ensemble [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(1): 7-14.
[9] ZHONG Zhiyan, WEN Zhiqiang, ZHANG Xiaoyun, YE Degang. Neighborhood similarity descriptor used in halftone image [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(3): 58-64.
[10] XIAO Qiao, PEI Jihong, WANG Lixia, GONG Zhicheng. Ship detection in remote sensing image based on the fuzzy fusion of multi-channel Gabor filtering [J]. Journal of Shandong University(Engineering Science), 2015, 45(5): 29-35.
[11] LU Dan, ZHOU Yiqi. Vibration analysis of excavator seat based on EEMD and CWT [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2015, 45(3): 58-64.
[12] ZHOU Zhe, SHANG Lin. A sentiment analysis method based on dynamic lexicon and three-way decision [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2015, 45(1): 19-23.
[13] YU Hai-jing1,2, LI Gui-ju1*. Color smoke image recognition based on differential box-counting fractal dimension algorithm [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2014, 44(1): 35-40.
[14] LI Hui1,2, HU Yun1,3, LI Cun-hua1. The technique of gas disaster information feature extraction based on rough set theory [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2012, 42(5): 91-95.
[15] CAO Hong-gen1, YUAN Bao-hua1, ZHU Hui-sheng2. Recognition of intersected face based on contrast information and  local binary pattern [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2012, 42(4): 29-34.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] LONG Zhi-Jian, ZHANG Chang-Qiao. Synthesis and properties of associating DRA by binary copolymerization based on lauryl methacrylate[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(5): 128 -132 .
[2] LI Shan-ping,HU Zhen,SUN Yi-ming*,ZHEN Bo-ru,ZHANG Qi-lei,CAO Han-lin . Preparation and evaluation of the electro-catalytic characteristics of novel lead Ti-based dioxide electrodes[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(3): 109 -113 .
[3] LUO Yun-hu,WU Xu-wen,PAN Shuang-lai,DONG Er-ling,SUN Xiu-juan,WANG Chuan-jiang,WU Na . Coordination of two kinds of interruptible loads of demand side and reserve capacity of generation side[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(6): 66 -70 .
[4] ZHAI Xin-Xian, CHEN Dong-Hai, GUO Nian-Bo, GOU Pan-Feng. The laws of strata behavior for the gob-side tailgate in the  Jining No.3 Coal Mine[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(4): 92 -96 .
[5] LIU Dian-rui,ZHAO Hui-hong,ZHONG Mai-ying . The H∞ fault estimation for linear discrete time-varying systems[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 11 -16 .
[6] QIAO Xiaoyan. Automatic recognition method of microscopic image of harmful algae[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(3): 1 -6 .
[7]

L〖AKU¨] Guo-ren,YAN Shu-ming,BAI Shu-feng,JIA Ning,MA Liang

. Impact property research on a new type of guardrail terminal for freeway[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 47 -52 .
[8] WANG Huiqing, SUN Hongwei, ZHANG Jianhui. Time series similarity searching algorithm based on Map/Reduce[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(1): 15 -21 .
[9] ZHANG Qin-song, LI Shu-cai, HAN Hong-wei, GE Yan-hui, LIU Ren-tai, ZHANG Xiao. Study on risk evaluation and water inrush disaster preventingtechnology during construction of karst tunnels[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(3): 106 -110 .
[10] LIU Yong,GU Naijie,REN Kaixin,LIU Gang, . Novel omegabased selfrouting multicast networks[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(4): 37 -43 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd