Journal of Shandong University(Engineering Science) ›› 2022, Vol. 52 ›› Issue (1): 1-8.doi: 10.6040/j.issn.1672-3961.0.2021.312

• Machine Learning & Data Mining •     Next Articles

Lightweight face super-resolution network based on asymmetric U-pyramid reconstruction

Tongyu JIANG1(),Fan CHEN2,Hongjie HE1,*()   

  1. 1. Sichuan Key Laboratory of Signal and Information Processing, Southwest Jiaotong University, Chengdu 611756, Sichuan, China
    2. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, Sichuan, China
  • Received:2021-06-28 Online:2022-02-21 Published:2022-02-21
  • Contact: Hongjie HE E-mail:jty971018@msn.cn;hjhe@swjtu.edu.cn

RichHTML

18

PDF (PC)

323

Abstract:

A lightweight face super-resolution network was proposed in order to solve the problem that the model of deep convolutional neural network was complicated and difficult to be applied in the face super-resolution task. The coder composed of rescoding blocks was used for feature extraction, and pyramid reconstruction was introduced into the decoder to achieve fast and accurate super-resolution. To reduce the parameter number of the up-sampling operation in the decoding block, a non-uniform channel widening strategy based on resolution selection was adopted. To avoid adding extra branches, the prior knowledge of the face was introduced through heatmap loss. Experimental results showed that the model proposed in this paper could achieve light and accurate super-resolution reconstruction of ultra-low resolution face images that achieved better visual quality than the state-of-the-art method with lower model complexity.

Key words: deep learning, face super-resolution, asymmetric encoder-decoder, pyramid reconstruction, heatmap loss, generative adversarial networks

CLC Number: 

  • TP391

Fig.1

Network architectures of the AUP-FSRNet"

Fig.2

The structure of basic block"

Fig.3

Feature maps and images with different resolution levels"

Fig.4

The structure of discriminator"

Table 1

The objective evaluation results"

方法 参数量/106 浮点运算量/GFLOPs PSNR/dB SSIM LPIPS NRMSE
Bicubic — — 23.66 0.637 9 0.557 0 0.113 3
RDN[21] 22.4 6.45 26.79 0.775 0 0.219 5 0.048 6
SRFBN[22] 7.9 279.91 26.92 0.779 1 0.185 4 0.047 9
FSRNet[6] — — 26.06 0.763 3 0.201 3 0.048 7
FSRGAN[6] — — 24.70 0.716 8 0.135 3 0.045 2
PFSR[10] 9.0 3.64 24.68 0.687 4 0.105 1 0.046 1
DIC[7] 21.8 14.76 27.15 0.789 6 0.171 1 0.046 0
DICGAN[7] 21.8 14.76 26.05 0.744 3 0.085 5 0.043 8
AUP-FSRNet 4.2 2.81 26.96 0.784 3 0.172 2 0.044 7
AUP-FSRGAN 4.2 2.81 25.87 0.732 6 0.087 4 0.043 1

Fig.5

SR face images reconstructed by different methods"

Fig.6

Models used for ablation experiments"

Table 2

Objective evaluation of ablation experiment"

方法 参数量/106 浮点运算量/GFLOPs PSNR/dB PSNR差值/dB SSIM SSIM差值
AUP-FSRNet 4.2 2.81 25.73 0 0.759 0 0
UP-FSRNet 13.0 23.83 26.03 0.30 0.768 1 0.009 1
P-FSRNet 4.2 7.60 25.92 0.19 0.766 3 0.007 3
AU-FSRNet 4.2 2.81 25.16 -0.57 0.739 0 -0.020 0
AUP-FSRNet(w/o conv1×1) 8.4 2.98 25.56 -0.17 0.750 4 -0.008 6

Fig.7

Subjective evaluation of ablation experiment"

Fig.8

Effect of λheatmap on the performance of AUP-FSRNet"

1 BAI Yancheng, ZHANG Yongqiang, DING Mingli, et al. Finding tiny faces in the wild with generative adversarial network[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 21-30.
2 BULAT A, TZIMIROPOULOS G. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230, 000 3d facial landmarks)[C]//Proceeding of the IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 1021-1030.
3 DONG Chao, LOY C C, HE Kaiming, et al. Learning a deep convolutional network for image super-resolution[C]//Proceeding of the European Conference on Computer Vision (ECCV). Zurich, Switzerland: Springer, 2014: 184-199.
4 KIM J, KWON Lee J, MU Lee K. Accurate image super-resolution using very deep convolutional networks[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 1646-1654.
5 LAI W S, HUANG J B, AHUJA N, et al. Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE, 2017: 624-632.
6 CHEN Yu, TAI Ying, LIU Xiaoming, et al. FSRNet: end-to-end learning face super-resolution with facial priors[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 2492-2501.
7 MA Cheng, JIANG Zhenyu, RAO Yongming, et al. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation[C]//Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [S. l. ]: IEEE, 2020: 5569-5578.
8 ZHANG Yunchen, WU Yi, CHEN Liang. MSFSR: a multi-stage face super-resolution with accurate facial representation via enhanced facial boundaries[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). [S. l. ]: 2020: 504-505.
9 BULAT A, TZIMIROPOULOS G. Super-FAN: inte-grated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 109-117.
10 KIM D, KIM M, KWON G, et al. Progressive face super-resolution via attention to facial landmark[C]//Proceeding of the British Machine Vision Conference (BMVC). Cardiff, UK: 2019.
11 RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmen-tation[C]//Proceeding of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Munich, Germany: Springer, 2015: 234-241.
12 LIU Ziwei, LUO Ping, WANG Xiaogang, et al. Deep learning face attributes in the wild[C]//Proceeding of the IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015: 3730-3738.
13 LE V, BRANDT J, LIN Z, et al. Interactive facial feature localization[C]//Proceeding of the European Conference on Computer Vision (ECCV). Florence, Italy: Springer, 2012: 679-692.
14 GUO Yong, CHEN Jian, WANG Jingdong, et al. Closed-loop matters: dual regression networks for single image super-resolution[C]//Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [S. l. ]: IEEE, 2020: 5407-5416.
15 SHI Wenzhe, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 1874-1883.
16 YU Xin, PORIKLI F. Ultra-resolving face images by discriminative generative networks[C]//Proceeding of the European Conference on Computer Vision (ECCV). Amsterdam, The Netherlands: Springer, 2016: 318-333.
17 CHEN Chaofeng , GONG Dihong , WANG Hao , et al. Learning spatial attention for face super-resolution[J]. IEEE Transactions on Image Processing, 2020, 30, 1219- 1231.
18 WU Xiang , HE Ran , SUN Zhenan , et al. A light cnn for deep face representation with noisy labels[J]. IEEE Transactions on Information Forensics and Security, 2018, 13 (11): 2884- 2896.
19 WANG Zhou , BOVIK A C , SHEIKH H R , et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612.
20 ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 586-595.
21 ZHANG Yulun, TIAN Yapeng, KONG Yu, et al. Residual dense network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 2472-2481.
22 LI Zhen, YANG Jinglei, LIU Zheng, et al. Feedback network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE, 2019: 3867-3876.
[1] Yue YUAN,Yanli WANG,Kan LIU. Named entity recognition model based on dilated convolutional block architecture [J]. Journal of Shandong University(Engineering Science), 2022, 52(6): 105-114.
[2] Jianqing WU,Xiuguang SONG. Review on development of simultaneous localization and mapping technology [J]. Journal of Shandong University(Engineering Science), 2021, 51(5): 16-31.
[3] YANG Xiuyuan, PENG Tao, YANG Liang, LIN Hongfei. Adaptive multi-domain sentiment analysis based on knowledge distillation [J]. Journal of Shandong University(Engineering Science), 2021, 51(3): 15-21.
[4] Qingfa CHAI,Shoujing SUN,Jifu QIU,Ming CHEN,Zhen WEI,Wei CONG. Prediction method of power grid emergency supplies under meteorological disasters [J]. Journal of Shandong University(Engineering Science), 2021, 51(3): 76-83.
[5] LIAO Jinping, MO Yuchang, YAN Ke. Model and application of short-term electricity consumption forecast based on C-LSTM [J]. Journal of Shandong University(Engineering Science), 2021, 51(2): 90-97.
[6] Guoyong CAI,Xinhao HE,Yangyang CHU. Visual sentiment analysis based on spatial attention mechanism and convolutional neural network [J]. Journal of Shandong University(Engineering Science), 2020, 50(4): 8-13.
[7] LIU Shuai, WANG Lei, DING Xutao. Emotional EEG recognition based on Bi-LSTM [J]. Journal of Shandong University(Engineering Science), 2020, 50(4): 35-39.
[8] Chunyang LI,Nan LI,Tao FENG,Zhuhe WANG,Jingkai MA. Abnormal sound detection of washing machines based on deep learning [J]. Journal of Shandong University(Engineering Science), 2020, 50(2): 108-117.
[9] Delei CHEN,Cheng WANG,Jianwei CHEN,Yiyin WU. GRU-based collaborative filtering recommendation algorithm with active learning [J]. Journal of Shandong University(Engineering Science), 2020, 50(1): 21-27,48.
[10] Zhifu CHANG,Fengyu ZHOU,Yugang WANG,Dongdong SHEN,Yang ZHAO. A survey of image captioning methods based on deep learning [J]. Journal of Shandong University(Engineering Science), 2019, 49(6): 25-35.
[11] Peng WAN. Object detection of 3D point clouds based on F-PointNet [J]. Journal of Shandong University(Engineering Science), 2019, 49(5): 98-104.
[12] Ji ZHANG,Cui JIN,Hongyuan WANG,Shoubing CHEN. Pedestrian recognition based on singular value decomposition pedestrian alignment network [J]. Journal of Shandong University(Engineering Science), 2019, 49(5): 91-97.
[13] Zhixiang LIANG,Xiaoming LIU,Ying MU,Yutian LIU. Prediction method of wind power and PV ramp event based on deep learning [J]. Journal of Shandong University(Engineering Science), 2019, 49(5): 24-28.
[14] Yutian LIU, Runjia SUN, Hongtao WANG, Xueping GU. Review on application of artificial intelligence in power system restoration [J]. Journal of Shandong University(Engineering Science), 2019, 49(5): 1-8.
[15] Lizhao LI,Guoyong CAI,Jiao PAN. A microblog rumor events detection method based on C-GRU [J]. Journal of Shandong University(Engineering Science), 2019, 49(2): 102-106, 115.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] ZHANG Ying,LANG Yongmei,ZHAO Yuxiao,ZHANG Jianda,QIAO Peng,LI Shanping . Research on technique of aerobic granular sludge cultivationby seeding EGSB anaerobic granular sludge[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(4): 56 -59 .
[2] LIU Xin 1, SONG Sili 1, WANG Xinhong 2. [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(2): 98 -100 .
[3] HUANG Le-jian,WANG Jian-ming . Dynamic analysis of the stabilized smoothing nodal integration meshfree method[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(5): 68 -72 .
[4] WU Hao,TIAN Guo-hui,HUANG Bin .

Research on the collaboration strategy of multi-robot for exploring unknown environment

[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 27 -31 .
[5] FANG Ting,YANG Zhong,SHEN Chun-Lin . Multiple targets accurate tracking on UAV formation video sequences[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(4): 22 -26 .
[6] LI Meng-li, WANG Wei-qiang ，XU Shu-gen ， SONG Ming-da. Possibility analysis on chemical explosion of material causing urea  reactor cylinder fracture[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(6): 1 -6 .
[7] NIU Xiu-ming,FU Chun-hua . The effect of carbon on organic wastewater degradation in the process of pulse discharge[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2008, 38(1): 121 -126 .
[8] ZHANG Ming-Liang, LI Fan-Chang. A new search method for a game tree[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2009, 39(6): 1 -7 .
[9] RUAN Jiu-Hong, LI Yi-Bin, YANG Fu-An, RONG Hua-Wen. On manned AWID-AWIS vehicle dynamics control[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2010, 40(1): 10 -14 .
[10] TANG Qingshun, JIN Lu, LI Guodong, WU Chunfu. Robotic manipulators tracking control based on adaptive terminal sliding mode controller[J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(5): 45 -53 .
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd