您的位置:山东大学 -> 科技期刊社 -> 《山东大学学报(工学版)》

山东大学学报 (工学版) ›› 2022, Vol. 52 ›› Issue (1): 1-8.doi: 10.6040/j.issn.1672-3961.0.2021.312

• 机器学习与数据挖掘 •    下一篇

基于非对称U型金字塔重建的轻量级人脸超分辨率网络

蒋桐雨1(),陈帆2,和红杰1,*()   

  1. 1. 西南交通大学信号与信息处理四川省重点实验室, 四川 成都 611756
    2. 西南交通大学计算机与人工智能学院, 四川 成都 611756
  • 收稿日期:2021-06-28 出版日期:2022-02-21 发布日期:2022-02-21
  • 通讯作者: 和红杰 E-mail:jty971018@msn.cn;hjhe@swjtu.edu.cn
  • 作者简介:蒋桐雨(1997—),男,辽宁抚顺人,硕士研究生,主要研究方向为数字图像处理和计算机视觉。E-mail: jty971018@msn.cn
  • 基金资助:
    国家自然科学基金资助项目(U1936113);国家自然科学基金资助项目(61872303)

Lightweight face super-resolution network based on asymmetric U-pyramid reconstruction

Tongyu JIANG1(),Fan CHEN2,Hongjie HE1,*()   

  1. 1. Sichuan Key Laboratory of Signal and Information Processing, Southwest Jiaotong University, Chengdu 611756, Sichuan, China
    2. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, Sichuan, China
  • Received:2021-06-28 Online:2022-02-21 Published:2022-02-21
  • Contact: Hongjie HE E-mail:jty971018@msn.cn;hjhe@swjtu.edu.cn

摘要:

为解决深度卷积神经网络在人脸超分辨率任务中模型复杂并难以实际应用的问题, 提出一种轻量级人脸超分辨率网络。利用残差编码块构成的编码结构进行特征提取, 在解码结构中引入金字塔重建从而实现快速准确的超分辨率。为降低解码块中上采样操作的参数量, 采用基于分辨率选择的非一致通道扩宽策略。为避免增加分支, 通过热图损失引入人脸先验知识。试验结果表明, 本研究提出的模型轻量有效地实现了超低分辨率人脸图像的超分辨重建, 以较低的模型复杂度, 重建出视觉质量优于其他先进方法的超分辨率人脸图像。

关键词: 深度学习, 人脸超分辨率, 非对称编解码, 金字塔重建, 热图损失, 生成对抗网络

Abstract:

A lightweight face super-resolution network was proposed in order to solve the problem that the model of deep convolutional neural network was complicated and difficult to be applied in the face super-resolution task. The coder composed of rescoding blocks was used for feature extraction, and pyramid reconstruction was introduced into the decoder to achieve fast and accurate super-resolution. To reduce the parameter number of the up-sampling operation in the decoding block, a non-uniform channel widening strategy based on resolution selection was adopted. To avoid adding extra branches, the prior knowledge of the face was introduced through heatmap loss. Experimental results showed that the model proposed in this paper could achieve light and accurate super-resolution reconstruction of ultra-low resolution face images that achieved better visual quality than the state-of-the-art method with lower model complexity.

Key words: deep learning, face super-resolution, asymmetric encoder-decoder, pyramid reconstruction, heatmap loss, generative adversarial networks

中图分类号: 

  • TP391

图1

AUP-FSRNet模型结构"

图2

基础块的结构"

图3

不同分辨率级的特征图和图像"

图4

判别器结构"

表1

客观评价结果"

方法 参数量/106 浮点运算量/GFLOPs PSNR/dB SSIM LPIPS NRMSE
Bicubic 23.66 0.637 9 0.557 0 0.113 3
RDN[21] 22.4 6.45 26.79 0.775 0 0.219 5 0.048 6
SRFBN[22] 7.9 279.91 26.92 0.779 1 0.185 4 0.047 9
FSRNet[6] 26.06 0.763 3 0.201 3 0.048 7
FSRGAN[6] 24.70 0.716 8 0.135 3 0.045 2
PFSR[10] 9.0 3.64 24.68 0.687 4 0.105 1 0.046 1
DIC[7] 21.8 14.76 27.15 0.789 6 0.171 1 0.046 0
DICGAN[7] 21.8 14.76 26.05 0.744 3 0.085 5 0.043 8
AUP-FSRNet 4.2 2.81 26.96 0.784 3 0.172 2 0.044 7
AUP-FSRGAN 4.2 2.81 25.87 0.732 6 0.087 4 0.043 1

图5

不同方法重建得到的超分辨率人脸图像"

图6

用于消融试验的模型"

表2

消融试验的客观评价"

方法 参数量/106 浮点运算量/GFLOPs PSNR/dB PSNR差值/dB SSIM SSIM差值
AUP-FSRNet 4.2 2.81 25.73 0 0.759 0 0
UP-FSRNet 13.0 23.83 26.03 0.30 0.768 1 0.009 1
P-FSRNet 4.2 7.60 25.92 0.19 0.766 3 0.007 3
AU-FSRNet 4.2 2.81 25.16 -0.57 0.739 0 -0.020 0
AUP-FSRNet(w/o conv1×1) 8.4 2.98 25.56 -0.17 0.750 4 -0.008 6

图7

消融试验主观评价"

图8

λheatmap对AUP-FSRNet重建质量的影响"

1 BAI Yancheng, ZHANG Yongqiang, DING Mingli, et al. Finding tiny faces in the wild with generative adversarial network[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 21-30.
2 BULAT A, TZIMIROPOULOS G. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230, 000 3d facial landmarks)[C]//Proceeding of the IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 1021-1030.
3 DONG Chao, LOY C C, HE Kaiming, et al. Learning a deep convolutional network for image super-resolution[C]//Proceeding of the European Conference on Computer Vision (ECCV). Zurich, Switzerland: Springer, 2014: 184-199.
4 KIM J, KWON Lee J, MU Lee K. Accurate image super-resolution using very deep convolutional networks[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 1646-1654.
5 LAI W S, HUANG J B, AHUJA N, et al. Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE, 2017: 624-632.
6 CHEN Yu, TAI Ying, LIU Xiaoming, et al. FSRNet: end-to-end learning face super-resolution with facial priors[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 2492-2501.
7 MA Cheng, JIANG Zhenyu, RAO Yongming, et al. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation[C]//Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [S. l. ]: IEEE, 2020: 5569-5578.
8 ZHANG Yunchen, WU Yi, CHEN Liang. MSFSR: a multi-stage face super-resolution with accurate facial representation via enhanced facial boundaries[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). [S. l. ]: 2020: 504-505.
9 BULAT A, TZIMIROPOULOS G. Super-FAN: inte-grated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 109-117.
10 KIM D, KIM M, KWON G, et al. Progressive face super-resolution via attention to facial landmark[C]//Proceeding of the British Machine Vision Conference (BMVC). Cardiff, UK: 2019.
11 RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmen-tation[C]//Proceeding of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Munich, Germany: Springer, 2015: 234-241.
12 LIU Ziwei, LUO Ping, WANG Xiaogang, et al. Deep learning face attributes in the wild[C]//Proceeding of the IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015: 3730-3738.
13 LE V, BRANDT J, LIN Z, et al. Interactive facial feature localization[C]//Proceeding of the European Conference on Computer Vision (ECCV). Florence, Italy: Springer, 2012: 679-692.
14 GUO Yong, CHEN Jian, WANG Jingdong, et al. Closed-loop matters: dual regression networks for single image super-resolution[C]//Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [S. l. ]: IEEE, 2020: 5407-5416.
15 SHI Wenzhe, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 1874-1883.
16 YU Xin, PORIKLI F. Ultra-resolving face images by discriminative generative networks[C]//Proceeding of the European Conference on Computer Vision (ECCV). Amsterdam, The Netherlands: Springer, 2016: 318-333.
17 CHEN Chaofeng , GONG Dihong , WANG Hao , et al. Learning spatial attention for face super-resolution[J]. IEEE Transactions on Image Processing, 2020, 30, 1219- 1231.
18 WU Xiang , HE Ran , SUN Zhenan , et al. A light cnn for deep face representation with noisy labels[J]. IEEE Transactions on Information Forensics and Security, 2018, 13 (11): 2884- 2896.
19 WANG Zhou , BOVIK A C , SHEIKH H R , et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612.
20 ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 586-595.
21 ZHANG Yulun, TIAN Yapeng, KONG Yu, et al. Residual dense network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 2472-2481.
22 LI Zhen, YANG Jinglei, LIU Zheng, et al. Feedback network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE, 2019: 3867-3876.
[1] 周晓昕,廖祝华,刘毅志,赵肄江,方艺洁. 融合历史与当前交通流量的信号控制方法[J]. 山东大学学报 (工学版), 2023, 53(4): 48-55.
[2] 于畅,伍星,邓秋菊. 基于深度学习的多视角螺钉缺失智能检测算法[J]. 山东大学学报 (工学版), 2023, 53(4): 104-112.
[3] 宋佳芮,陈艳平,王凯,黄瑞章,秦永彬. 基于Affix-Attention的命名实体识别语义补充方法[J]. 山东大学学报 (工学版), 2023, 53(2): 70-76.
[4] 袁钺,王艳丽,刘勘. 基于空洞卷积块架构的命名实体识别模型[J]. 山东大学学报 (工学版), 2022, 52(6): 105-114.
[5] 李旭涛,杨寒玉,卢业飞,张玮. 基于深度学习的遥感图像道路分割[J]. 山东大学学报 (工学版), 2022, 52(6): 139-145.
[6] 孟令灿,聂秀山,张雪. 基于遮挡目标去除的公交车拥挤度分类算法[J]. 山东大学学报 (工学版), 2022, 52(4): 83-88.
[7] 杨霄,袭肖明,李维翠,杨璐. 基于层次化双重注意力网络的乳腺多模态图像分类[J]. 山东大学学报 (工学版), 2022, 52(3): 34-41.
[8] 王心哲,邓棋文,王际潮,范剑超. 深度语义分割MRF模型的海洋筏式养殖信息提取[J]. 山东大学学报 (工学版), 2022, 52(2): 89-98.
[9] 吴建清,宋修广. 同步定位与建图技术发展综述[J]. 山东大学学报 (工学版), 2021, 51(5): 16-31.
[10] 杨修远,彭韬,杨亮,林鸿飞. 基于知识蒸馏的自适应多领域情感分析[J]. 山东大学学报 (工学版), 2021, 51(3): 15-21.
[11] 柴庆发,孙守晶,邱吉福,陈明,魏振,丛伟. 气象灾害条件下电网应急物资预测方法[J]. 山东大学学报 (工学版), 2021, 51(3): 76-83.
[12] 张俊三,程俏俏,万瑶,朱杰,张世栋. MIRGAN: 一种基于GAN的医学影像报告生成模型[J]. 山东大学学报 (工学版), 2021, 51(2): 9-18.
[13] 廖锦萍,莫毓昌,YAN Ke. 基于C-LSTM的短期用电预测模型和应用[J]. 山东大学学报 (工学版), 2021, 51(2): 90-97.
[14] 张月芳,邓红霞,呼春香,钱冠宇,李海芳. 融合残差块注意力机制和生成对抗网络的海马体分割[J]. 山东大学学报 (工学版), 2020, 50(6): 76-81.
[15] 刘帅,王磊,丁旭涛. 基于Bi-LSTM的脑电情绪识别[J]. 山东大学学报 (工学版), 2020, 50(4): 35-39.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] 张英,郎咏梅,赵玉晓,张鉴达,乔鹏,李善评 . 由EGSB厌氧颗粒污泥培养好氧颗粒污泥的工艺探讨[J]. 山东大学学报(工学版), 2006, 36(4): 56 -59 .
[2] 刘新1 ,宋思利1 ,王新洪2 . 石墨配比对钨极氩弧熔敷层TiC增强相含量及分布形态的影响[J]. 山东大学学报(工学版), 2009, 39(2): 98 -100 .
[3] 黄乐建,王建明 . 稳定相容节点积分无网格法动力学分析[J]. 山东大学学报(工学版), 2007, 37(5): 68 -72 .
[4] 吴 皓,田国会,黄 彬 . 未知环境探测的多机器人协作策略研究[J]. 山东大学学报(工学版), 2008, 38(4): 27 -31 .
[5] 方 挺,杨 忠,沈春林 . 无人机编队视频序列中的多目标精确跟踪[J]. 山东大学学报(工学版), 2008, 38(4): 22 -26 .
[6] 李梦丽 王威强 徐书根 宋明大 王功 苗光同. 物料化学爆炸引起尿塔塔体爆破可能性分析[J]. 山东大学学报(工学版), 2008, 38(6): 1 -6 .
[7] 牛秀明,傅春华 . 炭在脉冲放电过程中对污水中有机物的降解作用[J]. 山东大学学报(工学版), 2008, 38(1): 121 -126 .
[8] 张明亮 李凡长. 一种新的博弈树搜索方法[J]. 山东大学学报(工学版), 2009, 39(6): 1 -7 .
[9] 阮久宏, 李贻斌, 杨福广, 荣学文. 有人驾驶AWID-AWIS车辆动力学控制研究[J]. 山东大学学报(工学版), 2010, 40(1): 10 -14 .
[10] 唐庆顺,金璐,李国栋,吴春富. 基于自适应终端滑模控制器的机械手跟踪控制[J]. 山东大学学报(工学版), 2016, 46(5): 45 -53 .