基于非对称U型金字塔重建的轻量级人脸超分辨率网络

doi:10.6040/j.issn.1672-3961.0.2021.312

摘要/Abstract

摘要：

为解决深度卷积神经网络在人脸超分辨率任务中模型复杂并难以实际应用的问题, 提出一种轻量级人脸超分辨率网络。利用残差编码块构成的编码结构进行特征提取, 在解码结构中引入金字塔重建从而实现快速准确的超分辨率。为降低解码块中上采样操作的参数量, 采用基于分辨率选择的非一致通道扩宽策略。为避免增加分支, 通过热图损失引入人脸先验知识。试验结果表明, 本研究提出的模型轻量有效地实现了超低分辨率人脸图像的超分辨重建, 以较低的模型复杂度, 重建出视觉质量优于其他先进方法的超分辨率人脸图像。

关键词: 深度学习, 人脸超分辨率, 非对称编解码, 金字塔重建, 热图损失, 生成对抗网络

Abstract:

A lightweight face super-resolution network was proposed in order to solve the problem that the model of deep convolutional neural network was complicated and difficult to be applied in the face super-resolution task. The coder composed of rescoding blocks was used for feature extraction, and pyramid reconstruction was introduced into the decoder to achieve fast and accurate super-resolution. To reduce the parameter number of the up-sampling operation in the decoding block, a non-uniform channel widening strategy based on resolution selection was adopted. To avoid adding extra branches, the prior knowledge of the face was introduced through heatmap loss. Experimental results showed that the model proposed in this paper could achieve light and accurate super-resolution reconstruction of ultra-low resolution face images that achieved better visual quality than the state-of-the-art method with lower model complexity.

Key words: deep learning, face super-resolution, asymmetric encoder-decoder, pyramid reconstruction, heatmap loss, generative adversarial networks

中图分类号:

TP391

蒋桐雨, 陈帆, 和红杰. 基于非对称U型金字塔重建的轻量级人脸超分辨率网络[J]. 山东大学学报 (工学版), 2022, 52(1): 1-8.

Tongyu JIANG, Fan CHEN, Hongjie HE. Lightweight face super-resolution network based on asymmetric U-pyramid reconstruction[J]. Journal of Shandong University(Engineering Science), 2022, 52(1): 1-8.

图/表 10

图1

图2

图3

图4

表1

图5

图6

表2

图7

图8

参考文献 22

1	BAI Yancheng, ZHANG Yongqiang, DING Mingli, et al. Finding tiny faces in the wild with generative adversarial network[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 21-30.
2	BULAT A, TZIMIROPOULOS G. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230, 000 3d facial landmarks)[C]//Proceeding of the IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 1021-1030.
3	DONG Chao, LOY C C, HE Kaiming, et al. Learning a deep convolutional network for image super-resolution[C]//Proceeding of the European Conference on Computer Vision (ECCV). Zurich, Switzerland: Springer, 2014: 184-199.
4	KIM J, KWON Lee J, MU Lee K. Accurate image super-resolution using very deep convolutional networks[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 1646-1654.
5	LAI W S, HUANG J B, AHUJA N, et al. Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE, 2017: 624-632.
6	CHEN Yu, TAI Ying, LIU Xiaoming, et al. FSRNet: end-to-end learning face super-resolution with facial priors[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 2492-2501.
7	MA Cheng, JIANG Zhenyu, RAO Yongming, et al. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation[C]//Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [S. l. ]: IEEE, 2020: 5569-5578.
8	ZHANG Yunchen, WU Yi, CHEN Liang. MSFSR: a multi-stage face super-resolution with accurate facial representation via enhanced facial boundaries[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). [S. l. ]: 2020: 504-505.
9	BULAT A, TZIMIROPOULOS G. Super-FAN: inte-grated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 109-117.
10	KIM D, KIM M, KWON G, et al. Progressive face super-resolution via attention to facial landmark[C]//Proceeding of the British Machine Vision Conference (BMVC). Cardiff, UK: 2019.
11	RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmen-tation[C]//Proceeding of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Munich, Germany: Springer, 2015: 234-241.
12	LIU Ziwei, LUO Ping, WANG Xiaogang, et al. Deep learning face attributes in the wild[C]//Proceeding of the IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015: 3730-3738.
13	LE V, BRANDT J, LIN Z, et al. Interactive facial feature localization[C]//Proceeding of the European Conference on Computer Vision (ECCV). Florence, Italy: Springer, 2012: 679-692.
14	GUO Yong, CHEN Jian, WANG Jingdong, et al. Closed-loop matters: dual regression networks for single image super-resolution[C]//Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [S. l. ]: IEEE, 2020: 5407-5416.
15	SHI Wenzhe, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 1874-1883.
16	YU Xin, PORIKLI F. Ultra-resolving face images by discriminative generative networks[C]//Proceeding of the European Conference on Computer Vision (ECCV). Amsterdam, The Netherlands: Springer, 2016: 318-333.
17	CHEN Chaofeng , GONG Dihong , WANG Hao , et al. Learning spatial attention for face super-resolution[J]. IEEE Transactions on Image Processing, 2020, 30, 1219- 1231.
18	WU Xiang , HE Ran , SUN Zhenan , et al. A light cnn for deep face representation with noisy labels[J]. IEEE Transactions on Information Forensics and Security, 2018, 13 (11): 2884- 2896.
19	WANG Zhou , BOVIK A C , SHEIKH H R , et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612.
20	ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 586-595.
21	ZHANG Yulun, TIAN Yapeng, KONG Yu, et al. Residual dense network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE, 2018: 2472-2481.
22	LI Zhen, YANG Jinglei, LIU Zheng, et al. Feedback network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE, 2019: 3867-3876.

相关文章 15

[1]	李常刚,李宝亮,曹永吉,王佳颖. 人工智能在电力系统潮流计算中的应用综述及展望[J]. 山东大学学报 (工学版), 2025, 55(5): 1-17.
[2]	周群颖,隋家成,张继,王洪元. 基于自监督卷积和无参数注意力机制的工业品表面缺陷检测[J]. 山东大学学报 (工学版), 2025, 55(4): 40-47.
[3]	杨巨成,路开奎,王嫄. 基于生成对抗网络的知识蒸馏研究综述[J]. 山东大学学报 (工学版), 2025, 55(4): 56-71.
[4]	薛冰冰,王勇,杨维浩,王川,于迪,王旭. 基于ETC收费数据的高速公路交通流数据修复及实时预测[J]. 山东大学学报 (工学版), 2025, 55(3): 58-71.
[5]	董明书,陈俐企,马川义,张珠皓,孙仁娟,管延华,庄培芝. 沥青路面内部裂缝雷达图像智能判识算法研究[J]. 山东大学学报 (工学版), 2025, 55(3): 72-79.
[6]	贾轩,许吉凯,任艺婧,刘德才,许强,张利. 基于样本扩容和数据驱动的台区理论线损计算方法[J]. 山东大学学报 (工学版), 2025, 55(3): 158-164.
[7]	常新功,苏敏惠,周志刚. 基于进化集成的图神经网络解释方法[J]. 山东大学学报 (工学版), 2024, 54(4): 1-12.
[8]	索大翔,李波. 基于Gromov-Wasserstein最优传输的输电线路小目标检测方法[J]. 山东大学学报 (工学版), 2024, 54(3): 22-29.
[9]	宋辉,张轶哲,张功萱,孟元. 基于类权重和最小化预测熵的测试时集成方法[J]. 山东大学学报 (工学版), 2024, 54(3): 36-43.
[10]	刘新,刘冬兰,付婷,王勇,常英贤,姚洪磊,罗昕,王睿,张昊. 基于联邦学习的时间序列预测算法[J]. 山东大学学报 (工学版), 2024, 54(3): 55-63.
[11]	聂秀山,巩蕊,董飞,郭杰,马玉玲. 短视频场景分类方法综述[J]. 山东大学学报 (工学版), 2024, 54(3): 1-11.
[12]	李璐,张志军,范钰敏,王星,袁卫华. 面向冷启动用户的元学习与图转移学习序列推荐[J]. 山东大学学报 (工学版), 2024, 54(2): 69-79.
[13]	高泽文,王建,魏本征. 基于混合偏移轴向自注意力机制的脑胶质瘤分割算法[J]. 山东大学学报 (工学版), 2024, 54(2): 80-89.
[14]	陈成,董永权,贾瑞,刘源. 基于交互序列特征相关性的可解释知识追踪[J]. 山东大学学报 (工学版), 2024, 54(1): 100-108.
[15]	李家春,李博文,常建波. 一种高效且轻量的RGB单帧人脸反欺诈模型[J]. 山东大学学报 (工学版), 2023, 53(6): 1-7.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed

方法	参数量/10⁶	浮点运算量/GFLOPs	PSNR/dB	SSIM	LPIPS	NRMSE
Bicubic	—	—	23.66	0.637 9	0.557 0	0.113 3
RDN^[21]	22.4	6.45	26.79	0.775 0	0.219 5	0.048 6
SRFBN^[22]	7.9	279.91	26.92	0.779 1	0.185 4	0.047 9
FSRNet^[6]	—	—	26.06	0.763 3	0.201 3	0.048 7
FSRGAN^[6]	—	—	24.70	0.716 8	0.135 3	0.045 2
PFSR^[10]	9.0	3.64	24.68	0.687 4	0.105 1	0.046 1
DIC^[7]	21.8	14.76	27.15	0.789 6	0.171 1	0.046 0
DICGAN^[7]	21.8	14.76	26.05	0.744 3	0.085 5	0.043 8
AUP-FSRNet	4.2	2.81	26.96	0.784 3	0.172 2	0.044 7
AUP-FSRGAN	4.2	2.81	25.87	0.732 6	0.087 4	0.043 1

方法	参数量/10⁶	浮点运算量/GFLOPs	PSNR/dB	PSNR差值/dB	SSIM	SSIM差值
AUP-FSRNet	4.2	2.81	25.73	0	0.759 0	0
UP-FSRNet	13.0	23.83	26.03	0.30	0.768 1	0.009 1
P-FSRNet	4.2	7.60	25.92	0.19	0.766 3	0.007 3
AU-FSRNet	4.2	2.81	25.16	-0.57	0.739 0	-0.020 0
AUP-FSRNet(w/o conv1×1)	8.4	2.98	25.56	-0.17	0.750 4	-0.008 6