Journal of Shandong University(Engineering Science) ›› 2025, Vol. 55 ›› Issue (4): 18-28.doi: 10.6040/j.issn.1672-3961.0.2024.047

• Special Issue for Deep Learning with Vision • Previous Articles    

Face image inpainting based on texture and structure interaction

ZHOU Zunfu1,2, ZHANG Qian3*, SHI Jiliang1,2, YUE Shiqin4   

  1. ZHOU Zunfu1, 2, ZHANG Qian3*, SHI Jiliang1, 2, YUE Shiqin4(1. College of Data Science and Information Engineering, Guizhou Minzu University, Guiyang 550025, Guizhou, China;
    2. Key Laboratory of Pattern Recognition and Intelligent System, Guizhou Minzu University, Guiyang 550025, Guizhou, China;
    3. Academic Affairs Office, Guizhou Minzu University, Guiyang 550025, Guizhou, China;
    4. College of Automotive Engineering, Wuhan University of Technology, Wuhan 430070, Hubei, China
  • Published:2025-08-31

PDF (PC)

36

Abstract: Aiming at the issue of losing contextual semantic information when extracting deep features by learning-based face image inpainting methods, a generator with an efficient normalized attention mechanism was proposed, which extracted deep features from face images more effectively and better aggregated low-level and high-level features at multiple scales. To enhance the consistency of the generated images, a bi-level gated feature fusion module with residual main path transformation was introduced. This module further fused decoded texture and structure information, and incorporated an enhanced contextual feature aggregation module, in which an improved prompt generation block enabled prompt parameters to interact between features at multiple scales, guiding the dynamic adjustment of the inpainting network to generate realistic and believable face images. Experimental results on the CelebA-HQ datasets showed that this research method achieved 37.74 dB, 0.983 0, 0.24%, and 1.489 in terms of peak signal-to-noise ratio (RPSN), structural similarity (SSIM), mean absolute error(EMA), and Fréchet inception distance (DFI). On the LFW dataset, the RPSN, SSIM, EMA, and DFI of this research method achieved 39.19 dB, 0.987 7, 0.21%, and 3.555. Compared with five other mainstream methods, this research method achieved quite competitive results. Qualitative and quantitative experiments demonstrated that this research method could effectively restore corrupted facial structure and texture information.

Key words: face image inpainting, normalized attention module, prompt generation module, multi-scale feature fusion, two-stream discriminator

CLC Number: 

  • TP391.41
[1] BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 1-11.
[2] EFROS A A, FREEMAN W T. Image quilting for texture synthesis and transfer[C] //Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. New York, USA:ACM, 2001: 341-346.
[3] YU J, LIN Z, YANG J, et al. Free-form image inpainting with gated convolution[C] //Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul: IEEE, 2019: 4470-4479.
[4] YAN Z, LI X, LI M, et al. Shift-Net: image inpainting via deep feature rearrangement[C] //Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer, 2018: 3-19.
[5] LIU H, JIANG B, XIAO Y, et al. Coherent semantic attention for image inpainting[C] //Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul: IEEE, 2019: 4170-4179.
[6] LI J, HE F, ZHANG L, et al. Progressive reconstruction of visual structure for image inpainting[C] //Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul: IEEE, 2019: 5961-5970.
[7] WANG N, ZHANG Y, ZHANG L. Dynamic selection network for image inpainting[J]. IEEE Transactions on Image Processing, 2021, 30: 1784-1798.
[8] ZENG Y, FU J, CHAO H,et al. Aggregated contextual transformations for high-resolution image inpainting[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(7): 3266-3280.
[9] HUI S, ZHOU S, DENG Y, et al. Auxiliary loss reweighting for image inpainting[EB/OL].(2022-04-22)[2024-04-20]. https://arxiv.org/abs/2111.07279
[10] MISRA D. Mish: a self regularized non-monotonic neural activation function[EB/OL].(2019-08-26)[2024-04-20]. https://arxiv.org/abs/1908.08681
[11] LIU Y, SHAO Z, TENG Y, et al. NAM: normali-zation-based attention module[EB/OL].(2021-11-24)[2024-04-20]. https://arxiv.org/abs/2111.12419
[12] KARRAS T, AILA T, LAINE S, et al. Progressive growing of GANs for improved quality, stability, and variation[EB/OL].(2018-02-26)[2024-04-20]. https://arxiv.org/abs/1710.10196
[13] HUANG G B, MATTAR M, BERG T, et al. Labeled faces in the wild: a database for studying face recognition in unconstrained environments[EB/OL].(2008-09-16)[2024-04-20]. https://inria.hal.science/inria-00321923v1/document
[14] BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[C] //Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New Orleans, USA: ACM, 2000: 417-424.
[15] TSCHUMPERLÉ D, DERICHE R. Vector-valued image regularization with PDEs: a common framework for different applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(4): 506-517.
[16] DARABI S, SHECHTMAN E, BARNES C, et al. Image melding: combining inconsistent images using patch-based synthesis[J]. ACM Transactions on Graphics, 2012, 31(4): 1-10.
[17] BUYSSENS P, DAISY M, TSCHUMPERLÉ D, et al. Exemplar-based inpainting: technical review and new heuristics for better geometric reconstructions[J]. IEEE Transactions on Image Processing, 2015, 24(6): 1809-1824.
[18] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C] //Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada: Curran Associates, 2014: 2672-2680.
[19] NAZERI K, NG E, JOSEPH T, et al. EdgeConnect: generative image inpainting with adversarial edge learning[EB/OL].(2019-01-11)[2024-04-20]. https://arxiv.org/abs/1901.00212
[20] XIONG W, YU J, LIN Z,et al. Foreground-aware image inpainting[C] //Proceedings of the 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019: 5833-5841.
[21] REN Y, YU X, ZHANG R, et al. StructureFlow: image inpainting via structure-aware appearance flow[C] // Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul: IEEE, 2019: 181-190.
[22] ZENG Y H, FU J L, CHAO H Y, et al. Learning pyramid-context encoder network for high-quality image inpainting[C] //Proceedings of the 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019: 1486-1494.
[23] YU J, LIN Z, YANG J, et al. Generative image inpainting with contextual attention[C] //Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018: 5505-5514.
[24] ZHENG C X, CHAM T J, CAI J F. Pluralistic image completion[C] //Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019: 1438-1447.
[25] QIN J, BAI H, ZHAO Y.Multi-scale attention network for image inpainting[J]. Computer Vision and Image Understanding, 2021, 204: 103155.
[26] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
[27] GUO X, YANG H, HUANG D. Image inpainting via conditional texture and structure dual generation[C] //Proceedings of the 18th IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 14114-14123.
[28] POTLAPALLI V, ZAMIR S W, KHAN S, et al. PromptIR: prompting for all-in-one blind image restoration[C] //Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, USA: Curran Associates, 2023: 71275-71293.
[29] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C] //Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer, 2015: 234-241.
[30] IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
[31] LI C, WAND M. Precomputed real-time texture synthesis with Markovian generative adversarial networks[C] //Proceedings of the 14th European Conference on Computer Vision. Amsterdam, Netherlands: Springer, 2016: 702-716.
[32] LIU G, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C] //Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer, 2018: 89-105.
[33] DENG Y, HUI S, MENG R, et al. Hourglass attention network for image inpainting[C] //Proceedings of the 17th European Conference on Computer Vision. Tel Aviv, Israel: Springer, 2022: 483-501.
[1] Ye LIANG,Nan MA,Hongzhe LIU. Image-dependent fusion method for saliency maps [J]. Journal of Shandong University(Engineering Science), 2021, 51(4): 1-7.
[2] Baocheng LIU,Yan PIAO,Xuemei SONG. Adaptive fusion target tracking based on joint detection [J]. Journal of Shandong University(Engineering Science), 2020, 50(3): 51-57.
[3] Shiguang LIU,Hairong WANG,Jin LIU. Fast 4-points congruent sets for coarse registration of 3D point cloud [J]. Journal of Shandong University(Engineering Science), 2019, 49(2): 1-7.
[4] Guoxin WANG,Fengdong CHEN,Guodong LIU. Feature extraction method of color pseudo-random coded structured light [J]. Journal of Shandong University(Engineering Science), 2018, 48(5): 55-60.
[5] ZHAO Ye, HE Xiao, ZHOU Donghua. On least squares fault estimation with incorrect measurement noise statistics [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(5): 254-262.
[6] JIN Peipei, SUN Fengrong, LIU Fanglei, YAO Guihua. Cardiac cycle estimation of echocardiography with speckle tracking [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(2): 94-99.
[7] ZOU Guofeng, FU Guixia, LI Zhenmei, LI Haitao, WANG Kejun. Face image quality evaluation method based on the fusion of two level evaluation indexes [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(2): 6-13.
[8] WANG Na, CHEN Guodong, CHEN Yi. A bleeding simulation algorithm for skin surface based on improved SPH method [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2016, 46(1): 22-27.
[9] XIE Zhi-hua. A novel blood perfusion construction model and its application in infrared face recognition [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2013, 43(5): 1-5.
[10] LIN Zhe1, YAN Jing-wen2, YUAN Ye2. Multi-modality image fusion based on sparse representation and PCNN [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2013, 43(4): 13-17.
[11] LV Xing,SHI Zhong-ke . Design and implementation of a realtime motion detection method based on DirectShow [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(6): 5-9 .
[12] WANG Mei,WANG Guo-hong . New method the vehical license plate character segmentation based on concomitant and complementary color features [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(1): 31-34 .
[13] ZHANG Zhi-wen,SONG Shi-jun, . Circle locating method based on roundness [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(1): 19-22 .
[14] XUE Jian, ZHAO Lin, ZHANG Hao, YANG Lu, HAO Fanchang. Traffic sign detection algorithm of improving Faster R-CNN [J]. Journal of Shandong University(Engineering Science), 2024, 54(5): 34-41.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd