Journal of Shandong University(Engineering Science) ›› 2026, Vol. 56 ›› Issue (4): 52-64.doi: 10.6040/j.issn.1672-3961.0.2025.127

• Machine Learning & Data Mining • Previous Articles    

A spatial-frequency domain information guided algorithm for image inpainting

Shi Shujuan1, Ye Hailiang1*, Cao Feilong2   

  1. Shi Shujuan1, Ye Hailiang1*, Cao Feilong2(1. College of Sciences, China Jiliang University, Hangzhou 310018, Zhejiang, China;
    2. Institute of Mathematics and Cross-disciplinary Science, Zhejiang Normal University, Hangzhou 310012, Zhejiang, China
  • Published:2026-08-12

Abstract: Aiming at the issue of jointly optimizing spatial and frequency domain features and preserving global structural consistency during upsampling in image inpainting, a spatial-frequency domain information guided algorithm for image inpainting was proposed. Through three core stages of downsampling, joint feature extraction, and upsampling, the deep integration of spatial and frequency domain information was realized. In the downsampling stage, a frequency domain information-guided downsampling module was designed to preserve key structural information using frequency features. In the joint feature extraction stage, a spatial-frequency domain joint feature extraction module was introduced, employing a dual-branch parallel architecture to separately extract multi-scale local spatial features and global frequency domain features based on fast Fourier transform, realizing a synergistic representation of local details and global structures through feature fusion. In the upsampling stage, a frequency domain information-guided upsampling module was introduced, combining the advantages of sub-pixel convolution and bilinear interpolation while introducing frequency domain features to enhance the consistency of global structures, effectively balancing the delicacy and naturalness of the inpainting results. Experimental results on the CelebA-HQ, Places2, and Paris StreetView datasets demonstrated that the proposed method outperformed existing approaches in terms of peak signal-to-noise ratio, structural similarity index measure, and learned perceptual image patch similarity, effectively enhancing the texture coherence and visual authenticity of image inpainting.

Key words: deep learning, image inpainting, feature extraction, fast Fourier transform, frequency domain information

CLC Number: 

  • TP183
[1] Quan W Z, Chen J X, Liu Y L, et al. Deep learning-based image and video inpainting: a survey[J]. International Journal of Computer Vision, 2024, 132(7): 2367-2400.
[2] 李月龙, 高云, 闫家良, 等. 基于深度神经网络的图像缺损修复方法综述[J]. 计算机学报, 2021, 44(11):2295-2316. Li Yuelong, Gao Yun, Yan Jialiang, et al. Image inpainting methods based on deep neural networks: a review[J]. Chinese Journal of Computers, 2021, 44(11): 2295-2316.
[3] 王真言, 蒋胜丞, 宋奇鸿, 等. 基于Transformer的文物图像修复方法[J]. 计算机研究与发展, 2024, 61(3): 748-761. Wang Zhenyan, Jiang Shengcheng, Song Qihong, et al. Transformer-based image restoration method for cultural relics[J]. Journal of Computer Research and Develop-ment, 2024, 61(3): 748-761.
[4] Oh S W, Lee S, Lee J Y, et al. Onion-peel networks for deep video completion[C] //2019 IEEE/CVF Inter-national Conference on Computer Vision(ICCV). Seoul: IEEE, 2019: 4402-4411.
[5] Wan Z Y, Zhang B, Chen D, et al. Old photo restoration via deep latent space translation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(2): 2071-2087.
[6] Liu Y, Sun P, Wergeles N, et al. A survey and performance evaluation of deep learning methods for small object detection[J]. Expert Systems with Applications, 2021, 172: 114602.
[7] 王相海, 孙丽, 万宇, 等. 非局域样本填充和自适应曲率驱动模型的遥感图像修复算法[J]. 模式识别与人工智能, 2016, 29(8): 735-743. Wang Xianghai, Sun Li, Wan Yu, et al. Remote sensing image inpainting based on non-local sample filling and adaptive curvature driven diffusions model[J]. Pattern Recognition and Artificial Intelligence, 2016, 29(8): 735-743.
[8] Ghorai M, Samanta S, Mandal S, et al. Multiple pyramids based image inpainting using local patch statistics and steering kernel feature[J]. IEEE Transactions on Image Processing, 2019, 28(11): 5495-5509.
[9] He L T, Wang Y L. Iterative support detection-based split Bregman method for wavelet frame-based image inpainting[J]. IEEE Transactions on Image Processing, 2014, 23(12): 5470-5485.
[10] Liang X, Ren X, Zhang Z D, et al. Texture repairing by unified low rank optimization[J]. Journal of Computer Science and Technology, 2016, 31(3): 525-546.
[11] Wang Y F, Guo D S, Zhao H R, et al. Image inpainting via multi-scale adaptive priors[J]. Pattern Recognition, 2025, 162: 111410.
[12] Huang W L, Deng Y, Hui S Q, et al. Sparse self-attention Transformer for image inpainting[J]. Pattern Recognition, 2024, 145: 109897.
[13] Yu J H, Lin Z, Yang J M, et al. Free-form image inpainting with gated convolution[C] //2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul: IEEE, 2019: 4471-4480.
[14] Deng Y, Hui S Q, Zhou S P, et al. Learning contextual Transformer network for image inpainting[C] //Pro-ceedings of the 29th ACM International Conference on Multimedia. [S.l.] : ACM, 2021: 2529-2538.
[15] Wang J, Wang C, Huang Q M, et al. Image inpainting based on multi-frequency probabilistic inference model[C] //Proceedings of the 28th ACM International Conference on Multimedia. Seattle, USA: ACM, 2020: 1-9.
[16] Suvorov R, Logacheva E, Mashikhin A, et al. Resolution-robust large mask inpainting with Fourier convolutions[C] //2022 IEEE/CVF Winter Conference on Applications of Computer Vision(WACV). Waikoloa, USA: IEEE, 2022: 3172-3182.
[17] Lu Z Y, Jiang J J, Huang J Q, et al. GLaMa: joint spatial and frequency loss for general image inpainting[C] //2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). New Orleans, USA: IEEE, 2022: 1300-1309.
[18] Chu T Y, Chen J F, Sun J K, et al. Rethinking fast Fourier convolution in image inpainting[C] //2023 IEEE/CVF International Conference on Computer Vision(ICCV). Paris, France: IEEE, 2024: 23138-23148.
[19] Pathak D, Krähenbühl P, Donahue J, et al. Context encoders: feature learning by inpainting[C] //2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, USA: IEEE, 2016: 2536-2544.
[20] Liu G L, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions[C] //Computer Vision-ECCV 2018. Munich, Germany: Springer, 2018: 89-105.
[21] Yu J H, Lin Z, Yang J M, et al. Generative image inpainting with contextual attention[C] //2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018: 5505-5514.
[22] Liu H Y, Jiang B, Song Y B, et al. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations[C] //Computer Vision-ECCV 2020. Glasgow, UK: Springer, 2020: 725-741.
[23] Guo X F, Yang H Y, Huang D. Image inpainting via conditional texture and structure dual generation[C] //2021 IEEE/CVF International Conference on Computer Vision(ICCV). Montreal, Canada: IEEE, 2021: 14114-14123.
[24] 邵新茹, 叶海良, 杨冰, 等. 基于三阶段生成网络的图像修复[J]. 模式识别与人工智能, 2022, 35(12): 1047-1063. Shao Xinru, Ye Hailiang, Yang Bing, et al. Image inpainting with a three-stage generative network[J]. Pattern Recognition and Artificial Intelligence, 2022, 35(12): 1047-1063.
[25] Liu W H, Cun X D, Pun C M, et al. CoordFill: efficient high-resolution image inpainting via parameterized coordinate querying[C] //Proceedings of the AAAI Conference on Artificial Intelligence. Washington, DC, USA: AAAI, 2023: 1746-1754.
[26] Ko K, Kim C S. Continuously masked Transformer for image inpainting[C] //2023 IEEE/CVF International Conference on Computer Vision(ICCV). Paris, France: IEEE, 2024: 13123-13132.
[27] Yu Y C, Zhan F N, Lu S J, et al. WaveFill: a wavelet-based generation network for image inpainting[C] //2021 IEEE/CVF International Conference on Computer Vision(ICCV). Montreal, Canada: IEEE, 2022: 14094-14103.
[28] Li B, Zheng B W, Li H D, et al. Detail-enhanced image inpainting based on discrete wavelet transforms[J]. Signal Processing, 2021, 189:108278.
[29] Jain J, Zhou Y Q, Yu N, et al. Keys to better image inpainting: structure and texture go hand in hand[C] //2023 IEEE/CVF Winter Conference on Applications of Computer Vision(WACV). Waikoloa, USA: IEEE, 2023: 208-217.
[30] Cai X H, Lai Q X, Wang Y W, et al. Poly kernel inception network for remote sensing detection[C] //2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle, USA: IEEE, 2024: 27706-27716.
[31] Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C] //Computer Vision-ECCV 2018. Munich, Germany: Springer, 2018: 3-19.
[32] Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[C] //2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, USA: IEEE, 2016: 2818-2826.
[33] Yu W H, Zhou P, Yan S C, et al. InceptionNeXt: when inception meets ConvNeXt[C] //2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle, USA: IEEE, 2024: 5672-5683.
[34] Karras T, Aila T, Laine S, et al. Progressive growing of GANs for improved quality, stability, and variation[PP/OL]. V3.(2018-02-26)[2025-07-10]. https://arxiv.org/abs/1710.10196
[35] Zhou B L, Lapedriza A, Khosla A, et al. Places: a 10 million image database for scene recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6): 1452-1464.
[36] Doersch C, Singh S, Gupta A, et al. What makes Paris look like Paris?[J]. ACM Transactions on Graphics, 2012, 31(4):101.
[37] Kingma D P, Ba J. Adam: a method for stochastic optimization[PP/OL]. V9.(2017-01-30)[2025-07-10]. https://arxiv.org/abs/1412.6980
[38] Li J Y, Wang N, Zhang L F, et al. Recurrent feature reasoning for image inpainting[C] //2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle, USA: IEEE, 2020: 7757-7765.
[39] Zuo Z W, Zhao L, Li A L, et al. Generative image inpainting with segmentation confusion adversarial training and contrastive learning[C] //Proceedings of the AAAI Conference on Artificial Intelligence. Washington, DC, USA: AAAI, 2023: 3888-3896.
[40] Verma S, Sharma A, Sheshadri R, et al. GraphFill: deep image inpainting using graphs[C] //2024 IEEE/CVF Winter Conference on Applications of Computer Vision(WACV). Waikoloa, USA: IEEE, 2024: 4984-4994.
[41] Li Z, Zhang Y N, Du Y F, et al. STNet: structure and texture-guided network for image inpainting[J]. Pattern Recognition, 2024, 156: 110786.
[42] Cao F L, Xu Q J, Ye H L. Adaptive prior and long-range dependency-based learners for image inpainting[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35(11): 10742-10755.
[1] WANG Qian, ZHANG Ruimin, LI Mingjin, MENG Xianjing, GENG Leilei. Spatio-temporal series prediction based on frequency domain graph convolution network [J]. Journal of Shandong University(Engineering Science), 2026, 56(3): 84-92.
[2] WANG Xinjian, JING Zhibin, MENG Fancheng, SHI Jianguo, ZHANG Minhao, ZHANG Yifan, WANG Qinghua, ZHU Yankai. Ultra-short-term thermal power unit load forecasting based on D-Mamba model [J]. Journal of Shandong University(Engineering Science), 2026, 56(1): 169-178.
[3] LI Changgang, LI Baoliang, CAO Yongji, WANG Jiaying. Review and prospect on artificial intelligence application in power system power flow calculation [J]. Journal of Shandong University(Engineering Science), 2025, 55(5): 1-17.
[4] ZHOU Qunying, SUI Jiacheng, ZHANG Ji, WANG Hongyuan. Industrial product surface defect detection based on self supervised convolution and parameter free attention mechanism [J]. Journal of Shandong University(Engineering Science), 2025, 55(4): 40-47.
[5] ZHOU Zunfu, ZHANG Qian, SHI Jiliang, YUE Shiqin. Face image inpainting based on texture and structure interaction [J]. Journal of Shandong University(Engineering Science), 2025, 55(4): 18-28.
[6] DONG Mingshu, CHEN Liqi, MA Chuanyi, ZHANG Zhuhao, SUN Renjuan, GUAN Yanhua, ZHUANG Peizhi. Deep learning-based intelligent judgment for radar detection of pavement cracks [J]. Journal of Shandong University(Engineering Science), 2025, 55(3): 72-79.
[7] XUE Bingbing, WANG Yong, YANG Weihao, WANG Chuan, YU Di, WANG Xu. Real-time expressway traffic data imputation and state prediction based on ETC system data [J]. Journal of Shandong University(Engineering Science), 2025, 55(3): 58-71.
[8] Jiachun LI,Bowen LI,Jianbo CHANG. An efficient and lightweight RGB frame-level face anti-spoofing model [J]. Journal of Shandong University(Engineering Science), 2023, 53(6): 1-7.
[9] Ying LI,Jiankun WANG. The classification of mild cognitive impairment based on supervised graph regularization and information fusion [J]. Journal of Shandong University(Engineering Science), 2023, 53(4): 65-73.
[10] Yue YUAN,Yanli WANG,Kan LIU. Named entity recognition model based on dilated convolutional block architecture [J]. Journal of Shandong University(Engineering Science), 2022, 52(6): 105-114.
[11] Tongyu JIANG, Fan CHEN, Hongjie HE. Lightweight face super-resolution network based on asymmetric U-pyramid reconstruction [J]. Journal of Shandong University(Engineering Science), 2022, 52(1): 1-8.
[12] Jianqing WU,Xiuguang SONG. Review on development of simultaneous localization and mapping technology [J]. Journal of Shandong University(Engineering Science), 2021, 51(5): 16-31.
[13] Qingfa CHAI,Shoujing SUN,Jifu QIU,Ming CHEN,Zhen WEI,Wei CONG. Prediction method of power grid emergency supplies under meteorological disasters [J]. Journal of Shandong University(Engineering Science), 2021, 51(3): 76-83.
[14] YANG Xiuyuan, PENG Tao, YANG Liang, LIN Hongfei. Adaptive multi-domain sentiment analysis based on knowledge distillation [J]. Journal of Shandong University(Engineering Science), 2021, 51(3): 15-21.
[15] LIAO Jinping, MO Yuchang, YAN Ke. Model and application of short-term electricity consumption forecast based on C-LSTM [J]. Journal of Shandong University(Engineering Science), 2021, 51(2): 90-97.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd