Journal of Shandong University(Engineering Science) ›› 2026, Vol. 56 ›› Issue (5): 62-70.doi: 10.6040/j.issn.1672-3961.0.2025.224

• Machine Learning & Data Mining • Previous Articles    

Semantic description of ancient architecture images guided by visual cues

Chen Xin1, Yang Fan1, Cui Di2   

  1. Chen Xin1, Yang Fan1, Cui Di2(1. College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao 266590, Shandong, China;
    2. Qingdao Branch, Agricultural Bank of China, Qingdao 266300, Shandong, China
  • Published:2026-10-19

PDF (PC)

2

Abstract: In the field of digital services, users faced difficulties in accurately understanding the deep semantic information contained in cultural images such as ancient architectures. To effectively represent the semantic information of ancient architectural images, a method was proposed to generate descriptions of ancient architecture images using visual cues guided by a multi-modal large language model. First, the S2-wrapper mechanism was utilized to extract multi-scale visual features from images. A multi-label recognition network based on Transformer was constructed by introducing a label-mask training strategy with predefined label states, by which deep relationships between images and labels, as well as among labels themselves were established. Subsequently, the generated multiple labels were transformed into structured textual prompts and defined as visual cues. Finally, the visual cues and images were jointly input into the multi-modal large language model to generate a series of interpretations related to ancient architectural knowledge, without requiring any additional cross-modal training. Quantitative analysis was conducted to compare the recognition performance of different methods, and qualitative analysis was performed to evaluate the quality of the generated descriptions. The results showed that the proposed method achieved an average precision of 82.7%, a class-average F1 score of 72.7%, and an overall F1 score of 85.7% on our self-built ancient architecture image dataset. Furthermore, ablation experiments further verified the effectiveness of each proposed module.

Key words: building identification, Transformer, image description, multi-modal large model, visual cues

CLC Number: 

  • TP391.41
[1] 刘佳. 基于深度学习的建筑物风格识别研究[D]. 西安: 西安建筑科技大学, 2023: 5-7. LIU Jia. Research on Architectural Style Recognition Based on Deep Learning[D]. Xi'an: Xi'an University of Architecture and Technology, 2023: 5-7.
[2] Llamas J, Lerones P M, Medina R, et al. Classification of architectural heritage images using deep learning techniques[J]. Applied Sciences, 2017, 7(10): 992.
[3] Obeso A M, Benois-Pineau J, Acosta A Á R, et al.Architectural style classification of Mexican historical buildings using deep convolutional neural networks and sparse features[J]. Journal of Electronic Imaging, 2017, 26(1): 011016.
[4] Shi B F, Wu Z Y, Mao M L, et al. When do we not need larger vision models? [C] //Computer Vision-ECCV 2024. Cham, Switzerland: Springer, 2025: 444-462.
[5] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C] //Advances in Neural Information Processing Systems. Long Beach, USA:Curran Associates, Inc., 2017: 5998-6008.
[6] Wang B, Zhang S L, Zhang J F, et al. Architectural style classification based on CNN and channel-spatial attention[J]. Signal, Image and Video Processing, 2023, 17(1): 99-107.
[7] Sun M R, Zhang F, Duarte F, et al. Understanding architecture age and style through deep learning[J]. Cities, 2022, 128: 103787.
[8] Vinyals O, Toshev A, Bengio S, et al. Show and tell: a neural image caption generator[C] //2015 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Boston, USA: IEEE, 2015: 3156-3164.
[9] 赵永强, 金芝, 张峰, 等. 深度学习图像描述方法分析与展望[J]. 中国图象图形学报, 2023, 28(9): 2788-2816. Zhao Yongqiang, Jin Zhi, Zhang Feng, et al. Deep-learning-based image captioning: analysis and prospects[J]. Journal of Image and Graphics, 2023, 28(9): 2788-2816.
[10] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[PP/OL].(2021-02-26)[2025-01-08].https://doi.org/10.48550/arXiv.2103.00020
[11] Li J N, Li D X, Savarese S, et al. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models[C] //International Conference on Machine Learning. Honolulu, USA: PMLR, 2023: 19730-19742.
[12] Wang W, Lv Q, Yu W, et al. CogVLM: visual expert for pretrained language models[PP/OL].V2.(2024-02-04)[2025-01-13].https://doi.org/10.48550/arXiv.2311.03079
[13] 李嘉鑫, 汤鹏杰, 谭云兰, 等. CIC-CGT: 基于多模态大模型的漫画图像标题与描述生成[J]. 井冈山大学学报(自然科学版), 2024, 45(6): 77-86. Li Jiaxin, Tang Pengjie, Tan Yunlan, et al. Cic-cgt: comic image captioning and description with multimodal large scale model[J]. Journal of Jinggangshan University(Natural Sciences Edition), 2024, 45(6): 77-86.
[14] Koh J Y, Salakhutdinov R, Fried D. Grounding language models to images for multimodal inputs and outputs[C] //Proceedings of the 40th International Conference on Machine Learning. Honolulu, USA: PMLR, 2023, 36: 17283-17300.
[15] Zhu D, Chen J, Shen X, et al. MiniGPT-4: enhancing vision-language understanding with advanced large language models[PP/OL]. V2.(2023-10-02)[2025-01-13]. https://doi.org/10.48550/arXiv.2304.10592
[16] Bai J Z, Bai S, Yang S S, et al. Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond[PP/OL].V2.(2023-10-13)[2025-01-13]. https://doi.org/10.48550/arXiv.2308.12966
[17] Tsimpoukelli M, Menick J, Cabi S, et al. Multimodal few-shot learning with frozen language modelsd[PP/OL].V2.(2021-07-03)[2025-01-13]. https://doi.org/10.48550/arXiv.2106.13884
[18] Berrios W, Mittal G, Thrush T, et al. Towards language models that can see: computer vision through the LENS of natural languaged[PP/OL].(2023-07-28)[2025-01-13]. https://doi.org/10.48550/arXiv.2306.16410
[19] Li Z, Yang B, Liu Q, et al. Monkey: image resolution andtext label are important things for large multi-modal models[C] //2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2024: 26753-26763.
[20] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C] //2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, USA: IEEE, 2016: 770-778.
[21] Yu F, Tang J J, Yin W C, et al. ERNIE-ViL: knowledge enhanced vision-language representations through scene graphs[PP/OL] V3.(2021-05-19)[2025-01-13]. https://doi.org/10.48550/arXiv.2006.16934
[22] Zhipu AI, Tsinghua University. VisualGLM-6B: an open-source Chinese-English multimodal conversational language model[R/OL]. 2023-06-25. https://github.com/THUDM/VisualGLM-6B
[23] OpenAI. GPT-4o system card[EB/OL].(2024-05-13)[2025-01-13]. https://openai.com/index/gpt-4o-system-card/
[24] Team G, Georgiev P, Lei V I, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context[PP/OL]. V5.(2024-12-16))[2025-01-13]. https://doi.org/10.48550/arXiv.2403.05530
[25] Wang J, Yang Y, Mao J H, et al. CNN-RNN: a unified framework for multi-label image classification[C] //2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 2285-2294.
[26] Wang Z X, Chen T S, Li G B, et al. Multi-label image recognition by recurrently discovering attentional regions[C] //2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 464-472.
[27] Lanchantin J, Wang T L, Ordonez V, et al. General multi-label image classification with transformers[C] //2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE, 2021: 16473-16483.
[28] Liu S, Zhang L, Yang X, et al. Query2label: a simple transformer way to multi-label classification[PP/OL].(2021-07-22)[2025-01-13]. https://doi.org/10.48550/arXiv.2107.10834
[29] Zhou Wei, Xia Zhiwu, Dou Peng, et al. Double attention based on graph attention network for image multi-label classification[J]. ACM Transactions on Multimedia Computing, Communications, and Applica-tions, 2023, 19(1): 1-23.
[1] LI Wei, REN Qiwen, CAO Yongji, YU Sen, LI Changgang, KAN Rui, LIU Ziqi. Model-data hybrid driven estimation method of the state of charge for the lithium battery [J]. Journal of Shandong University(Engineering Science), 2026, 56(3): 156-165.
[2] REN Hongwei, MENG Fei, WANG Jikai, TIAN Weiyang, WEI Mingzhao, CHENG Zhiheng, DU Cong, WU Jianqing. Intelligent detection method for subgrade disease based on deep learning [J]. Journal of Shandong University(Engineering Science), 2026, 56(3): 93-105.
[3] LI Junliang, JIANG Yuan, WU Longxue, LIU Yu. Enhanced Graph Transformer with node and edge feature fusion [J]. Journal of Shandong University(Engineering Science), 2026, 56(3): 118-126.
[4] ZHENG Xiao, CHEN He, ZHOU Dongao, GONG Yongshun. Video anomaly detection method based on video caption augmentation and dual-stream feature fusion [J]. Journal of Shandong University(Engineering Science), 2025, 55(5): 110-119.
[5] JIA Xuan, XU Jikai, REN Yijing, LIU Decai, XU Qiang, ZHANG Li. Calculation method of theoretical line loss in transformer districts based on sample expansion and data-driven [J]. Journal of Shandong University(Engineering Science), 2025, 55(3): 158-164.
[6] . Earth fault protection method based on frequency band power factor [J]. Journal of Shandong University(Engineering Science), 2021, 51(1): 76-86.
[7] Zhiyong LIN,Damin ZHANG,Xungao ZHONG,Jianbin ZENG,Qiang ZHANG. Analysis of insulation state of transformer based on aging factor [J]. Journal of Shandong University(Engineering Science), 2019, 49(1): 114-119.
[8] WEI Bengang, GUO Ruochen, HUANG Hua, WANG Zhen, LIU Pengcheng, ZHANG Yuying, LI Kejun, LOU Jie. Simulation and analysis of two dimensional temperature field of the discrete cooling system transformer based on the finite element method [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2017, 47(6): 63-69.
[9] DU Zheng-wang. Study of the reasons of the over-voltage of PT in the distribution network based on a time-varying vector [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2012, 42(2): 124-129.
[10] MIN Yue,XIAO Jing . Analysis method of the 3D transformer electric field [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2007, 37(6): 54-57 .
[11] ZHANG Qingfan,CUI Naxin . The winding group analysis of threephase transformerbased on the voltage vector [J]. JOURNAL OF SHANDONG UNIVERSITY (ENGINEERING SCIENCE), 2006, 36(3): 81-85 .
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright 2018 © Editorial office of Journal of Shandong University (Engineering Science)
Supported by Beijing Magtech Co.Ltd