视觉线索引导的古建筑图像语义描述

doi:10.6040/j.issn.1672-3961.0.2025.224

摘要/Abstract

摘要： 在数字服务领域,用户面对古建筑等富含文化信息的图像时,有时难以正确把握深层语义。为有效反映古建筑图像语义信息,提出利用视觉线索引导多模态大模型生成古建图像描述的方法。首先使用S²-wrapper机制对图像进行多尺度特征提取,引入设定标签状态的标签掩膜训练策略,构建基于Transformer多标签识别网络,建立图像与标签间、标签之间的深层次联系;将生成的多个标签转化为结构化的文本提示,并定义为视觉线索;最后将视觉线索和图像一同输入至多模态大模型中,生成一系列关于古建筑知识的解读,并且无需任何额外的跨模态训练。通过对比不同方法的识别效果,并结合定性分析来评估生成描述的质量。结果表明,所提方法在自建古建筑图像数据集上平均精度、类别平均F₁分数和总体F₁分数分别达到82.7%、72.7%和85.7%,消融试验也进一步验证各个模块的有效性。

关键词: 建筑物识别, Transformer, 图像描述, 多模态大模型, 视觉线索

Abstract: In the field of digital services, users faced difficulties in accurately understanding the deep semantic information contained in cultural images such as ancient architectures. To effectively represent the semantic information of ancient architectural images, a method was proposed to generate descriptions of ancient architecture images using visual cues guided by a multi-modal large language model. First, the S²-wrapper mechanism was utilized to extract multi-scale visual features from images. A multi-label recognition network based on Transformer was constructed by introducing a label-mask training strategy with predefined label states, by which deep relationships between images and labels, as well as among labels themselves were established. Subsequently, the generated multiple labels were transformed into structured textual prompts and defined as visual cues. Finally, the visual cues and images were jointly input into the multi-modal large language model to generate a series of interpretations related to ancient architectural knowledge, without requiring any additional cross-modal training. Quantitative analysis was conducted to compare the recognition performance of different methods, and qualitative analysis was performed to evaluate the quality of the generated descriptions. The results showed that the proposed method achieved an average precision of 82.7%, a class-average F1 score of 72.7%, and an overall F₁ score of 85.7% on our self-built ancient architecture image dataset. Furthermore, ablation experiments further verified the effectiveness of each proposed module.

Key words: building identification, Transformer, image description, multi-modal large model, visual cues

中图分类号:

TP391.41

陈欣,杨凡,崔笛. 视觉线索引导的古建筑图像语义描述[J]. 山东大学学报 (工学版), 2026, 56(5): 62-70.

Chen Xin, Yang Fan, Cui Di. Semantic description of ancient architecture images guided by visual cues[J]. Journal of Shandong University(Engineering Science), 2026, 56(5): 62-70.

参考文献

[1] 刘佳. 基于深度学习的建筑物风格识别研究[D]. 西安: 西安建筑科技大学, 2023: 5-7. LIU Jia. Research on Architectural Style Recognition Based on Deep Learning[D]. Xi'an: Xi'an University of Architecture and Technology, 2023: 5-7.
[2] Llamas J, Lerones P M, Medina R, et al. Classification of architectural heritage images using deep learning techniques[J]. Applied Sciences, 2017, 7(10): 992.
[3] Obeso A M, Benois-Pineau J, Acosta A Á R, et al.Architectural style classification of Mexican historical buildings using deep convolutional neural networks and sparse features[J]. Journal of Electronic Imaging, 2017, 26(1): 011016.
[4] Shi B F, Wu Z Y, Mao M L, et al. When do we not need larger vision models? [C] //Computer Vision-ECCV 2024. Cham, Switzerland: Springer, 2025: 444-462.
[5] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C] //Advances in Neural Information Processing Systems. Long Beach, USA:Curran Associates, Inc., 2017: 5998-6008.
[6] Wang B, Zhang S L, Zhang J F, et al. Architectural style classification based on CNN and channel-spatial attention[J]. Signal, Image and Video Processing, 2023, 17(1): 99-107.
[7] Sun M R, Zhang F, Duarte F, et al. Understanding architecture age and style through deep learning[J]. Cities, 2022, 128: 103787.
[8] Vinyals O, Toshev A, Bengio S, et al. Show and tell: a neural image caption generator[C] //2015 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Boston, USA: IEEE, 2015: 3156-3164.
[9] 赵永强, 金芝, 张峰, 等. 深度学习图像描述方法分析与展望[J]. 中国图象图形学报, 2023, 28(9): 2788-2816. Zhao Yongqiang, Jin Zhi, Zhang Feng, et al. Deep-learning-based image captioning: analysis and prospects[J]. Journal of Image and Graphics, 2023, 28(9): 2788-2816.
[10] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[PP/OL].(2021-02-26)[2025-01-08].https://doi.org/10.48550/arXiv.2103.00020
[11] Li J N, Li D X, Savarese S, et al. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models[C] //International Conference on Machine Learning. Honolulu, USA: PMLR, 2023: 19730-19742.
[12] Wang W, Lv Q, Yu W, et al. CogVLM: visual expert for pretrained language models[PP/OL].V2.(2024-02-04)[2025-01-13].https://doi.org/10.48550/arXiv.2311.03079
[13] 李嘉鑫, 汤鹏杰, 谭云兰, 等. CIC-CGT: 基于多模态大模型的漫画图像标题与描述生成[J]. 井冈山大学学报(自然科学版), 2024, 45(6): 77-86. Li Jiaxin, Tang Pengjie, Tan Yunlan, et al. Cic-cgt: comic image captioning and description with multimodal large scale model[J]. Journal of Jinggangshan University(Natural Sciences Edition), 2024, 45(6): 77-86.
[14] Koh J Y, Salakhutdinov R, Fried D. Grounding language models to images for multimodal inputs and outputs[C] //Proceedings of the 40th International Conference on Machine Learning. Honolulu, USA: PMLR, 2023, 36: 17283-17300.
[15] Zhu D, Chen J, Shen X, et al. MiniGPT-4: enhancing vision-language understanding with advanced large language models[PP/OL]. V2.(2023-10-02)[2025-01-13]. https://doi.org/10.48550/arXiv.2304.10592
[16] Bai J Z, Bai S, Yang S S, et al. Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond[PP/OL].V2.(2023-10-13)[2025-01-13]. https://doi.org/10.48550/arXiv.2308.12966
[17] Tsimpoukelli M, Menick J, Cabi S, et al. Multimodal few-shot learning with frozen language modelsd[PP/OL].V2.(2021-07-03)[2025-01-13]. https://doi.org/10.48550/arXiv.2106.13884
[18] Berrios W, Mittal G, Thrush T, et al. Towards language models that can see: computer vision through the LENS of natural languaged[PP/OL].(2023-07-28)[2025-01-13]. https://doi.org/10.48550/arXiv.2306.16410
[19] Li Z, Yang B, Liu Q, et al. Monkey: image resolution andtext label are important things for large multi-modal models[C] //2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2024: 26753-26763.
[20] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C] //2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, USA: IEEE, 2016: 770-778.
[21] Yu F, Tang J J, Yin W C, et al. ERNIE-ViL: knowledge enhanced vision-language representations through scene graphs[PP/OL] V3.(2021-05-19)[2025-01-13]. https://doi.org/10.48550/arXiv.2006.16934
[22] Zhipu AI, Tsinghua University. VisualGLM-6B: an open-source Chinese-English multimodal conversational language model[R/OL]. 2023-06-25. https://github.com/THUDM/VisualGLM-6B
[23] OpenAI. GPT-4o system card[EB/OL].(2024-05-13)[2025-01-13]. https://openai.com/index/gpt-4o-system-card/
[24] Team G, Georgiev P, Lei V I, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context[PP/OL]. V5.(2024-12-16))[2025-01-13]. https://doi.org/10.48550/arXiv.2403.05530
[25] Wang J, Yang Y, Mao J H, et al. CNN-RNN: a unified framework for multi-label image classification[C] //2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 2285-2294.
[26] Wang Z X, Chen T S, Li G B, et al. Multi-label image recognition by recurrently discovering attentional regions[C] //2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 464-472.
[27] Lanchantin J, Wang T L, Ordonez V, et al. General multi-label image classification with transformers[C] //2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE, 2021: 16473-16483.
[28] Liu S, Zhang L, Yang X, et al. Query2label: a simple transformer way to multi-label classification[PP/OL].(2021-07-22)[2025-01-13]. https://doi.org/10.48550/arXiv.2107.10834
[29] Zhou Wei, Xia Zhiwu, Dou Peng, et al. Double attention based on graph attention network for image multi-label classification[J]. ACM Transactions on Multimedia Computing, Communications, and Applica-tions, 2023, 19(1): 1-23.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed