Main Article Content

Abstract

Open museum collections allow deep learning to support the documentation of historical textiles, but photographs taken over decades carry conventions that a model can learn in place of the objects. This study presents an explainable deep learning framework for the intelligent documentation of silk textile heritage and applies it to 11,388 open silk textile images from The Metropolitan Museum of Art and the Cleveland Museum of Art. Images of nine cultural origins were represented with frozen features from a residual network (ResNet-50), a vision transformer (ViT-B/16), and two foundation models, DINOv2 and CLIP. Ridge multinomial classifiers were evaluated with five repetitions of cross-validation grouped by object and near-duplicate photograph, and the ridge estimates for the main settings and backbones were reproduced in R. Gradient-weighted class activation mapping (Grad-CAM), supported by a deletion test, showed that the initial model attended in some classes to accession labels, color calibration charts, and the black-and-white format of archive photographs. After restricting the data to color photographs and trimming image borders, ResNet-50 features recognized origin with a balanced accuracy of 0.577 (95% CI 0.561 to 0.593) (chance 0.111). DINOv2 and CLIP features raised this to 0.655 and 0.641, respectively, and partial fine-tuning of ResNet-50 to 0.624. Baselines built on date, museum, image size, or background reached at most 0.389, and weaving technique was recognized with a balanced accuracy of 0.722 across four classes (0.709 after the same audit). Explainability auditing should therefore precede the use of open museum images in heritage recognition and design tools.

Keywords

Explainable artificial intelligence Deep learning Grad-CAM Transfer learning Cultural heritage Silk textiles

Article Details

How to Cite
Li, Y., Zhu, L., & Sikakaew, P. (2026). An explainable deep learning framework for intelligent documentation of silk textile heritage: cross-cultural pattern recognition and shortcut auditing on open museum collections. Future Technology, 6(1), 51–66. Retrieved from https://fupubco.com/futech/article/view/1248
Bookmark and Share

References

  1. M. Fiorucci, M. Khoroshiltseva, M. Pontil, A. Traviglia, A. Del Bue, and S. James, "Machine learning for cultural heritage: A survey," Pattern Recognition Letters, vol. 133, pp. 102-108, 2020. DOI: https://doi.org/10.1016/j.patrec.2020.02.017
  2. E. Cetinic and J. She, "Understanding and creating art with AI: Review and outlook," ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 18, no. 2, pp. 1-22, 2022. DOI: https://doi.org/10.1145/3475799
  3. E. Cetinic, T. Lipic, and S. Grgic, "Fine-tuning convolutional neural networks for fine art classification," Expert Systems with Applications, vol. 114, pp. 107-118, 2018. DOI: https://doi.org/10.1016/j.eswa.2018.07.026
  4. T. Mensink and J. van Gemert, "The Rijksmuseum Challenge: Museum-centered visual recognition," in Proceedings of International Conference on Multimedia Retrieval, 2014, pp. 451-454. DOI: https://doi.org/10.1145/2578726.2578791
  5. G. Strezoski and M. Worring, "OmniArt: A large-scale artistic benchmark," ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 14, no. 4, pp. 1-21, 2018. DOI: https://doi.org/10.1145/3273022
  6. S. He, "Research on a segmentation algorithm for the Tujia brocade images based on unsupervised Gaussian mixture clustering," Frontiers in Neurorobotics, vol. 15, Art. no. 739077, 2021. DOI: https://doi.org/10.3389/fnbot.2021.739077
  7. M. Ni, Q. Huang, N. Ni, H. Zhao, and B. Sun, "Research on the design of Zhuang brocade patterns based on automatic pattern generation," Applied Sciences, vol. 14, no. 13, Art. no. 5375, 2024. DOI: https://doi.org/10.3390/app14135375
  8. K. Meng, H. Li, Y. Shu, M. Chen, X. Han, and L. He, "Database construction and remodeling method on traditional Yi nationality patterns of China with GAN model," npj Heritage Science, vol. 13, no. 1, Art. no. 181, 2025. DOI: https://doi.org/10.1038/s40494-025-01707-w
  9. L. B. Mahanta, D. R. Mahanta, T. Rahman, and C. Chakraborty, "Handloomed fabrics recognition with deep learning," Scientific Reports, vol. 14, no. 1, Art. no. 7974, 2024. DOI: https://doi.org/10.1038/s41598-024-58750-z
  10. A. Das, A. Deka, K. Medhi, and M. J. Saikia, "Deep learning to authenticate traditional handloom textile," Information, vol. 15, no. 8, Art. no. 465, 2024. DOI: https://doi.org/10.3390/info15080465
  11. K. Arai, J. Shimazoe, and M. Oda, "A method for maintaining a unique Kurume Kasuri pattern of woven textile classified by EfficientNet by means of LightGBM-based prediction of misalignments," Information, vol. 15, no. 8, Art. no. 434, 2024. DOI: https://doi.org/10.3390/info15080434
  12. S. Tena, R. Hartanto, and I. Ardiyanto, "Content-based image retrieval for traditional Indonesian woven fabric images using a modified convolutional neural network method," Journal of Imaging, vol. 9, no. 8, Art. no. 165, 2023. DOI: https://doi.org/10.3390/jimaging9080165
  13. The Metropolitan Museum of Art, "MetObjects.csv," The Metropolitan Museum of Art Open Access data set, file version downloaded 24 September 2026, SHA-256 de617b9c947458e426111207f81a65bd1379a151c0077d3ce29cfc22fc0b9183. [Online]. Available: https://github.com/metmuseum/openaccess
  14. The Cleveland Museum of Art, "data.csv," The Cleveland Museum of Art Open Access data set, file version downloaded 24 September 2026, SHA-256 28abe722b33ec1c0cc397583ec4ef1ef59c0c9676c407168ab0ebb44fb871b9b. [Online]. Available: https://github.com/ClevelandMuseumArt/openaccess
  15. K. Chen, D. Lu, Z. Jin, M. Su, and J. Jin, "Song brocade in the Ming and Qing dynasties," Clothing and Textiles Research Journal, vol. 38, no. 4, pp. 285-297, 2020. DOI: https://doi.org/10.1177/0887302X20932657
  16. M. Dorozynski, D. Clermont, and F. Rottensteiner, "Multi-task deep learning with incomplete training samples for the image-based prediction of variables describing silk fabrics," ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. IV-2/W6, pp. 47-54, 2019. DOI: https://doi.org/10.5194/isprs-annals-IV-2-W6-47-2019
  17. M. Dorozynski and F. Rottensteiner, "Deep descriptor learning with auxiliary classification loss for retrieving images of silk fabrics in the context of preserving European silk heritage," ISPRS International Journal of Geo-Information, vol. 11, no. 2, Art. no. 82, 2022. DOI: https://doi.org/10.3390/ijgi11020082
  18. L. Rei, D. Mladenic, M. Dorozynski, F. Rottensteiner, T. Schleider, R. Troncy, J. S. Lozano, and M. Gaitán Salvatella, "Multimodal metadata assignment for cultural heritage artifacts," Multimedia Systems, vol. 29, no. 2, pp. 847-869, 2023. DOI: https://doi.org/10.1007/s00530-022-01025-2
  19. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, "Shortcut learning in deep neural networks," Nature Machine Intelligence, vol. 2, no. 11, pp. 665-673, 2020. DOI: https://doi.org/10.1038/s42256-020-00257-z
  20. J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann, "Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study," PLOS Medicine, vol. 15, no. 11, Art. no. e1002683, 2018. DOI: https://doi.org/10.1371/journal.pmed.1002683
  21. A. J. DeGrave, J. D. Janizek, and S.-I. Lee, "AI for radiographic COVID-19 detection selects shortcuts over signal," Nature Machine Intelligence, vol. 3, no. 7, pp. 610-619, 2021. DOI: https://doi.org/10.1038/s42256-021-00338-7
  22. J. K. Winkler, C. Fink, F. Toberer, A. Enk, T. Deinlein, R. Hofmann-Wellenhof, L. Thomas, A. Lallas, A. Blum, W. Stolz, and H. A. Haenssle, "Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition," JAMA Dermatology, vol. 155, no. 10, pp. 1135-1141, 2019. DOI: https://doi.org/10.1001/jamadermatol.2019.1735
  23. R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-CAM: Visual explanations from deep networks via gradient-based localization," in Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618-626. DOI: https://doi.org/10.1109/ICCV.2017.74
  24. S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, "Unmasking Clever Hans predictors and assessing what machines really learn," Nature Communications, vol. 10, no. 1, Art. no. 1096, 2019. DOI: https://doi.org/10.1038/s41467-019-08987-4
  25. W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders, and K.-R. Müller, "Explaining deep neural networks and beyond: A review of methods and applications," Proceedings of the IEEE, vol. 109, no. 3, pp. 247-278, 2021. DOI: https://doi.org/10.1109/JPROC.2021.3060483
  26. J. Xiang, N. Zhang, R. Pan, and W. Gao, "Fabric image retrieval system using hierarchical search based on deep convolutional neural network," IEEE Access, vol. 7, pp. 35405-35417, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2898906
  27. J. Xiang, R. Pan, and W. Gao, "An efficient retrieval system framework for fabrics based on fine-grained similarity," Entropy, vol. 24, no. 9, Art. no. 1319, 2022. DOI: https://doi.org/10.3390/e24091319
  28. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision," in Proceedings of the 38th International Conference on Machine Learning, PMLR, vol. 139, 2021, pp. 8748-8763. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html
  29. M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, "DINOv2: Learning robust visual features without supervision," Transactions on Machine Learning Research, 2024. [Online]. Available: https://openreview.net/forum?id=a68SUt6zFt
  30. M. V. Conde and K. Turgutlu, "CLIP-Art: Contrastive pre-training for fine-grained art classification," in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 3951-3955. DOI: https://doi.org/10.1109/CVPRW53098.2021.00444
  31. A. Baldrati, M. Bertini, T. Uricchio, and A. Del Bimbo, "Exploiting CLIP-based multi-modal approach for artwork classification and retrieval," in The Future of Heritage Science and Technologies: ICT and Digital Heritage, Communications in Computer and Information Science, vol. 1645, Cham: Springer, 2022, pp. 140-149. DOI: https://doi.org/10.1007/978-3-031-20302-2_11
  32. A. Toumarkine and C. Vidal-Gorène, "Beyond accuracy: Investigating vision model perception on 19th-century decorative arts," in Computational Humanities Research 2025, Anthology of Computers and the Humanities, vol. 3, 2025, pp. 1544-1562. DOI: https://doi.org/10.63744/k4Jy3Lr4rdK5
  33. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "CNN features off-the-shelf: An astounding baseline for recognition," in 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014, pp. 512-519. DOI: https://doi.org/10.1109/CVPRW.2014.131
  34. M. Oquab, L. Bottou, I. Laptev, and J. Sivic, "Learning and transferring mid-level image representations using convolutional neural networks," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1717-1724. DOI: https://doi.org/10.1109/CVPR.2014.222
  35. S. Kornblith, J. Shlens, and Q. V. Le, "Do better ImageNet models transfer better?," in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2656-2666. DOI: https://doi.org/10.1109/CVPR.2019.00277
  36. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248-255. DOI: https://doi.org/10.1109/CVPR.2009.5206848
  37. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778. DOI: https://doi.org/10.1109/CVPR.2016.90
  38. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations (ICLR), 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy
  39. A. Barredo Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera, "Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI," Information Fusion, vol. 58, pp. 82-115, 2020. DOI: https://doi.org/10.1016/j.inffus.2019.12.012
  40. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, "Learning deep features for discriminative localization," in 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2921-2929. DOI: https://doi.org/10.1109/CVPR.2016.319
  41. C. Rudin, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead," Nature Machine Intelligence, vol. 1, no. 5, pp. 206-215, 2019. DOI: https://doi.org/10.1038/s42256-019-0048-x
  42. V. Petsiuk, A. Das, and K. Saenko, "RISE: Randomized input sampling for explanation of black-box models," in Proceedings of the British Machine Vision Conference (BMVC), 2018. [Online]. Available: http://bmvc2018.org/contents/papers/1064.pdf
  43. A. Torralba and A. A. Efros, "Unbiased look at dataset bias," in CVPR 2011, 2011, pp. 1521-1528. DOI: https://doi.org/10.1109/CVPR.2011.5995347
  44. M. A. Badgeley, J. R. Zech, L. Oakden-Rayner, B. S. Glicksberg, M. Liu, W. Gale, M. V. McConnell, B. Percha, T. M. Snyder, and J. T. Dudley, "Deep learning predicts hip fracture using confounding patient and healthcare variables," npj Digital Medicine, vol. 2, no. 1, Art. no. 31, 2019. DOI: https://doi.org/10.1038/s41746-019-0105-1
  45. S. Kapoor and A. Narayanan, "Leakage and the reproducibility crisis in machine-learning-based science," Patterns, vol. 4, no. 9, Art. no. 100804, 2023. DOI: https://doi.org/10.1016/j.patter.2023.100804
  46. S. Varma and R. Simon, "Bias in error estimation when using cross-validation for model selection," BMC Bioinformatics, vol. 7, no. 1, Art. no. 91, 2006. DOI: https://doi.org/10.1186/1471-2105-7-91
  47. K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann, "The balanced accuracy and its posterior distribution," in 2010 20th International Conference on Pattern Recognition, 2010, pp. 3121-3124. DOI: https://doi.org/10.1109/ICPR.2010.764
  48. R. Wightman, "PyTorch image models," software repository, version 1.0.15 used, 2019. [Online]. Available: https://github.com/huggingface/pytorch-image-models
  49. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "PyTorch: An imperative style, high-performance deep learning library," in Advances in Neural Information Processing Systems 32, 2019, pp. 8024-8035. [Online]. Available: https://papers.nips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
  50. L. Breiman, "Random forests," Machine Learning, vol. 45, no. 1, pp. 5-32, 2001. DOI: https://doi.org/10.1023/A:1010933404324
  51. C. Cortes and V. Vapnik, "Support-vector networks," Machine Learning, vol. 20, no. 3, pp. 273-297, 1995. DOI: https://doi.org/10.1007/BF00994018
  52. T. Cover and P. Hart, "Nearest neighbor pattern classification," IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21-27, 1967. DOI: https://doi.org/10.1109/TIT.1967.1053964
  53. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, no. 85, pp. 2825-2830, 2011. [Online]. Available: https://jmlr.org/papers/v12/pedregosa11a.html
  54. J. Friedman, T. Hastie, and R. Tibshirani, "Regularization paths for generalized linear models via coordinate descent," Journal of Statistical Software, vol. 33, no. 1, pp. 1-22, 2010. DOI: https://doi.org/10.18637/jss.v033.i01
  55. M. N. Wright and A. Ziegler, "ranger: A fast implementation of random forests for high dimensional data in C++ and R," Journal of Statistical Software, vol. 77, no. 1, pp. 1-17, 2017. DOI: https://doi.org/10.18637/jss.v077.i01
  56. B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY, USA: Chapman and Hall/CRC, 1994. DOI: https://doi.org/10.1201/9780429246593
  57. C. Nadeau and Y. Bengio, "Inference for the generalization error," Machine Learning, vol. 52, no. 3, pp. 239-281, 2003. DOI: https://doi.org/10.1023/A:1024068626366
  58. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, "On calibration of modern neural networks," in Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321-1330. [Online]. Available: https://proceedings.mlr.press/v70/guo17a.html
  59. B. Zadrozny and C. Elkan, "Transforming classifier scores into accurate multiclass probability estimates," in Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002, pp. 694-699. DOI: https://doi.org/10.1145/775047.775151
  60. T. Viering and M. Loog, "The shape of learning curves: A review," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 7799-7819, 2023. DOI: https://doi.org/10.1109/TPAMI.2022.3220744
  61. B. Frénay and M. Verleysen, "Classification in the presence of label noise: A survey," IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 5, pp. 845-869, 2014. DOI: https://doi.org/10.1109/TNNLS.2013.2292894