Main Article Content
Abstract
Open museum collections allow deep learning to support the documentation of historical textiles, but photographs taken over decades carry conventions that a model can learn in place of the objects. This study presents an explainable deep learning framework for the intelligent documentation of silk textile heritage and applies it to 11,388 open silk textile images from The Metropolitan Museum of Art and the Cleveland Museum of Art. Images of nine cultural origins were represented with frozen features from a residual network (ResNet-50), a vision transformer (ViT-B/16), and two foundation models, DINOv2 and CLIP. Ridge multinomial classifiers were evaluated with five repetitions of cross-validation grouped by object and near-duplicate photograph, and the ridge estimates for the main settings and backbones were reproduced in R. Gradient-weighted class activation mapping (Grad-CAM), supported by a deletion test, showed that the initial model attended in some classes to accession labels, color calibration charts, and the black-and-white format of archive photographs. After restricting the data to color photographs and trimming image borders, ResNet-50 features recognized origin with a balanced accuracy of 0.577 (95% CI 0.561 to 0.593) (chance 0.111). DINOv2 and CLIP features raised this to 0.655 and 0.641, respectively, and partial fine-tuning of ResNet-50 to 0.624. Baselines built on date, museum, image size, or background reached at most 0.389, and weaving technique was recognized with a balanced accuracy of 0.722 across four classes (0.709 after the same audit). Explainability auditing should therefore precede the use of open museum images in heritage recognition and design tools.
Keywords
Article Details
References
- M. Fiorucci, M. Khoroshiltseva, M. Pontil, A. Traviglia, A. Del Bue, and S. James, "Machine learning for cultural heritage: A survey," Pattern Recognition Letters, vol. 133, pp. 102-108, 2020. DOI: https://doi.org/10.1016/j.patrec.2020.02.017
- E. Cetinic and J. She, "Understanding and creating art with AI: Review and outlook," ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 18, no. 2, pp. 1-22, 2022. DOI: https://doi.org/10.1145/3475799
- E. Cetinic, T. Lipic, and S. Grgic, "Fine-tuning convolutional neural networks for fine art classification," Expert Systems with Applications, vol. 114, pp. 107-118, 2018. DOI: https://doi.org/10.1016/j.eswa.2018.07.026
- T. Mensink and J. van Gemert, "The Rijksmuseum Challenge: Museum-centered visual recognition," in Proceedings of International Conference on Multimedia Retrieval, 2014, pp. 451-454. DOI: https://doi.org/10.1145/2578726.2578791
- G. Strezoski and M. Worring, "OmniArt: A large-scale artistic benchmark," ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 14, no. 4, pp. 1-21, 2018. DOI: https://doi.org/10.1145/3273022
- S. He, "Research on a segmentation algorithm for the Tujia brocade images based on unsupervised Gaussian mixture clustering," Frontiers in Neurorobotics, vol. 15, Art. no. 739077, 2021. DOI: https://doi.org/10.3389/fnbot.2021.739077
- M. Ni, Q. Huang, N. Ni, H. Zhao, and B. Sun, "Research on the design of Zhuang brocade patterns based on automatic pattern generation," Applied Sciences, vol. 14, no. 13, Art. no. 5375, 2024. DOI: https://doi.org/10.3390/app14135375
- K. Meng, H. Li, Y. Shu, M. Chen, X. Han, and L. He, "Database construction and remodeling method on traditional Yi nationality patterns of China with GAN model," npj Heritage Science, vol. 13, no. 1, Art. no. 181, 2025. DOI: https://doi.org/10.1038/s40494-025-01707-w
- L. B. Mahanta, D. R. Mahanta, T. Rahman, and C. Chakraborty, "Handloomed fabrics recognition with deep learning," Scientific Reports, vol. 14, no. 1, Art. no. 7974, 2024. DOI: https://doi.org/10.1038/s41598-024-58750-z
- A. Das, A. Deka, K. Medhi, and M. J. Saikia, "Deep learning to authenticate traditional handloom textile," Information, vol. 15, no. 8, Art. no. 465, 2024. DOI: https://doi.org/10.3390/info15080465
- K. Arai, J. Shimazoe, and M. Oda, "A method for maintaining a unique Kurume Kasuri pattern of woven textile classified by EfficientNet by means of LightGBM-based prediction of misalignments," Information, vol. 15, no. 8, Art. no. 434, 2024. DOI: https://doi.org/10.3390/info15080434
- S. Tena, R. Hartanto, and I. Ardiyanto, "Content-based image retrieval for traditional Indonesian woven fabric images using a modified convolutional neural network method," Journal of Imaging, vol. 9, no. 8, Art. no. 165, 2023. DOI: https://doi.org/10.3390/jimaging9080165
- The Metropolitan Museum of Art, "MetObjects.csv," The Metropolitan Museum of Art Open Access data set, file version downloaded 24 September 2026, SHA-256 de617b9c947458e426111207f81a65bd1379a151c0077d3ce29cfc22fc0b9183. [Online]. Available: https://github.com/metmuseum/openaccess
- The Cleveland Museum of Art, "data.csv," The Cleveland Museum of Art Open Access data set, file version downloaded 24 September 2026, SHA-256 28abe722b33ec1c0cc397583ec4ef1ef59c0c9676c407168ab0ebb44fb871b9b. [Online]. Available: https://github.com/ClevelandMuseumArt/openaccess
- K. Chen, D. Lu, Z. Jin, M. Su, and J. Jin, "Song brocade in the Ming and Qing dynasties," Clothing and Textiles Research Journal, vol. 38, no. 4, pp. 285-297, 2020. DOI: https://doi.org/10.1177/0887302X20932657
- M. Dorozynski, D. Clermont, and F. Rottensteiner, "Multi-task deep learning with incomplete training samples for the image-based prediction of variables describing silk fabrics," ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. IV-2/W6, pp. 47-54, 2019. DOI: https://doi.org/10.5194/isprs-annals-IV-2-W6-47-2019
- M. Dorozynski and F. Rottensteiner, "Deep descriptor learning with auxiliary classification loss for retrieving images of silk fabrics in the context of preserving European silk heritage," ISPRS International Journal of Geo-Information, vol. 11, no. 2, Art. no. 82, 2022. DOI: https://doi.org/10.3390/ijgi11020082
- L. Rei, D. Mladenic, M. Dorozynski, F. Rottensteiner, T. Schleider, R. Troncy, J. S. Lozano, and M. Gaitán Salvatella, "Multimodal metadata assignment for cultural heritage artifacts," Multimedia Systems, vol. 29, no. 2, pp. 847-869, 2023. DOI: https://doi.org/10.1007/s00530-022-01025-2
- R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, "Shortcut learning in deep neural networks," Nature Machine Intelligence, vol. 2, no. 11, pp. 665-673, 2020. DOI: https://doi.org/10.1038/s42256-020-00257-z
- J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann, "Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study," PLOS Medicine, vol. 15, no. 11, Art. no. e1002683, 2018. DOI: https://doi.org/10.1371/journal.pmed.1002683
- A. J. DeGrave, J. D. Janizek, and S.-I. Lee, "AI for radiographic COVID-19 detection selects shortcuts over signal," Nature Machine Intelligence, vol. 3, no. 7, pp. 610-619, 2021. DOI: https://doi.org/10.1038/s42256-021-00338-7
- J. K. Winkler, C. Fink, F. Toberer, A. Enk, T. Deinlein, R. Hofmann-Wellenhof, L. Thomas, A. Lallas, A. Blum, W. Stolz, and H. A. Haenssle, "Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition," JAMA Dermatology, vol. 155, no. 10, pp. 1135-1141, 2019. DOI: https://doi.org/10.1001/jamadermatol.2019.1735
- R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-CAM: Visual explanations from deep networks via gradient-based localization," in Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618-626. DOI: https://doi.org/10.1109/ICCV.2017.74
- S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, "Unmasking Clever Hans predictors and assessing what machines really learn," Nature Communications, vol. 10, no. 1, Art. no. 1096, 2019. DOI: https://doi.org/10.1038/s41467-019-08987-4
- W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders, and K.-R. Müller, "Explaining deep neural networks and beyond: A review of methods and applications," Proceedings of the IEEE, vol. 109, no. 3, pp. 247-278, 2021. DOI: https://doi.org/10.1109/JPROC.2021.3060483
- J. Xiang, N. Zhang, R. Pan, and W. Gao, "Fabric image retrieval system using hierarchical search based on deep convolutional neural network," IEEE Access, vol. 7, pp. 35405-35417, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2898906
- J. Xiang, R. Pan, and W. Gao, "An efficient retrieval system framework for fabrics based on fine-grained similarity," Entropy, vol. 24, no. 9, Art. no. 1319, 2022. DOI: https://doi.org/10.3390/e24091319
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision," in Proceedings of the 38th International Conference on Machine Learning, PMLR, vol. 139, 2021, pp. 8748-8763. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html
- M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, "DINOv2: Learning robust visual features without supervision," Transactions on Machine Learning Research, 2024. [Online]. Available: https://openreview.net/forum?id=a68SUt6zFt
- M. V. Conde and K. Turgutlu, "CLIP-Art: Contrastive pre-training for fine-grained art classification," in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 3951-3955. DOI: https://doi.org/10.1109/CVPRW53098.2021.00444
- A. Baldrati, M. Bertini, T. Uricchio, and A. Del Bimbo, "Exploiting CLIP-based multi-modal approach for artwork classification and retrieval," in The Future of Heritage Science and Technologies: ICT and Digital Heritage, Communications in Computer and Information Science, vol. 1645, Cham: Springer, 2022, pp. 140-149. DOI: https://doi.org/10.1007/978-3-031-20302-2_11
- A. Toumarkine and C. Vidal-Gorène, "Beyond accuracy: Investigating vision model perception on 19th-century decorative arts," in Computational Humanities Research 2025, Anthology of Computers and the Humanities, vol. 3, 2025, pp. 1544-1562. DOI: https://doi.org/10.63744/k4Jy3Lr4rdK5
- A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "CNN features off-the-shelf: An astounding baseline for recognition," in 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014, pp. 512-519. DOI: https://doi.org/10.1109/CVPRW.2014.131
- M. Oquab, L. Bottou, I. Laptev, and J. Sivic, "Learning and transferring mid-level image representations using convolutional neural networks," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1717-1724. DOI: https://doi.org/10.1109/CVPR.2014.222
- S. Kornblith, J. Shlens, and Q. V. Le, "Do better ImageNet models transfer better?," in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2656-2666. DOI: https://doi.org/10.1109/CVPR.2019.00277
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248-255. DOI: https://doi.org/10.1109/CVPR.2009.5206848
- K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778. DOI: https://doi.org/10.1109/CVPR.2016.90
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations (ICLR), 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy
- A. Barredo Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera, "Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI," Information Fusion, vol. 58, pp. 82-115, 2020. DOI: https://doi.org/10.1016/j.inffus.2019.12.012
- B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, "Learning deep features for discriminative localization," in 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2921-2929. DOI: https://doi.org/10.1109/CVPR.2016.319
- C. Rudin, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead," Nature Machine Intelligence, vol. 1, no. 5, pp. 206-215, 2019. DOI: https://doi.org/10.1038/s42256-019-0048-x
- V. Petsiuk, A. Das, and K. Saenko, "RISE: Randomized input sampling for explanation of black-box models," in Proceedings of the British Machine Vision Conference (BMVC), 2018. [Online]. Available: http://bmvc2018.org/contents/papers/1064.pdf
- A. Torralba and A. A. Efros, "Unbiased look at dataset bias," in CVPR 2011, 2011, pp. 1521-1528. DOI: https://doi.org/10.1109/CVPR.2011.5995347
- M. A. Badgeley, J. R. Zech, L. Oakden-Rayner, B. S. Glicksberg, M. Liu, W. Gale, M. V. McConnell, B. Percha, T. M. Snyder, and J. T. Dudley, "Deep learning predicts hip fracture using confounding patient and healthcare variables," npj Digital Medicine, vol. 2, no. 1, Art. no. 31, 2019. DOI: https://doi.org/10.1038/s41746-019-0105-1
- S. Kapoor and A. Narayanan, "Leakage and the reproducibility crisis in machine-learning-based science," Patterns, vol. 4, no. 9, Art. no. 100804, 2023. DOI: https://doi.org/10.1016/j.patter.2023.100804
- S. Varma and R. Simon, "Bias in error estimation when using cross-validation for model selection," BMC Bioinformatics, vol. 7, no. 1, Art. no. 91, 2006. DOI: https://doi.org/10.1186/1471-2105-7-91
- K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann, "The balanced accuracy and its posterior distribution," in 2010 20th International Conference on Pattern Recognition, 2010, pp. 3121-3124. DOI: https://doi.org/10.1109/ICPR.2010.764
- R. Wightman, "PyTorch image models," software repository, version 1.0.15 used, 2019. [Online]. Available: https://github.com/huggingface/pytorch-image-models
- A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "PyTorch: An imperative style, high-performance deep learning library," in Advances in Neural Information Processing Systems 32, 2019, pp. 8024-8035. [Online]. Available: https://papers.nips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
- L. Breiman, "Random forests," Machine Learning, vol. 45, no. 1, pp. 5-32, 2001. DOI: https://doi.org/10.1023/A:1010933404324
- C. Cortes and V. Vapnik, "Support-vector networks," Machine Learning, vol. 20, no. 3, pp. 273-297, 1995. DOI: https://doi.org/10.1007/BF00994018
- T. Cover and P. Hart, "Nearest neighbor pattern classification," IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21-27, 1967. DOI: https://doi.org/10.1109/TIT.1967.1053964
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, no. 85, pp. 2825-2830, 2011. [Online]. Available: https://jmlr.org/papers/v12/pedregosa11a.html
- J. Friedman, T. Hastie, and R. Tibshirani, "Regularization paths for generalized linear models via coordinate descent," Journal of Statistical Software, vol. 33, no. 1, pp. 1-22, 2010. DOI: https://doi.org/10.18637/jss.v033.i01
- M. N. Wright and A. Ziegler, "ranger: A fast implementation of random forests for high dimensional data in C++ and R," Journal of Statistical Software, vol. 77, no. 1, pp. 1-17, 2017. DOI: https://doi.org/10.18637/jss.v077.i01
- B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY, USA: Chapman and Hall/CRC, 1994. DOI: https://doi.org/10.1201/9780429246593
- C. Nadeau and Y. Bengio, "Inference for the generalization error," Machine Learning, vol. 52, no. 3, pp. 239-281, 2003. DOI: https://doi.org/10.1023/A:1024068626366
- C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, "On calibration of modern neural networks," in Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321-1330. [Online]. Available: https://proceedings.mlr.press/v70/guo17a.html
- B. Zadrozny and C. Elkan, "Transforming classifier scores into accurate multiclass probability estimates," in Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002, pp. 694-699. DOI: https://doi.org/10.1145/775047.775151
- T. Viering and M. Loog, "The shape of learning curves: A review," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 7799-7819, 2023. DOI: https://doi.org/10.1109/TPAMI.2022.3220744
- B. Frénay and M. Verleysen, "Classification in the presence of label noise: A survey," IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 5, pp. 845-869, 2014. DOI: https://doi.org/10.1109/TNNLS.2013.2292894
References
M. Fiorucci, M. Khoroshiltseva, M. Pontil, A. Traviglia, A. Del Bue, and S. James, "Machine learning for cultural heritage: A survey," Pattern Recognition Letters, vol. 133, pp. 102-108, 2020. DOI: https://doi.org/10.1016/j.patrec.2020.02.017
E. Cetinic and J. She, "Understanding and creating art with AI: Review and outlook," ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 18, no. 2, pp. 1-22, 2022. DOI: https://doi.org/10.1145/3475799
E. Cetinic, T. Lipic, and S. Grgic, "Fine-tuning convolutional neural networks for fine art classification," Expert Systems with Applications, vol. 114, pp. 107-118, 2018. DOI: https://doi.org/10.1016/j.eswa.2018.07.026
T. Mensink and J. van Gemert, "The Rijksmuseum Challenge: Museum-centered visual recognition," in Proceedings of International Conference on Multimedia Retrieval, 2014, pp. 451-454. DOI: https://doi.org/10.1145/2578726.2578791
G. Strezoski and M. Worring, "OmniArt: A large-scale artistic benchmark," ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 14, no. 4, pp. 1-21, 2018. DOI: https://doi.org/10.1145/3273022
S. He, "Research on a segmentation algorithm for the Tujia brocade images based on unsupervised Gaussian mixture clustering," Frontiers in Neurorobotics, vol. 15, Art. no. 739077, 2021. DOI: https://doi.org/10.3389/fnbot.2021.739077
M. Ni, Q. Huang, N. Ni, H. Zhao, and B. Sun, "Research on the design of Zhuang brocade patterns based on automatic pattern generation," Applied Sciences, vol. 14, no. 13, Art. no. 5375, 2024. DOI: https://doi.org/10.3390/app14135375
K. Meng, H. Li, Y. Shu, M. Chen, X. Han, and L. He, "Database construction and remodeling method on traditional Yi nationality patterns of China with GAN model," npj Heritage Science, vol. 13, no. 1, Art. no. 181, 2025. DOI: https://doi.org/10.1038/s40494-025-01707-w
L. B. Mahanta, D. R. Mahanta, T. Rahman, and C. Chakraborty, "Handloomed fabrics recognition with deep learning," Scientific Reports, vol. 14, no. 1, Art. no. 7974, 2024. DOI: https://doi.org/10.1038/s41598-024-58750-z
A. Das, A. Deka, K. Medhi, and M. J. Saikia, "Deep learning to authenticate traditional handloom textile," Information, vol. 15, no. 8, Art. no. 465, 2024. DOI: https://doi.org/10.3390/info15080465
K. Arai, J. Shimazoe, and M. Oda, "A method for maintaining a unique Kurume Kasuri pattern of woven textile classified by EfficientNet by means of LightGBM-based prediction of misalignments," Information, vol. 15, no. 8, Art. no. 434, 2024. DOI: https://doi.org/10.3390/info15080434
S. Tena, R. Hartanto, and I. Ardiyanto, "Content-based image retrieval for traditional Indonesian woven fabric images using a modified convolutional neural network method," Journal of Imaging, vol. 9, no. 8, Art. no. 165, 2023. DOI: https://doi.org/10.3390/jimaging9080165
The Metropolitan Museum of Art, "MetObjects.csv," The Metropolitan Museum of Art Open Access data set, file version downloaded 24 September 2026, SHA-256 de617b9c947458e426111207f81a65bd1379a151c0077d3ce29cfc22fc0b9183. [Online]. Available: https://github.com/metmuseum/openaccess
The Cleveland Museum of Art, "data.csv," The Cleveland Museum of Art Open Access data set, file version downloaded 24 September 2026, SHA-256 28abe722b33ec1c0cc397583ec4ef1ef59c0c9676c407168ab0ebb44fb871b9b. [Online]. Available: https://github.com/ClevelandMuseumArt/openaccess
K. Chen, D. Lu, Z. Jin, M. Su, and J. Jin, "Song brocade in the Ming and Qing dynasties," Clothing and Textiles Research Journal, vol. 38, no. 4, pp. 285-297, 2020. DOI: https://doi.org/10.1177/0887302X20932657
M. Dorozynski, D. Clermont, and F. Rottensteiner, "Multi-task deep learning with incomplete training samples for the image-based prediction of variables describing silk fabrics," ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. IV-2/W6, pp. 47-54, 2019. DOI: https://doi.org/10.5194/isprs-annals-IV-2-W6-47-2019
M. Dorozynski and F. Rottensteiner, "Deep descriptor learning with auxiliary classification loss for retrieving images of silk fabrics in the context of preserving European silk heritage," ISPRS International Journal of Geo-Information, vol. 11, no. 2, Art. no. 82, 2022. DOI: https://doi.org/10.3390/ijgi11020082
L. Rei, D. Mladenic, M. Dorozynski, F. Rottensteiner, T. Schleider, R. Troncy, J. S. Lozano, and M. Gaitán Salvatella, "Multimodal metadata assignment for cultural heritage artifacts," Multimedia Systems, vol. 29, no. 2, pp. 847-869, 2023. DOI: https://doi.org/10.1007/s00530-022-01025-2
R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, "Shortcut learning in deep neural networks," Nature Machine Intelligence, vol. 2, no. 11, pp. 665-673, 2020. DOI: https://doi.org/10.1038/s42256-020-00257-z
J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann, "Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study," PLOS Medicine, vol. 15, no. 11, Art. no. e1002683, 2018. DOI: https://doi.org/10.1371/journal.pmed.1002683
A. J. DeGrave, J. D. Janizek, and S.-I. Lee, "AI for radiographic COVID-19 detection selects shortcuts over signal," Nature Machine Intelligence, vol. 3, no. 7, pp. 610-619, 2021. DOI: https://doi.org/10.1038/s42256-021-00338-7
J. K. Winkler, C. Fink, F. Toberer, A. Enk, T. Deinlein, R. Hofmann-Wellenhof, L. Thomas, A. Lallas, A. Blum, W. Stolz, and H. A. Haenssle, "Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition," JAMA Dermatology, vol. 155, no. 10, pp. 1135-1141, 2019. DOI: https://doi.org/10.1001/jamadermatol.2019.1735
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-CAM: Visual explanations from deep networks via gradient-based localization," in Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618-626. DOI: https://doi.org/10.1109/ICCV.2017.74
S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, "Unmasking Clever Hans predictors and assessing what machines really learn," Nature Communications, vol. 10, no. 1, Art. no. 1096, 2019. DOI: https://doi.org/10.1038/s41467-019-08987-4
W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders, and K.-R. Müller, "Explaining deep neural networks and beyond: A review of methods and applications," Proceedings of the IEEE, vol. 109, no. 3, pp. 247-278, 2021. DOI: https://doi.org/10.1109/JPROC.2021.3060483
J. Xiang, N. Zhang, R. Pan, and W. Gao, "Fabric image retrieval system using hierarchical search based on deep convolutional neural network," IEEE Access, vol. 7, pp. 35405-35417, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2898906
J. Xiang, R. Pan, and W. Gao, "An efficient retrieval system framework for fabrics based on fine-grained similarity," Entropy, vol. 24, no. 9, Art. no. 1319, 2022. DOI: https://doi.org/10.3390/e24091319
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision," in Proceedings of the 38th International Conference on Machine Learning, PMLR, vol. 139, 2021, pp. 8748-8763. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html
M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski, "DINOv2: Learning robust visual features without supervision," Transactions on Machine Learning Research, 2024. [Online]. Available: https://openreview.net/forum?id=a68SUt6zFt
M. V. Conde and K. Turgutlu, "CLIP-Art: Contrastive pre-training for fine-grained art classification," in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 3951-3955. DOI: https://doi.org/10.1109/CVPRW53098.2021.00444
A. Baldrati, M. Bertini, T. Uricchio, and A. Del Bimbo, "Exploiting CLIP-based multi-modal approach for artwork classification and retrieval," in The Future of Heritage Science and Technologies: ICT and Digital Heritage, Communications in Computer and Information Science, vol. 1645, Cham: Springer, 2022, pp. 140-149. DOI: https://doi.org/10.1007/978-3-031-20302-2_11
A. Toumarkine and C. Vidal-Gorène, "Beyond accuracy: Investigating vision model perception on 19th-century decorative arts," in Computational Humanities Research 2025, Anthology of Computers and the Humanities, vol. 3, 2025, pp. 1544-1562. DOI: https://doi.org/10.63744/k4Jy3Lr4rdK5
A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "CNN features off-the-shelf: An astounding baseline for recognition," in 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014, pp. 512-519. DOI: https://doi.org/10.1109/CVPRW.2014.131
M. Oquab, L. Bottou, I. Laptev, and J. Sivic, "Learning and transferring mid-level image representations using convolutional neural networks," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1717-1724. DOI: https://doi.org/10.1109/CVPR.2014.222
S. Kornblith, J. Shlens, and Q. V. Le, "Do better ImageNet models transfer better?," in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2656-2666. DOI: https://doi.org/10.1109/CVPR.2019.00277
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248-255. DOI: https://doi.org/10.1109/CVPR.2009.5206848
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778. DOI: https://doi.org/10.1109/CVPR.2016.90
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations (ICLR), 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy
A. Barredo Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera, "Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI," Information Fusion, vol. 58, pp. 82-115, 2020. DOI: https://doi.org/10.1016/j.inffus.2019.12.012
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, "Learning deep features for discriminative localization," in 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2921-2929. DOI: https://doi.org/10.1109/CVPR.2016.319
C. Rudin, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead," Nature Machine Intelligence, vol. 1, no. 5, pp. 206-215, 2019. DOI: https://doi.org/10.1038/s42256-019-0048-x
V. Petsiuk, A. Das, and K. Saenko, "RISE: Randomized input sampling for explanation of black-box models," in Proceedings of the British Machine Vision Conference (BMVC), 2018. [Online]. Available: http://bmvc2018.org/contents/papers/1064.pdf
A. Torralba and A. A. Efros, "Unbiased look at dataset bias," in CVPR 2011, 2011, pp. 1521-1528. DOI: https://doi.org/10.1109/CVPR.2011.5995347
M. A. Badgeley, J. R. Zech, L. Oakden-Rayner, B. S. Glicksberg, M. Liu, W. Gale, M. V. McConnell, B. Percha, T. M. Snyder, and J. T. Dudley, "Deep learning predicts hip fracture using confounding patient and healthcare variables," npj Digital Medicine, vol. 2, no. 1, Art. no. 31, 2019. DOI: https://doi.org/10.1038/s41746-019-0105-1
S. Kapoor and A. Narayanan, "Leakage and the reproducibility crisis in machine-learning-based science," Patterns, vol. 4, no. 9, Art. no. 100804, 2023. DOI: https://doi.org/10.1016/j.patter.2023.100804
S. Varma and R. Simon, "Bias in error estimation when using cross-validation for model selection," BMC Bioinformatics, vol. 7, no. 1, Art. no. 91, 2006. DOI: https://doi.org/10.1186/1471-2105-7-91
K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann, "The balanced accuracy and its posterior distribution," in 2010 20th International Conference on Pattern Recognition, 2010, pp. 3121-3124. DOI: https://doi.org/10.1109/ICPR.2010.764
R. Wightman, "PyTorch image models," software repository, version 1.0.15 used, 2019. [Online]. Available: https://github.com/huggingface/pytorch-image-models
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "PyTorch: An imperative style, high-performance deep learning library," in Advances in Neural Information Processing Systems 32, 2019, pp. 8024-8035. [Online]. Available: https://papers.nips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
L. Breiman, "Random forests," Machine Learning, vol. 45, no. 1, pp. 5-32, 2001. DOI: https://doi.org/10.1023/A:1010933404324
C. Cortes and V. Vapnik, "Support-vector networks," Machine Learning, vol. 20, no. 3, pp. 273-297, 1995. DOI: https://doi.org/10.1007/BF00994018
T. Cover and P. Hart, "Nearest neighbor pattern classification," IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21-27, 1967. DOI: https://doi.org/10.1109/TIT.1967.1053964
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, no. 85, pp. 2825-2830, 2011. [Online]. Available: https://jmlr.org/papers/v12/pedregosa11a.html
J. Friedman, T. Hastie, and R. Tibshirani, "Regularization paths for generalized linear models via coordinate descent," Journal of Statistical Software, vol. 33, no. 1, pp. 1-22, 2010. DOI: https://doi.org/10.18637/jss.v033.i01
M. N. Wright and A. Ziegler, "ranger: A fast implementation of random forests for high dimensional data in C++ and R," Journal of Statistical Software, vol. 77, no. 1, pp. 1-17, 2017. DOI: https://doi.org/10.18637/jss.v077.i01
B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY, USA: Chapman and Hall/CRC, 1994. DOI: https://doi.org/10.1201/9780429246593
C. Nadeau and Y. Bengio, "Inference for the generalization error," Machine Learning, vol. 52, no. 3, pp. 239-281, 2003. DOI: https://doi.org/10.1023/A:1024068626366
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, "On calibration of modern neural networks," in Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321-1330. [Online]. Available: https://proceedings.mlr.press/v70/guo17a.html
B. Zadrozny and C. Elkan, "Transforming classifier scores into accurate multiclass probability estimates," in Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002, pp. 694-699. DOI: https://doi.org/10.1145/775047.775151
T. Viering and M. Loog, "The shape of learning curves: A review," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 7799-7819, 2023. DOI: https://doi.org/10.1109/TPAMI.2022.3220744
B. Frénay and M. Verleysen, "Classification in the presence of label noise: A survey," IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 5, pp. 845-869, 2014. DOI: https://doi.org/10.1109/TNNLS.2013.2292894