AI and Cultural Heritage Image Collections: Opportunities and challenges

Abstract

This paper contributes to the discussion about the opportunities and challenges of applying computer vision and machine learning to archival image collections of significant cultural heritage value. We explore these questions from an institutional perspective. Our case study is a pilot project developed at Dumbarton Oaks, a research institute and library, museum, and historic garden affiliated with Harvard University and located in Washington, DC. The project focused on a collection of 10,000 images of Syrian monuments in the institution’s Image Collections and Fieldwork Archives (ICFA). Drawing on that project, as well as the broader landscape of AI-based categorisation efforts in the fields of art and architecture, we will share our insights on the potential of AI to facilitate and enhance archival image access and recording. Many of the Syrian sites in the Dumbarton Oaks collection have been inaccessible to researchers and the public for over a decade and/or have been damaged or destroyed. The pilot project, undertaken in 2019-2020, was a collaboration between Dumbarton Oaks; a commercial tech partner, ArthurAI Inc.; and a computer science research team from the University of Maryland. For Dumbarton Oaks, the primary goal was to explore whether AI can improve the speed and efficiency of sharing collections and allow for more sophisticated curation by subject experts who, thanks to automation, would be relieved of the burden of rote processing. For the technology partners, the experimental value of the project lay in the availability of a collection that could be shared open access (no privacy or copyright issues) and was focused enough to yield a domain-specific training set. The methods and techniques explored included multi-label classification, multi-task classification, unsupervised image clustering, and explainability.

REFERENCES

1. G. Amato, F. Falchi, and L. Vadicamo (2016). Visual Recognition of Ancient Inscriptions Using Convolutional Neural Network and Fisher Vector. Journal on Computing and Cultural Heritage, 9(4),21.

2. A. Belhi, A. Bouras, and S. Foufou (2018) Leveraging Known Data for Missing Label Prediction in Cultural Heritage Context. Applied Sciences, 8(10), 1768.

3. J. Choi (2019) Exploring Art with Open Access and AI: What’s Next? https://www.metmuseum.org/blogs/now-at-themet/2019/met-microsoft-mit-exploring-art-open-access-ai-whats-next (retrieved 13 March 2021).

4. R. Cordell (2020) Machine Learning + Libraries: A Report on the State of the Field. Library of Congress, Washington, D.C.

5. A. Elgammal, M. Mazzone, B. Liu, D. Kim, and M. Elhoseiny (2018) The Shape of Art History in the Eyes of the Machine. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018. AAAI Press, New Orleans.

6. P. Kumar, F. Ofli, M. Imran, and C. Castillo (2020) Detection of Disaster-Affected Cultural Heritage Sites from Social Media Images Using Deep Learning Techniques. Journal on Computing and Cultural Heritage, 13(3), 23.

7. A. Lamas, S. Tabik, P. Cruz, R. Montes, A. Martinez-Sevilla, T. Cruz, and F. Herrera (2021) MonuMAI: Dataset, Deep Learning Pipeline and Citizen Science Based App for Monumental Heritage Taxonomy and Classification. Neurocomputing, 420, pp. 266–80.

8. L. Le, Y. Zheng, G. Carneiro, and L. Yang (eds.) (2017). Deep Learning and Convolutional Neural Networks for Medical Image Computing: Precision Medicine, High Performance and Large-Scale Datasets. Springer International Publishing, Cham.

9. Y. LeCun, Y. Bengio, and G. Hinton (2015) Deep Learning. Nature, 521 (7553), pp. 436-444.

10. J. Llamas, P. M. Lerones, R. Medina, E. Zalama and J. Gómez-García-Bermejo (2017) Classification of Architectural Heritage Images Using Deep Learning Techniques. Applied Sciences, 7, 992.

11. C. Mirachi, B. Di Martino, A. Pavan, and A. Esposito (2020) Impact of Industry 4.0 in Architecture and Cultural Heritage: Artificial Intelligence and Semantic Web Technologies to Empower Interoperability and Data Usage. In C. M. Bolognesi and C. Santagati (eds.), Impact of Industry 4.0 on Architecture and Cultural Heritage. IGI Global, Hershey.

12. V. Palma (2019) Towards Deep Learning for Architecture: A Monument Recognition Mobile App. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (XLII-2/W9 2019), Bergamo, Italy, 6-8 February 2019, 551–556. Bergamo.

13. C. Sandoval, E. Pirogova, and M. Lech (2019). Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings. IEEE Access, 7, pp. 41770-41781.

14. S. Yang, Mook Oh., D. Merchant, B. Howe, and J. West (2019) Classifying Digitized Art Type and Time Period. Proceedings of International Journal for Digital Art History (DSDAH 2018), London, UK, August 2018. London.