Researchers at BSC are training artificial intelligence models to recognise the symbology and content of the Prado Museum's paintings, and even detect details that may go unnoticed by the human eye.
- The aim of this project is to generate detailed descriptions of the Prado Museum's paintings that will help to better understand our cultural and historical heritage
- After a first pilot project, the challenge will be a global analysis of the entire collection, with more than 35,000 works of art, of which 8,000 belong to one of the most outstanding collections of paintings in Europe, with paintings by Velázquez, Goya and El Greco
- "With the project we also seek to provoke reflection on the potential of AI to recognise the past, culture and symbols," say the BSC researchers, while claiming that it will serve to create new cultural and dissemination initiatives that enhance the value of cultural heritage
The Barcelona Supercomputing Center - Centro Nacional de Supercomputación (BSC-CNS) and the Museo Nacional del Prado have presented the FrAI Angelico project, which shows how artificial intelligence (AI) can help to bring cultural heritage closer to the public thanks to the ability of machine learning systems to recognise the content of paintings and their historical and iconographic context, so that it is possible to obtain a detailed description of each painting without human intervention.
In recent years, the Museo Nacional del Prado has been developing various AI projects. In 2019, it implemented an augmented reading of the descriptive texts of the works published on the website through a Natural Language Understanding (NLU) engine that allows it to recognise the entities and concepts treated in them to facilitate a contextualised reading for the user. Now, with the support of the Barcelona Supercomputing Center - Centro Nacional de Supercomputación (BSC-CNS), it will go a step further in the application of AI for the study and dissemination of its collections and will implement a system that will facilitate the recognition of objects on the images of the works. Javier Pantoja, Head of Digital Development at the Museo Nacional del Prado explains that "With NLU we help our users to better understand the explanatory texts of the works thanks to their augmented reading. With this project, our intention is, among other things, to support the work of documentalists and specialists in the recognition and description of objects, figures and themes of the works".
AI is a powerful tool that can be of great use to cultural heritage professionals because of its ability to analyse large amounts of data. Researchers at BSC's Computational Applications for Science and Engineering (CASE) department have developed an object-detection AI model that they have trained with descriptions of thousands of paintings to be able to accurately analyse the content of paintings in the Prado Museum, to the point of detecting details or objects that may go unnoticed by the human eye.
FrAI Angelico is a prototype based on the technology of a similar earlier EU-funded project called Saint George on a Bike, coordinated by the BSC in collaboration with the Europeana Foundation. In the first phase, 25 works will be analysed, but the aim is to establish a more stable collaboration that will allow a global analysis of the entire collection of the Prado Museum, considered the most important cultural institution in the country with more than 35,000 works of art, of which 8,000 belong to the collection of paintings, one of the most important in Europe with works by artists such as Velázquez, Goya, El Greco, Rubens, Titian and Raphael, among others.
Mobile phones in 16th century paintings
Object recognition systems are trained with images that show the different elements that surround us. Once trained, these AI models can detect new shapes and interpret them in their temporal context. This has been one of the main challenges of the project, as until now systems trained in a given context made interpretation errors in different environments, leading them to identify mobile phones or baseball bats in paintings from the 16th or 17th centuries.
Similarly, the new models are also able to detect objects that are common in pictorial representations of the past, such as a skull, a crown of thorns, angels or demons. The next challenge is to teach these AI systems to interpret abstract concepts in classical European painting, such as that two naked bodies plus an angel probably represent Adam and Eve or a dove accompanied by an angel and a person represent the Holy Spirit.
"With the FrAI Angelico project we also seek to provoke reflection on the potential of AI to recognise the past, culture or symbols. Comparing the analysis of the new models trained at the BSC with the results of other current object recognition systems is an exercise that invites us to think about how we look at the past with the eyes of the present and revives the fascination produced by the contemplation of a painting and the discovery of what is not apparent," says Joaquim Moré, researcher in the CASE group at the BSC.
More accessible works of art for the visually impaired
The ability of AI systems to detect objects in paintings will help the Prado Museum and other institutions in the cultural heritage sector to describe and classify their works of art automatically, facilitating the interpretation of the meaning of the paintings.
The results of FrAI Angelico can be used to study the thematic relationships between thousands of elements of different works simultaneously, to develop iconographic documentation on the Museum's works and to support the classification of works by themes and objects represented. All with the ultimate aim of increasing the public's interest in art and cultural heritage. Also, as an example, this is especially useful for visually impaired people, who will benefit from a more accessible format of the pictorial works thanks to more detailed descriptions of the images.
"Thanks to this initiative, it will be possible to analyse the paintings, detect objects that the visitor may not have noticed, infer themes, discover relationships between their elements or interpret their symbolism in order to create cultural and dissemination initiatives such as virtual exhibitions with related paintings from all over the world. And, above all, this work will serve as the basis for a new way of studying and understanding our cultural heritage," concludes Maria Cristina Marinescu, researcher in the CASE group at the BSC.
This article was first published on 27 June by Barcelona Supercomputing Center.
Receive the Funding Newswire each Tuesday, our Policy Bulletin each Thursday, and news about bridging Europe’s east-west innovation gap twice a month in The Widening. 
