The AI that finds similarities between works of art: here is the algorithm developed by MIT and Microsoft

MIT Artificial Intelligence - Innovate Your Strategy with AI‎

Researchers of the Computer Science and Artificial Intelligence Laboratory (CSAIL) MIT in collaboration with Microsoft worked on the development of a new algorithm that combines the paintings in the Metropolitan Museum of Art with those exhibited at the Rijksmuseum in Amsterdam based on possible hidden stylistic connections, which may not be evident to the eye of the observer.

Mosaic, the name of the algorithm, draws inspiration from the exhibition "Rembrandt and Velazquez" of the Rijksmuseum, based on the juxtaposition of quards that may seem different but are actually linked by a subtle common element, by style or interpretation. For example, in the exhibition the San Serapio by Francisco de Zurbarán is exposed next to the threatened Swan by Jan Asselijn because the two subjects although profoundly different (a martyr and a swan, in fact) show postural connections.

MIT Artificial Intelligence - Innovate Your Strategy with AI‎

Based on this concept, researchers developed an algorithm to find images from a query search. For example, asking the system "which musical instrument is more like this painting than a blue and white dress?" They discovered a white and blue porcelain violin, which allowed them to outline cultural exchanges between Holland and China.

From a certain point of view Mosaic is similar to the experimental project X Degrees of Separation by Google, which connects two images or two works of art through a series of paintings and other works of art. Mosaic however requires only one image to find other similar stylistic examples. The algorithm tries to find artistic correspondences in different cultures.

Mark Hamilton, the main author of the publication and student at MIT, explained that the creation of the algorithm proved to be rather laborious since the goal of the researchers was to match relevant images not only for style or colors, but also for meaning and theme represented.

This AI can discover the hidden links between great works of art

Hamilton and colleagues then used a moving K-Nearest Neighbor data structure that places similar images in a tree structure and runs through it to locate the nearest result. This algorithm was then applied to the free-access collections of the two museums.

Researchers have also found that this method can be applied to highlight the limitations of deepfake algorithms based on GAN (Generative Adversarial Network) and where they fail in their task. It is not clear, however, whether the algorithm, whose operation is detailed here, can be able to differentiate deepfakes from genuine images.