Reality, realistically augmented

A new augmented reality computer programme can edit colours and materials in video streams in realtime

Augmented reality (AR) has once again come into the spotlight since the mobile phone game Pokémon Go attracted more than 75 million fans within the space of a few days. In AR, images or videos of a real scene are enhanced or altered by overlaying the image with computer-generated information. Scientists from the Max Planck Institute for Informatics have now developed a programme that can edit materials and shading/lighting in videos in a very realistic way. The Saarbrücken-based researchers believe that they have achieved an important step forward for AR applications.

Pokémon Go, a game for portable devices like Smartphones and tablets, has given a fresh boost to augmented reality. The Saarbrücken-based computer scientists have taken the technology to a new level of quality in that they no longer overlay real digital image with visual information. Instead, the image is realistically altered. Using this approach, they can impart a different colour or texture to a surface, making it appear different to what it looks like in reality.

That sounds easier than it is: “The colour value a camera assigns to individual pixels is, in simple terms, always the product of reflection and illumination,” explains Christian Theobalt, Leader of the Graphics, Vision & Video Group at the Max Planck Institute for Informatics on the Saarland Informatics Campus in Saarbrücken. “However, the problem is that the two elements in this calculation are only indirectly contained in the image. This means that they can’t be measured,” he adds.

Only the illumination or the reflection is altered

Together with the two Max Planck researchers Abhimitra Meka and Michael Zollhöfer as well as Christian Richardt from the University of Bath and the Intel Visual Computing Institute, he solved this problem in a unique way. “By estimating the illumination and degree of reflection for each pixel in a fraction of a second and changing only one of the two factors, we’re able to preserve a realistic impression,” Theobalt explains. This estimate is based on mathematical optimization processes. Because the software makes these calculations in parallel, the changes are applied so quickly that even the live stream of a conventional webcam can be processed.

Using a surprisingly simple setup, the researchers demonstrate what their programme can do: A young woman is standing in a room in front of a mobile white board and a large plant. She is wearing a red T-shirt emblazoned with a large white logo. A webcam films her, but the T-shirt that appears on the screen of the connected monitor is not red, but blue and after the material modification still interacts realistically with the scene illumination. And that is the only difference between the scene in the room and the scene on the screen.

A cotton shirt becomes a velvet one

Moreover, the programme can change more than just the colours. The scientists can even mimic other materials by modifying the degree of reflection. For example, the software can transform a cotton shirt into a velvet one.

“At present, no other technique can achieve this so quickly and with the help of a simple camera,” says Abhimitra Meka, who developed the software as part of his doctoral dissertation at Saarland University. The Max Planck scientists are therefore curious as to the future applications of the algorithm. Some obvious applications are in the computer games and the fashion industry. Theobalt believes that online retailers could also benefit: “Imagine, you’re considering a new sofa in a different colour. Our programme will allow you to check whether the colour you have in mind will match the lighting in your living room. You don’t even have to leave your seat.”

gob