Realistic 3D models for fashion, film and healthcare
Meshcapade licenses technologies for creating avatars
The start-up company Meshcapade, a spin-off from the Max Planck Institute for Intelligent Systems, automates creating realistic digital 3D bodies of people, also known as 3D avatars. Parametric 3D models like SMPL (Skinned Multi-Person Linear Model) are used to create 3D avatars of humans with accurate body shape and motion, using statistics and machine learning. Newer versions of the SMPL Model include SMPL-X (SMPL-eXpressive), which extends SMPL to provide hand motion and facial expressions, and SMIL (Skinned Multi-Infant Linear Model), which enables creation of realistic 3D avatars for infants. Another technology called SMPLify-X even allows generating avatars of people from a single image and produces detailed facial expressions, subtle hand gestures and realistic body movement and has great application potential in the fashion, gaming and film industry. This method can also be used together with SMIL Model to enable 3D movement detection of babies, and thus help with the early detection of developmental disorders. These and other innovations are based on proprietary research work of the Max-Planck-Institute for Intelligent Systems that has now been licensed by Meshcapade from Max Planck Innovation.
The creation of realistic and movable 3D models of the human body has previously required the use of complex methods and techniques. Motion capture (mocap) is one of these methods. Here, several cameras record the movements of 30 to 40 reflective markers that are attached to a person. The recorded marker data are usually converted to a 3D skeleton with days of manual work of multiple engineers. Finally, this skeleton is connected to a 3D model, which also takes many days to create. The SMPL model together with many of the automated body estimation technologies, licensed and further developed by Meshcapade, allow the calculation of an accurate 3D model and motion of the person within minutes from images, videos, depth cameras and even motion capture.
Since subtle gestures and facial expressions are often important for human interaction and communication, an update to the SMPL Model, called SMPL-X, makes it possible to create a complex, expressive 3D model of the body. In addition to realistic body poses, this model also includes detailed hand poses and facial expressions. The basis for this is a comprehensive database of high-resolution 3D scans of thousands of people of various body shapes, and in various body poses. SMPL-X is a statistical model trained to learn the difference between body shapes of real people and to create realistic pose and facial expressions for different body types.
The recently licensed technology called SMPLify-X now makes it possible for the first time to create these realistic 3D models using a single two-dimensional image. SMPLify-X estimates expressive 3D models by recognizing key features (such as joints-locations, facial features) in an image and uses the SMPL-X model’s statistical distribution to create realistic body shape, pose and expression that best fits to these detected key-points. This development represents a significant simplification for the acquisition and creation of realistic body models and opens up completely new possibilities for the use of 3D avatars. The models can be used to create convincing animated characters in films and computer games. Online clothing retailers can offer their customers a virtual try-on after simply uploading a picture. In addition, many other fields of application for the avatars are conceivable in our increasingly digitized world.
Another technology that was developed jointly with the Max Planck Institute for Intelligent Systems and the Fraunhofer Society describes a virtual infant body model (SMIL, Skinned Multi-infant Linear Model). On the basis of video sequences recorded from an infant using a depth camera or standard video, SMIL, together with SMPLify-X, can create an infant-specific 3D avatar with realistic movement parameters. This three-dimensional measurement of the spontaneous movement of infants can allow automated detection of pathological spontaneous motor skills on the basis of machine learning algorithms. Automated, inexpensive and easy-to-use screening with the aid of SMIL opens up completely new opportunities for pediatric screening.
Customers can license SMPL, SMPL-X, SMIL and SMPLify-X from Meshcapade and use them in their applications. “With our statistical models we want to improve the world of 3D modelling, sizing and animation of the human body. Our models are extremely versatile and can be used in many areas from fashion and entertainment to healthcare and fitness. The basis of our innovations is in particular the knowledge of the Max Planck Institute for Intelligent systems about the principles of perception, learning and action in machine-learning based 3D systems, ”says Naureen Mahmood, Managing Director of Meshcapade GmbH. In addition to these technologies, Meshcapade has licensed additional technologies from the Max Planck institute for Intelligent Systems in the past twelve months that facilitate the creation of 3D models and provide great benefits in many areas of application. "With Meshcapade we got a very successful commercial partner understanding both customer needs and licensed technologies and constantly advancing them for the best results", adds Bernd Ctortecka, Senior Patent and Licensing Manager at Max Planck Innovation, the technology transfer organization of the Max Planck Society.
