The age of social media and immersive technologies has created a growing need for processing detailed visual representations of ourselves as virtual and augmented reality is growing into the next generation platform for online communication, connecting hundreds of millions of users. A realistic simulation of our presence in a mixed reality environment is unthinkable without a compelling and directable 3D digitization of ourselves. With the wide availability of mobile cameras and internet images, we introduce a technology that can build a realistic 3D avatar from a single photograph. This textured 3D face model includes hair and can be instantly animated by anyone in real-time through natural facial performances captured from a regular RGB camera. Immediate applications include personalized gaming and VR-enabled social networks using automatically digitized 3D avatars, as well as mobile apps such as video messengers (e.g., Snapchat) with face-swapping capabilities. As opposed to existing solutions, our technology enables the automatic generation of a complete head model from a fully unconstrained image.
Our objective is to build a fully textured blendshape model of a subject’s face (arbitrary lighting, facial expressions, etc.) from a single unconstrained image including a hair mesh. While the fitting of 3D face models to images has been extensively explored in the vision and graphics community, the digitization of hair models has only been possible recently with some manual input. One of the key steps for hair modeling requires a pixel-level segmentation of hair regions. As opposed to faces, whose geometry and appearances can be approximated effectively using linear models, the dimensionality and complexity of hairstyle variations are substantially larger.
In summary, we introduce the first system that can automatically build a complete avatar model that includes face and hair meshes from a single input image. Our hair modeling algorithm combines recent advances in deep learning-based segmentation and data-driven shape modeling.