Introduction:

The project aims to develop a real-time geometry capture approach to digital face replacement for a dynamic performance. Digital face replacement has major applications in visual effects for motion pictures as well as interactive applications such as video games and simulation and training environments. This project looks into extending the current 3D face scanning technology developed at the ICT Graphics Lab to support seamless face replacement along with separated diffuse and specular albedo textures and surface normals for high quality post-production relighting of the captured performance. Such an approach goes beyond the traditional scope of face replacement techniques that are either completely image based and hence view-dependent or typically capture a performance under a fixed lighting condition and hence cannot be re-used for other performances.

Method:

Our approach builds upon the high resolution 3D face scanning technology developed at the ICT Graphics Lab for capturing dynamic facial performance. This includes capture of high resolution performance geometry and textures with high speed cameras (stereo pair of Phantom V10) and active illumination using a high speed MULE projector and spherical gradient illumination using Light Stage 5. Furthermore, our approach involves tracking of the facial pose during performance (with and without markers) in order to be able to composite the digital face on the captured background plate (e.g., a stunt performance) in a view-independent manner.
 

Another aspect of the project looks into techniques for obtaining separate specular and diffuse albedo textures and surface normals for high quality relighting of the captured facial performance to match the lighting of a given background plate. While the 3D face scanning technique for static expressions exploits polarization of light for this purpose [Ma et al. 2007], such a technique cannot be used while capturing a dynamic performance due to limitations in the hardware setup [Ma 2008a]. Thus, the project looks into alternative separation techniques of diffuse and specular reflectance based on color-space analysis and computing separate diffuse and specular normals based on such separation as a post-process.
 
 
 

Tracking and Alignment:

In order to align the dynamic geometry of the subject with the dynamic background plate, we perform a 3D tracking based on a small set of 15-25 painted marker points on both the subject and the background plate actor. We manually select a number of markers that can be reliably tracked automatically in both of the input videos. Using the tracking data of the subject, we stabilize its performance by inverting the rigid movement. The stabilized geometry is then transformed dynamically to the 3D face position in the background plate.

Rendering:

The captured facial performance is finally rendered with incident illumination from a light probe that is also captured on-site along with the background plate in order to match the appearance of the digital face for compositing with the background plate.

Here, we sample the captured light probe into a set of point lights using the Median Cut algorithm [Debevec 2005] and employ the hybrid normal map technique [Ma et al. 2007] for efficiently relighting the performance with a local shading model using the separated diffuse and specular normals and albedo textures.

Material:

ICT Technical Report 2008:

• ICT-TR-04-2008-FaceReplacement.pdf, Adobe Acrobat Document, 2.54 MB.

SIGGRAPH Technical Talk 2008 Videos:

• Final.mov, 28.5 MB. ( Quicktime )
• Cleanplate.mov, 25.1 MB. ( Quicktime )
• Mike.mov, 11.1 MB. ( Quicktime )
• FloatingHead.mov, 2.16 MB. ( Quicktime )
• Stereo_Patterns_H264.mov, 21.7 MB. ( Quicktime )

SIGGRAPH Technical Talk 2008 Slides:

• GradientMocap_S2008Slides, Microsoft Powerpoint, 121 MB.

Related Projects: