Multi-View Stereo on Consistent Face Topology
Graham Fyffe*    Koki Nagano*    Loc Huynh    Shunsuke Saito 2   
Jay Busch    Andrew Jones    Hao Li 2    Paul Debevec   
Joint First Authors *    USC Institute for Creative Technologies    University of Southern California 2

(a) (b) (c) (d)
Figure 1: Our method automatically reconstructs dynamic facial models from multi-view stereo with consistent parameterization. (a) Facialreconstruction with artist-quality mesh topology, Reconstructed facial mesh overlaid on the input video. (c) Reconstructed face modelwith a displacement map estimated from details in the images. (d) Close-up of fine details, such as pores and dynamic wrinkles from (c).

We present a multi-view stereo reconstruction technique that directly produces a complete high-fidelity head model with consistent facial mesh topology. While existing techniques decouple shape estimation and facial tracking, our framework jointly optimizes for stereo constraints and consistent mesh parameterization. Our method is therefore free from drift and fully parallelizable for dynamic facial performance capture. We produce highly detailed facial geometries with artist-quality UV param-eterization, including secondary elements such as eyeballs, mouth pockets, nostrils, and the back of the head. Our approachconsists of deforming a common template model to match multi-view input images of the subject, while satisfying cross-view,cross-subject, and cross-pose consistencies using a combination of 2D landmark detection, optical flow, and surface and vol-umetric Laplacian regularization. Since the flow is never computed between frames, our method is trivially parallelized byprocessing each frame independently. Accurate rigid head pose is extracted using a PCA-based dimension reduction and de-noising scheme. We demonstrate high-fidelity performance capture results with challenging head motion and complex facialexpressions around eye and mouth regions. While the quality of our results is on par with the current state-of-the-art, our ap-proach can be fully parallelized, does not suffer from drift, and produces face models with production-quality mesh topologies.


Full Paper
XimeaRiver_EG_2017.pdf, (4.17MB)

Multiviewstereo 24fps 1920x1080 V10.m4v, (131MB)

Presentation Power Point
MultiviewStereoOnConsistentFaceTopology-EG2017_Web.pptx, (530MB)