Recent advances in single-view 3D hair digitization have made the creation of high-quality CG characters scalable and accessible to end-users, enabling new forms of personalized VR and gaming experiences. To handle the complexity and variety of hair structures, most cutting-edge techniques rely on the successful retrieval of a particular hair model from a comprehensive hair database. Not only are the aforementioned data-driven methods storage intensive, but they are also prone to failure for highly unconstrained input images, complicated hairstyles, and failed face detection. Instead of using a large collection of 3D hair models directly, we propose to represent the manifold of 3D hairstyles implicitly through a compact latent space of a volumetric variational autoencoder (VAE). This deep neural network is trained with volumetric orientation field representations of 3D hair models and can synthesize new hairstyles from a compressed code. To enable end-to-end 3D hair inference, we train an additional embedding network to predict the code in the VAE latent space from any input image. Strand-level hairstyles can then be generated from the predicted volumetric representation. Our fully automatic framework does not require any ad-hoc face fitting, intermediate classification and segmentation, or hairstyle database retrieval. Our hair synthesis approach is significantly more robust and can handle a much wider variation of hairstyles than state-of-the-art data-driven hair modeling techniques with challenging inputs, including photos that are low-resolution, overexposured, or contain extreme head poses. The storage requirements are minimal and a 3D hair model can be produced from an image in a second. Our evaluations also show that successful reconstructions are possible from highly stylized cartoon images, non-human subjects, and pictures taken from behind a person. Our approach is particularly well suited for continuous and plausible hair interpolation between very different hairstyles.
The 3D acquisition of human hair has become an active research
area in computer graphics in order to make the creation of digital
humans more efficient, automated, and cost effective. High-end hair
capture techniques based on specialized hardware [Beeler et al. 2012;
Echevarria et al. 2014; Herrera et al. 2012; Jakob et al. 2009; Luo et al.
2013; Paris et al. 2008; Xu et al. 2014] can already produce highquality
3D hair models, but can only operate in well-controlled
studio environments. More consumer-friendly techniques, such
as those that only require a single input image [Chai et al. 2015,
2016; Hu et al. 2015, 2017], are becoming increasingly popular and
important as they can facilitate the mass adoption of new 3D avatardriven
applications, including personalized gaming, communication
in VR [Li et al. 2015; Olszewski et al. 2016; Thies et al. 2018], and
social media apps [FaceUnity 2017; itSeez3D: Avatar SDK 2017;
Myidol 2017; Pinscreen 2017]. Existing single-view hair modeling
methods all rely on a large database containing hundreds of 3D
airstyles, which is used as shape prior for further refinement and
to handle the complex variations of possible hairstyles.
This paradigm comes with several fundamental limitations: (1)
the large storage footprints of the hair model database prohibit
their deployment on resource-constrained platforms such as mobile
devices; (2) the search steps are usually slow and difficult to scale
as the database grows to handle increasingly various hairstyles; (3)
these techniques also rely on well-conditioned input photographs
and are susceptible to the slightest failures during the image preprocessing
and analysis step, such as failed face detection, incorrect
head pose fitting, or poor hair segmentation. Furthermore, these
data-driven algorithms are based on hand-crafted descriptors and
do not generalize well beyond their designed usage scenarios. They
often fail in practical scenarios, such as those with occluded face/hair,
poor resolution, degraded quality, or artistically stylized input.
To address the above challenges, we propose an end-to-end singleview
3D hair synthesis approach using a deep generative model to
represent the continuous space of hairstyles.We implicitly model the
continuous space of hairstyles using a compact generative model so
that plausible hairstyles can be effectively sampled and interpolated,
and hence, eliminate the need for a comprehensive database. We
also enable end-to-end training and 3D hairstyle inference from
a single input image by learning deep features from a large set of
To effectively model the space of hairstyles, we introduce the use of volumetric occupancy and flow fields to represent 3D hairstyles for our generative hair modeling framework. We present a variant of volumetric variational autoencoder (VAE) [Kingma and Welling 2014] to learn the mapping from a compact latent space to the space of hairstyles represented by a volumetric representation of a large database of hairstyles [Hu et al. 2015]. To achieve end-to-end 3D hair inference, we train an additional hair embedding neural network to predict the code in the learned VAE latent space from input images. Instead of direct prediction to the latent space, we perform Principled Component Analysis (PCA) in the latent space for an embedding subspace to achieve better generalization performance via prediction to this subspace. In addition, we apply Iterative Error Feedback (IEF) [Carreira et al. 2016] to our embedding network to further facilitate generalization. We include an ablation study of different algorithmic components to validate our proposed architecture (Section 4). We show that our method can synthesize faithful 3D hairstyles from a wide range of input images with various occlusions, degraded image quality, extreme lighting conditions, uncommon hairstyles, and significant artistic abstraction (see Fig 1 and Section 5).We also compare our technique to the latest algorithm for single-view 3D hair modeling [Chai et al. 2016] and show that our approach is significantly more robust on challenging input photos. Using our learned generative model, we further demonstrate that plausible hairstyles can be interpolated effectively between drastically different ones, while the current state-of-the-art method [Weng et al. 2013] fails.
Our main contributions are:
• The first end-to-end framework for synthesis of 3D hairstyles from a single input image without requirement of face detection or hair segmentation. Our approach can handle a wider range of hairstyles and is significantly more robust for challenging input images than existing data-driven techniques.
• A variational autoencoder using a volumetric occupancy and flow field representation. The corresponding latent space is compact and models the wide range of possible hairstyles continuously. Plausible hairstyles can be sampled and interpolated effectively using this VAE-based generative model, and converted into a strand-based hair representation.
• A hair embedding network with robust generalization performance using PCA embedding and an iterative error feedback technique.
In this section, we describe the entire pipeline of our algorithm for single-view 3D hair modeling (Figure 3). We first explain our hair data representation using volumetric occupancy and flow fields (Section 3.1). Using a dataset of more than two thousand different 3D hairstyles, we train a volumetric variational autoencoder to obtain a compact latent space, which encodes the immerse space of plausible 3D hairstyles (Section 3.2). To enable end-to-end single-view 3D hairstyle modeling, we train an additional embedding network to help predict the volumetric representation from an input image (Section 3.3). Finally, we synthesize hair strands by growing them from the scalp of a head model based on the predicted volume. If a face can be detected or manually fitted from the input image, we can optionally refine the output strands to better match the single-view input (Section 3.4).
Single-View Hair Modeling. We show single-view 3D hairstyle
modeling results from a variety of input images in Figures 1 and 5.
For each image, we show the predicted occupancy field with colorcoded
local orientation as well as synthesized strands with manually
specified color. Note that none of these test images are used to train
our hair embedding network. Our method is end-to-end and does
not require any user interactions such as manually fitting a head
model and drawing guiding strokes. Moreover, several input images
in Figure 5 are particularly challenging, because they are either
over-exposed (the third row), have low contrast between the hair
and the background (the fourth row), have low resolution (the fifth
row and the sixth row), or are illustrated in a cartoon style (the last
two rows). Although our training dataset for the hair embedding
network only consists of examples modeled from normal headshot
photographs without any extreme cases (e.g. poorly illuminated
images or pictures of dogs), our method generalizes very well due
to the robustness of deep image features. A typical face detector
will fail to detect a human face from the third, the fifth and the sixth
input images in Figure 5, which will prevent existing automatic hair
modeling method [Hu et al. 2017] from generating any meaningful
results. In Figure 5, only the first image can be handled by the
system proposed by Chai et al. , since their algorithm requires both
successful face detection and high-quality hair segmentation.
In Figure 6, we compare our method to a state-of-the-art automatic
single-view hair modeling technique [Chai et al. 2016] on a
variety of input images. Our results are comparable to those by Chai
et al.  on those less challenging input of typical hairstyles
(Figure 6(a) - (f) and (l)). For these challenging cases (Figure 6(g)
- (k)), we can generate more faithful modeling output, since the
method of Chai et al.  relies on accurate hair segmentation
which can be difficult to achieve with partial occlusions or less
See link to paper for complete description.