Registration algorithms are an essential component of many computer graphics and computer vision systems. With recent technological advances in RGBD sensors (color plus depth), an active area of research is in techniques combining color, geometry, and learnt priors for robust real-time registration. The goal of this course is to introduce the mathematical foundations and theoretical explanation of registration algorithms, in addition to the practical tools to design systems that leverage information from RGBD devices. We present traditional methods for correspondence computation derived from geometric first principles, along with modern techniques leveraging pre-processing of annotated datasets (e.g. deep neural networks). To illustrate the practical relevance of the theoretical content, we discuss applications including static and dynamic scanning/reconstruction as well as real-time tracking of hands and faces.

Recent technological advances in RGB-D sensing devices, such as the Microsoft Kinect, facilitate numerous new and exciting applications. While affordable and accessible, consumer-level RGB-D devices typically exhibit high noise levels in the acquired data. Moreover, difficult lighting situations and geometric occlusions commonly occur in many application settings, potentially leading to a severe degradation in data quality. This necessitates a particular emphasis on the robustness of image and geometry processing algorithms. The combination of geometry (3D) and image (2D) registration is one important aspect in the design of robust applications based on RGB-D devices. This course introduces the main concepts of 2D and 3D registration and explains how to combine them efficiently. To enable dense correspondence computation and non-rigid registration between shapes of significant deformations and shape variations, we present a deep learning framework based on convolutional neural networks.

We formulate the energy measuring the quality of the 2D and 3D alignment as follow

The first term is the matching energy presented in Section 2.1. The second term is similar to the 2D matching energy presented in Section 2.2. The only difference is the additional function f: that projects a 3D point z_i to the 2D image J. For example this function could be a perspective projection of the form

Figure 3: A blendshape model composed of 48 expressions.

In this course, we introduced 2D/3D registration algorithms and show their applications for data captured with RGB-D devices, such as the Microsoft Kinect or the Intel RealSense. Image and geometry registration algorithms are an essential component of many computer graphics and computer vision systems. With recent technological advances in RGB-D sensors, robust algorithms that combine 2D image and 3D geometry registration have become an active area of research. The goal of this course was to introduce the basics of 2D/3D registration algorithms and to provide theoretical explanations and practical tools to design robust computer vision and computer graphics systems based on RGBD devices. We have shown that 2D and 3D registration can be expressed and combined in a common framework. We also presented a deep learning framework that can infer accurate dense correspondences between partial shapes of objects with extremely large intra-class shape variations or deformations. Numerous application based on RGB-D devices can benefit from this formulation that allows to combine different priors in an easy manner. To illustrate the theory and demonstrate practical relevance, we briefly discuss three applications: rigid scanning, non-rigid modeling, realtime face tracking, and human performance capture.