SoftRas
ICCV 2019

Figure 1: Our differentiable Soft Rasterizer R (upper) can render mesh silhouette that faithfully approximates that generated by a standard rasterizer R' (below). R' renders a pixel as solid once it is covered by a projected triangle, leading to a discrete and non-differentiable process. We propose to approximate the rasterized triangles {D'i} with a “soft” continuous representation {Di} based on signed distance field. We further fuse {Di} with a differentiable aggregate function A(•), which is essentially a logical or operator, so that the entire framework is differentiable.
Soft Rasterizer (SoftRas)

This repository contains the code (in PyTorch) for "Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning" (ICCV'2019 Oral) by Shichen Liu, Tianye Li, Weikai Chen and Hao Li.


Introduction

Soft Rasterizer (SoftRas) is a truly differentiable renderer framework with a novel formulation that views rendering as a differentiable aggregating process that fuses probabilistic contributions of all mesh triangles with respect to the rendered pixels. Thanks to such "soft" formulation, our framework is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes (color, normal, etc.) from various forms of image representations, including silhouette, shading and color images.


Usage

The code is built on Python3 and PyTorch 1.1.0. CUDA is needed in order to install the module. Our code is extended on the basis of this repo.

To install the module, using

  python setup.py install



Applications

0. Rendering
We demonstrate the rendering effects provided by SoftRas. Realistic rendering results (1st and 2nd columns) can be achieved with a proper setting of sigma and gamma. With larger sigma and gamma, one can obtain renderings with stronger transparency and blurriness (3rd and 4th column).



  CUDA_VISIBLE_DEVICES=0 python examples/demo_render.py


1. 3D Unsupervised Single-view Mesh Reconstruction
By incorporating SoftRas with a simple mesh generator, one can train the network with multi-view images only, without requiring any 3D supervision. At test time, one can reconstruct the 3D mesh, along with the mesh texture, from a single RGB image. Below we show the results of single-view mesh reconstruction on ShapeNet.


Download shapenet rendering dataset provided by NMR:


  bash examples/recon/download_dataset.sh

To train the model:


  CUDA_VISIBLE_DEVICES=0 python examples/recon/train.py -eid recon

To test the model:


  CUDA_VISIBLE_DEVICES=0 python examples/recon/test.py -eid recon \
      -d 'data/results/models/recon/checkpoint_0200000.pth.tar'


2. Image-based Shape Deformation
SoftRas provides strong supervision for image-based mesh deformation. We visualize the deformation process from a sphere to a car model and then to a plane given supervision from multi-view silhouette images.



  CUDA_VISIBLE_DEVICES=0 python examples/demo_deform.py

The optimized mesh is included in data/obj/plane/plane.obj



3. Pose Optimization for Rigid Objects
With scheduled blurry renderings, one can obtain smooth energy landscape that avoids local minima. Below we demonstrate how a color cube is fitted to the target image in the presence of large occlusions. The blurry rendering and the corresponding rendering losses are shown in the 3rd and 4th columns respectively.


4. Non-rigid Shape Fitting
We fit the parametric body model (SMPL) to a target image where the part (right hand) is entirely occluded in the input view.


Contacts

Shichen Liu: liushichen95@gmail.com
Any discussions or concerns are welcomed!

Citation 

 
 @article{liu2019softras,
   title={Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning},
   author={Liu, Shichen and Li, Tianye and Chen, Weikai and Li, Hao},
   journal={The IEEE International Conference on Computer Vision (ICCV)},
   month = {Oct},
   year={2019}
 }


License

MIT License

Copyright (c) 2017 Hiroharu Kato
Copyright (c) 2018 Nikos Kolotouros
Copyright (c) 2019 Shichen Liu

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.




Footer With Address And Phones