Deep Reflectance Fields - High-Quality Facial Reflectance Field Inference from Color Gradient Illumination

Abhi Meka
Christian Haene
Michael Zollhöfer
Graham Fyffe
Xueming Yu
Jason Dourgarian
Peter Denny
Sofien Bouaziz
Peter Lincoln
Matt Whalen
Geoff Harvey
Jonathan Taylor
Shahram Izadi
Paul Debevec
Christian Theobalt
Julien Valentin
Christoph Rhemann
SIGGRAPH (2019)
Google Scholar

Abstract

Photo-realistic relighting of human faces is a highly sought after feature with many applications ranging from visual effects to truly immersive virtual experiences. Despite tremendous technological advances in the field, humans are often capable of distinguishing real faces from synthetic renders. Photo-realistically relighting any human face is indeed a challenge with many difficulties going from modelling sub-surface scattering and blood flow to estimating the interaction between light and individual strands of hair. We introduce the first system that combines the ability to deal with dynamic performances to the realism of 4D reflectance fields, enabling photo-realistic relighting of non-static faces. The core of our method consists of a Deep Neural network that is able to predict full 4D reflectance fields from two images captured under spherical gradient illumination. Extensive experiments not only show that two images under spherical gradient illumination can be easily captured in real time, but also that these particular images contain all the information needed to estimate the full reflectance field, including specularities and high frequency details. Finally, side by side comparisons demonstrate that the proposed system outperforms the current state-of-the-art in terms of realism and speed.