Single-image inverse lighting of faces with a virtual light stage

This dissertation addresses the problem of inverse lighting from a single image of a face. No information is given about the face or the imaging conditions, yet, the goal is to estimate a physically plausible lighting that reproduces plain and harsh illumination effects with respect to the appearance of the face in the given image. First, a 3D Morphable Model is fit to the 2D input face. Then, a generating set of images is rendered under all the same conditions as the input image, but different lights. Each image is rendered under a single light source with unit intensity. The light sources build a fixed set that is called a Virtual Light Stage in this dissertation. We assume that the input image belongs to the synthetic illumination cone that this generating set spans. We estimate the coefficients, so that the linear combination of the generating set is as similar as possible to the input image. To aim for more realistic illumination effects, this thesis uses a non-Lambertian reflectance that considers Fresnel specular highlights at grazing angles. Analysis and synthesis of cast shadows under complex lighting conditions is another important subject of the thesis. For the parameter estimation, two probabilistic modeling approaches are proposed. A hierarchical Bayesian model automatically suppresses inconsistencies between the generative model and the input. The nonnegative optimization algorithm finds the optimal spectral intensities of the Virtual Light Stage light sources for the input face. To enhance the performance of the algorithm on complex illumination effects, such as cast shadows, the hyperparameters of the hierarchical approach are controlled by constraints. This dissertation is a contribution to single image face and environment modeling and analysis with applications in realistic scene reconstruction, intrinsic face model decomposition, relighting and lighting design.