Improving Imaging Pipelines by Sensor Optimization and Low-Level Image Enhancement

Modern digital imaging pipelines transform incident light from the physical world into high-level semantic information for a variety of applications, such as medical imaging, robotics, visualization, and autonomous driving. They consist of multiple stages, including the focusing of light on an image sensor by optical systems, the measurement on the sensor itself, and further processing by an image signal processor (ISP) that also reduces noise and other errors from the measurement process, before the image is consumed by a task-specific algorithm such as a deep neural network. Conventionally, the training of the network is largely decoupled from the rest of the components, which are typically optimized for general-purpose use cases or human perception. This thesis investigates how imaging pipelines can be improved (i) as a whole by jointly optimizing sensor parameters and downstream learning-based algorithms, and (ii) by developing efficient low-level reconstruction and error-removal methods to handle the unavoidable imperfections introduced in the measurement process.

The first part of this thesis focuses on the task-driven design of sensors. It develops methods to optimize the physical position of pixels on a sensor and the shape of spectral color filters, jointly with a downstream neural network to solve specific tasks. Although learned sensor layouts show a significant improvement in a variety of tasks compared to uniform grids, no such improvement could be found for learned spectral color filters in experiments for the specific task of semantic segmentation in autonomous driving, suggesting that, in this setting, the information gained from the spatial positioning of pixels is more impactful than fine-grained spectral information.

The second part of the thesis improves the handling of imperfect measurements through several methods. It includes a provably convergent plug-and-play variational scheme with learned priors for solving inverse problems, a generic depth image enhancement method utilizing normal consistency constraints, and an outlier-robust point-based rendering algorithm that can directly render very noisy point clouds produced by depth sensors.