Informed Learning for Image Restoration and Understanding
Open Access
- Author:
- Tofighi, Mohammad
- Graduate Program:
- Electrical Engineering
- Degree:
- Doctor of Philosophy
- Document Type:
- Dissertation
- Date of Defense:
- September 12, 2019
- Committee Members:
- Vishal Monga, Dissertation Advisor/Co-Advisor
Vishal Monga, Committee Chair/Co-Chair
William Evan Higgins, Committee Member
Jeffrey Louis Schiano, Committee Member
Leonid V Berlyand, Outside Member
Kultegin Aydin, Program Head/Chair - Keywords:
- Machine Learning
Cell Detection
Image Processing
Image Deblurring
Deep Learning
Video Deblurring - Abstract:
- Images captured by variety of imaging devices are often corrupted by artifacts. Hence, they require quality enhancement and/or analysis techniques to remove the artifacts to be visually appealing and amenable for analysis. Many of these approaches are model based methods, which use physically inspired formulations to address the problems. On the other hand, there are learning based approaches which act as a black-box and learn a mapping between input data and desired output without using any explicit knowledge about the structure of data. In this dissertation, we pursue the combination of learning based methods and domain knowledge towards important real-world image processing and vision problems. Specifically, for blind image deconvolution we model the problem as a rank-1 matrix and use structured sparse representations to recover the image and the blur kernel. In the second part, to enhance cell nucleus detection in microscopic imagery, we incorporate shape prior information to regularize the learning of a convolutional neural network. In the last part, for blind video deblurring problem, we propose a solution which combines the interpretablity merits of model-based iterative algorithms with data-driven enhanced performance and fast inference merits of deep learning approaches by unrolling an iterative algorithm to construct a neural network. In the first part of the dissertation, we propose an algorithm based on structured sparse representations for image deblurring. Blind image deblurring is a particularly challenging inverse problem where the blur kernel is unknown and must be estimated en route to recover the deblurred image. The problem is of strong practical relevance since many imaging devices such as cellphone cameras must rely on deblurring algorithms to yield satisfactory image quality. Despite significant research effort, handling large motions remains an open problem. Here, we develop a new method called Blind Image Deblurring using Row-Column Sparsity (BD-RCS) to address this issue. Specifically, we model the outer product of kernel and image coefficients in certain transformation domains as a rank-one matrix, and recover it by solving a rank minimization problem. Our central contribution then includes solving {\em two new} optimization problems involving row and column sparsity to automatically determine blur kernel and image support sequentially. The kernel and image can then be recovered through a singular value decomposition (SVD). The second part of the dissertation addresses accurate cell nucleus detection in medical images. Nuclei detection is a challenging research topic because of limitations in cellular image quality and diversity of nuclear morphology, i.e. varying nuclei shapes, sizes, and overlaps between multiple cell nuclei. This has been a topic of enduring interest with promising recent success shown by deep learning methods. These methods train Convolutional Neural Networks (CNNs) with a training set of input images and known, labeled nuclei locations. Many such methods are supplemented by spatial or morphological processing. Using a set of canonical cell nuclei shapes, prepared with the help of a domain expert, we develop a new approach that we call Shape Priors with Convolutional Neural Networks (SP-CNN). We further extend the network to introduce a shape prior (SP) layer and then allowing it to become trainable (i.e. optimizable). We call this network tunable SP-CNN (TSP-CNN). In summary, we present new network structures that can incorporate `expected behavior' of nucleus shapes via two components: {\em learnable} layers that perform the nucleus detection and a {\em fixed} processing part that guides the learning with prior information. Analytically, we formulate two new regularization terms that are targeted at: 1) learning the shapes, 2) reducing false positives while simultaneously encouraging detection inside the cell nucleus boundary. While neural networks have achieved vastly enhanced performance over traditional iterative methods in many cases, they are generally empirically designed and the underlying structures are difficult to interpret. In recent years, an approach called algorithm unrolling has helped connect iterative algorithms to neural network architectures by incarnating each iteration as one layer and train the network in a cascade manner. However, such connections have not been made yet for blind image/video deblurring. In the last part of this dissertation, we propose a neural network architecture that advances this idea. We first present an iterative algorithm based on total-variation regularization method on the gradient domain, and subsequently unroll the half-quadratic splitting algorithm to construct a neural network. Key algorithm parameters are learned with the help of training data using backpropagation, for which we derive analytically simple forms that are amenable to fast implementation. Considering that the blur in consecutive frames is often unequal, some frames may remain sharp. Hence, our network incorporates spatio-temporal features to enhance the deblurring performance. We call this approach Algorithm Unrolling for Deep Video Deblurring (AUDVD), which achieves practical performance gains while enjoying interpretability at the same time.