A multi-task approach to face deblurring

2.1 Image prior and edge prediction

Image deblurring problem is often formulated as an ill-posed problem, which is solved via constraining by assuming of latent prior. A relevant method is based on statistical prior (i.e., heavy-tail distribution). Fergus et al. [1] adopt a mixture of Gaussian to model the statistical prior of the image. Furthermore, Shan et al. [2] and Levin et al. [7] propose to describe the gradient distribution as Hyper Laplacian prior and a piecewise function for image deblurring respectively. To solve the problem, here the maximum a posteriori (MAP) method utilizes the sparse statistical priors to constrain the model.

Except for statistical latent prior, numerous different priors also have been proposed for describing the latent properties. Krishnan et al. [17] introduce a sparsity prior of the clear images. Xu et al. [18] propose to describe the gradient prior via a L₀ constraint. In [19], Michaeli and Irani utilize the patch recurrence to model the image prior. These methods often estimate the blur kernel and clear image alternately via the MAP-based method. These methods often via MAP to estimate the blur kernel and clear image alternately. In addition to the image priors, some methods estimate the blur kernel and clear images via explicitly obtaining salient edge [6, 18, 20]. Summarizing the above discussion, these methods depend on generic priors, statistic of natural images. While the utilized coarse-to-fine model is computationally expensive, and it may not perform well that images do not contain such substantially information in some subjects.

2.2 Convolutional neural networks

Recently, convolutional neural networks have been widely used in image processing. Compared with the aforementioned methods, CNN-based image deblurring methods could be summarized as follows. First, learn effective priors for image deconvolution. Schuler et al. [9] adopt a multi-layer perception (MLP) to process the images with defocus blur. Xu et al. [10] propose a singular value decomposition (SVD)-based network to achieve deconvolution with outlier, but it needs to fine-tune the sub-network for each case. Zhang et al. [11] use the convolutional neural network to learn the effective prior of images and deblur the image in a half-quadratic optimization. Second, in contrast to the non-blind deblurring problem, estimating the blur kernel via a CNN-based model also be proposed in some methods. Yan et al. [21] learn a classifier to distinguish the type of the blur kernels and then estimate the parameter of the kernels for each type in two sub-network. Sun et al. [12] propose to use classification network to describe the linearly non-uniform blur kernel and combine Markov random field (MRF) to optimize these patch-wised blur kernels in an approximatively traversed way. Third, in terms of an end-to-end model, CNN-based methods achieve speed gained which is time-consuming in some existing algorithms. Chakrabarti et al. [13] and Schuler et al. [14] propose to estimate the blur kernel and achieve deblur images in a two-stage framework in space and frequency domain respectively. Furthermore, in view of text and face deblurring, such end-to-end models [15] also are introduced to solve these specific tasks. Due to less texture, such state-of-the-art methods do not perform well on face deblurring.

2.3 Multi-task learning

Transfer learning bridging tasks in such domains to target domain is utilized in machine learning and computer vision. This algorithm where a model trained on the source domain or data is purposed to refine the target model. It is exploited to assist the generalization in source task to improve a significant performance in target task. The transfer learning is classified to inductive transfer learning, transductive transfer learning, and unsupervised transfer learning based on the kind of source and target tasks (domains). Multi-task learning is an inductive transfer learning method to solve multiple tasks at the same time. It can result in improving the learning efficiency and prediction accuracy of multiple tasks in the model. Multi-task learning has been widely used in many examples in computer vision (i.e., semantic segmentation [22], classification [23], detection [24, 25], and depth regression). Inspired by these works, we exploit multi-task framework for multiple features learning. We demonstrate that the proposed multi-task learning method would present better constraint compared to the single-task learning.

3.1 Multi-task learning

To describe the following multi-task learning model, We first summarize the single-task learning method. Most machine learning task is a single task. As shown in Fig. 1, each task has its own training model, with training on the specific data, the trained model which is appropriate to a task is independent of others. In a multi-task model, it can optimize more than one task in paralleled learning. For an inductive transfer learning model, the inductive bias is introduced to optimize the model. In the case of multiple tasks model, not only the target task but also the source tasks will impact upon the inductive bias. With the help of domain-specific information that related tasks achieved from the training set, the main (target) tasks utilize this inductive bias to improve the generalization performance of this main task.

Single-task learning which constrained by a L₁ or a L₂ norm for the image content will converge to the homologous solution of the image. In our CNN-based deblurring method, we propose a multi-task framework to propagate the image and structure into the network simultaneously. In contrast to the one task model, we learn a network via sharing weights between multiple tasks, it facilitates to enforce the sparsity across tasks through norm regularization.

As shown in Fig. 2, the main network (orange and blue layers) shares the weights in image domain and its structure domain. For a single-task model, it focuses on learning features only for one specific task. To achieve the goal of multiple feature learning, the multi-task model learns the feature representation via constraining each task simultaneously. In this way, the latent feature will be estimated with rich structure prior.

3.2 Multi-scale face deblurring network

The aim of face deblurring is to restore clear images with more explicit structure and facial details. Most state-of-the-art methods are employed to estimate the blur kernel and latent images alternately. Here, a coarse-to-fine framework is applied to solve the iterative problem. As the multi-scale model has been embedded in such framework and it succeeds in gaining implicit or explicit prior from each scale. To address this problem, we employ the multi-scale model to the convolutional neural network framework. For the multi-scale model, the degraded image will be downsampled to a coarse size, and the introduced blur will meanwhile be rescaled to alleviative. It shows in Fig. 2 that we first deblur the image on a coarse scale; it learns to present the features on a lower level. For the upper scale, when combining the coarse reconstruct result, the upper level will accept the lower feature and formulate the coarse-to-fine architecture.

3.3 Synthesis loss function

4.1 Implementation detail

We use a multi-scale network for the deblurring problem as shown in Fig. 2. This network has two scales which share weight between each level. The basic model for each scale begins with three convolutional layers; we add six ResBlocks and three convolutional layers after the beginning unit. Especially, except the first convolutional layer sets the kernel size to 11×11 to increase the receptive field, all the convolutional layers have the kernel size of 5×5 with 64 channels. The input of each scale contains six channels, that is to say, we combine the blurred image and deblurred result of lower scale as the input of the upper scale. Here, note that we copy the blurred image (i.e., six channels) as the input of the first scale. In this multi-scale model, we utilize a deconvolutional layer to achieve upsampling. For this multi-task framework, except the last three convolutional layers which utilized to perform reconstruction, the basic deblurring network shares the weight between two tasks (i.e., structure deblurring sub-network and image deblurring sub-network).

We implement the multi-task deblurring network on the Pytorch platform and train the network on NVIDIA Titan X GPU. We set the batch size of 16 and learning rate to 1e⁻⁴. To guarantee the convergence of the multi-task framework, we firstly train the multi-task deblurring network with a content loss for about 5 days. we then add the perceptual loss and adversarial loss individually for joint training. Specifically, first, we train this multi-task deblurring network using the loss (2) for 100,000 iterations. Second, we embed the perceptual loss (3) for 50,000 iterations. Finally, we add the adversarial loss (4) and jointly train this network for 50,000 iterations.

4.2 Datasets

We use images with the size of 128×128×3 in all our training and testing experiments. For the multi-scale framework, the image will be downsampled to 64×64×3 for the first level of the network. Here, we collect 6464 face images from the Helen dataset [33] (2000 images), CMU PIE dataset [34] (2164 images), and CelebA [35] dataset (2300 images) for training. To generate images for training and testing this deblurring network, we synthesize 20000 motion blur kernels based on random 3D camera trajectories generative model [36]. We set the blur kernel size from 13×13 to 27×27, and convolute them with images to generate blur images. That is to say, we utilize 130 million face images to train this network. For testing this deblurring network, we choose other 200 images from CelebA and Helen datasets respectively, synthesize other 80 blur kernels, and generate 16,000 blur images to test this model.

4.3 Ablation study

4.3.1 Multi-scale learning

In our work, we use a multi-scale network to formulate this coarse-to-fine deblurring model. To validate the performance of the proposed model, we evaluate the effect of the multi-scale framework. Here, we first train a baseline model which only with a single-scale network for face deblurring. We then build another multi-scale model with two level and compare these two networks. These models are optimized only by content loss. Figure 3 shows the deblurring results for the proposed model with different scales. As shown in Fig. 3c, the deblurring result which obtained via a single-scale model produces fewer detail results. The proposed multi-scale architecture is designed as a coarse-to-fine model. Especially, the first level is utilized as a coarse deblurring model; it can achieve a slight reconstruction for the degraded images. In addition, with the help of recovered implicit or explicit feature acquired from the lower level, we feed the original blur images and the upsampled deblurring result into the second level of the model and optimized it with the same constraint. Therefore, the model is sufficient to achieve a great convergence. Figure 3d shows that the coarse-to-fine model optimized by the content loss and coarse deblurred image lead to an accurate convergence with facial features. Table 1 also demonstrates that the multi-scale framework performs well than the single-task model.

Model	Loss	Helen		CelebA
		PSNR	SSIM	PSNR	SSIM
Multi-scale (1 scale)	Content loss	22.70	0824	22.19	0.844
Multi-scale (2 scales)	Content loss	23.30	0.847	22.57	0.856
Multi-task (2 scales)	Content loss	23.69	0.852	23.04	0.859

4.3.2 Transfer learning

To further demonstrate the potential of the proposed multi-task learning model, we conduct an analysis on ablation study. Here, we set the multi-scale model constrained by the content loss as the baseline model and train another multi-task model which guides original images and corresponding latent structure simultaneously. Figure 4c illustrates an inevitably produced smooth and ambiguous deblurring result considering the obtained implicit or explicit feature from the blurred images. The goal of image deblurring is to restore a sharp image with more details (facial features, accurate structures, etc.). If more accurate structures could be obtained, it will constrain the network to a better convergence. We train the network with two goals which include images and structures. The main network learns the capacity of recovering the content and structures simultaneously. The following deblurring sub-network will utilize these features to reconstruct an accurate deblurring image with more details. As shown in Fig. 4d and Table 1, the proposed method performs well for face deblurring. For example, Fig. 4 shows that deblurring face images via the multi-task model can reconstruct more accurate shape of the facial feature; furthermore, more texture can also be restored.

4.3.3 Additional synthesis analysis

In addition to the proposed multi-scale and multi-task model, by constraining the network with perceptual loss and adversarial loss, it will achieve a more realistic result. The visual examples are provided in Fig. 5, and a quantitative evaluation is also shown in Table 2. If the network is just optimized via content loss (L₁), it will achieve a solution with high PSNR or SSIM. However, it will fall to recover more texture and high frequency content from the degraded images. Here, we utilize the pre-trained VGG19 [30] network which can express the sufficient and efficient features of images to constrain the network in feature domain. To extract the semantic features from such specific layers of the VGG19 [30] network, the perceptual loss can assist to preserve more details and texture from the blur images. In addition to the perceptual loss, we also improve to learn the reconstruction by employing the feedback of adversarial loss. As shown in Fig. 5 and Table 2, the perceptual loss and adversarial loss encourage the deblurring process to a better optimized solution. Figure 5e also demonstrates that it will increase more details (e.g., the tooth and eyes).

Approach	Helen		CelebA
	PSNR	SSIM	PSNR	SSIM
Content loss	23.69	0.852	23.04	0.859
+ Perceptual loss	24.21	0.857	23.48	0.864
+ Adversarial loss	24.25	0.864	23.56	0.871

5.1 Comparisons with state-of-the-arts

We first evaluate the image quality on PSNR and SSIM. We provide seven deblurring algorithms to compare with the proposed method. As shown in Figs. 6 and 7, we compare with the state-of-the-art methods on different size of blur kernels. We evaluate the result on Helen and CelebA dataset, the proposed algorithm performs favorably against the state-of-the-art methods. Here, we also provide a qualitative comparison which is present in Fig. 8. As the MAP-based methods [2, 6, 17, 18, 37, 38] synthesize the deblurred results based on the latent prior. That is to say, an unsatisfactory implicit or explicit prior will directly introduce a failure deblurring result. For example, as shown in Fig. 8d, the face deblurring method [37] depends on the structure to constrain the iterative solution. The inaccurate prior will lead to an ambiguous result. Furthermore, due to the deconvolution process, the deblurring result will lead to heavy ringing artifacts. We also compare our algorithm with other CNNs-based method (e.g., Fig 8i). Our method performs well on the latent structure and details, as the robust features which are obtained via the multi-task model.

5.2 Face recognition

To further demonstrate the potential of the proposed face deblurring method, we conduct an analysis of the facial feature. We first exploit the identity distance [39] to evaluate the consistency between deblurred face images and their corresponding ground truth images. It is defined as the Euclidean distance of identity feature acquired via a deep convolution model. Except for PSNR and SSIM, it can also evaluate the consistency (i.e., similarity) of the deblurred result. The curve in Fig. 9 depicts the identity distance of the state-of-the-art algorithms and the proposed method. The lower distance (error) demonstrates that our method can better match the original images with more accurate facial features. In addition, we also test our method for face recognition. For probe data, we choose 100 face images (8000 deblurred results) with different identities from the CelebA dataset. For each identity, we randomly choose other 9 images (i.e., 900 face images) to generate a gallery data. As the deblurred images do not perform well or reconstruct with such artifacts, it will impact on the recognition. For example, Fig. 8e, h shows that deblurring the image via the state-of-the-art methods will lead to such ringing or extremely smooth artifacts; the limited facial feature will be faded. During the recognition, the face detection process and similarity evaluation based on identity distance and the ungraded features will result in the poor performance. As shown in Fig. 8j and Table 3, our deblurring result with less artifacts and preserving more details is effective for face detection and recognition.

Method	Detection (%)	Top 1 (%)	Top 3 (%)	Top 5 (%)
Clear images	100	71	84	89
Blurred images	77.4	29.1	43.4	51.3
Krishnan et al. [17]	80.0	33.8	48.9	56.6
Pan et al. [37]	78.9	42.0	55.7	62.2
Shan et al. [2]	76.0	32.4	46.9	54.0
Xu et al. [18]	82.5	41.1	55.4	62.1
Cho and Lee [6]	52.2	17.2	27.3	32.5
Zhong et al. [38]	69.5	27.6	41.6	48.5
Nah et al. [15]	86.0	40.1	55.3	62.4
Ours	92	55	69	75

5.3 Execution time