Skip to main content

TV2++: a novel spatial-temporal total variation for super resolution with exponential-type norm

Abstract

Recently, many super-resolution algorithms have been proposed to recover high-resolution images to improve visualization and help better analyze images. Among them, total variation regularization (TV) methods have been proven to have a good effect in retaining image edge information. However, these TV methods do not consider the temporal correlation between images. Our algorithm designs a new TV regularization (TV2++) to take advantage of the time dimension information of the images, further improving the utilization of useful information in the images. In addition, the union of global low rank regularization and TV regularization further enhances the image super-resolution recovery. And we extend the exponential-type penalty (ETP) function on singular values of a matrix to enhance low-rank matrix recovery. A novel image super-resolution algorithm based on the ETP norm and TV2++ regularization is proposed. And the alternating direction method of multipliers (ADMM) is applied to solve the optimization problems effectively. Numerous experimental results prove that the proposed algorithm is superior to other algorithms.

Introduction

The super-resolution technology [14] is to restore high-frequency details lost during hardware acquisition of images, thereby improving image quality, making the image more rich in texture and providing better visualization. Super-resolution technology is widely applied to face recognition [1], CT diagnosis [2], high-definition television [3], remote sensing image [4, 5], etc. These studies based on image processing are of great significance to the construction of the Internet of Things (IoT) [6] and smart cities. Although SR technology has been researched and improved a lot, its efficiency is still far from the requirements of real-time applications. The research and implementation of cloud computing [7] provides new ideas for the improvement of SR technology. Unfortunately, the system based on cloud environment has many shortcomings [8, 9].

At present, the mainstream of super-resolution algorithms can be divided into two types of categories, namely, the reconstruction-based methods [1013] and the learning-based methods [1417]. Recently, learning-based methods have gained huge attentions because of its efficiency and generalization. The learning-based methods acquire the co-occurrence prior knowledge between the low-resolution and high-resolution image blocks through the learning process. From sparse representation [15, 18], anchored neighborhood regression [16] to later transformed self-exemplars [17, 19] and deep convolutional networks [2022], by learning from examples based on huge databases consisting of low-resolution and high-resolution image pairs, these methods can recover details and enhance texture information. Extracting the high frequency information based on local features from the training images and then adding the high frequency detail for low-resolution image can guide a high-resolution image reconstruction [14]; however, it also results that these methods deeply rely on selected data and sometimes may bring some artifacts. Moreover, learning-based methods usually require a large number of training sets and a large amount of time to complete the training, which makes it impossible to meet the requirements in real time. Because the model relies too heavily on the data set, it does not have independence [17, 19]. A lot of time and pretreatment also make this method inefficient so that it cannot achieve the desired effect. The edge calculation based on big data [23, 24] can improve the processing efficiency of the system model to a certain extent. We propose to use ADMM to optimize the calculation of the model, which greatly improves the running speed and efficiency.

Conversely, the bulk of reconstructed based methods usually get high-resolution image from a sequence of low-resolution images by optimizing the cost function which have prior restriction [10]. The reconstruction based methods are to invert high-resolution images from the low-resolution image through the image degradation model, which is to solve the inverse problem of the image, and often requires prior knowledge of gradient [25], sparse [15, 26, 27], spatial-temporal feature [28, 29] and so on. Iterative back projection method (IP) proposed by Pejman et al. [11] obtained the high-resolution image through iterative back projection for deviation of simulated low-resolution image and observed image. Non-local means (NLM) [12, 24] is a SR method based on the non-local property. It explains that analogous pixels are not limited to a local area, such as long edges and texture structures. As seen in [13, 30], a framework of maximum a posteriori (MAP) estimation is proposed to performs joint blur identification and high-resolution image reconstruction. And the algorithm addresses blind image super resolution by fusing multiple blurred low-resolution images to render a high-resolution image. In recent years, many popular super-resolution algorithms based on total variation regularization (TV) [10, 3134] have emerged through reconstruction perspective. The TV function is usually defined as the integral of absolute value of image gradient, as shown in [31, 35]; the total variation of the signal is used as a regularizing functional. Since the signal with too much and possibly false details has a high total variation, i.e., the integral of the absolute gradient of the signal is high. According to this principle, the total variation of the signal is reduced to closely match the original signal, removing unnecessary details while retaining important details such as edges. In the case of low signal-to-noise ratio, total variation denoising is also very effective in retaining edges while smoothing out noise in flat areas. This also leads to a total deterioration that will make the restored image too smooth, and in some more detailed images, the restored image will lose details. Therefore, it is the focus of research to keep more details while removing noise.Farsiu et al [33,36] proposed a bilateral total variation (BTV) regularization prior algorithm, which used the same smoothing coefficient to process adjacent pixels at the same position, and smoothed the images in all directions while retaining certain edge information. In [5,37], the authors formulated a new time dependent convolutional model for super resolution based on the TV regularization. And in [38], a spatially weighted TV image super-resolution algorithm is proposed, in which the spatial information distributed in different image regions is added to constrain the super-resolution process. In addition, Li et al. [39] further improved the above algorithm based on TV regularization. They proposed a new regularization term named steering kernel regression total variation (SKRTV), in which the local structural regularity properties can be fully exploited. Furthermore, in [32,40], a partial differential equation (PDE) model is proposed for SR based on a constrained variational model which uses the nonlocal total variation (NLTV) as a regularization term. As known from these algorithms, TV regularization has a good effect on retaining image edge information [36,41].

However, these TV-based regularization methods act only on the space dimension and does not take into account the dependence of the time dimension [42,43]. And there are many images with temporal similarities between frames, such as sequence images [44] and medical CT images [45,46]. In addition, for the time dimension in TV regularization, Li et al. [47] proposed that one can directly treat the time component as an additional space variable and form the corresponding seminorm. In this form, all dimensions are coupled, and the expected behavior is uniform in all directions.The time component is directly regarded as a spatial variable, which ignores the similarity in the time dimension and the difference between the temporal variable and the spatial variable. Without considering time and space separately, picture details and edge information will be ignored. Therefore, considering the differing behaviors of the spatial and temporal components, we decouple the spatial dimensions from the temporal one when constructing TV2++. Moreover, TV2++ could allow for more flexibility in penalizing. Admittedly, TV regularization is so susceptible to noise that often causes staircase effects. To weaken this flaw, we added a global low-rank constraint on the image, as Feng et al. [10] did. The global low-rank constraint in the model is to enhance global consistency and improve noise immunity. In addition, we introduce the exponential-type penalty (ETP) norm [48] to constrain the image with low-rank. Non-convex penalty functions such as ETP are extended on singular values of a matrix as surrogate functions of -norm to enhance low-rank matrix recovery more effectively, which has been proven in [49]. The ETP is characterized by a positive parameter, which establishes a connection with the 0 and 1 penalties.

In this paper, we introduce the ETP global low-rank regularization [48,50] and design a TV regularization on the space dimension and the time dimension (TV2++) to construct the basic framework of the algorithm. Overall, our contributions are threefold:

  • The algorithm, which designs TV2++ regularization based on spatio-temporal, utilizes image time dimension similarity and further strengthens image detail recovery.

  • A novel framework based on TV2++ regularization and ETP norm low-rank regularization is constructed; TV2++ regularization helps retain image edge information; ETP norm low-rank regularization does well in global consistency and improving noise immunity.

  • We use an efficient alternating direction method of multipliers (ADMM) [51]-based approach for problem optimization.

Background

In this part, we first introduce a basic model in the TV methods, and then we introduce the exponential-type penalty function and IRNN algorithm for solving the nonconvex low-rank minimization problem.

Basic model

We first suggest the basic model in the TV methods. In this model, we take noise and blur into consideration. Then, we solve the SR problem through this model. The basic observation model is:

$$\begin{array}{@{}rcl@{}} T=DSX+N \end{array} $$
(1)

where T means the image we observed and X is the HR image we expect to recover. D symbolizes the down-sampling operator and S is the blurring operator. At the same time, N represents the noise which influencing our observation results.

We want the HR image to be as perfect as possible.In mathematics, we can get the optimal solution in the basic model. So, we deform the above model to build a new cost function as follows:

$$\begin{array}{@{}rcl@{}} \min_{X}\quad \left\|DSX-T\right\|^{2} \end{array} $$
(2)

When we solve the optimization problem, then we can get the optimal X, which we can define as X0. The cost function consist of one penalizing term about LR image T and HR image X.

To better solve this ill-posed problem, we can add regularization terms.

$$\begin{array}{@{}rcl@{}} \min_{X}\quad \left\|DSX-T\right\|^{2}+\lambda \rho(X) \end{array} $$
(3)

where ρ(X) means the regularization term. This term usually depends on prior knowledge and we will illustrate this term in the following part. λ is the parament controlling the weigh of regularization terms in the whole model.

Nonconvex low-rank minimization

Variable selection problems are typically addressed under a penalized optimization framework. Nonconvex penalties such as the exponential-type penalty (ETP) [48,52] have been demonstrated to have the properties of sparsity practically and theoretically. The exponential-type penalty is characterized by a positive parameter, which establishes a connection with the and penalties. More specifically,the limits of ETP are the 0 and 1 penalties when this parameter approaches and 0 respectively. And it is easy to extend the exponential-type penalty functions on singular values of a matrix as surrogate functions of -norm to enhance low-rank matrix recovery. In addition, matrix completion algorithm [53] is based on the assumption that recovered matrix has low rank and then the missing entries can be filled in by minimizing the difference between the input matrix and the estimated one.

However, different from convex optimization, solving the nonconvex low-rank minimization problem is much more challenging than the nonconvex sparse minimization problem. Iteratively reweighted least squares (IRLS) algorithm [54] has been recently extended to handle the nonconvex Schatten-p norm penalty. Actually, it solves a relaxed smooth problem which may require many iterations to achieve a low-rank solution. It cannot solve the general nonsmooth problem.

Recently, Lu et al. [49] propose an iteratively reweighted nuclear norm (IRNN) algorithm to solve the nonconvex nonsmooth low-rank minimization problem. The IRNN algorithm takes advantage of the following properties of exponential-type penalty function: the exponential-type penalty function is concave and monotonically increasing on [0,], and its gradient is a decreasing function. IRNN algorithm iteratively solves a weighted singular value thresholding (WSVT) problem. By setting the weight vector as the gradient of the concave penalty function, the WSVT problem has a closed form solution.

$$ \begin{aligned} X^{k+1}&=\arg\min_{X}\ \sum_{i=1}^{m}w_{i}^{k}\sigma_{i} +f\left(X^{k}\right) +\langle \nabla f\left(X^{k}\right),X-X^{k}\rangle +\frac{\mu}{2}\left\|X-X^{k}\right\|_{F}^{2}\\ &=\min_{X}\sum_{i=1}^{m}\sigma_{i}+\frac{\mu}{2}\left\|X-\left(X^{k}-\frac{1}{\mu}\nabla f\left(X^{k}\right)\right)\right\|_{F}^{2} \end{aligned} $$
(4)

where w is the weight coefficient, σ is the singular value, and f(Xk) is the super gradient.

Our method

In this part, we will propose our own model, namely LRTV2++. As we know, rank is the important property, often calculated by the minimum of columns and rows containing non-zero elements. Rank is often thought about the richness of information contained in images. In regularization, if a matrix has low rank property, we can recover some losing columns or rows linearly expressed by some known rows and columns. This theory is widely used in matrix completion. We often represent the low rank function about matrix X from the certain norm, in this paper, we use ETP function to approximate L0 norm.

$$\begin{array}{@{}rcl@{}} {R(X)=\lambda_{1} \sum_{i=1}^{N}\alpha_{i} \left\|X\right\|_{ETP}} \end{array} $$
(5)

where X is the number of dimensions, and αi is the weigh. Especially, αi satisfies two constraints:

$$\begin{array}{@{}rcl@{}} \alpha_{i} \geq 0;\sum_{i=1}^{N}\alpha_{i}=1 \end{array} $$
(6)

Gao et al. [17] proposed a non-convex continuous surrogate function called exponential type penalty (ETP), as follows:

$$\begin{array}{@{}rcl@{}} \left\|X\right\|_{ETP}=\frac{\lambda}{1-e^{-\gamma}}\left(1-e^{-\gamma X}\right) \end{array} $$
(7)

And its super gradient is defined as:

$$\begin{array}{@{}rcl@{}} \partial \left\|X\right\|_{ETP}=\frac{\lambda \gamma}{\left(1-e^{-\gamma}\right)} e^{-\gamma X} \end{array} $$
(8)

As we know, TV function on the vertical horizontal is usually defined as:

$$\begin{array}{@{}rcl@{}} \text{TV}(X)={\iint}_{\Omega} \mid \nabla X \mid dxdy \end{array} $$
(9)

where Ω is spatial domain, and ΩR2. And then, we consider temporal dependency and get the difference term of the time dimension tX. Its formula is as follows:

$$\begin{array}{@{}rcl@{}} \partial_{t} X=X_{t1}-X_{t2} \end{array} $$
(10)

Therefore, TV2++ can be expressed as:

$$\begin{array}{@{}rcl@{}} \lambda_{1}\left\|\sqrt{\left(\partial_{x} X\right)^{2}+\left(\partial_{y} X\right)^{2}}\right\|_{1}+\lambda_{2} \left\|\partial_{t} X\right\|_{1} \end{array} $$
(11)

The first of Eq. (11)is the spatial regularization term, and the latter is the temporal regularization term. Then, adding the ETP low rank regularization and TV2++ regularization, the super-resolution reconstruction model can be further expressed as:

$$ \begin{aligned} &\min_{x}\left\|DSX-T\right\|^{2}+\alpha \left\|X\right\|_{ETP}\\ &+\lambda_{1}\left\|\sqrt{\left(\partial_{x} X\right)^{2}+\left(\partial_{y} X\right)^{2}}\right\|_{1} +\lambda_{2}\left\|\partial_{t} X\right\|_{1} \end{aligned} $$
(12)

In this method, we separate tX from xX and yX, separating spatial dimensions from temporal dimensions, considering the difference of the temporal dimension and spatial dimensions.

The optimization

To solve the optimization problem of the cost function Eq. (11), we use the alternating direction method of multipliers (ADMM) algorithm. ADMM is proved to be an efficient algorithm and a widely used optimization method for constrained problems when a cost function includes multiple variables. This algorithm solve one variable through other dual variables and iterate this variable to solve other variables. Repeat this process, then we can complete the optimization problem of our cost functions. We will show how to use ADMM to optimize our cost functions in the following contents.

$$ \begin{aligned} \min_{x}&\left\|DSX-T\right\|^{2}\,+\,\alpha\left\|M\right\|_{ETP}\\ &\,+\,\lambda_{1}\left\| \sqrt{\left(\partial_{x} X\right)^{2}\,+\,(\partial_{y} X)^{2}}\right\|_{1}\,+\,\lambda_{2}\left\|U\right\|_{1}\\ \mathrm{s.t.}\quad & X=M; \\ & \partial_{t} X=U; \end{aligned} $$
(13)

where M and U are the auxiliary variables. Further derivation of Eq. (13), the augmented Lagrangian function can be obtained as follows:

$$ \begin{aligned} &L(X,M,\psi)=\left\|DSX-T\right\|^{2}+\alpha\left\|M\right\|_{ETP}\\ &+\lambda_{1}\left\|\sqrt{\left(\partial_{x} X\right)^{2}+\left(\partial_{y} X\right)^{2}}\right\|_{1}+\lambda_{2}\left\|U\right\|_{1} +\langle \psi_{1},X-M\rangle\\ &+\frac{\rho_{1}}{2}\left\|X-M\right\|_{F}^{2}+\langle \psi_{2},\partial_{t} X-U\rangle +\frac{\rho_{2}}{2}\left\|\partial_{t} X-U\right\|_{F}^{2} \end{aligned} $$
(14)

Sub-problem 1: updating X

$$ \begin{aligned} \min_{X}&\left\|DSX-T\right\|^{2}+\lambda_{1}\left\|\sqrt{\left(\partial_{x} X\right)^{2}+\left(\partial_{y} X\right)^{2}}\right\|_{1}\\ &+\langle \psi_{1},X-M\rangle+\frac{\rho_{1}}{2}\left\|X-M\right\|_{F}^{2}+\langle \psi_{2},\partial_{t} X-U\rangle \\ &+\frac{\rho_{2}}{2}\left\|\partial_{t} X-U\right\|_{F}^{2} \end{aligned} $$
(15)

equivalent to:

$$ \begin{aligned} &\left\|DSX-T\right\|^{2}+\lambda_{1}\left\|\sqrt{\left(\partial_{x} X\right)^{2}+\left(\partial_{y} X\right)^{2}}\right\|_{1} \\ &+\frac{\rho_{1}}{2}\left\|X-M+\frac{\psi_{1}}{\rho_{1}}\right\|_{F}^{2}+\frac{\rho_{2}}{2}\left\|\partial_{t} X-U+\frac{\psi_{1}}{\rho_{2}}\right\|_{F}^{2} \end{aligned} $$
(16)

Take the derivative of the above function :

$$ \begin{aligned} &\frac{X^{n+1}-X^{n}}{dt}=2\cdot S^{T} \otimes D^{T}(DSX-T)\\ &+\lambda_{1} \frac{\partial_{x} X\cdot \partial_{x}+\partial_{y} X\cdot\partial_{y}}{\sqrt{\left(\partial_{x} X\right)^{2}+\left(\partial_{y} X\right)^{2}}}+\rho_{1} \left(X-M+\frac{\psi_{1}}{\rho_{1}}\right)\\ &+\rho_{2} \cdot \partial_{t}^{T} \left(\partial_{t} X-U+\frac{\psi_{2}}{\rho_{2}}\right) \end{aligned} $$
(17)

From Eq. (17), we can easily get the equation of X in TV2++ method:

$$ \begin{aligned} X^{n+1}&=X^{n}+dt(2\cdot S^{T} \otimes D^{T}(DSX-T)\\ &+\lambda_{1} \frac{\partial_{x} X\cdot \partial_{x}+\partial_{y} X\cdot\partial_{y}}{\sqrt{\left(\partial_{x} X\right)^{2}+\left(\partial_{y} X\right)^{2}}}+\rho_{1} \left(X-M+\frac{\psi_{1}}{\rho_{1}}\right) \\ &+\rho_{2} \cdot \partial_{t}^{T} \left(\partial_{t} X-U+\frac{\psi_{2}}{\rho_{2}}\right) \end{aligned} $$
(18)

Sub-problem 2: updating M

$$\begin{array}{@{}rcl@{}} \min_{M} \alpha \left\|M\right\|_{ETP}+\langle \psi_{1},X-M\rangle +\frac{\rho_{1}}{2}\left\|X-M\right\|_{F}^{2} \end{array} $$
(19)

equivalent to:

$$\begin{array}{@{}rcl@{}} \min_{M} \alpha \left\|M\right\|_{ETP} +\frac{\rho_{1}}{2}\left\|X-M+\frac{\psi_{1}}{\rho_{1}}\right\|_{F}^{2} \end{array} $$
(20)

It can be further transformed into:

$$\begin{array}{@{}rcl@{}} \min_{M} \frac{\alpha}{\rho_{1}} \sum_{i=1}{r} w_{i} \sigma_{i} (M)+\frac{1}{2}\left\|M-\left(X+\frac{\psi_{1}}{\rho_{1}}\right) \right\|_{F}^{2} \end{array} $$
(21)

where,

$$ \begin{aligned} w_{i}&=\frac{\xi \kappa}{1-exp(-\kappa)} exp\left[-\kappa \sigma_{i}(M)\right],\\ M&=USV^{T}, \sigma_{i}(M)=Diag(S_{i}). \end{aligned} $$
(22)

As we all know, a globally optimal solution to the above problem is given by the weighted singular value thresholding:

$$\begin{array}{@{}rcl@{}} M^{i+1}=US\left(\sum\right)V^{T} \end{array} $$
(23)

where,

$$\begin{array}{@{}rcl@{}} Y=\left(X+\frac{\psi_{1}}{\rho_{1}}\right) =U\sum V^{T} \end{array} $$
(24)
$$\begin{array}{@{}rcl@{}} S(\sum)=Diag\left[max\left(\sum_{ii} -\frac{\alpha}{\rho_{1}} w_{i},0 \right) \right] \end{array} $$
(25)

Eq. (24) is the SVD of Y.

Sub-problem 3: update U

$$\begin{array}{@{}rcl@{}} \min_{U} \lambda_{2}\left\|U\right\|+\langle \psi_{2},\partial_{t} X-U\rangle+\frac{\rho_{2}}{2}\left\|\partial_{t} X-U\right\|_{F}^{2} \end{array} $$
(26)

equivalent to:

$$\begin{array}{@{}rcl@{}} \min_{U} \lambda_{2}\left\|U\right\|+\frac{\rho_{2}}{2}\left\|\partial_{t} X-U+\frac{\psi_{2}}{\rho_{2}}\right\|_{F}^{2} \end{array} $$
(27)

which can be solved efficiently via a soft shrinkage operator:

$$\begin{array}{@{}rcl@{}} U=shrink\left(\partial_{z} X+\frac{\psi_{2}}{\rho_{2}},\frac{\lambda_{2}}{\rho_{2}}\right) \end{array} $$
(28)

where,

$$\begin{array}{@{}rcl@{}} shrink(\zeta,\mu)=\frac{\zeta}{|\zeta|}*max(\zeta-\mu,0) \end{array} $$
(29)

Experimental results and analysis

We take CT images of the HNSCC-3DCT-RT dataset as original images to perform the experiment. And the low-resolution images are all obtained by MATLAB simulation. There are four comparison algorithms used in the experiment: L1TV, L2TV, NLM [12], and LRTV [10]. In addition, we employed two important characters to estimate the recovery quality: peak signal-to-noise ratio (PSNR) and structure similarity index (SSIM). PSNR is widely used to evaluate quality of SR recovery and its unit is decibels (dB). The mathematical formula of PSNR is defined as:

$$\begin{array}{@{}rcl@{}} PSNR=10*{log}_{10}\lgroup \frac{(2^{n} -1)^{2}}{MSE}\rgroup \end{array} $$
(30)

where MSE is the mean square error between original HR image and recovered HR image. Aside from PSNR, SSIM is another important character we used to evaluate the similarity between images. SSIM is usually considered to accord with the visual perception of human naked eyes. Its mathematical formula can be written as:

$$\begin{array}{@{}rcl@{}} SSIM(x,y)=\frac{\left(2\mu_{x} \mu_{y}+c_{1}\right)\left(2\sigma_{xy}+c_{2}\right)}{\left(\mu_{x}^{2}+\mu_{y}^{2}+c_{1}\right)\left(\sigma_{x}^{2}+\sigma_{y}^{2}+c^{2}\right)} \end{array} $$
(31)

where μx is the average value of the original HR image x and μy is the average value of the recovered y, σx is the variance of x and σx is the variance of y. σxy is the covariance of original HR image and recovered image. c1 and c2 are the balance parameters. SSIM is ranged from 0 to 1. When the recovered image is exactly same to original HR image, SSIM equals 1.

Performance under noise environment

In this part, we add Gaussian noise to the images (4-HEADNECK 2.0 B30s-50686) in order to test the robustness of super-resolution reconstruction. Gaussian noise is a special kind of noise whose probability density function obeys normal distribution. We set mean of Gaussian noise to be 0 and its variance to be 0.001, 0.003, 0.005, 0.006, 0.007. Each algorithm is tested in the same experimental environment, and the experimental results are shown in curves charts.

Figure 1 and Fig. 2 show the PSNR and SSIM trends for all algorithm results as the variance of Gaussian noise increases from 0.001 to 0.007, where sigma represents the variance of the Gaussian noise. In Fig. 1 and Fig. 2, we can see that the proposed method has overwhelmingly advantage compared with other algorithms. When sigma increased gradually, the advantage tends to emerge. We can find that the index data curve of the proposed algorithm is much better than the others, which means that the proposed algorithm has better image super-resolution recovery ability.

Fig. 1
figure1

PSNR of different methods with increasing noise variance from 0.001 to 0.007

Fig. 2
figure2

SSIM of different methods with increasing noise variance from 0.001 to 0.007

Figure 3 shows recovered images of different methods and enlarged regions. Compared to original HR image, recovered images of L1TV, L2TV, and NLM appear to be blurred while the recovered image of the proposed method is clearer and the details are better restored. This is because we enhance the efficiency of low rank regular terms by replacing the tracking norm with ETP non-convex penalty function constraints. In addition, the TV2++ regular term utilizes the temporal dimensional similarity of the image to further enhance the detail recovery capability of the image.

Fig. 3
figure3

The performance of recovered images of different methods and enlarged regions (4-HEADNECK 2.0 B30s-50686)

Recovery performance comparison

In this part, we will use more medical images from our dataset and show results by using different methods. We mainly show two other CT images in different part of bodies to evaluate the recovery performance of our proposed methods. We first chose one 3D CT of 12-08-2009-RTHEADNECK Adult-25471 series. By using the above methods, we will put results and recovered image and close-up view of selected regions in each image in the following part for comparison. From the Table 1, we can see that the proposed method gets best PSNR and SSIM index values. The recovered images and details in selected location are shown in Fig. 4. We can find that the result of the proposed method is clearer than other methods.

Fig. 4
figure4

The performance of recovered images of different methods and enlarged regions (12-08-2009-RTHEADNECK Adult-25471)

Table 1 PSNR and SSIM index values of each method for image (12-08-2009-RTHEADNECK Adult-25471)

When we compare these results to original image, we also find the proposed method is more similar to original image, resulting in high SSIM.

We further chose HEADNECK CT images to test and verify experimental results in the subsequent part. Processed images are shown in Fig. 5. We also show specific data of their PSNR and SSIM in the Table 2.

Fig. 5
figure5

The performance of recovered images of different methods and enlarged regions(HEADNECK)

Table 2 PSNR and SSIM index values of each method for image (HEADNECK)

We also tested other images in our database and got similar results to what we show by testing two examples. In general, the proposed method gets highest SSIM and PSNR than all other comparison methods. In addition, under blur interference, our proposed method is clearer than other methods.By selecting a close-up view of the location, this method displays the details better than other methods.

Ablation analysis

We use adult images 4-HEADNECK 2.0 B30s-50686 as the test image for parameter optimization experiments. For the parameter of low-rank term, we set \(\alpha _{1}=\alpha _{2}=\frac {1}{2}\), which means weigh controlling each matrix unfolded in each dimensions plays the same role. We estimate the error for each iteration by calculating XkXk−1. When the above error is less than 1e−5, we stop the optimization of the parameters.

First, we set λ2 to 0.0001. We experiment with λ2 being set between 0.0001 and 0.1, respectively. Through running our observed Fig. 6, it can be seen that the value of λ1 is near 0.001 when the PSNR reaches a peak. Continuing to increase the value of λ1, the PSNR shows a monotonous decreasing trend. As can be seen from Fig. 7, when λ1 increases between intervals 0 and 0.1, the value of SSIM shows a tendency to decrease slightly. Taking into account the performance of the above indicators, we believe that the experimental result obtained is the best when the value of λ1 is set to 0.009.

Fig. 6
figure6

PSNR of λ1 in LRTV2++ in parameter optimization

Fig. 7
figure7

SSIM of λ1 in LRTV2++ in parameter optimization

Then, we fixed the parameter λ1 and performed a parameter optimization experiment on the parameter λ2. We experiment with λ2 being set between 0.0001 and 0.1, respectively. Through running our program, we acquire the value of PSNR and SSIM when λ2 is 0.0001, 0.0002, 0.0005, 0.0007, 0.01, 0.1, and 0.5. And the line charts are shown in Fig. 8 and Fig. 9.

Fig. 8
figure8

PSNR of λ2 in LRTV2++ in parameter optimization

Fig. 9
figure9

SSIM of λ2 in LRTV2++ in parameter optimization

As can be seen from Fig. 8, the line chart has both a rising interval and a falling interval, and when λ2 is around 0.1, the PSNR value reaches a peak. For the SSIM in Fig. 9, it can be seen that as λ2 increases, the value of SSIM continues to decrease. Taking into account the performance of the above indicators, we believe that the experimental result obtained is the best when the value of λ2 is set to 0.1.

Discussion

In this paper, we propose a super-resolution reconstruction algorithm based on TV2++ regularization and ETP norm low-rank regularization and optimize it by ADMM method. TV2++ regularization is based on spatio-temporal and utilizes image time dimension similarity. Considering the differing behaviors of the spatial and temporal components, we decouple the spatial dimensions from the temporal one which also allows for more flexibility in penalizing. On the other hand, ETP norm are extended on singular values of a matrix as surrogate functions of L0-norm to enhance low-rank matrix recovery more effectively. Then, the experimental results reveals that the proposed method has better capacity of resisting noise and blur. In addition, our model is more complex than NLM, L1TV, and L2TV, although we use the ADMM algorithm for optimization. But it still takes more time than NLM, L1TV, and L2TV. The reason may be that the program takes a lot of time to solve the TV problem.

Availability of data and materials

Not applicable

Abbreviations

TV:

Total variation

ETP:

Exponential-type penalty

ADMM:

Alternating direction method of multipliers

CT:

Computed tomography

IBP:

Iterative back projection method

NLM:

Non-local means

MAP:

Maximum a posteriori

BTV:

Bilateral total variation

SKRTV:

Steering kernel regression total variation

PDE:

Partial differential equation

NLTV:

Nonlocal total variation

SR:

Super-resolution

HR:

High resolution

LR:

Low resolution

IRLS:

Iteratively reweighted least squares

IRNN:

Iteratively reweighted nuclear norm

WSVT:

Weighted singular value thresholding

SVD:

Singular value decomposition

MATLAB:

Matrix laboratory

L1TV:

L1 norm and total variation

L2TV:

L2 norm and total variation

LRTV:

Low-rank and total variation

PSNR:

Peak signal-to-noise ratio

SSIM:

Structure similarity index

dB:

Decibels

MSE:

Mean square error

References

  1. 1

    J. Jiang, R. Hu, Z. Han, Z. Wang, Low-resolution and low-quality face super-resolution in monitoring scene via support-driven sparse coding. J Signal Proc. Syst.75(3), 245–256 (2014). doi:10.1007/s11265-013-0804-9.

    Article  Google Scholar 

  2. 2

    C. You, Y. Zhang, X. Zhang, G. Li, S. Ju, Z. Zhao, Z. Zhang, W. Cong, P. K. Saha, G. Wang, CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE). IEEE Trans. Med. Imaging. 39(1), 188–203 (2019). doi:10.1109/TMI.2019.2922960.

    Article  Google Scholar 

  3. 3

    M. Sugie, S. Gohshi, H. Takeshita, C. Mori, in 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS). Subjective assessment of super-resolution 4k video using paired comparison, (2014), pp. 042–047. doi:10.1109/ISPACS.2014.7024422.

  4. 4

    Z. Pan, J. Yu, H. Huang, S. Hu, A. Zhang, H. Ma, W. Sun, Super-resolution based on compressive sensing and structural self-similarity for remote sensing images. IEEE Trans. Geosci. Remote Sens.51(9), 4864–4876 (2013). doi:10.1109/TGRS.2012.2230270.

    Article  Google Scholar 

  5. 5

    G. Xu, S. Khan, H. Zhu, L. Han, M. K. Ng, H. Yan, Discriminative tracking via supervised tensor learning. Neurocomputing. 315:, 33–47 (2018). doi:10.1016/j.neucom.2018.05.108.

    Article  Google Scholar 

  6. 6

    J. Zhou, J. Sun, P. Cong, Z. Liu, S. Hu, Security-critical energy-aware task scheduling for heterogeneous real-time MPSoCs in IoT. IEEE Trans. Serv. Comput.13(4), 745–758 (2020). doi:10.1109/TSC.2019.2963301.

    Article  Google Scholar 

  7. 7

    X. Xu, R. Mo, F. Dai, W. Lin, S. Wan, W. Dou, Dynamic resource provisioning with fault tolerance for data-intensive meteorological workflows in cloud. IEEE Trans. Ind. Inform.16(9), 6172–6181 (2020). doi:10.1109/TII.2019.2959258.

    Article  Google Scholar 

  8. 8

    L. Qi, Y. Chen, Y. Yuan, S. Fu, X. Xu, A QoS-aware virtual machine scheduling method for energy conservation in cloud-based cyber-physical systems. World Wide Web. 23(2), 1275–1297 (2020). doi:10.1007/s11280-019-00684-y.

    Article  Google Scholar 

  9. 9

    Y. Xu, L. Qi, W. Dou, J. Yu, Privacy-preserving and scalable service recommendation based on simhash in a distributed cloud environment. Complexity. 2017:, 1–9 (2017). doi:10.1155/2017/3437854.

    Google Scholar 

  10. 10

    F. Shi, J. Cheng, L. Wang, P. T. Yap, D. Shen, LRTV: MR image super-resolution with low-rank and total variation regularizations. IEEE Trans. Med. Imaging. 34(12), 2459–2466 (2015). doi:10.1109/TMI.2015.2437894.

    Article  Google Scholar 

  11. 11

    P. Rasti, H. Demirel, G. Anbarjafari, in 2014 22nd Signal Processing and Communications Applications Conference (SIU). Improved iterative back projection for video super-resolution, (2014), pp. 552–555. doi:10.1109/SIU.2014.6830288.

  12. 12

    J. Manjon, P. Coupé, A. Buades, V. Fonov, L. Collins, M. Robles, Non-local MRI upsampling. Med. Image Anal.14(6), 784–792 (2010). doi:10.1016/j.media.2010.05.010.

    Article  Google Scholar 

  13. 13

    Y. He, K. -H. Yap, L. Chen, L. -P. Chau, Blind super-resolution image reconstruction using a maximum a posteriori estimation, (2006). doi:10.1109/ICIP.2006.312715.

  14. 14

    Z. Lu, C. Wu, D. Chen, Y. Qi, C. Wei, in The 26th Chinese Control and Decision Conference (2014 CCDC). Overview on image super resolution reconstruction, (2014), pp. 2009–2014. doi:10.1109/CCDC.2014.6852498.

  15. 15

    L. Y. Zhou, S. U. Cai-Xia, Y. F. Cao, Image super-resolution via sparse representation. IEEE Trans. Image Process.19(11), 2861–2873 (2010). doi:10.1109/TIP.2010.2050625.

    MathSciNet  Article  Google Scholar 

  16. 16

    R. Timofte, V. DeSmet, L. VanGool, in Asian Conference on Computer Vision. A+: adjusted anchored neighborhood regression for fast super-resolution, (2014), pp. 111–126. doi:10.1007/978-3-319-16817-3\_8.

  17. 17

    J. B. Huang, A. Singh, N. Ahuja, in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Single image super-resolution from transformed selfexemplars, (2015), pp. 5197–5206. doi:10.1109/CVPR.2015.7299156.

  18. 18

    S. Khan, M. Nawaz, G. Xu, H. Yan, Image correspondence with CUR decomposition based graph completion and matching. IEEE Trans. Circ. Syst. Video Technol.30(9), 3054–3067 (2020). doi:10.1109/TCSVT.2019.2935838.

    Article  Google Scholar 

  19. 19

    J. Zhou, X. S. Hu, Y. Ma, J. Sun, S. Hu, Improving availability of multicore real-time systems suffering both permanent and transient faults. IEEE Trans. Comput.68(12), 1785–1801 (2019). doi:10.1109/TC.2019.2935042.

    MATH  Article  Google Scholar 

  20. 20

    D. Chao, C. L. Chen, K. He, X. Tang, in European Conference on Computer Vision. Learning a deep convolutional network for image super-resolution, (2014), pp. 184–199. doi:10.1007/978-3-319-10593-2\_13.

  21. 21

    H. Zhu, Y. Qiao, G. Xu, L. Deng, Y. Yu-Feng, DSPNet: a lightweight dilated convolution neural networks for spectral deconvolution with self-paced learning. IEEE Trans. Ind. Inform.16(12), 7392–7401 (2020). doi:10.1109/TII.2019.2960837.

    Article  Google Scholar 

  22. 22

    J. Zhou, J. Sun, X. Zhou, T. Wei, M. Chen, S. Hu, X. S. Hu, Resource management for improving soft-error and lifetime reliability of real-time MPSoCs. IEEE Trans. Comput. Aided Des. Integr. Circ. Syst.38(12), 2215–2228 (2018). doi:10.1109/TCAD.2018.2883993.

    Article  Google Scholar 

  23. 23

    X. Xu, C. He, Z. Xu, L. Qi, S. Wan, M. Z. A. Bhuiyan, Joint optimization of offloading utility and privacy for edge computing enabled IoT. IEEE Internet of Things J.7(4), 2622–2629 (2019). doi:10.1109/JIOT.2019.2944007.

    Article  Google Scholar 

  24. 24

    Y. -F. Yu, G. Xu, M. Jiang, H. Zhu, D. -Q. Dai, H. Yan, Joint transformation learning via the L2,1-norm metric for robust graph matching. IEEE Trans. Cybern., 1–13 (2019). doi:10.1109/TCYB.2019.2912718.

  25. 25

    J. Sun, Z. Xu, H. -Y. Shum, Gradient profile prior and its applications in image super-resolution and enhancement. IEEE Trans. Image Process.20(6), 1529–1542 (2010). doi:10.1109/TIP.2010.2095871.

    MathSciNet  MATH  Google Scholar 

  26. 26

    Y. Yu, D. Dai, C. Ren, K. Huang, Discriminative multi-scale sparse coding for single-sample face recognition with occlusion. Pattern Recognit.66:, 302–312 (2017). doi:10.1016/j.patcog.2017.01.021.

    Article  Google Scholar 

  27. 27

    H. Liu, H. Kou, C. Yan, L. Qi, Link prediction in paper citation network to construct paper correlation graph. EURASIP J. Wirel. Commun. Netw.2019(1), 1–12 (2019). doi:10.1186/s13638-019-1561-7.

    Article  Google Scholar 

  28. 28

    L. Qi, X. Zhang, S. Li, S. Wan, Y. Wen, W. Gong, Spatial-temporal data-driven service recommendation with privacy-preservation. Inf. Sci.515:, 91–102 (2020). doi:10.1016/j.ins.2019.11.021.

    Article  Google Scholar 

  29. 29

    L. Deng, H. Zhu, C. Tao, Y. Wei, Infrared moving point target detection based on spatial-temporal local contrast filter. Infrared Phys. Technol.76:, 168–173 (2016). doi:10.1016/j.infrared.2016.02.010.

    Article  Google Scholar 

  30. 30

    W. Zhong, X. Yin, X. Zhang, S. Li, W. Dou, R. Wang, L. Qi, Multi-dimensional quality-driven service recommendation with privacy-preservation in mobile edge environment. Comput. Commun.157:, 116–123 (2020). doi:10.1016/j.comcom.2020.04.018.

    Article  Google Scholar 

  31. 31

    A. Marquina, S. Osher, Image super-resolution by TV-regularization and Bregman iteration. J. Sci. Comput.37:, 367–382 (2008). doi:10.1007/s10915-008-9214-8.

    MathSciNet  MATH  Article  Google Scholar 

  32. 32

    W. Zeng, X. Lu, A robust variational approach to super-resolution with nonlocal TV regularisation term. Imaging Sci. J.61:, 268–278 (2013). doi:10.1179/1743131X11Y.0000000064.

    Article  Google Scholar 

  33. 33

    S. Farsiu, M. D. Robinson, M. Elad, P. Milanfar, Fast and robust multiframe super resolution. IEEE Trans. Image Process.13:, 1327–1344 (2004). doi:10.1109/TIP.2004.834669.

    Article  Google Scholar 

  34. 34

    X. Chi, C. Yan, H. Wang, W. Rafique, L. Qi, Amplified locality-sensitive hashing-based recommender systems with privacy protection. Concurr. Comput. Pract. Experience, e5681 (2020). doi:10.1002/cpe.5681.

  35. 35

    X. Yuan, L. Han, S. Qian, G. Xu, H. Yan, Singular value decomposition based recommendation using imputed data. Knowl.-Based Syst.163:, 485–494 (2019). doi:10.1016/j.knosys.2018.09.011.

    Article  Google Scholar 

  36. 36

    L. Qi, Q. He, F. Chen, X. Zhang, W. Dou, Q. Ni, Data-driven web APIs recommendation for building web applications. IEEE Trans. Big Data, 1–1 (2020). doi:10.1109/TBDATA.2020.2975587.

  37. 37

    A. Marquina, S. J. Osher, Image super-resolution by TV-regularization and Bregman iteration. J. Sci. Comput.37:, 367–382 (2008). doi:10.1007/s10915-008-9214-8.

    MathSciNet  MATH  Article  Google Scholar 

  38. 38

    Q. Yuan, L. Zhang, H. Shen, Multiframe super-resolution employing a spatially weighted total variation mode. IEEE Trans. Circ. Syst. Video Technol.22:, 379–392 (2012). doi:10.1109/TCSVT.2011.2163447.

    Article  Google Scholar 

  39. 39

    L. Li, Y. Xie, W. Hu, W. Zhang, Single image super-resolution using combined total variation regularization by split Bregman Iteration. Neurocomputing. 142:, 551–560 (2014). doi:10.1016/j.neucom.2014.02.045.

    Article  Google Scholar 

  40. 40

    G. Xu, H. Zhu, L. Deng, L. Han, Y. Li, H. Lu, Dilated-aware discriminative correlation filter for visual tracking. World Wide Web. 22:, 791–805 (2019). doi:10.1007/s11280-018-0555-4.

    Article  Google Scholar 

  41. 41

    J. Lu, B. Wu, Single-image super-resolution with joint-optimization of TV regularization and sparse representation. Optik. 125:, 2497–2504 (2014). doi:10.1016/j.ijleo.2013.10.093.

    Article  Google Scholar 

  42. 42

    H. Schaeffer, Y. Yi, S. Osher, Space-time regularization for video decompression. SIAM J. Imaging Sci.8:, 373–402 (2015). doi:10.1137/140977400.

    MathSciNet  MATH  Article  Google Scholar 

  43. 43

    L. Qi, X. Wang, X. Xu, W. Dou, S. Li, Privacy-aware cross-platform service recommendation based on enhanced locality-sensitive hashing. IEEE Trans. Netw. Sci. Eng., 1–1 (2020). doi:10.1109/TNSE.2020.2969489.

  44. 44

    X. J. Chen, G. Q. Han, Z. Li, X. Liao, in 2013 International Conference on Wavelet Analysis and Pattern Recognition. Image super-resolution via multi-resolution image sequence, (2013), pp. 178–183. doi:10.1109/ICWAPR.2013.6599313.

  45. 45

    Y. Wang, Q. Teng, X. He, J. Feng, T. Zhang, CT-image of rock samples super resolution using 3D convolutional neural network. Comput. Geosci.133:, 104314 (2019). https://doi.org/10.1016/j.cageo.2019.104314.

    Article  Google Scholar 

  46. 46

    X. Wang, L. T. Yang, Y. Wang, L. Ren, M. J. Deen, ADTT: a highly-efficient distributed tensor-train decomposition method for IIoT big data. IEEE Trans. Ind. Inform., 1–1 (2020). doi:10.1109/TII.2020.2967768.

  47. 47

    C. Li, W. Yin, H. Jiang, Y. Zhang, An efficient augmented lagrangian method with applications to total variation minimization. Comput. Optim. Appl.53:, 507–530 (2013). doi:10.1007/s10589-013-9576-1.

    MathSciNet  MATH  Article  Google Scholar 

  48. 48

    C. Gao, N. Wang, Q. Yu, Z. Zhang, in Aaai. A feasible nonconvex relaxation approach to feature selection, (2011), pp. 356–361. doi:10.5555/29004232900479.

  49. 49

    C. Lu, J. Tang, S. Yan, Z. Lin, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Generalized nonconvex nonsmooth low-rank minimization, (2014), pp. 4130–4137. doi:10.1109/CVPR.2014.526.

  50. 50

    H. Zhu, S. Liu, L. Deng, Y. Li, F. Xiao, Infrared small target detection via low-rank tensor completion with top-hat regularization. 1004–1016. 58: (2019). doi:10.1109/TGRS.2019.2942384.

  51. 51

    G. Ongie, M. Jacob, in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). A fast algorithm for structured low-rank matrix recovery with applications to undersampled MRI reconstruction, (2016), pp. 522–525. doi:10.1109/ISBI.2016.7493322.

  52. 52

    H. Liu, H. Kou, C. Yan, L. Qi, Keywords-driven and popularity-aware paper recommendation based on undirected paper citation graph. Complexity. 2020(1), 1–15 (2020). doi:10.1155/2020/2085638.

    Google Scholar 

  53. 53

    C. Fu, X. Ji, Y. Zhang, Q. Dai, in Data Compression Conference Proceedings. A single frame super-resolution method based on matrix completion, (2012), pp. 297–306. doi:10.1109/DCC.2012.36.

  54. 54

    K. Mohan, M. Fazel, in 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton). Iterative reweighted least squares for matrix rank minimization, (2010), pp. 653–661. doi:10.1109/ALLERTON.2010.5706969.

Download references

Acknowledgements

Not applicable

Funding

This work is supported by the National Natural Science Foundation of China under Grant 61701259, 61572503, 61872424, 6193000388, and NUPTSF (Grant No. NY218001).

Author information

Affiliations

Authors

Contributions

The authors have contributed jointly to all parts on the preparation of this manuscript, and all authors read and approved the final manuscript.

Corresponding author

Correspondence to Bing-Kun Bao.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Deng, L., Zhou, Z., Xu, G. et al. TV2++: a novel spatial-temporal total variation for super resolution with exponential-type norm. J Wireless Com Network 2020, 223 (2020). https://doi.org/10.1186/s13638-020-01815-0

Download citation

Keywords

  • ADMM
  • ETP
  • Super-resolution
  • TV2++