Visual sensor image enhancement based on non-sub-sampled shearlet transform and phase stretch transform

Abstract

Acquiring clear images is a requisite in visual sensor networks. Image enhancement is an effective way to improve image quality. In this paper, non-sub-sampled shearlet transform (NSST) multi-scale analysis is combined with phase stretch transform (PST) to nonlinearly enhance the images captured by visual sensors. The components of different scales after NSST multi-scale decomposition are processed by nonlinear models with different thresholds. The thresholds of the enhanced model are determined by the local standard deviation of PST feature map. The noise is well suppressed, and the detail features are enhanced obviously. Experiments show that the proposed algorithm can improve image distortion, clear details, and enhance image contrast effectively.

1 Introduction

Wireless sensor network technology has developed rapidly in recent years [1,2,3]. It has been widely used in military, intelligent transportation, medical care, and other fields [4, 5]. Visual sensor is a kind of commonly used sensor in wireless sensor networks. Visual sensor is an instrument that uses optical elements to obtain image information of external environment. A visual sensor network is constructed by optical sensor devices distributed in space. Visual sensor networks are very helpful in applications involving environmental monitoring, area detection, and trail tracking. The purpose of computer vision is to extract meaningful descriptive information from images or image sequences. So the images obtained by sensor networks must be clear and easy to be identified. While in the process of acquiring digital images in visual sensor networks, it often leads to blurred images, low contrast, and difficult identification due to some objective factors, such as illumination, camera angles, and imaging characteristics of sensors.

Image enhancement is a crucial pre-treatment step for many image processing applications, because it is an effective way to improve the image quality. Image enhancement refers to the use of certain methods that can highlight the interested features in an image according to some specific needs, meanwhile weakening or removing irrelevant information. By this way, it can transform the image into a more suitable style for human eye recognition and computer perception.

Image enhancement algorithms can be roughly grouped into two kinds: space domain image enhancement and frequency domain image enhancement. Space domain image enhancement algorithm is mainly based on grayscale transformation, histogram processing [6], mean filtering, and other methods. Its basic principle is to adjust the gray value of images in order to achieve local contrast enhancement and noise suppression. Frequency domain image enhancement is also called transform domain image enhancement. Its basic idea is to use multi-resolution analysis tools to modify the transform coefficients at different scales of the image so as to indirectly remove noise and enhance details of some interested features. Nowadays, the transform domain image enhancement algorithm has gradually become a research hotspot. Common multi-scale analysis tools for image enhancement include wavelet transform [7], contourlet transform, and shearlet transform. Literature [8] proposed an algorithm for image enhancement based on non-sub-sampled contourlet transform (NSCT) and adaptive threshold. Literature [9] combined contrast limited adaptive histogram equalization and wavelet transform to enhance the local details of an image. Shearlet transform appeared in recent years. It has good results in image processing applications, such as image fusion, enhancement, and denoising. In our early research, we have applied non-sub-sampled shearlet transform (NSST) to enhance images adaptively and fuse images in compressive domain [10, 11].

Image phase information plays a crucial role in the procedure of human perception of images. Phase information can reflect the space information of images and has a strong ability to interpret images. Phase correlation is often applied to image registration [12, 13]. Literature [14] proposed a phase information-based contrast enhancement scheme that can transform phase changes into amplitude variations so as to reveal the inhibited structural details in high-frequency components.

From the above analysis, it is supposed that if we can combine the accuracy of multi-scale analysis with the stability of phase analysis in the process of image enhancement, we will get a more ideal enhancement performance. In this paper, we propose an image enhancement method which takes the advantages of NSST and phase stretch transform (PST) to nonlinearly enhance the images captured by visual sensors. Experiments show that the enhanced images processed by our algorithm have clear details and contrast without any artifacts.

2 Design of the algorithm

2.1 NSST multiscale analysis

NSST is a multi-scale and multi-directional analysis tool. It is an optimal approximation, which is a “true” sparse representation of the image in all directions and at various scales. In addition, unlike traditional shearlet transform, NSST does not perform down-sampling when performing multi-scale decomposition of an image. It combines a non-down-sampling tower transform with a non-sub-sampled directional filter bank and transforms. The size of the directional sub-bands of each scale is the same as that of the original image. The transform has shift invariance and can overcome the pseudo-Gibbs phenomenon when the image is reconstructed.

2.2 Feature extraction with PST

PST is a new image processing method proposed by Asghari and Jalali [15]. It is said that PST has superior performance in feature extraction, especially edge extraction. The PST processing can be divided into several steps: Firstly, smooth the image with a low pass filter to reduce noise. Secondly, apply the PST kernel which has a non-linear frequency-dependent transfer function to the image. The output of the transform is the phase in space domain. The applied phase is frequency dependent with the original image, it means that higher amount of phase corresponds to higher frequency image features. Finally, find image sharp transitions by thresholding the phase and apply binary morphological operations, so as to enhance the features and clean the transformed image. PST is defined in frequency domain [15] as (1).

$$ A\left[n,m\right]=\angle \left\langle IFFT2\;\left\{\tilde{K}\left[p,q\right]\cdot \tilde{L}\left[p,q\right]\cdot FFT2\left\{B\left[n,m\right]\right\}\right\}\right\rangle $$

(1)

where n and m are the space domain variables, A[n,m] is the output phase image, B[n,m] is the input image, FFT2 is the two-dimensional fast Fourier transform, IFFT2 is the two-dimensional inverse fast Fourier transform. p and q are the frequency domain variables. The function $\tilde{K}\left[p,q\right]$ is the warped phase kernel, and $\tilde{L}\left[p,q\right]$ is the frequency response of the localization kernel.

2.3 Nonlinear enhancement model

Image information can be roughly classified into several parts, such as edges, details, background, and noise. The edges of image can be subdivided into sharp edges and fuzzy edges. When selecting the enhancement model, we need to take into account the features of different kinds of information. It is appropriate to preserve clear edges, enhance fuzzy edges, and suppress noise effectively. Therefore, nonlinear function is a good choice. The enhancement function selected in this paper was presented by A. F. Laine. After that, some researchers applied this function to image enhancement and obtained good performances. The enhancement function is shown in (2).

$$ f(x)=a\left[ sigm\left(c\left(x-b\right)\right)- sigm\left(-c\left(x+b\right)\right)\right] $$

(2)

where $a=\frac{1}{sigm\left(c\left(1-b\right)\right)- sigm\left(-c\left(1+b\right)\right)}$, sigmis defined as$sigm(x)=\frac{1}{1+{e}^{-x}}$。.

In this function, there are two parameters of b and c, where b is used to control the enhancement range, and its value is generally between 0 and 1. c is used to control the enhancement strength, and it usually uses the fixed value between 20 and 50. However, our algorithm does not pre-set fixed values for b and c. Instead, we set two thresholds of the function to determine the shape of the function curve, and then obtain the values of b and c. After the procedure of NSST decomposition, the image has been decomposed into different scales and directions, so we can set various thresholds for different scales of the coefficients and enhance them respectively. Therefore, the enhancement process would be more refined.

2.4 Algorithm flow and steps

NSST transform is applied to decompose the original image. After that, the NSST coefficients of different scales and directions can be obtained. Since image detail information often exists in high-frequency coefficients, we choose high scale coefficients for processing, while keep the low frequency coefficients unchanged. The purpose of image enhancement is to enhance detail information and suppress noise. Therefore, we need to use the appropriate method to distinguish high-frequency signals.

Considering that PST algorithm has obvious advantage in extracting features of image, the proposed method extracts the details of the original image by PST and uses its output as the image feature map. Figure 1 is the feature map of the Lena image processed by PST. From the feature map, we can clearly distinguish the region where the detail information is concentrated. So it can be used as the reference of image enhancement.

Fig. 1

Lena image and its feature map processed by PST. a Lena b PST feature of Lena

After PST, the local standard deviation (LSD) of each pixel in the feature map is calculated according to (3). The LSD reflects the contrast change of the local area in an image. Therefore, the pixels with larger LSD in feature map will be rich in the detail textures around it, so they should be enhanced, while the region with smaller value of the LSD can be mildly enhanced or preserved.

$$ {\displaystyle \begin{array}{c}{\mathrm{LSD}}_n\left(x,y\right)=\sqrt{\frac{1}{{\left(2n+1\right)}^2}\sum \limits_{i=-n}^n\sum \limits_{j=-n}^n{\left[p\left(x+i,\kern0.5em y+j\right)-{\mathrm{mean}}_n\left(x,y\right)\right]}^2}\\ {}{\mathrm{mean}}_n\left(x,y\right)=\frac{1}{{\left(2n+1\right)}^2}\sum \limits_{i=-n}^n\sum \limits_{j=-n}^np\left(x+i,\kern0.5em y+j\right)\end{array}} $$

(3)

where mean_n(x, y) is the mean value of the pixels in the local window centered at p(x, y) and the size of the local window is (2n + 1) ∗ (2n + 1).

The enhancement of the NSST high scale coefficients depends on the output of the nonlinear enhancement model. In order to facilitate the calculation, the values of the input need to be normalized. The nonlinear enhancement function in Fig. 2 contains two thresholds. One threshold is the first nonzero intersection of the enhancement function and the line of y = x. It is the threshold that characterizes the feature coefficients from noise coefficients. It is called as T1. If the abscissa value of a certain point on the function curve is less than T1, then the value of its ordinate will be reduced nonlinearly, that is suppression (applicable to noise information); while if the abscissa value of a certain point on the function curve is greater than T1, the value of its ordinate will be nonlinearly stretched, that is enhancement (applicable to detail information). The other threshold is the first intersection of the enhancement function and the line of y = 1. It is the threshold that determines the extent of enhancement. We call it T2. If the abscissa of a certain point on the enhancement function curve is larger than T2, it is considered as strong edge, so the ordinate value of this part will be set to 1 after enhancement.

Fig. 2

The nonlinear enhancement function

As shown in Fig. 2, the abscissa value in the graph is set to be the LSD of the corresponding pixels in the feature map. We choose k1 times of the minimum value of the LSD as T1 and k2 times of the maximum as T2. The values of k1 and k2 depend on the quality of the original image and the decomposition scales. In our algorithm, k1 is an integer proportional to the decomposition level n, and k2 is an integer inversely proportional to the decomposition level, as is shown in (4).

$$ {\displaystyle \begin{array}{l}k1=k\ast n\\ {}k2=k/n\end{array}} $$

(4)

In our experiments, we choose 15 as an empirical value of k. If the image contains noise, the value of k should be enlarged properly. And smaller k will be more suitable for low contrast image. When the two thresholds are determined, the values of b and c can be obtained by solving the nonlinear equation group (5). Hence, the enhancement function is formed.

$$ \left\{\begin{array}{c}f\left({T}_1\right)={T}_1\\ {}f\left({T}_2\right)=1\end{array}\right. $$

(5)

After that, if we bring the pixel’s LSD of the PST feature map into the enhancement function, the enhancement ratio at the pixel can be read in the enhancement curve. It is the ratio of the ordinate value to the abscissa value. The NSST coefficients of the each scale can be multiplied with the enhancement ratio, and then the new coefficients after the enhancement processing can be obtained. Finally, the enhanced image is formed by NSST inverse transform.

The flow of the algorithm can be concluded as Fig. 3. The enhancement steps are as follows:

• 1) Decompose the original image with NSST. The source image can be decomposed into a low-frequency sub-band and a series of high-frequency sub-bands. In our experiment, the image is decomposed into three scales and the coefficients of the higher two layers will be enhanced.

• 2) The original image is processed by PST, and the image feature map is obtained.

• 3) Calculate the LSD of each pixel in the feature map. The value represents the complexity of image detail, and it can be used as the reference of the enhancement extent. That is, the enhancement extent of NSST coefficient is determined according to the LSD of each pixel in the PST feature map.

• 4) The two thresholds of the nonlinear enhancement function are set up according to the maximum and minimum value of the LSD in the feature map. Different k values are set for different scales of NSST coefficients, and then the enhancement models are constructed for different decomposition scales.

• 5) The LSD of each pixel in the feature map is taken into the enhancement model, and the enhancement ratio of each coefficient can be obtained. Multiply the NSST coefficients of each layer with the enhancement ratios, so as to obtain the enhanced coefficients.

• 6) The NSST inverse transformation is performed on the enhanced higher two-scale coefficients and the unprocessed coefficients of the first scale to reconstruct the enhanced image.

Fig. 3

The flow of the algorithm

3 Experiments and simulation

3.1 Experiment settings

The experiments are carried out on four images. They are standard images Lena and Barbara, image Car shot by optical sensor, and a local screenshot of a radar image. Five representative algorithms including the proposed method are selected to enhance the images respectively. All experiments are carried out on Matlab R2016b platform.

3.2 Simulation and evaluation

The enhancement simulation results are shown in Figs. 4, 5, 6, and 7, among which (a) is the original image, (b) is enhanced using Dynamic Stretching-based Brightness Preservation [16] (DSBP), (c) is the enhancement effect via Tuned Fuzzy Intensification operators [17] (Fuzzy_INT), (d) is the image enhanced by NSCT adaptive enhancement [8], (e) is the enhancement result with NSST [10], and (f) is the performance of the proposed algorithm.

Fig. 4

Enhancement results of Lena. a Original image b DSBP c Fuzzy_INT. d NSCT e NSST f Proposed

Fig. 5

Enhancement results of Barbara. a Original image b DSBP c Fuzzy_INT d NSCT e NSST f Proposed

Fig. 6

Enhancement results of car. a Original image b DSBP c Fuzzy_INT d NSCT e NSST f Proposed

Fig. 7

Enhancement results of the SAR image. a Original image b DSBP c Fuzzy_INT d NSCT e NSST f Proposed

Generally, it is difficult for human beings to perceive the mild differences among the enhancement results of various algorithms. Hence, the paper adopts two widely used objective evaluation parameters, Edge Preserved Index (EPI) [18] and Contrast Improvement Index (CII) [19] to evaluate the image enhancement effects. EPI measures the edge preserving ability of the enhanced image. CII is used to measure the contrast of the processed image, so as to reflect the details of the image and the enhancement effect. The greater values of EPI and CII indicate the better quality of the image.

4 Results and discussion

By subjective evaluation, it can be seen that our algorithm has significant effects on detail and edge enhancement. The enhanced images have no pseudo-Gibbs phenomenon, and the details are clear without obvious distortion. It has better enhancement effects on images with complex details.

In terms of the objective metrics, Tables 1, 2, 3, and 4 show that the enhancement effects of our algorithm are more ideal than the comparison algorithms, especially in edge preservation. The EPI of the proposed algorithm is much higher than the other algorithms. For contrast improvement, we still have potentiality to do better. We will try different ways to set the thresholds of the enhancement model in our future work aiming to improve the image contrast more obviously.

Table 1 Objective evaluation parameters for Lena image set

Table 2 Objective evaluation parameters for Barbara image set

Table 3 Objective evaluation parameters for car image set

Table 4 Objective evaluation parameters for the SAR image set

The proposed algorithm is suitable for the enhancement of optical images captured by visual sensors. And it is also applicable for radar image enhancement. We have tried to extend our method to improve images captured by infrared sensors, but we have not achieved good performance until now. Our algorithm depends largely on the detail features of images, so it is less effective in infrared images enhancement which are not rich in detail textures.

5 Conclusion

This paper describes an approach for accomplishing an image enhancement algorithm, which can improve the clarity of the gray images captured by visual sensors. The proposed method combines NSST and PST and takes the advantages of multi-scale analysis and image phase analysis to perform nonlinear enhancement of images. The algorithm can enhance image details and edge features without amplifying noise. And there are no ringing artifacts near edges. Experiments on Matlab platform prove that the algorithm is better than the other algorithms in terms of accurate edge preservation and adequate contrast enhancement.