NSST multiscale analysis
NSST is a multi-scale and multi-directional analysis tool. It is an optimal approximation, which is a “true” sparse representation of the image in all directions and at various scales. In addition, unlike traditional shearlet transform, NSST does not perform down-sampling when performing multi-scale decomposition of an image. It combines a non-down-sampling tower transform with a non-sub-sampled directional filter bank and transforms. The size of the directional sub-bands of each scale is the same as that of the original image. The transform has shift invariance and can overcome the pseudo-Gibbs phenomenon when the image is reconstructed.
Feature extraction with PST
PST is a new image processing method proposed by Asghari and Jalali [15]. It is said that PST has superior performance in feature extraction, especially edge extraction. The PST processing can be divided into several steps: Firstly, smooth the image with a low pass filter to reduce noise. Secondly, apply the PST kernel which has a non-linear frequency-dependent transfer function to the image. The output of the transform is the phase in space domain. The applied phase is frequency dependent with the original image, it means that higher amount of phase corresponds to higher frequency image features. Finally, find image sharp transitions by thresholding the phase and apply binary morphological operations, so as to enhance the features and clean the transformed image. PST is defined in frequency domain [15] as (1).
$$ A\left[n,m\right]=\angle \left\langle IFFT2\;\left\{\tilde{K}\left[p,q\right]\cdot \tilde{L}\left[p,q\right]\cdot FFT2\left\{B\left[n,m\right]\right\}\right\}\right\rangle $$
(1)
where n and m are the space domain variables, A[n,m] is the output phase image, B[n,m] is the input image, FFT2 is the two-dimensional fast Fourier transform, IFFT2 is the two-dimensional inverse fast Fourier transform. p and q are the frequency domain variables. The function \( \tilde{K}\left[p,q\right] \) is the warped phase kernel, and \( \tilde{L}\left[p,q\right] \) is the frequency response of the localization kernel.
Nonlinear enhancement model
Image information can be roughly classified into several parts, such as edges, details, background, and noise. The edges of image can be subdivided into sharp edges and fuzzy edges. When selecting the enhancement model, we need to take into account the features of different kinds of information. It is appropriate to preserve clear edges, enhance fuzzy edges, and suppress noise effectively. Therefore, nonlinear function is a good choice. The enhancement function selected in this paper was presented by A. F. Laine. After that, some researchers applied this function to image enhancement and obtained good performances. The enhancement function is shown in (2).
$$ f(x)=a\left[ sigm\left(c\left(x-b\right)\right)- sigm\left(-c\left(x+b\right)\right)\right] $$
(2)
where \( a=\frac{1}{sigm\left(c\left(1-b\right)\right)- sigm\left(-c\left(1+b\right)\right)} \), sigmis defined as\( sigm(x)=\frac{1}{1+{e}^{-x}} \)。.
In this function, there are two parameters of b and c, where b is used to control the enhancement range, and its value is generally between 0 and 1. c is used to control the enhancement strength, and it usually uses the fixed value between 20 and 50. However, our algorithm does not pre-set fixed values for b and c. Instead, we set two thresholds of the function to determine the shape of the function curve, and then obtain the values of b and c. After the procedure of NSST decomposition, the image has been decomposed into different scales and directions, so we can set various thresholds for different scales of the coefficients and enhance them respectively. Therefore, the enhancement process would be more refined.
Algorithm flow and steps
NSST transform is applied to decompose the original image. After that, the NSST coefficients of different scales and directions can be obtained. Since image detail information often exists in high-frequency coefficients, we choose high scale coefficients for processing, while keep the low frequency coefficients unchanged. The purpose of image enhancement is to enhance detail information and suppress noise. Therefore, we need to use the appropriate method to distinguish high-frequency signals.
Considering that PST algorithm has obvious advantage in extracting features of image, the proposed method extracts the details of the original image by PST and uses its output as the image feature map. Figure 1 is the feature map of the Lena image processed by PST. From the feature map, we can clearly distinguish the region where the detail information is concentrated. So it can be used as the reference of image enhancement.
After PST, the local standard deviation (LSD) of each pixel in the feature map is calculated according to (3). The LSD reflects the contrast change of the local area in an image. Therefore, the pixels with larger LSD in feature map will be rich in the detail textures around it, so they should be enhanced, while the region with smaller value of the LSD can be mildly enhanced or preserved.
$$ {\displaystyle \begin{array}{c}{\mathrm{LSD}}_n\left(x,y\right)=\sqrt{\frac{1}{{\left(2n+1\right)}^2}\sum \limits_{i=-n}^n\sum \limits_{j=-n}^n{\left[p\left(x+i,\kern0.5em y+j\right)-{\mathrm{mean}}_n\left(x,y\right)\right]}^2}\\ {}{\mathrm{mean}}_n\left(x,y\right)=\frac{1}{{\left(2n+1\right)}^2}\sum \limits_{i=-n}^n\sum \limits_{j=-n}^np\left(x+i,\kern0.5em y+j\right)\end{array}} $$
(3)
where meann(x, y) is the mean value of the pixels in the local window centered at p(x, y) and the size of the local window is (2n + 1) ∗ (2n + 1).
The enhancement of the NSST high scale coefficients depends on the output of the nonlinear enhancement model. In order to facilitate the calculation, the values of the input need to be normalized. The nonlinear enhancement function in Fig. 2 contains two thresholds. One threshold is the first nonzero intersection of the enhancement function and the line of y = x. It is the threshold that characterizes the feature coefficients from noise coefficients. It is called as T1. If the abscissa value of a certain point on the function curve is less than T1, then the value of its ordinate will be reduced nonlinearly, that is suppression (applicable to noise information); while if the abscissa value of a certain point on the function curve is greater than T1, the value of its ordinate will be nonlinearly stretched, that is enhancement (applicable to detail information). The other threshold is the first intersection of the enhancement function and the line of y = 1. It is the threshold that determines the extent of enhancement. We call it T2. If the abscissa of a certain point on the enhancement function curve is larger than T2, it is considered as strong edge, so the ordinate value of this part will be set to 1 after enhancement.
As shown in Fig. 2, the abscissa value in the graph is set to be the LSD of the corresponding pixels in the feature map. We choose k1 times of the minimum value of the LSD as T1 and k2 times of the maximum as T2. The values of k1 and k2 depend on the quality of the original image and the decomposition scales. In our algorithm, k1 is an integer proportional to the decomposition level n, and k2 is an integer inversely proportional to the decomposition level, as is shown in (4).
$$ {\displaystyle \begin{array}{l}k1=k\ast n\\ {}k2=k/n\end{array}} $$
(4)
In our experiments, we choose 15 as an empirical value of k. If the image contains noise, the value of k should be enlarged properly. And smaller k will be more suitable for low contrast image. When the two thresholds are determined, the values of b and c can be obtained by solving the nonlinear equation group (5). Hence, the enhancement function is formed.
$$ \left\{\begin{array}{c}f\left({T}_1\right)={T}_1\\ {}f\left({T}_2\right)=1\end{array}\right. $$
(5)
After that, if we bring the pixel’s LSD of the PST feature map into the enhancement function, the enhancement ratio at the pixel can be read in the enhancement curve. It is the ratio of the ordinate value to the abscissa value. The NSST coefficients of the each scale can be multiplied with the enhancement ratio, and then the new coefficients after the enhancement processing can be obtained. Finally, the enhanced image is formed by NSST inverse transform.
The flow of the algorithm can be concluded as Fig. 3. The enhancement steps are as follows:
-
1)
Decompose the original image with NSST. The source image can be decomposed into a low-frequency sub-band and a series of high-frequency sub-bands. In our experiment, the image is decomposed into three scales and the coefficients of the higher two layers will be enhanced.
-
2)
The original image is processed by PST, and the image feature map is obtained.
-
3)
Calculate the LSD of each pixel in the feature map. The value represents the complexity of image detail, and it can be used as the reference of the enhancement extent. That is, the enhancement extent of NSST coefficient is determined according to the LSD of each pixel in the PST feature map.
-
4)
The two thresholds of the nonlinear enhancement function are set up according to the maximum and minimum value of the LSD in the feature map. Different k values are set for different scales of NSST coefficients, and then the enhancement models are constructed for different decomposition scales.
-
5)
The LSD of each pixel in the feature map is taken into the enhancement model, and the enhancement ratio of each coefficient can be obtained. Multiply the NSST coefficients of each layer with the enhancement ratios, so as to obtain the enhanced coefficients.
-
6)
The NSST inverse transformation is performed on the enhanced higher two-scale coefficients and the unprocessed coefficients of the first scale to reconstruct the enhanced image.
