Aircraft detection in remote sensing images based on saliency and convolution neural network

Abstract

New algorithms and architectures for the current industrial wireless sensor networks shall be explored to ensure the efficiency, robustness, and consistence in variable application environments which concern different issues, such as the smart grid, water supply, and gas monitoring. Object detection automatic in remote sensing images has always been a hot topic. Using the conventional deep convolution network based on region proposal for detection, there are many negative samples in the generated region proposal, which will affect the model detection precision and efficiency. Saliency uses the human visual attention mechanism to achieve the bottom-up object detection. Since replacing the selective search with saliency can greatly reduce the number of proposal areas, we will get some region of interests (RoIs) and their position information by using the saliency algorithm based on the background priori for the remote sensing image. And then, the position information is mapped to the feature vector of the whole image obtained by deep convolution neural network. Finally, the each RoI will be classified and fine-tuned bounding box. In this paper, our model is compared with Fast-RCNN that is the current state-of-the-art detection model. The mAP of our model reaches 99%, which is 12.4% higher than that of Fast-RCNN. In addition, we also study the effect of different iterations on model and find the model of 10,000 iterations already has a higher accuracy. Finally, we compare the results of different number of negative samples and find the detection accuracy is highest when the number of negative samples reaches 400.

Introduction

With the wireless sensor networks booming, various researches based on wireless sensor networks have made great progress [1,2,3,4,5,6,7,8,9,10], such as the physical sensors [11], architecture for the service computing [12], fault-tolerant optimization [13], data gathering and compression [14], and smart data analysis [15]. Although the wireless sensor networks have been proposed, studied, and developed for more than a decade of years, there are still a lot of challenging issues especially in various industrial scenarios [16, 17], including the object detection for the remote sensing images that are gathered by the industrial wireless sensor networks. We can obtain a large number of high resolution remote sensing images obtained using various satellites and all kinds of sensors. For the object detection in these high-resolution remote sensing images, the traditional manual detection cannot meet the actual needs. Especially for the small target objects in the image, it is hard to detect it quickly and accurately by artificial detection [18]. In the field of real-time image processing, automatic analysis and detection for remote sensing images will be highly cost-effective.

In the field of remote sensing image processing, it is of great military value for the automatic detection of aircraft in the airport [19]. The texture, color, and other characteristics of aircraft are very similar to those of the background in the remote sensing image because the airports are often built in desert or remote spaces, where the color is single and often gray [20]. It often fails to detect the aircraft using a general automatic detection method based on low-level image features. Later, the researchers have introduced the machine learning method into the field of remote sensing image automatic detection, good results have been achieved. Considering the implementation principle, algorithms for aircraft recognition can be briefly summarized in three categories. The first category is mainly based on statistical feature learning method. It usually constructs rotation-invariant and scale-invariant features based on shape, texture, and geometric features. These features include Hu moments [21], Zernike moments [22], wavelet moments [23], the Fourier descriptor [24, 25], and sift [26], which are accompanied by the generally adopted classifiers, e.g., SVMs and neural networks.

The second category is the template matching method for aircraft detection. Considering that the size and shape of the aircraft is similar between the same types of aircraft, researcher design the various shape templates for each type of aircraft and compute a similarity measurement between the template and aircraft. In [27], the authors propose a template-matching method using high-level features of aircraft shape. The method integrates a coarse-to-fine process. The first coarse stage is the rough estimation phase. In this stage, the estimation score is given according to the matching degree of the single type of aircraft. The second fine stage is the refinement stage. In this stage, the aircraft shape parameters are obtained by using the combined features and kernel functions. In [28], Wu et al. propose a robust aircraft detection method without the shape and silhouette extraction of the aircraft. The authors firstly apply a direction estimation method to align the aircraft to the same direction for further matching and employed a reconstruction similarity measure to transform type recognition into a reconstruction problem. Then, a jigsaw matching pursuit algorithm is proposed to solve the reconstruction problem. Although the template-matching method is easy to implement, there are also the following problems [29]:

• (1) The template-matching method is only applicable to the rare samples or low resolution images. If the resolution of the image is higher or the image size is larger, more feature points will be generated, which result in slower matching. If the number of templates for the target sample is larger and all these templates are matched one by one with the samples in images, the computation is very large and it is difficult to guarantee the speed of calculation. If only a few of the samples are selected as the reference samples, it leads to uncovering all features of the sample and waste of information resources. Meanwhile, how to select target samples is also a difficult point.

• (2) Remote sensing image is photographed in complex environments, which not only includes light, viewpoint, and scale variant but also includes the influence of atmospheric and cloud cover, complex background, and similar ground environment. These disadvantages cause images blurring. The insufficient samples and obvious difference between the templates and the identified images will lead to detection accuracy decreased even a failure to match.

• (3) The traditional method of object detection based on feature matching can only detect and recognize one type object by a given template, but cannot model and classify multi type objects at the same time.

The last category is deep learning. Since 2012, Hinton [30] gets significant achievements in the ImageNet classification using the deep learning algorithm: the deep learning methods have achieved very good results in various fields such as object classification and recognition. In 2014, Girshick et al. proposed the RCNN [31] that first uses depth convolution neural network for object detection. The authors first use selective search [32] to extract the region proposal from the original image, and then use the multi-layer convolutional neural network to extract the features of objects in the region proposal. Finally, SVM or softmax is used for classification and positioning. The method achieves the best effect of the object detection in that year.

The authors have applied this method to aircraft detection in remote sensing images since the depth learning has higher accuracy in object detection field. But there are so many region proposals, about 2000, extracted by the RCNN algorithm [31] using the selective search [32]. Extracting features from so many region proposals using convolution network will result in very low efficiency. In addition, in the 2000 region proposals, only a very small number of them are regions of interest (RoIs), and most of the regions are background. So many negative samples in the classification will affect the detection accuracy.

The paper will base on how to extract the appropriate number of region proposals, so as to improve the efficiency of object detection. Saliency algorithm has received great attention and has been widely used in image segmentation, object recognition, etc. since Itti [33] proposed a saliency algorithm for simulating the attention mechanism of the human visual system in 1998. The saliency of human vision is that they first notice the most significant part of the image and the background information independent on the foreground information is automatically deleted when people look at an image. Itti proposes an up-bottom image preprocessing algorithm that simulates the human visual system based on this principle. The algorithm uses multi-scale pyramid model to extract the texture, direction, color, and other features of the image. According to these feature maps, it computes three saliency maps using center-surround differences, which are combined together to form the final master saliency map. Later, a lot of research efforts have been made to design various saliency features characterizing salient objects or regions.

If the saliency algorithm is used to preprocess the image, it can greatly reduce the number of region proposals and improve the efficiency and accuracy of subsequent feature extraction and classification because the saliency algorithm can quickly find detected objects in the image. In this paper, we first get some region proposals by using the saliency algorithm to preprocess the original image. Then, the bounding box of these region proposals are obtained by thresholding calculation and connected domain method. The area that the bounding box contains is the RoI that we need to be detected. Finally, the Fast-RCNN [34] model is used to extract, map, and classify these RoIs, and the final test result is obtained by region proposal regression operation. As Fig. 1 shows, our detection network takes as input an entire image and a set of object proposals that are obtained with the saliency algorithm. The network first processes the whole image with several convolutional and max pooling layers to produce a conv feature map. Then, for each object proposal a region of interest pooling layer extracts a fixed-length feature vector from the conv feature map. Each feature vector is fed into a sequence of fully connected (fc) layers that finally branch into two sibling output layers: one that produces softmax probability estimates over “aircraft” class and “background” class and another layer that outputs four real-valued numbers, which encode refined bounding-box positions for each of the “aircraft” class.

Fig. 1

Network structure diagram

It is obvious that the aircrafts area is small in proportion to the whole image and the color and texture features of the aircrafts are similar to those of the background in the remote sensing image. So it usually leads to low accuracy using the conventional saliency algorithms, i.e., saliency algorithm based on regions and saliency algorithm based on frequency, to calculate the salient map because many small planes are filtered out as background chip. In [35], the robust saliency algorithm based on background priors can solve this problem. However, the algorithm uses statistical object area method, which is very difficult to calculate because the target object is irregular shape in images, to compute salient objects. Our algorithm uses statistical pixel number instead of statistical area method based on [35]. Experiment shows that this method can achieve good results. In addition, our algorithm is also compared with the state-of-the-art algorithms, e.g., IT [33], HC [36], RC [36], FT [37], CA [38], LC [39], SR [40], DSR [41], BL [42], and our saliency algorithm has good robustness to the remote sensing image.

We also compare our object detection model, which combined saliency algorithm with depth detection algorithm, and the state-of-the-art detection model. Our model gets higher recognition rate and detection accuracy than the latter.

The paper is organized as follows:. in Section 2, we introduce related works. Thereafter in the Section 3, we demonstrate the saliency algorithm based on background priori and the detection network. Experiments are presented in Section 4. We conclude with a discussion in Section 5.

Related works

The focus of attention, being closely related to how we perceive and process unconscious visual stimuli, which scans the scene both in a rapid, bottom-up, saliency-driven, and task-independent manner. Itti et al. [33], which are inspired by the visual neuron structure of primate [43], propose a saliency model based on the local contrast. The model uses the Gaussian pyramid to generate multi-scale images, and then calculates the color, brightness, and direction characteristics of the image to obtain saliency image by center-surrounding. Harel et al. [44], considering graph theory, introduce the Markov chain to calculate the steady-state distribution of Markov chain as the final saliency graph based on Itti’s study. Liu et al. [45] used the linear fusion of the Gaussian pyramid to compute the multi-scale local features based on the MZ method. Considering distance factor, Goferman et al. [46], fuse multi-scale images, e.g., 1, 1/2 and 1/3 three scales, to get saliency image. Different from previous methods of analysis in color space, Borji [47] uses the bag-of-words (BOW) to learn the sparse coding of image blocks from original image and obtains saliency image by analyzing the local rarity of image blocks based on sparse coding. It receives a higher saliency score in both marginal and noisy regions based on the local contrast methods, so that the entire saliency area cannot be significant.

Considering the disadvantages of computing saliency image using local contrast, we can obtain the saliency image using global contrast method. Zhai and Shah et al. [40] use the MZ algorithm to calculate the contrast of each pixel in the whole image. Inspired by this, Fenget et al. [48] used the sliding window to calculate the global contrast. Margolin et al. [49] obtain the saliency of image blocks by using the statistical information of image and linearly fusing the color contrast of image blocks.

Cheng et al. [37], calculate the histogram contrast of the whole image to obtain the saliency image. It means the wider the distribution of a color in the image, the lower the probability of the saliency region that contains this color. The color space distribution can be used to describe the salient area.

A color is more widely distributed in the image and the probability of the saliency including this color is smaller. So, the special color space distribution can be used to describe the saliency region. Perazzi et al. [50] obtain saliency image by computing the variance of each super-pixel’s color spatial distribution, which is got by the adaptive super-pixel segmentation method. The global contrast method, which calculates the whole image, will produce a saliency image of uniform density. However, the computational complexity increases using the global image processing. And if reducing the resolution or reducing the feature dimension is applied to address the problem, it leads to minutiae deletion and sensitive to noise.

Both local contrast and global contrast methods belong to the airspace mode. In order to improve the robustness of model and improve the checking efficiency of model, some people transform the image from the spatial domain to the frequency domain for calculating image contrast. Hou et al. [41] find that there are some similar spectrums among the similarity images in frequency domain. Based on it, Hou et al. propose a spectral residual method to compute salient image. The detail of the method as follows. Firstly, the original image is transformed from spatial domain to frequency domain. Then, the amplitude spectrum, phase spectrum, and amplitude spectrum residuals are computed in the frequency domain. Finally, the final saliency image is obtained by inverse Fourier transform. Different from Hou’s analysis of amplitude spectrum, Guo et al. [51] get the saliency image by analyzing phase spectrum. Achanta et al. [38] find that the high frequency part of the image reflects the overall information of the image, and the low frequency part reflects the detail information of the image. That is, the saliency image is larger probability in low frequency.

In addition, with the wide application of deep learning in recent, it achieves a good result by introducing the depth learning method into the computation of saliency image. Li et al. [52] get the saliency image by calculating the contrast of high-level features that are extracted from three different scale images by the CNNs (deep convolutional neural networks). Zhang et al. [53] think that it cannot obtain a saliency image accurately in spatial domain if the original image has low resolution, and the features of the image are extracted totally dependent on CNNs. So they propose the saliency model based on CNNs of spatial-temporal.

The traditional object detection method uses a sliding window to divide an image into millions of sub windows with different positions and scales, and then, each sub window is determined by the classifier whether it is the target object.

Traditional methods use a sliding window frame to break down an image into millions of sub-windows with different locations and different scales, and then use the classifier for each window to determine whether the target object is included. For each type of object, traditional methods extract its unique features and design specific classifiers, e.g., face detection algorithm is usually Haar feature +Adaboosting classifier [54]. Pedestrian detection algorithm is HOG feature (histogram of gradients) + SVM (support vector machine) [55]. And the detection algorithm of general objects is the HOG feature +DPM (deformable part model) [56, 57].

Recently, most of object detection algorithms are based on the deep learning frameworks. These algorithms are mainly classified into two groups: one is object detection method based on region proposal [58,59,60], which is a mainstream algorithm, e.g., RCNN [31], SPP-Net [61], Fast-RCNN [34], Faster-RCNN [62], and MSRA recently proposes algorithm R-FCN [63]. The other is not using the region proposal method to detection, e.g., YOLO [64] and SSD [65].

For the first object detection method, it firstly selects the RoIs, i.e., the RoIs are generated from the original image by the selective search [32], edge box [66], or RPN [62]. Then, the feature extraction is carried out for every RoI by CNNs. Finally, the target object with bbox is obtained using softmax classifier to classify objects. In [31], about 2000 RoIs, which followed are extracted and classified by CNNs, are generated using selective search. Due to the fact that there is a large amount of overlap between these RoIs, redundant calculations result in inefficiencies. SSP-Net [61] and Fast-RCNN [34] propose a shared feature method that is extracted only one time for the whole image for this problem. And then, about 2000 RoIs are mapped according to their location information to the feature vector of the whole image to obtain the features of each RoI, so it greatly improves the speed of calculation because the feature extraction calculations of different RoI can be shared.

Whether it is SSP-Net or Fast-RCNN, although they reduce the number of CNN operations, still need to generate the region proposal box for each image using selective search, which takes 2 s/image. The time consumption of selective search on the CPU is far greater than that of the CNN on the GPU. Therefore, the bottleneck of object detection is the region proposal operation.

In order to save selective search operating time, Faster-RCNN inputs the convolution feature of the image to the RPN (region proposal network) layer and obtains the region proposal. After the RPN layer, the region proposal is required to be categorized and fine-tuned through softmax classifier and bounding box. Experiments show that Faster-RCNN is not only more faster but also has higher detection accuracy. Considering the full connection operation is also a time-consuming process of RoIs in Faster-RCNN, R-FCN Faster also incorporates this process, which shares computing for different RoIs, into the forward computation of the network. So R-FCN is faster than Faster-RCNN.

The latter object detection method does not use region proposal for object detection. YOLO divides the original image into S*S cells. If the center of an object falls into a cell, the corresponding cell is responsible for detecting the object and gives confidence score for each cell. The score reflects the possibility of the target object and the accuracy of location of the target predicted by bounding box. Since YOLO did not use the region proposal, but directly conduct convolution operation on the image, it is much faster in detection than Faster-RCNN but the accuracy is less than that of Faster-RCNN. SSD also uses a single convolutional neural network to convolute the image, and predicts a series of bounding boxes with different sizes and aspect ratios at each position of the feature vector. During the testing phase, SSD predicts the possibility that each bounding box contains objects is a target object and adjusts the bounding box of the object to accommodate the size. G-CNN [67] regards the object detection as a process of changing size of the bounding box from fixed cell to the real borders of the object. Firstly, SSD divides the whole image into some cells with different size to obtain the initial detection bounding box and convolutes the whole image to get a feature vector of it. Then, the feature vector corresponding to the initial bounding box is transformed into a fixed-size feature vector by Fast-RCNN. Finally, a more accurate border of the target object is got through a regression process.

In short, the performance of the first type of detection is better, but slower, while that of the other is slightly worse, but faster [68]. Recently, some researchers carry out the object detection from other perspectives. For example, many researchers regard that the lower layer of the neural network usually retain more fine-grained, while the higher layer of it usually has better semantic features. To improve the accuracy of detection, the features of different layer are fused [59] [69,70,71,72,73]. Other researchers combine the object detection with other applications of image processing, e.g., Kaiming [74] combine the object detection with image segmentation and gets good results.

Model introduction

In order to obtain an efficient and robust detection model, we change the selective search method of Fast-RCNN to a more efficient saliency method. Since the saliency method has fewer region proposals and more accurate location, the computation of bbox regression is reduced in followed Fast-RCNN. In addition, it improves accuracy of detection because of reducing the number of negative samples. As Fig. 2 shows, the algorithm consists of two parts: region proposal and object detection network. The region proposal mainly generates some object proposals with the saliency algorithm, and the object detection network carries out the detection function of the object based on Fast-RCNN. Our algorithm firstly processes the whole image with several convolutional and max pooling layers to produce a conv feature map. Then, for each object proposal that is generated with the saliency algorithm a RoI pooling layer extracts a fixed-length feature vector from the conv feature map. Each feature vector is fed into a sequence of fully connected (fc) layers that finally branch into two sibling output layers: one that produces softmax probability estimates over aircraft class and background class, another layer outputs the bounding-box positions for each of the aircraft. The algorithm flowchart is shown in Fig. 2.

Fig. 2

Algorithm structure diagram

Region proposal acquisition

Currently most of the models use the selective search method to obtain region proposals, which solves the time-consuming and resource problems caused by the previous exhaustive search method. The method includes two steps. Firstly, 1000 ~ 3000 region proposals are randomly generated in an image. And then, these region proposals are merged through certain strategies, e.g., similarity measures of texture or color. The selective search method has a good result on the images with gradation distinction of clear color or texture feature. However, for objects such as aircrafts, whose features have similar background features in the remote sensing image, the effect is poor.

In addition, Fast-RCNN obtains about 2000 RoIs that include target objects and non-target objects method. Most of these RoIs are negative samples, which not only reduce the detection efficiency, but also reduce the detection accuracy. For this problem, we propose using saliency method to calculate the RoIs whose number is much smaller than those of Fast-RCNN.

But the common saliency methods are often fail to high-quality saliency image due to the small size of the aircraft in the remote sensing image. In [18], researchers obtain the saliency image through the recognition of the foreground based on background prior. The method proposed in the paper [35] mainly to determine whether the object is a saliency object by the ratio of the area of target object to the area of the part connecting to the image boundary. But because remote sensing image has low resolution and the size of object is small, it cannot calculate the area of object and fails to generate the saliency image for small object. Our algorithm calculate saliency image by counting the number of pixels of target object and those of the part connecting to the image boundary based on background prior proposed in [35]. Specific implementations are as follows:

• (1) Background perception

  We observe that object and background regions in images are quite different in their spatial layout, i.e., object regions are much less connected to image boundaries than background ones. As shown in Fig. 3, there are four regions: blue, white, red, and green. The blue region and white region are the backgrounds since they significantly touch the image boundary. Only a small amount of the red region touches the image boundary, but as its size is also small, so it looked as a partially cropped object. The green region is clearly a salient object as it is large, compact, and only slightly touches the image boundary.

• (2) We can obtain salient objects by the ratio of the number of pixels of the part touching the image boundary to those of the entire object field. The part with the smallest ratio is the salient object as shown in Eqs. (1)~(3).

Fig. 3

An example of background perception. The image consists of four regions with their boundary connectivity values. The boundary connectivity is large for background regions and small for object regions

$$ \mathrm{BndCon}(p)=\frac{\mathrm{NumPixel}(p)}{\mathrm{TotalPixel}(p)}, $$

(1)

$$ \mathrm{NumPixel}(p)=\left\{p|p\in R,p\in \mathrm{Bnd}\right\}, $$

(2)

$$ \mathrm{TotalPixel}(p)=\left\{p|p\in R\right\}, $$

(3)

where p is an image pixel, R is an object region, and Bnd is the set of image boundary pixels.

• (3) Pixel calculation

  Because it is difficult to segment the image accurately, it is difficult, in the practical application, to count the number of pixels of the part touching the image boundary and the total number of pixels of the entire object. We use super-pixel instead of pixels. Firstly, the superpixels are obtained using SLIC [75], and then, we calculate the shortest path between all adjacent superpixel to construct similarity of them as in Eq. (4).

$$ {d}_{\mathrm{geo}}\left(p,q\right)=\underset{p_1=p,{p}_2,\dots, {p}_n=q}{\min}\sum \limits_{i=1}^{n-1}{d}_{\mathrm{app}}\left({p}_i,{p}_{i+1}\right), $$

(4)

Where d_geo is the shortest path between superpixel p and superpixel q. For convenience, we define d_geo(p, p) = 0.Finally, Eq. (5) computes the number of superpixels of the region that superpixel p belongs to.

$$ \mathrm{TotalPixel}(p)=\sum \limits_{i=1}^n\exp \left(-\frac{d_{\mathrm{geo}}^2\left(p,{p}_i\right)}{2{\sigma}_{\mathrm{clr}}^2}\right)=\sum \limits_{i=1}^nS\left(p,{p}_i\right), $$

(5)

where n is total number of superpixels. We use the Gauss function to map the similarity of two superpixels to (0, 1]. When p _i is similar to p, d_geo = 0and s = 1, which ensures the number of super pixels increases 1 when p and p _i are similar. Otherwise, On the contrary, when p and p _i are not similar, i.e.,d_app(∗, ∗) >  > 3σ _clr , s = 0.Similarly, the number of pixels touched the boundary can be obtained as in Eq. (6).

$$ \mathrm{NumPixel}(p)=\sum \limits_{i=1}^nS\left(p,{p}_i\right)\cdot \delta \left({p}_i\in \mathrm{Bnd}\right), $$

(6)

where δ is 1 for superpixels on the image boundary, and 0 otherwise. In experiment, we take 200 superpixels for a typical 300 × 400 resolution image and find that the effect reaches best when σ_clr is 10.

• (4) Algorithm updating

  In order to ensure that the saliency objects can be completely segmented, we regard the partially cropped objects, which are located in the middle of the image and untouched the image boundary, as the saliency objects, i.e., NumPixel(p) = 0 and TotalPixel(p) are relatively small. This method can guarantee the number of negative samples, which improve training accuracy of the following object detection model, while not missing the real objects.

Object detection model

Girshick proposed a RCNN model for object detection in 2014 and achieved the best detection results in that year. The RCNN firstly uses selective search method to generate many region proposals (RoIs), and then uses CNNs to extract feature of each RoI. Finally, the RoIs are classified and its bounding boxes are regressed by the softmax classifier and bbox regressor. The RCNN has the very low detection efficiency because about 2000 RoIs produced by selective search are extracted feature by CNNs. Later, Girshick proposed the Fast-RCNN model based on the RCNN. That model firstly extracted the feature of whole image using CNNs. Then, the 2000 RoIs are mapped the feature vector according to the bounding box of those RoIs, and do not extract its features. Since the feature extraction operation is carried out only one time during the whole detection period, the detection efficiency has been greatly improved. The detection accuracy of Fast-RCNN is lower because most of them are negative samples in 2000 RoIs. As shown in Fig. 4, we use saliency method to produce some saliency objects as the RoIs and send these RoIs into detection network to detect objects. The detection network extracts the feature of the whole image by five CNN layers and some pooling layers. And then, the bounding box of the saliency object is mapped to this feature vector in RoI pooling layer. Finally, the saliency object is classified and its bounding box is regressed.

• (1) RoI pooling layer

  The bounding box information of RoIs produced by saliency method mainly consists of a four-tuple (r, c, h, w) that specifies its top-left corner (r, c) and its height and width (h, w). In addition, since the area of RoIs is any size, we divide the h × w RoI into h/7 × w/7 sub-regions of 7 × 7 size for calculation convenience. Then, max-pooling the values in each sub-region into the corresponding output grid cell. The RoI pooling layer is simply the special-case of the only one spatial pyramid pooling layer used in SPPnets [22].

• (2) Fine-tuning

  In the back-propagation, the traditional detection networks, e.g., RCNN or SPPnets, are highly inefficient because the number of RoIs is large and the entire image also is processed when each RoI needs to be retrained.

  Our method based on Fast-RCNN uses a share parameter way, which shares the feature between back-propagation networks and forward pass by stochastic gradient descent (SGD), during training. Fast-RCNN has proved the hierarchical sampling method (i.e., each batch data contains 2 images and 128 RoIs for each image) is effective during training. However, due to the fact that the number of RoIs produced by the saliency method is small and there are less than 128 ROIs in each image, we combine two types of RoIs, one of which is obtained by the saliency algorithm and the other is randomly generated, to improve the detection accuracy in the actual training process.

  In addition to the hierarchical sampling, Fast R-CNN uses a streamlined training process with one fine-tuning stage that jointly optimizes a softmax classifier and bounding-box regressors, rather than training a softmax classifier, SVMs, and regressors in three separate stages. Our method also adopts this method to improve efficiency.

• (3) Loss function

  There are two output layers in the detection model: one output is used to describe the probability distribution of each RoI, p = (p₀, p₁, …, p _u ), and p is computed by a softmax over the u + 1 outputs of a fully connected layer. Since we detect only one type of object in remote sensing image, so we set u = 1. The other output layer is the bounding box regression offsets, t^u = (t _x ^u, t _y ^u, t _w ^u, t _h ^u), for each of the u object classes, indexed by u. t^u specifies a scale-invariant translation and log-space height/width shift relative to an object proposal.

  Each training RoI is labeled with a ground-truth class u and a ground-truth bounding-box regression target v. We use a multi-task loss function L:

  $$ L\left(p,u,{t}^u,v\right)={L}_{\mathrm{cls}}\left(p,u\right)+\lambda \left[u\ge 1\right]{L}_{\mathrm{loc}}\left({t}^u,v\right), $$

  (7)

  where L_cls(p, u) =  − log p _u is the log loss of the real category u. The second regression loss function L _loc is the loss between the real object position information v = (v _x , v _y , v _w , v _h ) and the predicted object position informationt^u = (t _x ^u, t _y ^u, t _w ^u, t _h ^u). The super parameter λ balances two loss function, and we take λ = 1 in the experiment.

  Experiment shows that, for remote sensing images, L₁ regularization is better than L₂ regularization, so we use L₁ smoothing loss function for L _loc :

  $$ {L}_{\mathrm{loc}}\left({t}^u,v\right)=\sum \limits_{i\in \left\{x,y,w,h\right\}}{\mathrm{smooth}}_{L_1}\left({t}^u-v\right), $$

  (8)

  in which

  $$ {\mathrm{smooth}}_{L_1}(x)=\left\{\begin{array}{l}0.5{x}^2\kern2.75em \mathrm{if}\left|x\right|<1\\ {}\left|x\right|-0.5\kern2em \mathrm{otherwise}\end{array}\right.. $$

  (9)

• (4) Mini-batch sampling

  During fine tuning, we enter two images every time, randomly select 64 RoIs for each image. We take 25% of the RoIs from object proposals that have intersection over union (IoU) overlap with a ground truth bounding box of at least 0.5. These are the positive samples, i.e., u > = 1. The remaining RoIs are sampled from object proposals that have a maximum IoU with ground truth in the interval [0.1, 0.5). These are the negative samples, i.e., u = 0. The lower threshold of the negative sample is 0.1 because of a heuristic for hard example mining. During training, the image has a probability level of 0.5 to flip. The table of sample allocation is shown in Table 1.

• (5) Scale-invariance

  Fast-RCNN explores two ways of achieving scale invariant object detection: (1) via “brute force” (single-scale) learning and (2) by using image pyramids (multi-scale). A single-scale directly pre-defined pixel size during both training and testing. The network must directly learn scale-invariant object detection from the training data.

  During multi-scale training, we randomly sample a pyramid scale each time an image is sampled, and then train the image object with an approximate scale invariant sampled. At test-time, the image pyramid is used to approximately scale-normalize each object proposal.

  Experiment of Fast-RCNN shows that the mAP of single-scale is lower 1.2 to 1.5% than that of multi-scale for the AlexNet and VGG_CNN_M_1024 model, but the former is faster than the latter in terms of detection speed.

  So we can see that the deep neural network is not sensitive to the scale of the object. Multi-scale image is only slightly higher in detection accuracy than a single-scale image, but the efficiency is greatly reduced. Therefore, we use the single-scale approach (i.e., the short edge is limited to 300 pixels, the long edge is limited to 500 pixels) to training and testing the image.

• (6) Object detection

  Once the detection network is fine-tuned, the detection amounts to little more than running a forward pass. The network takes as input an image and a list of RoIs to score. During the testing phase, all 400 RoIs will be scaled to 224 × 224 pixels when the image pyramid is used to extract the features of the RoIs.

  For each test RoI r, the network outputs a posterior probability distribution p and a series of predicted bbox offsets relative to r (each of the u classes gets its own refined bounding-box prediction). Then, we assign a detection confidence to r for each object class u using the estimated probability P _r (class = u| r) = P _u . Finally, we perform non-maximum suppression independently for each class.

Fig. 4

Aircraft sample set. a The background is mainly made up of grassland. b The background is mainly made up of land. c The background is mainly made up of cement. d A single aircraft in each image. e Multiple aircrafts in each image

Table 1 The table of sample allocation

Experiments

Data set acquisition

We sample 1200 images containing the aircraft from google earth, and the resolution of each image is between 300 × 300 pixels and 700 × 700 pixels. Since the airport is in the remote places, there are three main types of background: grassland, land, and cement. As shown in Fig. 4, to ensure data sets cover all types of data, we sample lots of samples from those different backgrounds. Besides, in our data set, there are some images containing a different number of aircraft: one is the image with a single aircraft, and the other is with multiple aircrafts.

Saliency experiment

We compare our algorithm with the state-of-the-art saliency algorithms, e.g., IT [33], HC [36], RC [36], FT [37], CA [38], LC [39], SR [40], DSR [41] and BL [42] in the five types of data set. IT is the most cited algorithm. HC calculates saliency image based on global histograms. RC obtains saliency image based on local contrast method. CA combines local contrast with global contrast to calculate saliency image. FT calculates saliency image in the frequency domain. LC uses the combination of time domain and frequency domain to obtain saliency image. SR calculates it in frequency domain. And BL adopts the machine learning method for saliency image. In the experiment, all of algorithms employ the source provided by the author except the LC using the program provided by the paper [36] because its author did not provide source code. Running results of all algorithms in different types of image set are shown in Fig. 5. IT is the earliest to use computer to simulate the biological visual attention mechanism, which mainly contrasts the color, brightness and direction of the image with the background to get a saliency image. The result of IT is the effect to be bad actually because of the similarity between the background color, texture, and brightness in remote sensing image. The HC algorithm, which is proposed a global histogram method, cannot obtain the better saliency image based on global histogram due to the worse discrimination of the color feature in remote sensing image. RC, relative to the HC, uses local contrast to obtain a saliency image, and the effect is better than that of HC in the all types of background except grassland. CA combines the global and local contrast methods, which can distinguish between background and foreground. But from Fig. 5, it is difficult to obtain a high quality binary image through a suitable thresholds, CA can not accurately box the aircraft objects.

Fig. 5

Comparison of saliency images. (a) Grassland. (b) Land. (c) Cement. (d) Only a single aircraft in each image. (d) Multiple aircrafts in each image. (1) The original remote sensing image. (2) IT [33]. (3) HC [36]. (4) RC [36]. (5) FT [37]. (6) CA [38]. (7) LC [39]. (8) SR [40]. (9) DSR [41]. (10) BL [42]. (11) Our algorithm

The other algorithms different from the past in the time domain, e.g., FT, etc., achieve a better saliency image in the frequency domain except the small objects in remote sensing image as shown in Fig. 5b. Through the above analysis, we can see that the conventional saliency algorithms cannot receive better results only by local contrast calculation, global contrast calculation, or biological vision-based saliency calculation because the remote sensing image has the characteristics of low resolution, the small target object, and low color contrast between foreground and background. Our algorithm, according to the object area accounts for a large proportion of the whole image as the background and the remains is foreground, has better robustness for remote sensing image processing based on background prior.

To obtain objects with bounding boxes, we need to box the saliency image after obtaining the saliency image. Since the foreground has been separated from the background when we get the saliency image, a simple connected domain method can box the saliency images. And to ensure the entire aircraft can be boxed, the overlapping bounding box is merged according to the maximum principle of merger.

RCNN, Fast-RCNN, and Faster-RCNN are currently the state-of-the-art three object detection methods. The first two methods, i.e., RCNN and Fast-RCNN, will generate about 2000 RoIs during the object detection and the last method also generates at least 300 RoIs. However, as shown in Fig. 6, these RoIs, whose number is smaller than 2000 RoIs that obtained by Fast-RCNN, are put into detection network to train model and the training model time is much smaller than it by Fast-RCNN.

Fig. 6

The bounding box of saliency image. If the bounding boxes are overlapped, these bounding boxes are merged and remain the maximum border

Object detection

We compare our algorithm with Fast-RCNN in this section. Fast-RCNN firstly obtains 2000 RoIs using selective search, and then the bounding box information of those RoIs are mapped to feature vector of the whole image by CNNs. Finally, these RoIs are classified by softmax and its bounding boxes are fine-tuned by bbox. To analyze the influence of the number of iterations on model checking effect and training time in CNNs, we compare mAP with the different number of iterations.

As shown in Table 2, the more the number of training, the greater the mAP, which is consistent with the basic theory of deep convolution network. In addition, we also can find that the mAP of our network is about 10% higher than that of Fast-RCNN from the above Table 2. The loss chart of training of the two algorithms is shown in Fig. 7.

Table 2 The mAP with different iteration time

Fig. 7

Loss chart. a Our algorithm. b Fast-RCNN

As shown in Fig. 7, we can find that the loss rate is basically stable when the number of iterations reached 10,000 and the average loss rate is only reduced about 1% when the number of iterations is increased to 30,000. So, we can use the model of 10,000 iterations to detect object in the real-time application.

The saliency method, which can highlight the suspect object to generate RoIs in image preprocessing, is completely different from the Fast-RCNN. The threshold has a great influence on the number of salient objects during using saliency method to generate the salient maps. Although we adjust the different thresholds as much as possible to minimize the impact, the total number of RoIs in each remote sensing image is still small and the number of negative samples is less in such a small amount of RoIs. To examine the effect of different number of negative samples on the detection accuracy, we compare the results obtained with the different number of negative samples in experiment. Firstly, we directly put the RoIs that are generated by saliency algorithm into network to train and find the loss rate shows a divergent state (shown in Figure 8). So the trained model fails to detect objects.

Fig. 8

Loss presented the state of dispersing

Then, we randomly generate 200 and 400 RoIs for saliency processed images respectively. To ensure all of generated RoIs are negative samples, those that overlapping reaches 50% with the original RoIs will be discarded. Figure 9 shows the result of different number of negative samples. The detection effect and the calibration accuracy of the bounding box have been very accurate when the negative samples increase to 400. Many objects that belong to the background image are labeled as target objects, and some bounding boxes of objects are boxed inaccurately because the number of negative samples is low that leads to model training inadequate.

Fig. 9

Results of different number of negative samples. a Original image. b 400 negative samples. c 200 negative samples

Results and discussion

Since the RoIs generated by saliency method are less than those generated by select search method, our algorithm can greatly improve the detection precision and reduce the detection time than the other state-of-the-art algorithms, e.g., RCNN, Fast-RCNN, and Faster-RCNN. But our saliency algorithm based on background prior has a poor effect when the boundaries are fuzzy between the foreground and the background, so it is difficult to obtain some clear salient map if the boundaries are fuzzy, which leads to failure of subsequent object detection.

Conclusions

Since remote sensing images have low resolution and the target objects in it are very small, the conventional automatic object detection method is very difficult to detect objects accurately. The current object detection algorithm based on deep learning is required to obtain a large number of RoIs, and most of those RoIs are negative samples. These redundant negative samples not only reduce the detection accuracy, but also increase the training time of the model. We propose a very robust and efficient detection method by combining background prior-based saliency detection algorithm with the CNNs based object detection algorithm.

For future work, we believe that investigating more sophisticated techniques for improving the saliency accuracy, including the deep convolution neural network, will be beneficial. Moreover, since it is a time-consuming for fitting parameters of model, we propose the share feature method between saliency detection and object detection using CNNs in order to further improve the efficiency of model training.