Deep multimodal fusion for ground-based cloud classification in weather station networks

Abstract

Most existing methods only utilize the visual sensors for ground-based cloud classification, which neglects other important characteristics of cloud. In this paper, we utilize the multimodal information collected from weather station networks for ground-based cloud classification and propose a novel method named deep multimodal fusion (DMF). In order to learn the visual features, we train a convolutional neural network (CNN) model to obtain the sum convolutional map (SCM) by using a pooling operation across all the feature maps in deep layers. Afterwards, we employ a weighted strategy to integrate the visual features with multimodal features. We validate the effectiveness of the proposed DMF on the multimodal ground-based cloud (MGC) dataset, and the experimental results demonstrate the proposed DMF achieves better results than the state-of-the-art methods.

1 Introduction

Clouds, as one of the major meteorological phenomena, play a profound role in climate predictions and services [1, 2]. Cloud classification is a crucial task for cloud observation, and it is currently undertaken by the professional observers [3]. However, manual observation is time-consuming and labor-intensive. Furthermore, the observation results are unreliable due to large dependency on subjective judgements. Hence, there is a high demand for automatic classification of ground-based cloud.

In recent years, many attempts have been made to classify the ground-based cloud. One trend is to develop the ground-based sky imagers such as whole-sky imager (WSI) [4], total-sky imager (TSI) [5], infrared cloud imager (ICI) [6], all-sky imager (ASI) [7, 8], whole-sky infrared cloud measuring system (WSIRCMS) [9], and day/night whole sky imagers (D/N WSIs) [10]. Benefiting from these devices, a number of ground-based cloud images are available for developing automatic classification algorithms. Calbo et al. [1] extracted statistical texture features based on Fourier transform to classify the cloud images into eight categories. Heinle et al. [11] distinguished seven sky conditions based on twelve kinds of features and k-nearest neighbor classifier. Ghonima et al. [12] treated the pixel red-blue ratio (RBR) between the test image and clear sky image as the feature. Zhuo et al. [13] combined texture and structure features for cloud representation and obtained a high classification accuracy. Kazantzidis et al. [14] took into account the statical color, the solar zenith angle, and the existence of raindrops in sky images. Cheng et al. [15] divided cloud images into several blocks and conducted the classification task on blocks. Xiao et al. [16] fused the texture, structure, and color features as the multi-view cloud visual features.

It is observed that appearance of clouds can be treated as a kind of natural texture. Therefore, it is reasonable to describe cloud appearances using texture and image descriptors. Sun et al. [17] utilized the local binary pattern (LBP) to classify the cloud images into five predefined types. Liu et al. [18–20] proposed several algorithms for extracting texture and image descriptors, such as the multiple random projections, the salient local binary pattern, and the group pattern learning. Recently, convolutional neural networks (CNNs) have shown remarkable performances in several fields, such as visual classification [21], object detection [22], and speech recognition [23]. The success of CNNs is attributed to their ability to learn rich representations. CNNs could achieve some degree of shift and deformation invariance by using local receptive fields, shared weights, and spatial subsampling. In particular, the shared weights are supportive for improving the generalization of CNNs, because they reduce the number of parameters of CNNs. Several researchers have resorted to training CNNs using cloud images for ground-based cloud classification. For example, Ye et al. [24] utilized cloud visual features from the convolutional layers of the network. Afterward, they employed Fisher vector encoding to further improve the cloud classification results. Shi et al. [25] extracted visual features from both the shallow and deep convolutional layers of the network. They also evaluated the performance of fully connected (FC) layer for cloud classification.

However, it is difficult to solve the problems of ground-based cloud classification by using one kind of sensor, i.e., image sensors. This is because the cloud type is determined by many factors, such as temperature, humidity, pressure, and wind speed. We are inspired by the recent development in weather station networks which is a kind of wireless sensor network (WSN) [26–28], and we consider to classify the ground-based cloud by using weather station networks. The weather station networks consist of many kinds of sensors [29], for example, image sensors, thermal sensors, moisture sensors, and wind speed sensors. These sensors have the abilities to obtain multimodal information of clouds. The visual, thermal, moisture, and wind speed information could collect more complete information for ground-based cloud information with the help of weather station networks, so the limitation of each kind of information could be compensated.

In this paper, we propose a novel method named deep multimodal fusion (DMF) for ground-based cloud classification in weather station networks. Concretely, we first fine-tune the pre-trained CNN model to adjust the parameters for visual cloud information. Then, the visual features are extracted from convolutional layers. Different from the other activation-based features, we apply a pooling strategy across all the feature maps to reserve the spatial information of cloud images. Furthermore, we also evaluate the performance of FC features. After obtaining the visual information of clouds, we fuse the multimodal information collected from weather station networks, e.g., temperature, humidity, pressure, and wind speed, into the final representations. This fusion strategy learns the complementary information between visual and multimodal features, which could further improve the performance. Finally, support vector machine (SVM) [30] is selected as the classifier.

The rest of this paper is organized as follows. Section 2 introduces the proposed approach in details, Section 3 illustrates the experimental results, and Section 4 draws the conclusion for this paper.

2 Method

In this section, we present the proposed method in detail, and the flowchart is illustrated in Fig. 1. The cloud images are first utilized to train a CNN model, and then the sum pooling is applied to aggregate the feature maps of one convolutional layer so as to obtain the visual features. Afterwards, the visual features and multimodal cloud features are integrated. Finally, we utilize the SVM to train the classification model.

Fig. 1

The flowchart of the proposed method. a Input cloud image. b The CNN model. c Convolutional activation-based features. d Sum pooling strategy for convolutional activation-based features. e Fusing visual features with the multimodal information. f The multimodal information. g The SVM classifier

2.1 Deep convolutional neural networks

In recent years, CNNs have achieved great success in classification task [21] due to large-scale databases and efficient computational abilities. Simonyan and Zisserman [31] proposed very deep neural networks which consist of more convolutional and pooling layers. The deep CNNs not only obtain outstanding performance on imagenet large-scale visual recognition challenge (ILSVRC), but also show promising performance on other classification tasks. Hence, we employ a deep CNN to extract features for cloud images.

Training a CNN model needs a very large number of annotated images to learn millions of parameters [32, 33]. Thus, it is running the risk of overfitting to train a CNN model from scratch by only utilizing a few thousand cloud images. To overcome this drawback, we first fine-tune a deep CNN model named imagenet-vgg-f [34] to transfer the cloud information to the deep model. The imagenet-vgg-f comprises five convolutional layers and three fully connected layers, and the detail configuration is shown in Table 1. The last FC layer has 1000 dimensionality, and we replace it with a new one with N dimensionality to start the fine-tuning procedure, where N is the number of cloud classes. Meanwhile, for the bias, all the parameters in the new FC layer are initialized to zero. For the weight, the parameters obey a Gaussian distribution. In addition, three max pooling layers follow the first, second, and fifth convolutional layer, respectively, with the size of 3×3 in conjunction with a downsampling factor 2. Moreover, two local response normalization layers are after the first two convolutional layers, respectively.

Table 1 The configuration of imagenet-vgg-f. convi denotes the i-th convolutional layer

2.2 Deep features for cloud images

The appearance of clouds can be treated as a kind of natural texture, and therefore, it is rational to describe cloud appearance using texture descriptors. The CNN models have been applied to capture texture information and have achieved promising results [35, 36]. The features extracted from deeper layers possess several desirable properties such as invariance and discrimination. On the contrary, the shallower layers tend to be more sensitive to small transformations, which is challenging for unpredictable and changeful cloud.

Based on the analysis mentioned above, we adopt deeper layers to extract the cloud features. For the convolutional layer, we aggregate the raw activations by sum pooling and then obtain the sum convolutional map (SCM). The activation value y _{i j} in SCM at position (i,j) is defined as

$$\begin{array}{@{}rcl@{}} \begin{aligned} y_{i j}=\sum_{k=1}^{C} {x_{i j}^{k}}, \end{aligned} \end{array} $$

(1)

where $x_{i j}^{k}$ is the activation at position (i,j) in the k-th feature map and C is the number of feature maps in the convolutional layer. Suppose the size of each feature map is H×W and the SCM is also with the size of H×W. The SCM preserves the spatial information because the pooling operation is conducted across all the feature maps, while the other traditional pooling operations aggregate one feature map into a feature. As a result, the convolutional activation-based features V _conv for each image is acquired by transforming the SCM into a vector

$$\begin{array}{@{}rcl@{}} \begin{aligned} V_{conv}&=\left[y_{1 1}, y_{2 1}, \cdots, y_{H 1}, y_{1 2}, y_{2 2}, \cdots, y_{H 2},\right.\\ & \qquad \left. \cdots, \cdots, y_{1 W}, y_{2 W}, \cdots, y_{H W}\right]^{\mathrm{T}}.\\ \end{aligned} \end{array} $$

(2)

The dimensionality of the vector is H×W. The above procedure is summarized in Fig. 2. On the other hand, we do not use any pooling strategies in FC layers. The FC layer-based features could be considered as a special case of convolutional layers. It utilizes rather smaller filter banks with the size of 1×1. The feature vector V _fc for a FC layer is indicated as

$$\begin{array}{@{}rcl@{}} \begin{aligned} V_{fc}=\left[ v_{1}, v_{2}, \cdots, v_{k}, \cdots, v_{K} \right]^{\mathrm{T}},\\ \end{aligned} \end{array} $$

(3)

Fig. 2

The sum pooling strategy for feature maps in a convolutional layer. a Input cloud image. b Feature maps of a certain convolutional layer. c The SCM feature. d The feature vector for a cloud image

where v _k is the output of the k-th neuron and K is the number of neurons in the FC layer. Finally, V _conv and V _fc are normalized by L₂-norm.

2.3 Multimodal fusion

To capture the complete cloud information, we integrate multimodal cloud information collected from weather station networks. The integration features can be formulated as

$$\begin{array}{@{}rcl@{}} \begin{aligned} Q=f (V,M),\\ \end{aligned} \end{array} $$

(4)

where V is the visual feature vector, i.e., activations-based or FC-based feature vector, and M=[m₁,m₂,⋯,m _p ]^T denotes the multimodal feature vector. For simplicity and efficiency, we directly catenate the visual feature vector with multimodal feature vector

$$\begin{array}{@{}rcl@{}} \begin{aligned} f(V,M)=\left[\alpha V^{\mathrm{T}}, \beta M^{\mathrm{T}}\right]\\ \end{aligned} \end{array} $$

(5)

where [ ·,·] indicates to concatenate two vectors, and α and β are the parameters to balance the importance between visual features and multimodal features. Note that the multimodal information M should be normalized by L₂-norm before fusion.

3 Experimental results

In this section, we conduct a series of experiments on the multimodal ground-based cloud (MGC) dataset to evaluate the effectiveness of the proposed DMF. We first introduce the MGC dataset and the implementation details of experiments. Then, we compare the proposed DMF with the other methods. Finally, we evaluate the influence of visual features extracted from different layers.

3.1 Dataset and experimental setup

The MGC dataset collected in China consists of cloud images and multimodal cloud information. The cloud images are captured by a sky camera with a fisheye lens under a variety of conditions. The fisheye lens could scan the sky with a wide angle. In the interim, we utilize a weather station to capture the multimodal information of clouds, that is to say temperature, humidity, pressure, and wind speed. We should note that the cloud images and multimodal information are collected at the same time. Therefore, each cloud image corresponds to a set of multimodal data. The MGC dataset is a challenging dataset, because it covers a wide range of sky conditions and possesses large intra-class variations. The MGC dataset comprises a total number of 1720 cloud data. According to the International cloud classification system criteria published in the World Meteorological Organization (WMO), considering the visual similarity in practice, the sky conditions are divided into seven classes, i.e., cumulus, cirrus, altocumulus, clear sky, stratus, stratocumulus, and cumulonimbus. Note that the clear sky is the condition that the cloud accounts for no more than 10% of the total sky. The number of cloud samples of each class varies from 140 to 350, and the detailed numbers are listed in Table 2. Herein, cloud classes are labeled using Arabic numerals from 1 to 7. Figure 3 shows some cloud samples from each class where each cloud image is with the size of 1056×1056.

Fig. 3

The cloud samples on the MGC dataset

Table 2 The sample number of each cloud class on the MGC dataset

The MGC dataset is randomly partitioned into 120 training samples for each class and the remaining ones as the test set. The partition process is implemented 10 times independently, and the final classification accuracy is reported as the average accuracy over these 10 random splits. For fair comparison, the same experimental setup is used for all the experiments. In the training stage, we first resize the original cloud images into 256×256 pixels with preserved aspect ratio by bilinear interpolation. Then, in order to learn more cloud information, we centrally crop the training images into 224×224 pixels. In addition, each training image subtracts the mean RGB values computed on training set from each pixel.

We shuffle the training set images to fine-tune the pre-trained imagenet-vgg-f model. To learn the parameters (weights and bias), we train the network using the backpropagation gradient-descent procedure [21] with a mini-batch size of 48. The fine-tuning procedure is terminated at 20 epochs. For the first 10 epochs, the learning rate is set to 0.001. While for the remaining 10 epochs, it is reduced by a factor of 10 to 0.0001. The weight decay is with the value of 0.0005. In the test stage, the images have the same pre-processing with those in the training stage. For the multimodal information fusion, we empirically set α and β in Eq. (5) to 1 and 0.8, respectively. The multimodal information vector M is [m₁,m₂,…,m₆]^T, where m₁,m₂,…,m₆ denotes the temperature, humidity, pressure, wind speed, average wind speed, and maximum wind speed in one minute, respectively. Finally, we treat SVM with radial basis function (RBF) kernel as the classifier.

3.2 Baselines

We compare the proposed method with the state-of-the-art methods which are listed as follows.

• (1) BoW [37] model: The bag-of-words (BoW) model represents cloud images as histograms over a discrete codebook of local features. We choose SIFT [38] descriptors as the local features. The codebook size for each cloud class is set to 200, which results in 1400 dimensionality histogram for each cloud image.

• (2) PBoW [39] model: The pyramid BoW (PBoW) model is that the BoW model incorporates with the spatial pyramid which could learn the spatial information of cloud images. We divide each cloud image into three levels, i.e., 1, 2, and 4, which results in 1, 4, and 16 cells, respectively. Thus, for each cloud image, it contains a total of 21 cells. The PBoW model also represents cloud images as histograms based on each cell. Herein, the codebook is obtained in the same way as BoW. Hence, the histogram for each cloud image is 29400 dimensionality.

• (3) LBP [40]: The local binary pattern (LBP) labels each pixel by computing the sign of the difference between the intensities of that pixel and its neighboring pixels. In our experiments, we utilize the uniform invariant LBP and set the parameter (P,R) to (8, 1), (16, 2), and (24, 3), respectively. Here, P is the total number of involved neighbors in a circle and R is the radius of the circle. Then we combine the representations from these three different conditions. Hence, the dimensionality of the representation for each cloud image is 54.

• (4) CLBP [41]: The completed LBP (CLBP) is an extension of LBP and has shown to perform well in image analysis and texture classification. In CLBP, a local region is represented by its center pixel, and the signs and magnitudes of the local differences. We combine these three components into joint distributions to obtain completed cloud representation. The parameter (P,R) is also set to (8, 1), (16, 2), and (24, 3), respectively. We concatenate the three scales into one feature vector resulting in a 2200 dimensional vector.

For all of the above feature extraction techniques, we use the same training set and test set as the proposed method. The only difference is that each cloud image is converted to gray scale with the size of 300×300.

3.3 Comparison with other methods

We first compare the deep visual features (DVF) learned from CNN with the other state-of-the-art methods, and the results are shown in Table 3. We extract the DVF from conv5, and therefore, the dimensionality of DVF is 169. From Table 3, we can see that the DVF obtains the best result. Especially, the classification accuracy of DVF is over 15% better than that of PBoW which achieves the second best result among all the compared methods. Furthermore, the dimensionality of DVF is much smaller than PBoW. The improvement of the proposed DVF is because the CNN model could learn more discriminative features than other learning-based methods (BoW and PBoW) and hand-crafted features (LBP and CLBP). Moreover, the sum pooling strategy ensures that the DVF possesses more cloud spatial information.

Table 3 Classification accuracy (%) using visual features

Then, we compare the proposed DMF with the other state-of-the-art methods for the multimodal information fusion, and the results are listed in Table 4. Note that “+M” indicates concatenating the visual features with multimodal information. From Table 4, we can see that the proposed DMF outperforms the other methods and the classification accuracy achieves over 86%. A comparison between Tables 3 and 4 shows that the classification accuracies in the latter are all better than those in the former. This demonstrates that the multimodal cloud information provides support for the ground-based cloud classification. This demonstrates that the multimodal cloud information is helpful for the ground-based cloud classification. The visual features and the multimodal cloud information are complementary and therefore fusing them could obtain the completed information of ground-based cloud. The improvement of DMF exceeds other methods, which verifies the effectiveness of the fusion algorithm.

Table 4 Classification accuracy (%) with multimodal information

In this paper, we focus on the feature representation of cloud, and any classifiers could be chosen, such as the 1-nearest neighbor (1NN) classifier. In Table 5, we summarize the classification accuracy with 1NN classifier. From the table, we can observe that the proposed method obtains the best results when utilizing 1NN classifier.

Table 5 Classification accuracy (%) with 1NN classifier for different methods

3.4 Influence of different parameters

In this subsection, we evaluate the performance of different layers in CNN for ground-based cloud classification. For the convolutional layers, the feature dimensionality is equal to that of SCM. For example, the size of SCM from conv5 is 13×13, and therefore, the dimensionality of V_cov5 is 169. For the FC layers, the feature dimensionality is equal to the number of neurons. For example, fc6 has 4096 neurons, and therefore, the dimensionality of V_fc6 is 4096. The dimensionality of visual features extracted from the trained CNN is concluded in Table 6. Then, we directly concatenate visual features with multimodal features to obtain the final features.

Table 6 The feature dimensionality of different layers

The classification performance of DVF and DMF are summarized in Table 7. Several conclusions can be drawn from the results presented in Table 7. First, the accuracy of conv5 achieves the best in both DVF and DMF. Second, comparing conv5 with other shallower convolutional layers, we can see that deeper convolutional layers could learn more semantic information. Third, the accuracy of conv5 is higher than that of fc6 and fc7, while the feature dimensionality of conv5 is much less than that of fc6 and fc7. It is because the sum pooling strategy across over all feature maps could keep more spatial information. Forth, the classification accuracies of DMF are all higher than that of DVF, which validates the effectiveness of the fusion between the visual features and the multimodal information.

Table 7 The classification accuracy (%) of DVF and DFM in different layers

Additionally, we compare the classification results of V_cov5 for different α and β settings. Since the ratio of α to β is important to fusion performance, we fix α and change β. The comparison results are listed in Table 8. From the table, we can see that when α and β are set to 1 and 0.8, respectively, the best classification accuracy is obtained. α is larger than β in the optimal situation which indicates that the visual features are more important than the multimodal features.

Table 8 Classification accuracy (%) of conv5 for different α and β settings

4 Conclusions

In this paper, the integration of the deep visual features and the multimodal information has been proposed for ground-based cloud classification in weather station networks. We first fine-tune the pre-trained deep CNN model using the cloud images, followed by extraction of deep visual features and then fused with the multimodal information. A series of comparative experiments have been conducted to test the effectiveness of the proposed DMF, and the results show that the accuracy of the proposed DMF is higher than the state-of-the-art methods.