An efficient method for meteorological nephogram recognition in cloud environment

Abstract

Nephogram could provide important information for meteorological business, and nephogram recognition is a kind of challenge in the meteorological industry. In this paper, a selective neural network ensemble method based on K-means and Hadoop processing technology is proposed. This method combines the neural network, K-means clustering and AdaBoost in cloud environment. The experimental results show that the recognition precision of the method proposed by this paper is higher than that of traditional method in a stand-alone environment.

1 Introduction

In recent years, with the wide application of data mining and image recognition technology, more and more applications based on image processing technology have appeared in people’s daily life. The innovation and development of these technologies, along with the huge amount of computation that comes with them, has led people to turn their perspectives into the same fast-growing cloud computing platform [1]. Data need to be offloaded to the remote cloud data centers [2–5]. However, because of the rapid rise in the volume of data, faster and more efficient computational techniques are required. A major benefit of big data is to provide timely information and proactive services for humans [6]. The advantages of cloud computing in data processing are obvious. It can meet different levels of requirements such as system changes, user needs and environmental changes. Cloud computing is an increase, use, and delivery model of Internet-based related services. In simple terms, cloud computing is to calculate, transform, and store all resources to form a giant cloud network data storage platform. Storage and computing tasks are ideal for recognizing algorithms [7–9].

In recent years, with the development of image processing and pattern recognition theory, research on automatic classification and recognition of nephogram has become a research hot spot of meteorological applications [10–13]. Common researches on the classification of ground-based nephogram mainly use the classification methods based on minimum distance classifier and neural network. The support vector machine is selected to classify the nephogram [14, 15]. Extracting texture features by means of autocorrelation, co-occurrence matrix as well as Laws energy has been applied, but there are only theory discussion and analysis. Literature [16, 17] used the minimum distance classification methods to achieve nephogram, but the method relied excessively on the selection of the initial center and easily fell into the local optimum. Literature [18, 19] proposed to use BP neural network classifier to automatically classify and identify nephogram, but this method does not overcome the shortcomings of the traditional BP neural network. And it is easy to fall into the local minimum which causes the algorithm to converge slowly, then result in unstable classification results. The traditional BP AdaBoost algorithm can be used for the classification of nephogram. Although it can overcome the disadvantage that a single BP neural network trapped into local minimum values and improve the accuracy of nephogram recognition, which weakened the performance of ultimately integrated classifier. A selective neural network ensemble method based on K-means clustering algorithm is proposed in this paper. Based on the BP neural network ensemble model, this method uses K-means clustering algorithm to select some individual neural networks with different measurements. Finally, multiple BP neural networks strong classifier with higher precision and diverse differences will come out through the integration of AdaBoost algorithm, which can be applied for the identification and classification of ground-based nephogram.

In this paper, this method combines cloud platform with BP neural network, K-means and AdaBoost algorithm, and it could implement the nephogram recognition based on cloud computing platform. On the basis of inheriting the high scalability and high efficiency of the cloud platform, the problems are existing in the traditional nephogram cloud recognition system in terms of computational efficiency and data processing capability can be efficiently solved.

2 Model description

In this section, we propose a nephogram recognition technology based on cloud computing platform, which is composed of cloud platform and a nephogram recognition algorithm based on selective neural network. Specifically, the algorithm is implemented by integrating the nephogram recognition algorithm into the cloud platform, aiming to use the cloud platform for computing, to improve the accuracy and efficiency of nephogram recognition.

Today, cloud computing is gradually becoming widely used, and the types of cloud computing methods are also increasing. Among them, Apache’s open-source distributed platform Hadoop has been widely used for its low price and high fault tolerance. Hadoop is mainly composed of three parts: file system HDFS [20], database Hbase, and distributed parallel computing Map Reduce. The most important computing and processing part of the platform is mainly realized by two key technologies: distributed file management system and parallel processing. As a processing platform for storage and computing, the core part of the Hadoop platform processing is the distributed file system HDFS and the parallel processing Map Reduce. The platform is well-structured and can be directly expanded for storage and computing without modification [21]. There are several key attributes of the Hadoop platform, such as scalability; the platform can be directly designed and deployed on low-cost hardware; the Hadoop platform automatically saves multiple copies of data and automatically restores allocations when errors occur. For Hadoop platforms, Map Reduce handles data in a bit-wise manner, making it safer and more efficient [22–25].

2.1 Nephogram recognition technology for stand-alone platforms

The nephogram recognition framework of the stand-alone platform is shown in Fig. 1.

• Image preprocessing module: Convert color images and store the converted grayscale images in memory to prepare for subsequent calculations.

  Fig. 1

  

  Nephogram recognition framework for stand-alone platforms. This is a figure recognition step diagram of a stand-alone platform. First, image preprocessing is performed, then feature extraction is performed, and then the classifier is trained to obtain image recognition results

• Image data calculation: Using the image data read by the relevant feature extraction module in the memory to perform operations, to ensure the consistency of the acquired image features and the nephogram features of the desired nephogram.

• Classifier training classification module: Through the application of K-means algorithm, BP neural network algorithm and AdaBoost algorithm, the module can train the acquired training data samples. The classifier obtained through the training is stored in the local file system for class determination of the nephogram.

When categorizing images, the traditional image classification takes a long time to extract image features. At the same time, it is necessary to read and write system memory and hard disk when the feature matrix is calculated with a large data volume, it has a negative impact on the correct rate of the program, and reduces system stability and reliability [26, 27].

2.2 Nephogram recognition algorithm based on cloud computing platform

The image recognition framework based on the cloud computing platform is shown in Fig. 2 [11].

Fig. 2

Nephogram recognition framework based on cloud platform. When the user submits a recognition request, the cloud platform will receive a request to initialize and assign an image classification job task, then the node will perform an image classification phase after acquiring the assigned task, and will obtain a nephogram classification result through the Reduce Task process, and finally the platform will complete the nephogram recognition task

• User submits request for nephogram recognition: The request will be firstly processed on Hadoop’s Job Tracker by using the Job Client nephogram recognition job ID. The project JAR package can be run in the image classification job by applying the Job Client to copy the configuration files and image feature classification data that the program depends on. After completing the above process, Job Client will automatically submit the job in Job Tracker. At first Job Tracker checks the related information of the job, inputs the information of the data, and obtains the job from the Hadoop distributed file system (HDFS) which prepares the job for execution.

• Initializing and assigning nephogram recognition job tasks: Job Tracker initializes the job after receiving the submitted job, and inserts the job into an internal task queue. And Hadoop’s job scheduler schedule tasks in this queue. Job Tracker can assign job tasks to cluster nodes by referring to the Job Tracker’s heartbeat situation. Because some feature data is stored in the Data Node, the Job Tracker will directly obtain input data when performing Map Task, and it will effectively reduce network loss during data transmission.

• Nephogram classification Map stage: When the Job Tracker obtains the assigned task, the program’s related data and JAR files will be automatically obtained from HDFS, and stored in the local file disk. Then, the JAR data and files will be loaded by the execution of the local virtual machine. The data block will be obtained in the instance of Task Tracker.

• Reduce tasks that used for nephogram classification: The nodes classify the feature values according to the corresponding key value Map Reduce framework. If the key values of the intermediate results are the same, then the feature values of these key-value pairs will be fully collated and merged, and the merged results are processed by Reduce Task. Output routing of Map Task will use the Reduce Task, and get the results of nephogram classification and write the results to HDFS.

• Complete the nephogram classification task: The completion of the image classification task is to use the job to identify and display the results, and the parameters of the job run can be obtained through the Job Tracker. Job Tracker can get all job status, Task Tracker can delete the intermediate results generated by the Map stage. The user can access the results files on HDFS.

2.3 A nephogram recognition method based on the selective neural network on the could computing platform

Literature [28] gives the Hadoop cloud platform system architecture (Fig. 3).

• Uploading data information on the Hadoop cloud platform. The data source is obtained from the HDFS, after the job is submitted, and the data is divided and processed by the data cluster configuration. Then the Reduce and Map jobs are classified and processed. The nodes information in the Reduce and Map processes can be input from the configurations.

  Fig. 3

  

  Hadoop cloud computing platform system architecture. The figure shows the architecture of the Hadoop cloud computing platform system, including the Tomcat server, web human-computer interface, and cloud computing platform

• Operation process. The nephogram samples stored in the HDFS are read into the system, and the data samples parameter types are converted at the same time. The genetic algorithm is used to optimize the converted the combined parameters. At the same time, the BP neural network algorithm, the K-means clustering algorithm and the AdaBoost algorithm function are called. They acquire the classification through the training of sample data, and establish a classification model of foundation nephogram, and pass the processing result to the Reduce process.

• 3) Implementing the Reduce process. This process will implement the Reduce operation process, which is used to classify and sort the data form key /value converted by the Map function, and input the processed data into the implementation path.

3 Methods

In this section, we describe the principle of BP neural network, K-means algorithm and AdaBoost algorithm in detail. Through these algorithms, we propose a nephogram recognition algorithm based on selective neural network integration which can effectively solve the problem of low recognition model accuracy.

3.1 Main idea of selective neural network

Differences in training sets and differences in initial weights and thresholds for each BP network are importantly influencing factors between individual networks. However, the traditional BP Adaboost integration methods may produce neural network classifiers that are very similar to each other because of the randomness of the learning algorithm, while integrating less differentiated classifiers may not necessarily reduce the integration generalization error. It is also possible that the generalization ability of the algorithm is weak. The higher the accuracy of each network that constitutes the integrated learning model, the greater the difference between the members, and the more favorable the reduction of the integrated learning generalization error [12]. In the case of limited training sets, the difference between training sets are obtained by repeating sampling technique, while the raw data may repeat in the sub-training set or not. The Bootstrap technique is used to sample N subsets of n training sets by means of the BP neural network to independently obtain the weights and thresholds of each network, and classify the obtained ownership thresholds and obtain the same clustering center as the base classifiers via the K-means algorithm [17]. Finally, the weights and thresholds of the individual classifiers close to each cluster center are selected as the initial weights and thresholds of the M base classifiers, thus forming M different base classifiers through the AdaBoost algorithm. The M base classifiers are cascaded into a final strong classifier to establish classification model for ground-based nephogram images.

3.2 Traditional BP AdaBoost classifier model

The traditional BP AdaBoost models, which consider the BP neural network as the weak classifier, train multiple BP neural network weak classifiers to classify samples under different sample distribution, which are combined into strong classifiers at last via the AdaBoost algorithm [13, 15, 19]. The model based on the traditional BP AdaBoost is used in this paper to recognize nephogram, and the algorithm flow is shown in Fig. 4.

Fig. 4

BP AdaBoost classifier for nephogram. This figure reflects the BP AdaBoost classifier nephogram recognition principle, input meteorological nephogram feature library, iterate through AdaBoost algorithm, and then output strong classifier

As the following, the steps of strong classifier algorithm trained by the BP AdaBoost are:

(1) Selecting data and initializing BP network

The m group training data are randomly selected from the cloud samples, and the distribution weight the AdaBoost algorithm $\mathop D\nolimits _{1} \left (i \right) = 1/m$ and the weight and threshold of the BP neural network are initialized. Then, the structure of the neural network is determined by the input and output dimensions of the samples nephogram images.

(2) Training weak classifiers

When training the t^th weak classifier, the training data of the distribution weights after the (t−1)^th training are used as the input of the BP neural network. ε_t is the same of the error of the classifier g_t, is calculated as follows:

$$\begin{array}{@{}rcl@{}} {\varepsilon_{t}} = \sum\limits_{i} {{D_{t}}\left(i \right)},i = 1,2, \ldots,m\left({{g_{t}}\left({{x_{i}}} \right) \ne {y_{i}}} \right) \end{array} $$

(1)

(3) Calculating the weight of the classifier

The weights of each classifier are mainly calculated according to the error of each classifier. The calculation formula of the t^th classifier weight is:

$$\begin{array}{@{}rcl@{}} {\alpha_{t}} = \frac{1}{2}\ln \left({\frac{{1 - {\varepsilon_{t}}} }{{{\varepsilon_{t}}}}} \right) \end{array} $$

(2)

(4) Adjusting the weights of training data

The weights of the (t+1)^th classifier training samples are adjusted by the weights α_t of the t^th classifier, the weights adjustment formula is:

$$\begin{array}{@{}rcl@{}} {D_{t + 1}}(i) = \frac{{{D_{t}}(i)}}{{{Z_{t}}}} \times \exp \left({ - {\alpha_{i}}{y_{i}}{g_{t}}({x_{i}})} \right),i = 1,2, \ldots \ldots,m \end{array} $$

(3)

In the formula Z_t is the normalization factors, so that the sum of the weights of the samples is 1.

(5) Generating a strong classifier for the nephogram

The AdaBoost algorithm obtains T BP neural network weak classifiers after training for T times. The final strong classifier h(x) is obtained by combining with T weak classifiers:

$$\begin{array}{@{}rcl@{}} h(x) = \frac{{{\alpha_{t}}}}{{\sum\limits_{t = 1}^{T} {{\alpha_{t}}} }}{g_{t}}(x) \end{array} $$

(4)

3.3 K-means clustering

The principle of K-means clustering is to group similar objects into the same cluster and group dissimilar objects into different clusters by means of similarity calculation methods. This method has specific geometric and statistical significance. In this paper, the trained samples based on neural network classifiers are clustered by this idea, and the individual classifiers with partial differences are integrated selectively while ensuring accuracy. The working flow of the K-means clustering algorithm are as follows:

Input: Training data set X={x₁,x₂,……,x_n}, the number of clusters k;

Output: k result clusters C_j,j=1,2,…,k;

Step 1. Set I=1, and select randomly k training samples as the initial cluster centers of k clusters m_j(I),j=1,2,…,k, I is the iteration times of the algorithm;

Step 2. In distance selection, Euclidean distance is adopted in this paper. Calculate the distance d(x_i,m_j(I)),i=1,2,…,n; j=1,2,…,k of each training sample from the centers of the k clusters, if d(x_i,m_i(I))= min{d(x_i,m_j(I)),j=1,2,…,k}, then set x_i∈C_j ;

Step 3. Calculate k new cluster centers:

$$\begin{array}{@{}rcl@{}} {m_{j}}\left({I + 1} \right) = \frac{1}{{{N_{j}}}},\sum\limits_{i = 1,{x_{i}} \in {C_{j}}}^{{N_{j}}} {{x_{i}}},j = 1,2, \ldots,{\mathrm{k}} \end{array} $$

(5)

Step 4. If m_j(I+1)≠m_j(I),j=1,2,…,k, set I=I+1, then jump to step 2; otherwise the algorithm ends.

4 Experimental results and analysis

In this section, the research and application of nephogram recognition technology based on cloud computing platform through the project of China Special Fund for Meteorological Research in the Public Interest was carried out. In the exploration and application, the test environment, test data, test results, analysis, and evaluation of experimental were carried out. The results show that the cloud recognition method based on cloud computing platform proposed in this paper has achieved good effect.

4.1 Experimental data set

In this paper, a total of 800 samples were selected. Each type of nephogram sample was 200, and 150 samples were selected as training samples for each type. The remaining samples were used as test samples to analyze and verify the method. Four kinds of typical clouds, including bald cumulonimbus, light cumulus, cirrus, and stratus, were selected to build the classified sample database. The data comes from the cloud map data base of each comprehensive observation base and the cloud map data base of each meteorological bureau official website.

4.2 Experimental context

Construction of the experimental platform: Each node in Hadoop has the same configuration as the host and the computer used by the single machine: the memory is 8 GB, DDR3 1333 MHz, 1T hard disk and Intel I5 4950 CPU. Ubuntu 14.10 operating system, eclipse-jee-juno-SR2 version and Hadoop 1.0.0 version IDE platform, Java implementation in JDK 1.7.0 07 environment.

4.3 Experimental results and analysis

According to the needs of meteorological observation and weather forecasting, the weather nephogram is divided into low, medium and high level by the height of the weather nephogram bottom from the ground. According to the macroscopic characteristics, physical structure and genesis of the weather nephogram, the meteorological nephogram is divided into ten genus and 29 species [29]. In this paper, four typical meteorological nephogram classes, such as cumulonimbus calvus, Cu hum, cirrus spissatus, and stratus meteorological nephogram, are selected to establish a classification sample library. The experimental samples are collected by meteorological observation base in the Nanjing University of Information Science and Technology, the meteorological nephogram data website of the Shanxi Meteorological Bureau, China personal digital camera, and the encyclopedia of the China Nephogram Images. The four typical nephogram samples are shown in Fig. 5.

Fig. 5

Meteorological nephogram sample images. The figure shows the characteristics of four typical meteorological nephogram samples, including stratus clouds, Cu hum, cirrus spissatus, and cumulonimbus calvus. a Stratus cloud. b Cu hum. c Cirrus spissatus. d Cumulonimbus calvus

Usually in Fig. 5 the cumulonimbus nephogram is thick, large and towering, with varying heights, dark bottoms, and smooth or even texture. The cumulus nephogram are generally cumulus clusters with well-defined contours and flat. The nephogram is not connected to each other, causing texture fluctuations and uneven boundaries. The texture of the cirrus is white filament or narrow strip, and the distribution is relatively uniform. The bottom of the layer nephogram is very low; the texture is evenly layered, like fog, but not connected to the ground; and the thickness and height of the nephogram top are very small.

In the experiment with SVM on the stand-alone version, it was found that the performance was poor compared with the integrated neural network, so we did not choose SVM in meteorological nephogram. In the algorithm model based on cloud computing platform proposed in this paper, the Bootstrap method is firstly used to generate a N(N=200) meteorological nephogram image training set with n size (n=150, sharing the same size as the original training set). Then, each group trains out a neural network weak classifier, and cluster the initial weights and thresholds of each weak classifier generated to obtain M class clusters via the K-means algorithm(M shares the same number with the number of base classifiers, which are experimentally explored from base classifiers in the integrated network in this paper, M=15). The initial weights and thresholds of individual classifiers close to the center of M clusters are selected, and their values are used as the initial weights of the integrated network individuals. The threshold is finally integrated into a strong classifier by the AdaBoost algorithm. The error rate comparison between the traditional BP AdaBoost algorithm model and the proposed algorithm model on the nephogram test samples is shown in Fig. 6 (the training error precision value of each base classifier is 0.003) when the number of base classifiers is the same.

Fig. 6

Error rate for the two methods. This picture shows the difference in error rate between the two algorithms on the nephogram test set when the number of base classifiers is the same. It can be seen from the figure that the algorithm model of this paper is better than the traditional BP AdaBoost algorithm

In the traditional BP AdaBoost integration mode, due to the generated randomly initial weights and thresholds of each base classifier, a large difference between the networks is hardly guaranteed that the differences of the various base classifiers generated by the training are not obvious enough. Integrating these base classifiers does not improve the generalization ability of the entire meteorological nephogram classifier, and the nephogram test set error rate curve shows a turbulent trend, as shown in Fig. 6. However, the K-means algorithm proposed in this paper selectively selects the weights and thresholds of individual networks to train the classifiers, so that the generated base classifiers have certain differences, and the individual networks integrating these differences are used for the Identification of nephogram identification. The error rate can continue to decrease steadily, which demonstrates in Fig. 6 that the error rate of the strong classifier generated by the algorithm model proposed in this paper is significantly lower than that of the traditional BP AdaBoost algorithm on the meteorological nephogram test set.

The accuracy formula in this paper is as follows:

$$\begin{array}{@{}rcl@{}} R = \frac{\text{nc}}{\text{nt}} \end{array} $$

(6)

Where nt is number of total samples, and nc is number of correct samples.

This paper also compares the performance of the model proposed in this paper is used in the nephogram classification in the single-machine environment and the Hadoop platform uses the model proposed in this paper as well as the model a single BP neural network model in different accuracy of training error in the base classifier. For the three models, we conducted a total of 4 time tests, with the training error accuracy value of each base classifier set with 0.04, 0.01, 0.008, and 0.003 in per test. Moreover, in the integration model, the number of basic classifiers is taken with 15, the experimental results are shown in Fig. 7, where the x-axis represents the experimental number, the y-axis is the error rate of the measured test sample. As shown in Fig. 7, the higher the accuracy of the various networks that comprise the integrated learning model, the greater the difference between the memberships, the more conducive to reduce the error in learning generalization integration.

Fig. 7

Error rate of different models. This figure shows the performance difference of three algorithm models in nephogram classification and recognition

The verification experiment mentioned above of cloud data demonstrates that method for selectively integrating BP neural network classification based on K-means algorithm based on cloud platform presented by this paper can obtain better recognition accuracy, more reliable stability, and also higher meteorological nephogram classification performance compared with traditional BP AdaBoost.

In addition, in terms of the comparison of program time, all samples need to be calculated on a single machine, so the time is relatively long. In the cloud platform environment, the task is scattered, so the time is relatively short.

4.4 Evaluation of the experimental performance

The main purpose of the experiment is to verify the correctness of the proposed method by comparing the recognition results of each method, and combine it with the actual nephogram recognition for comparison. This shows that the method proposed in this paper can effectively improve the performance of cloud classification.

5 Related work

In recent years, The huge amount of computation with innovation and development of image recognition technology has led people to turn their perspectives into the same fast-growing cloud computing platform [1]. Data need to be offloaded to the remote cloud data centers [2–5]. A major benefit of big data is to provide timely information and proactive services for humans [6]. The advantages of cloud computing in data processing are obvious. In simple terms, cloud computing is to calculate, transform, and store all resources to form a giant cloud network data storage platform. Storage and computing tasks are ideal for recognizing algorithms [7–9].

With the development of image processing and pattern recognition theory, research on automatic classification and recognition of nephogram has become a research hot spot of meteorological applications [10–13]. Common researches on the classification of ground-based nephogram mainly use the classification methods based on minimum distance classifier and neural network. The support vector machine is selected to classify the nephogram [14, 15]. Extracting texture features by means of autocorrelation, co-occurrence matrix as well as Laws energy has been applied, but there are only theory discussion and analysis. In literature [16, 17], it used the minimum distance classification methods to achieve nephogram, but the method relied excessively on the selection of the initial center and easily fell into the local optimum. In literature [18, 19], it proposed to use BP neural network classifier to automatically classify and identify nephogram, but this method does not overcome the shortcomings of the traditional BP neural network. And it is easy to fall into the local minimum which causes the algorithm to converge slowly, then result in unstable classification results. The traditional BP AdaBoost algorithm can be used for the classification of nephogram. Although it can overcome the disadvantage that a single BP neural network trapped into local minimum values and improve the accuracy of nephogram recognition, which weakened the performance of ultimately integrated classifier.

As far as we know, there are many researches on nephogram recognition algorithms, which aim to continuously improve the accuracy and efficiency of nephogram recognition. However, the above methods have many limitations and shortcomings, and the nephogram cannot be identified and classified accurately and efficiently.

6 Conclusion

In this paper, the image processing technology, a k-means algorithm-based selective neural network ensemble method and Hadoop processing technology are used, and the nephogram recognition technology based on cloud computing platform is proposed by combining the cloud platform with BP neural network algorithm, K-means algorithm and AdaBoost algorithm. This method is validated by a nephogram recognition instance. It is more advantageous than single-machine running algorithms and single-neural network algorithms.