- Open Access
Statistical wavelet-based anomaly detection in big data with compressive sensing
© Wang et al.; licensee Springer. 2013
- Received: 25 September 2013
- Accepted: 1 November 2013
- Published: 17 November 2013
Anomaly detection in big data is a key problem in the big data analytics domain. In this paper, the definitions of anomaly detection and big data were presented. Due to the sampling and storage burden and the inadequacy of privacy protection of anomaly detection based on uncompressed data, compressive sensing theory was introduced and used in the anomaly detection algorithm. The anomaly detection criterion based on wavelet packet transform and statistic process control theory was deduced. The proposed anomaly detection technique was used for through-wall human detection to demonstrate the effectiveness. The experiments for detecting humans behind a brick wall and gypsum based on ultra-wideband radar signal were carried out. The results showed that the proposed anomaly detection algorithm could effectively detect the existence of a human being through compressed signals and uncompressed data.
- Anomaly detection
- Big data
- Through-wall human detection
- Compressive sensing
Anomaly detection refers to finding inconsistency with the desired pattern in data, which is also known as novelty detection, anomaly mining, and noising mining. Anomaly detection has many applications, such as credit card fraud detection, medical diagnostics information anomaly detection, industrial equipment fault detection and structural defect detection, network intrusion detection, and a novel theme of text mining. Currently, anomaly detection methods include anomaly detection based on classification, anomaly detection based on the nearest neighbor method, anomaly detection based on clustering, statistical anomaly detection, anomaly detection based on information theory, spectral theory anomaly detection, and so on . All the anomaly detection techniques can operate in one of the three modes: supervised anomaly detection which assumed the availability of a training data set that labeled instances for normal as well as anomaly class, semi-supervised anomaly detection which only assumed the availability of a normal data set, and unsupervised anomaly detection which did not require training data.
‘Big data’ refers to large, diverse, complex, longitudinal, and distributed data sets generated from instruments, sensors, internet transactions, email, video, click streams, and other digital sources available today and in the future. Some core technologies are needed to solve the problem in big data, such as effective data collection and storage and data mining techniques and anomaly detection method. However, the existing anomaly detection algorithms are mainly based on the complete data. It greatly limits the application of anomaly detection algorithms in big data [2, 3]. The main reason is that big data acquisition and storage become increasingly difficult with the increasing amount of data because of the sampling bandwidth and storage space constraints. Meanwhile, anomaly detection based on the complete data would inevitably expose some sensitive data which the data owner did not want to be leaked, so the effective privacy protection is the problem needed to be overcome currently.
Compressive sensing (or compressed sampling, CS) theory suggested that a high-dimensional signal can be projected into a low-dimensional space with a random measurement matrix when the signal was sparse or compressible which was proposed by Donoho and Candès in 2006 [4, 5]. Then, the original signal can be reconstructed from the low-dimensional information by solving an optimization problem. In other words, the low-dimensional signal contained the main features of the original signal, so CS theory can provide an effective method for anomaly detection in big data.
Usually, the number of normal data instances is more than the number of anomaly data instances. This just meets the sparsity requirements of compressive sensing theory. In , a compressed sensing-based clone identification scheme for sensor networks has been proposed according to the sparsity of clone appearance in the network. The data was recovered at the data base station, and then the safety of sensor networks was monitored. The algorithm achieved the lowest communication overhead. In , the compressed sensing theory was used for wireless network data acquisition and achieved the anomaly detection during a signal recovery procedure based on the modified BP reconstruction algorithm. Moreover, considering the asymmetry of anomaly and energy consumption, an improvement was made by giving a distributed detection algorithm. However, the main purpose of compressive sensing theory in the above research was to reduce the pressure of data storage and transmission, but it did not realize the anomaly detection in the compressed domain and cannot overcome the large amount of data calculation and privacy protection issues. Saha used the compressed sensor network data for anomaly detection based on the spectrum theory method and obtained satisfactory detection results in the light of residual analysis of compressed data [8, 9]. In [10, 11], random projection in conjunction with principal component analysis (PCA) was implemented for anomaly detection in the compressed domain. An application of this proposed methodology to detect IP-level volume anomalies in the computer network traffic suggested a high relevance to practical problems. Although this type of methodologies can achieve the anomaly detection in the compressed domain and has privacy protection features, the algorithms were complex, computationally intensive, and insufficient in real-time detection.
According to the problem in anomaly detection of big data, an anomaly detection algorithm in the compressed domain is proposed which makes full use of the advantages of compressive sensing technology. The experiment for through-wall human detection with the provided algorithm is carried out to test the effectiveness of the algorithm. The remainder of the paper is organized as follows. In Section 2, the criterion and procedure of anomaly detection are deduced base on wavelet packet transform and statistical method. The compressive sensing theory is then introduced. Experimental results for the detection of a human being will be shown in Section 3. Conclusion and discussion are in Section 4.
2.1 Anomaly detection procedure
The conventional signal frequency spectrum analysis is built on the basis of Fourier transform (FT) which is a global transform. However, FT has a low-frequency resolution and cannot recognize the subtle changes of the frequency spectrum. Wavelet transform (WT) may be viewed as an extension of the traditional FT with adjustable window locations and sizes. Compared with the Fourier-based analyses that use global sine and cosine functions as bases, the basis wavelets are local functions, each of which is defined by two parameters: its scale (relating to frequency) and its position (relating to time). One possible drawback of the WT is that its frequency resolution is quite poor in the high-frequency region. The wavelet packet transform (WPT) is one extension of the WT that provides a complete level-by-level decomposition. The WPT enables the extraction of features from signals that combine stationary and nonstationary characteristics with an arbitrary time-frequency resolution.
According to the theory of WPT, each node signal contains information of the original signal in a specific time-frequency window. Hence Equation 2 illustrates that the node signal energy is the energy stored in the corresponding frequency band. Obviously, the frequency components will be varied when anomaly occurs in the original signal. Thus, the anomaly detection can be achieved by investigating the changing trend of .
where is the node signal energy ratio in the total signal energy and is the reference baseline of node signal energy ratio which is the mean value of node signal energy by measuring some subsequent signals. In order to eliminate the noise effect, the first m dominant nodes are retained. It can be deduced that anomalies in the signal would affect the wavelet node signal energies and subsequently alter the criterion ‘ADC.’ However, the anomalies are not the only factor that can affect the criterion which also can be influenced by measurement noise. It is essential to establish threshold values for the anomaly detection criterion so that the criterion can be used to extract anomalies from measurement noise with a large probability . In this paper, the threshold values would be fixed based on statistical process control (SPC) and statistical process control charts which are used to describe the output characteristics of the process in a coordinate graph.
SPC was proposed by Dr. W.A. Shewhart, and the statistical process control charts described the output characteristics of the process in a coordinate graph. The algorithm was proposed to monitor the manufacturing process so as to reduce variation and guarantee quality into the product. The control chart includes center line, upper control limit (UCL), and lower control limit (LCL). For anomaly detection applications, the core of the technique is to establish the control limits that can enclose variation of the extracted criterion due to measurement noises with a large probability.
where Z α is the value of a standard normal distribution with zero mean and unit variance such that the cumulative probability is 100 (1 - α)%. The level UCL α can be regarded as the threshold value of the criterion. Therefore, if no anomalies happen, the average criterion ADC of a followed subgroup would be in the range of UCL α with a high probability of 100 (1 - α)%. On the other hand, when the average criterion ADC of a consecutive subgroup is beyond the limit, it shows that there are some anomalies. However, it should indicate that the SPC is a statistical principle of hypothesis testing. So, there are two types of hypothesis testing errors. Usually, the confidence limit can be improved by increasing the sizes p and q. From the anomaly detection procedure, it can be seen that no training data is required to construct a mathematical mode for anomaly detection. That is to say that the proposed algorithm belongs to unsupervised anomaly detection and can achieve online anomaly detection.
2.2 Compressive sensing theory
Although the proposed algorithm can overcome the noise interference and achieve anomaly detection with high probability, this algorithm is based on the complete data which greatly limits the application in the big data field. Compressive sensing theory overwhelms the limitation of Nyquist sampling theory and can acquire and compress data simultaneously. The theory provides a feasible basis for the proposed anomaly detection algorithm in the big data field.
where Ψ is a M × N random measurement matrix and M < N. Here, (Φ, Ψ) is a pair of orthobases which followed the incoherence restriction.
If x is termed as K sparse in the orthonormal basis, then we only need to collect M = O(K log(N/K)) random measurements to recover the signal by l1-norm algorithm. Many recovery algorithms based on linear programming, convex relaxation, and greedy strategies have been proposed to solve Equation 8, such as matching pursuit (MP), orthogonal matching pursuit (OMP), StOMP, subspace pursuit (SP), and CoSaMP. Finally, the reconstruction signal x can be given by .
According to compressive sensing theory, the acquired low-dimensional signal contained the main features of the signal under the premise of appropriate measurement matrix. So, the frequency component features would be included in the gathered low-dimensional signal. Therefore, the proposed anomaly detection procedure for big data could be carried out in the compressed domain.
The performance of our approach was evaluated by using experiments of through-wall human detection.
3.1 Through-wall human detection based on UWB radar
Through-wall human detection is of great interest for many applications, such as military reconnaissance, anti-terrorism, and medical and natural disaster ambulance, which can penetrate non-metallic media to detect life signal in far-off areas. Due to the strong anti-interference ability, high-resolution performance, and good target recognition capabilities, the ultra-wideband (UWB) radar has emerged as one of the most optimal choices for through-wall human detection which can emit very short duration pulses to penetrate walls, bulkhead, and other obstacles. The through-wall detection of a human being is based on the fact that the human body is always in a state of motion even if it sleeps or is trapped because of breathing. These tiny human motions would cause the scattering and reflection changes of an electromagnetic wave which is emitted by the UWB radar and passes through walls to reach the human body target. Furthermore, human target detection, location, and tracking can be achieved through signal extraction and analysis of human characteristics from the radar echo signal [19–21]. Qilian Liang from the University of Texas at Arlington has investigated these topics deeply [22–29].
However, the UWB echo signals usually are the big data which increase the burden of data sampling and storage, and it is difficult for anomaly detection or through-wall human detection with complete UWB echo signal data. In this paper, we would use the proposed anomaly detection procedure in the compression domain for through-wall human detection to verify its feasibility.
3.2 Experimental results and analysis
In the experiments, we used the P220 UWB radar of Time Domain Company as the detection tool, and it worked in monostatic mode wherein the waveform pulses were transmitted from a single omnidirectional antenna and scattered waveforms were acquired by a collocated omnidirectional antenna. In the project, the P220 UWB radar was working with a center frequency of 4.3 GHz. In this project, data were collected at three different locations with different types of walls which were brick wall and gypsum wall. The radar parameters were the same and were as follows: hardware integration = 512, software integration = 2, pulse repetition frequency 9.6 MHz, step size 13 bin, and window size 10 ft. The echo signals were collected at two statuses: no person behind the wall and a still human being standing behind the wall.
To verify the feasibility of the proposed algorithm, each of the 30 sets of echo signals having a status of without a human being and with a human being was acquired. The signals were decomposed by an eight-layer wavelet packet with a ‘db10’ wavelet. The first 20 node signals were used to construct the criterion ADC. The length of subgroup is q = 5. The upper confidence level is 1 - α = 0.95. The random Gaussian measurement matrix was chosen for anomaly detection in the compression domain based on compressive sensing.
Through-wall human detection with compressed data for gypsum wall
Average criterion value of a subgroup
Compressive ratio 1:5 UCL α = 0.973
Without a human being
With a human being
Compressive ratio 1:3 UCL α = 1.289
Without a human being
With a human being
Through-wall human detection with compressed data for brick wall
Average criterion value of a subgroup
Compressive ratio 1:5 UCL α = 1.1419
Without a human being
With a human being
Compressive ratio 1:3 UCL α = 1.1938
Without a human being
With a human being
According to the proposed anomaly detection procedure, from the experimental results, it can be seen that the identification method could effectively detect the existence of a human being in complete data and compressed data with a high probability of 95%. Meanwhile, the results showed that the difference of the average criterion of a subgroup between without a human being and with a human being became more obvious in the compressed domain. It is mainly because more features of the original signal have been eliminated in the compressed domain and affected the reference baseline of the node signal energy . Otherwise, due to the random feature of the measurement matrix, we got the average criterion value of a subgroup by calculating three times and acquired the mean value as the criterion value.
Big data is a collection of data sets which are too large and complex to be processed using traditional data processing methods. Anomaly detection is an important problem that has been researched within diverse research areas and application domains. However, traditional anomaly detection techniques based on complete data are confronted with many difficulties for big data.
According to the principle that the frequency components would vary because of the existence of anomaly, the paper proposed an anomaly detection algorithm based on wavelet packet transform and statistic process control theory. Due to the sampling and storage burden of anomaly detection in big data, compressive sensing theory was used in the proposed anomaly detection method. Subsequently, we proved that through-wall human detection using compressed data could achieve equivalent performance as it did using the original uncompressed data and reduced the computational cost significantly based on the proposed anomaly detection technique.
The authors would love to thank Professor Qilian Liang from the University of Texas at Arlington for providing the UWB radar data. This research was supported by the Tianjin Younger Natural Science Foundation (12JCQNJC00400) and National Natural Science Foundation of China (61271411).
- Chandola V, Banerjee A, Kumar V: Anomaly detection: a survey. ACM Comput. Surv. 2009, 41(3):1-72.View ArticleGoogle Scholar
- Chou CT, Rana R, Hu W: Energy efficient information collection in wireless sensor networks using adaptive compressive sensing. IEEE 34th Conference on Local Computer Networks, Zurich, Switzerland, 20–23 October 2009 443-450.View ArticleGoogle Scholar
- Wang J, Tang S, Yin B, et al.: Data gathering in wireless sensor networks through intelligent compressive sensing. Proceedings IEEE INFOCOM, Orlando, FL, 25–30 March 2012 603-611.View ArticleGoogle Scholar
- Donoho DL: Compressed sensing. IEEE Trans. Inf. Theory 2006, 52(4):1289-1306.MathSciNetView ArticleMATHGoogle Scholar
- Candès E: Compressive sampling. In Proceedings of International Congress of Mathematicians. Madrid: European Mathematical Society Publishing House; 2006:1433-1452.Google Scholar
- Yu CM, Lu CS, Kuo SY: CSI: compressed sensing-based clone identification in sensor networks. 8th IEEE International Workshop on Sensor Networks and Systems for Pervasive Computing 2012, Lugano, 9–23 March 2012 296-301.Google Scholar
- Xia Y, Zhao Z, Zhang H: Distributed anomaly event detection in wireless networks using compressed sensing. Proc IEEE ISCIT 2011, Hangzhou, 12–14 October 2011 250-255.Google Scholar
- Budhaditya S, Pham DS, Lazarescu M, Venkatesh S: Effective anomaly detection in sensor networks data streams. 9th IEEE International Conference on Data Mining, Miami, FL, 6–9 December 2009 722-727.Google Scholar
- Pham DS, Venkatesh S, Lazarescu M, Budhadity S: Anomaly detection in large-scale data stream networks. Data Min. Knowl. Discov. 2012. doi: 10.1007/s10618-012-0297-3Google Scholar
- Ding Q, Kolaczyk ED: A compressed PCA subspace method for anomaly detection in high-dimensional data. IEEE Trans. Inf. Theory 2012, 59(11):7419-7422.View ArticleGoogle Scholar
- Pham DS, Saha B, Lazarescu M, Venkatesh S: Scalable network-wide anomaly detection using compressed data, Technical report. Institute for Multisensor Processing and Content Analysis, Department of Computing, Curtin University of Technology; 2009.Google Scholar
- Sun Z, Chang CC: Statistical wavelet-based method for structural health monitoring. J. Struct. Eng. 2004, 130(7):1055-1062. 10.1061/(ASCE)0733-9445(2004)130:7(1055)View ArticleGoogle Scholar
- Candès E, Wakin M: An introduction to compressive sampling. IEEE Signal Process Mag. 2008, 25: 21-30.View ArticleGoogle Scholar
- Xu L, Liang Q, Cheng X, Chen D: Compressive sensing in distributed radar sensor networks using pulse compression waveforms. EURASIP J. Wirel. Commun. Netw. 2013. doi: 10.1186/1687-1499-2013-36Google Scholar
- Xu L, Liang Q: Zero correlation zone sequence pair sets for MIMO radar. IEEE Trans. Aerosp. Electron. Syst. 2012, 48(3):2100-2113.MathSciNetView ArticleGoogle Scholar
- Xu L, Liang Q: Orthogonal pulse compression codes for MIMO radar system. IEEE Globecom, Miami, FL, 6–10 Dec. 2010Google Scholar
- Xu L, Liang Q: Waveform design and optimization in radar sensor network. IEEE Globecom, Miami, FL, 6–10 Dec. 2010Google Scholar
- Liang Q, Mendel JM: Design interval type-2 fuzzy logic systems using SVD-QR method: rule reduction. Int. J. Intell. Syst. 2000, 15(10):939-957. 10.1002/1098-111X(200010)15:10<939::AID-INT3>3.0.CO;2-GView ArticleMATHGoogle Scholar
- Singh S, Liang Q, Chen D, Sheng L: Sense through wall human detection using UWB radar. EURASIP J. Wirel. Commun. Netw. 2011, 2011: 20. 10.1186/1687-1499-2011-20View ArticleGoogle Scholar
- Kocur D, Rovňáková J, Švecová M: Through wall tracking of moving targets by M-sequence UWB radar. Stud. Comput. Intell. 2009, 243: 349-364. 10.1007/978-3-642-03737-5_25View ArticleGoogle Scholar
- Wang W, Zhou X, Zhang B, Mu J: Anomaly detection in big data from UWB radars. Secur. Comm. Network 2013. doi: 10.1002/sec.745Google Scholar
- Xu L, Liang Q: Radar sensor network using a set of new ternary codes: theory and application. IEEE Sensors J. 2011, 11(2):439-450.View ArticleGoogle Scholar
- Liang Q: Situation understanding based on heterogeneous sensor networks and human-inspired favor weak fuzzy logic system. IEEE Syst. J. 2011, 5(2):156-163.View ArticleGoogle Scholar
- Liang Q: Biologically-inspired target recognition in radar sensor networks. EURASIP J. Wirel. Commun. Netw. 2010, 2010: 523435.Google Scholar
- Liang Q, Cheng X, Samn S: NEW: network-enabled electronic warfare for target recognition. IEEE Trans. Aerosp. Electron. Syst. 2010, 46(22):558-568.View ArticleGoogle Scholar
- Liang Q, Cheng X: KUPS: knowledge-based ubiquitous and persistent sensor networks for threat assessment. IEEE Trans. Aerosp. Electron. Syst 2008, 44(3):1-7. 1060–1069View ArticleGoogle Scholar
- Liang Q: Automatic target recognition using waveform diversity in radar sensor networks. Pattern Recogn. Lett. 2008, 29(2):377-381.View ArticleGoogle Scholar
- Liang Q: Radar sensor networks: algorithms for waveform design and diversity with application to ATP with delay-doppler uncertainty. EURASIP J. Wirel. Commun. Netw. 2007, 1-9.Google Scholar
- Liang Q: Waveform design and diversity in radar sensor networks: theoretical analysis and application to automatic target recognition. IEEE Third Annual Conference on Sensor, Mesh and Ad Hoc Communications and Networks (SECON 2006), Reston, VA, 28–28 September 2006Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.