Research on real-time network data mining technology for big data

Abstract

The data distribution in big data environment is very different, and it is difficult to mine the data because of the strong interference of redundant data and frequent items. The traditional data mining algorithm uses closed frequent item feature extraction algorithm. Due to the uneven distribution of web data in big data environment, the mining accuracy of closed frequent item feature extraction is not high. A real-time web data mining model is proposed based on high order spectral feature fuzzy neural network learning in big data environment. The transmission channel model and statistical time series model of web data under big data environment are constructed, and the redundant information flow is removed and reprocessed, and the web data after redundant filtering is analyzed by fusion clustering. The feature of high order spectrum is extracted, and the optimal mining of web data is realized by using fuzzy neural network learning classification method. The simulation results show that this web data mining method has good timeliness, high mining precision, and superior performance.

1 Introduction

With the development of computer network information technology, all kinds of large websites are constantly established and updated in real time. In the network society, large-scale websites are information browsing and data publishing through the way of establishing web. Web has become an important platform for people to interact and interact with each other. It is also an important module for storing and transmitting massive data information. In the big data environment, a large number of images, sounds, data, and text information are stored and displayed in the client in order to provide users with information sharing and use. In the big data environment, the distribution of the data is more complex and varied, so it is difficult to mine the web data. Moreover, in the network big data environment, the web data adopts the form of packet-switched network data communication, which usually results in the interference of redundant data. It leads to data offset and error in web data mining and access, and reduces the accurate mining and access probability of data [1]. It is necessary to study the optimization mining model of web data based on massive web model to improve the ability of accessing and managing web data. People pay great attention to the related algorithms.

Traditionally, web data mining models in large data environment mainly adopt high order cumulant feature extraction, time-frequency analysis and feature extraction, wavelet analysis, support vector machine classification mining algorithm, and data mining algorithm based on rough set classification in large data environment; there are many drawbacks in the web data model, such as the inaccuracy of data, the lack of effectiveness [2], the error resulting in the rule pattern of the system coding, and the other most important thing is that in the process of mining the data [3], it is not possible to determine whether the system is safe or not, if the data is excavated into an unsafe system, not only the data are data, but the data are not the same in this case; in reference [4], a feature data mining algorithm is proposed based on the distributed feature partition extraction of mass web access time, and improved the web data by multi-layer autoregressive vector analysis. Classification mining ability, but the computation cost of the algorithm is large, and the time delay error occurs in web information retrieval. In reference [5], a data mining and text retrieval method is proposed based on the large data environment based on the decision time classification search engine. By constructing the web search engine, the text feature extraction is realized and the number of semantic matching is improved by strict semantic matching. According to the convergence ability of the mining, the problem of the algorithm is that the accuracy of data mining is limited when the efficiency of the data attribute classification is not obvious or the interference data of the approximate web is large, but the algorithm has poor convergence and complex computation [6].

In order to solve the above problems, this paper presents a real-time web data mining model based on higher-order spectral feature fuzzy neural network learning in big data environment. In this paper, the transmission channel model and statistical time series model of web data under big data environment are constructed, and the redundant information flow is removed and reprocessed, and the web data after redundant filtering is analyzed by fusion clustering. On this basis, the feature of high order spectrum is extracted, and the optimal mining of web data is realized by using fuzzy neural network learning classification method. Finally, the performance test is carried out through the simulation experiment, which shows the superior performance of this method in improving the ability of data mining.

2 Data distribution model based on big data and anti-interference preprocessing of web data information flow

2.1 Data distribution model in big data environment

In order to realize the web data mining under the big data environment, it is necessary to analyze the web data in the big data environment first. In the big data environment, a large amount of data is stored in the deep web database for data cloud storage. Information browse and result display are realized on the web through web server [7]. The mining of web data under big data environment is realized by establishing database fusion and data clustering model, and data feature extraction and fusion clustering to realize real-time data mining. In big data environment, a large amount of data information is sorted and matched by similarity degree. Search engine and deep web database search engine and deep web database through data links and query results to carry out intelligent retrieval [8]. The overall model of web data mining under big data environment is shown in Fig. 1.

Fig. 1

Overall structure model of web data mining in big data environment

According to the above design of the overall structure model of web data mining in big data environment, the distribution of web data in big data environment is analyzed. It is assumed that the web data to be mined is distributed in the web database through the topic crawler method. Firstly, the attributes of the continuous data set of the information flow sequence under the big data environment are discretized, A = {a₁, a₂,  … , a_n} is the initial vector of the information flow under the big data environment. The associated attribute set of web data in big data environment is B = {b₁, b₂,  … , b_m}. The mathematical model of the whole network database system is expressed as follows:

$$ \dot{x}=f\left(x,u\right) $$

(1)

where u is the bit rate of data access, x is the initial access scalar time series, and the first-order vector group x₁, x₂, ⋯x_n ∈ C^m (m-dimensional complex space) of the higher order cumulative vector of the data information is given, where:

$$ u=\left[{u}_1,{u}_2,\cdots, {u}_N\right]\in {R}^{mN} $$

(2)

According to the global search ability of chaotic differential evolution algorithm, the optimal value of clustering center is found [9]. If there is no G₃ = (M^α₃, M^β₃, Y₃), the inherent mode function of big data state space is:

$$ y(t)=\frac{1}{\pi }P\int \frac{x\left(\tau \right)}{t-\tau } d\tau =x(t)\ast \frac{1}{\pi t} $$

(3)

In order to reflect the diversity of the web data groups, the adaptive beamforming estimation of the web database information flow is carried out under the big data environment. The global convergence retrieval beam of the output is obtained as follows:

$$ l(t)=\left(\sum \limits_{m=1}^M{u}_m\right)\cos \left(2\pi {f}_0t\right)-\left(\sum \limits_{m-1}^M{v}_m\right)\sin \left(2\pi {f}_0t\right) $$

(4)

By the above processing, the network crawler method is used to focus the data stream information in the high dimensional feature space of web to form a cluster center of web data access, which improves the ability of data feature mining and web data mining.

2.2 Web statistical time series model construction and anti-interference pretreatment

In order to improve the ability of accessing and mining web data in big data environment and combining modern data information processing algorithm to construct the information flow of massive data, because the web data is interfered by adjacent web data, it is necessary to carry out anti-interference suppression [10]. After processing, the empirical mode decomposition and Hilbert spectrum analysis of the web data information model under the big data environment are carried out, and the state transfer equation of the data information flow distribution in the process of web data access is obtained as follows:

$$ x(n)=s(n)+v(n)={\omega}_{k-1}^{(i)}\frac{p\left({y}_k|{X}_k^{(i)},{Y}_{k-1}\right)p\left({x}_k^{(i)}|{X}_{k-1}^{(i)},{Y}_{k-1}\right)}{q\left({x}_k^{(i)}|.\right)} $$

(5)

In the above formula, s(n) is the distributed time sampling sequence of web data under big data environment, v(n) denotes interference component, and considering the phase difference of network differential characteristic behavior in the information source i of web node, the web number under big data environment is needed [11]. According to the information flow access process, the error square of delay estimation is:

$$ {\varepsilon}^2(k)={d}^2(k)-2d(k){X}^T(k)W+{W}^TX(k){X}^T(k)W $$

(6)

Thus, the web statistical time series model under the large data environment is constructed. The interference suppression filtering of web data is designed, and the interference suppression of web data is suppressed by the multipath adaptive cascade filtering method [12]. It is assumed that the time-varying multipath correlation dimension of the web data in the large data environment is expressed as follows:

$$ x(t)=\lambda \operatorname{Re}\left\{{a}_n(t){e}^{-j2\pi {f}_c{\tau}_n(t)}{s}_l\left(t-{\tau}_n(t)\right){e}^{-j2\pi {f}_ct}\right\} $$

(7)

An adaptive cascade tracking filter is designed to suppress interference [13]. The system transmission function of the filter is obtained as follows:

$$ H(z)= Am\cdot \frac{1+2{z}^{-1}+{z}^{-2}}{\left(1-\rho {e}^{j\phi}{z}^{-1}\right)\left(1-\rho {e}^{- j\phi}{z}^{-1}\right)} $$

(8)

By the above filtering process, assuming that the symbol width of web data in web is T_a, T_a = 1/R_a, the output amplitude of web data mining after interference suppression is:

$$ a(t)=\sum \limits_{n=0}^{\infty }{a}_n{g}_a\left(t-{nT}_a\right) $$

(9)

The big data is sampled by pseudorandom sequence by using binary phase shift keying (BPSK) modulation, and the sampling value is c(t). Thus, the mining accuracy of web data under big data environment can be improved effectively.

3 Methods

3.1 Learning algorithm of higher order spectrum feature fuzzy neural network for web data in data environment

In this paper, a real-time data mining model based on higher-order spectral feature fuzzy neural network learning in big data environment is proposed. High-order spectral features are extracted from the web data in big data environment, and the clustering fusion analysis is carried out by using the adaptive learning method. The higher-order spectral features of the big data are extracted by the Agung Rai Museum of Art (ARMA) model [14]. Assuming the spectral width of the Doppler time slice is T_c, T_c = 1/R_c, then:

$$ c(t)=\sum \limits_{n=0}^{N-1}{c}_n{g}_c\left(t-{nT}_c\right) $$

(10)

Based on higher-order spectral feature extraction method, the time-delay and scale estimation of p-dimensional vector in web data mining model is carried out, and the following binary hypothesis testing problems are obtained:

$$ \left\{\begin{array}{l}{H}_0:r(t)=n(t)\\ {}{H}_1:r(t)=g(t)+n(t)\end{array}\right.\kern0.5em t\in \left[0,T\right] $$

(11)

In the formula, r(t) is the unilateral exponential distribution of fusion clustering, g(t) is the center vector of data clustering, σ² is the color noise with zero mean value, and φ_mi is the variance. The analytical expression of phase deviation φ_mi of web data mining in web is obtained by taking mathematical expectation on both sides of the above formula:

$$ {\varphi}_{mi}=\frac{2\pi {r}_i}{\lambda}\left(\sqrt{1+\frac{m^2{d}^2}{{r_i}^2}-\frac{2 md\sin {\theta}_i}{r_i}}-1\right) $$

(12)

Based on the fuzzy neural network learning algorithm, the error tracking and fitting of web data mining is carried out, and the fitting state function is:

$$ p\left({Q}_s\right)=\frac{1}{\sqrt{2\pi }{\sigma}_s}\exp \left[-\frac{{\left({Q}_s-\left\langle {Q}_s\right\rangle \right)}^2}{2{\sigma_s}^2}\right] $$

(13)

$$ {\int}_{-\infty}^{\infty }p\left({Q}_s\right)d{Q}_s=1 $$

(14)

Where {η_i} is an independent and uniformly distributed fuzzy neural network learning tracker with a mean value of 0 and a variance of σ². The error of data mining is reduced and the mining accuracy is improved by the learning of the fuzzy neural network [15].

3.2 Implementation of data real-time mining model

On the basis of the data mining fusion clustering analysis and fuzzy neural network learning, the improved design of data mining model is carried out. The sampling interval of data mining is assumed to be n ∈ [n₁, n₂], and the data mining is studied by fuzzy neural network [16]. The phase characteristics of the web data are described as follows [17]:

$$ s(v)=\underset{0}{\overset{v}{\int }}\sin \left(\frac{\pi }{2}{x}^2\right) dx,\kern0.5em c(v)=\underset{0}{\overset{v}{\int }}\cos \left(\frac{\pi }{2}{x}^2\right) dx $$

(15)

In big data environment, the probability of any web data mining can be expressed as:

$$ x(t)=R\left(a(t){e}^{i\theta (t)}\right)=a(t)\cos \theta (t) $$

(16)

Based on decision tree classification, the semantic similarity information attributes of web data are extracted, and the feature classification functions of data mining under big data environment are obtained:

$$ {v}_1=\sqrt{BT}\frac{1+2\left(f-{f}_0\right)/B}{\sqrt{2}} $$

(17)

$$ {v}_2=\sqrt{BT}\frac{1-2\left(f-{f}_0\right)/B}{\sqrt{2}} $$

(18)

Based on the above processing, the high order spectrum feature of web data is studied by fuzzy neural network under the environment of big data, which improves the confidence and accuracy of data mining, reduces the false alarm probability, and realizes the real time mining and accurate mining of web data.

4 Experience

In order to test the application performance of this algorithm in the implementation of web data mining in big data environment, the simulation experiment is carried out. Based on Matlab platform, the simulation experiment is carried out. The computer simulation experiment platform is configured as Intel: core i5 processor, the main frequency is 2.8 GHz/4G memory and Windows 10 professional edition 32 Bit SP2 operating system. The test data is a deep web database under the big data environment of a large website. CWT200G data combination mode is used to start visa resource manager for data loading. More than 200,000 big data information in big data environment are obtained. The data collected are 16-bit vertical accuracy. The massive data were divided into training set and test set, assuming that the interference intensity in data mining was − 15 dB Gaussian color noise. The simulated dataset consists of two partitions of 25.2 MB in size, and the size distribution of the dataset for training and testing is shown in Table 1.

Table 1 Training data sets

According to the above simulation environment and parameter setting, the data mining under the big data environment is carried out, and the output of the data mining is shown in Fig. 2.

Fig. 2

Data mining output. Figure shows that this method can accurately mine the web data under the big data environment, and the feature expression ability of the data mining output is strong

Figure 2 shows that this method can accurately mine the web data under the big data environment, and the feature expression ability of the data mining output is strong. The convergence of different methods for web data mining is tested, and the comparison results are shown in Fig. 3. The execution time comparison of data mining is shown in Fig. 4. The analysis shows that the proposed method has better convergence, shorter execution time, and better real-time performance (Table 2).

Fig. 3

Comparison of convergence curves of data mining. Figure shows the convergence of different methods for web data mining is tested

Fig. 4

Comparison of time cost for data mining

Table 2 Test data set

5 Results and discussion

In this paper, a real-time web data mining model is proposed based on high order spectral feature fuzzy neural network learning in big data environment. The transmission channel model and statistical time series model of web data under big data environment are constructed, and the redundant information flow is removed and reprocessed, and the web data after redundant filtering is analyzed by fusion clustering. The feature of high order spectrum is extracted, and the optimal mining of Web data is realized by using fuzzy neural network learning classification method. The simulation results show that this web data mining method has good timeliness, high mining precision, and superior performance. This method has good application value in real-time data mining and feature extraction of web data.