RETRACTED ARTICLE: Data cleansing method of talent management data in wireless sensor network based on data mining technology

Abstract

Data mining technology is a very common computer technology, which has been widely used in many fields because of its superior performance. The method of talent management data cleaning in wireless sensor networks is studied based on data mining technology. The research status of data mining technology is first introduced at home and abroad, and the specific application forms of wireless sensor networks are analyzed. Then, the structure characteristics of wireless sensor networks are introduced, and a data cleansing technology is proposed based on clustering model. A cluster-based replication record deletion algorithm is proposed, and finally, the accuracy of data cleansing methods is verified. The results show that the research method of this paper is correct and effective.

1 Introduction

With the rapid development of the Internet and the popularity of computers, people have entered the era of network information (Zhu L et al. 2018) [1]. In particular, the rapid development of the World Wide Web has led to an exponential increase in the amount of online information (Sun W et al. 2018) [2]. The sources of information are very extensive. The largest sources of data are network information sources. In addition to relational databases, distributed databases, etc., they have all achieved considerable development (Shafiee S et al. 2018). [3]. The US Department of Health and Human Services’ Child Management Research Office (OCAR) conducted a survey of children in the USA, and record matching was used to identify each child to obtain more accurate survey results. Since then, the data cleaning field has attracted widespread attention (Q et al. 2018) [4]. The amount of data available is getting larger and larger, but at the same time, some related problems have come one after another. Common problems are data integration problems. Data integration is an important processing step in many areas (DW Upton et al. 2018) [5]. For a large data set, the quality of the relevant parameters such as the quality and accuracy of the data in the integration process is a criterion for judging whether the data integration is excellent (Mekikis P V et al. 2018) [6]. However, there are some mistakes that data cannot avoid during the integration process. Overall, the main reason for the problems in the integration process is that there are no agreed standards among the databases, and the format of the data is not the same, which will affect the data. The integration has caused some obstacles (Aygör D et al. 2018) [7]. Therefore, when entering large amounts of data, there will always be some errors and some inconsistent data (Alomari A et al. 2018) [8].

2 Start of the art

Wireless sensor network is an information acquisition and information processing technology, which is mainly composed of sensors, MEMS, and network systems. Compared with the traditional PC era, it has the characteristics of smaller size and lower price and breaks through the traditional computer limitations (Kumar L et al. 2018) [9]. For each sensor node, it can measure and analyze the signals in the surrounding environment through built-in multiple sensors and obtain the required data (Lee W K et al. 2018) [10]. The computational processing capability brought about by the universal network is even more difficult to measure. Compared to the traditional network, it is centered on data rather than transmission data. The wireless sensor has a wide range of functions, which not only can detect the environment and the state of the building, but also can control the smart home through certain technical means (Zhang W, et al. 2017) [11]. The application in the military is the origin of the sensor network research. In view of these characteristics of wireless sensor networks, this paper will conduct in-depth research on talent management data clean-up methods in wireless sensor networks based on data mining technology. Through this method, talent management data in wireless sensor networks can be implemented without affecting. In the case of its effectiveness, the overall data size is reduced. Lowering the scale of talent management data can not only improve the efficiency of the analysis process, but also improve the quality of the analysis results. Therefore, the research on data cleaning methods will be very meaningful.

3 Methodology

3.1 Wireless sensor network structure

Wireless sensor networks are mainly composed of sensor nodes, detection areas, and servers. Sensor nodes can be placed near objects that require measurement data through manual deployment. After deployment, these sensor nodes will self-organize in a certain way, and they will perceive the surrounding environment and objects in a cooperative manner so as to obtain the required data. This self-organizing form can form a corresponding network and relay all data back to the master node through the relay mode. Finally, all data in the entire node is transmitted to the server through the communication system. When users use wireless sensors to acquire data, they can effectively collect the required data through the management and control of the nodes. The architecture of a typical wireless sensor network is shown in Fig. 1.

Fig. 1

Topology structure of sensor networks

The sensor node consists of a sensor module, a processing module, a wireless communication module, and an energy supply module. The sensor module is responsible for information acquisition and data conversion. The processing module controls the operation of the entire sensor node, processes the data collected by itself and data sent by other nodes, and runs the network protocol to control the communication process of the node; the wireless communication module is specifically implemented with other sensor nodes. The data is sent and received; the energy supply module provides energy for the sensor nodes. For the network function, each sensor node in the wireless sensor needs to take into account both traditional network nodes and routers, not only to collect and process local information, but also to process the data transmitted from other nodes. In the process of data transmission, nodes need to have the ability to work together. At present, the hardware and software technology of the sensor node is the focus of the sensor network research, because it has a strong ability to process, store, and communicate the node data. By connecting the sensor and the Internet, the communication between different network protocols can be converted. At the same time, it can also distribute tasks to all nodes at the same time and transmit the collected data to the external network. For different applications, the composition of the sensor is also different, but almost all sensors have common characteristics, they generally include the sensor unit, processing unit, wireless communication unit, and energy supply unit, the traditional sensor data transmission. The process is shown in Fig. 2.

Fig. 2

Components of sensor node

The composition of the sensor unit is generally relatively simple and mainly consists of a sensor and an analog-to-digital conversion function module, which is mainly responsible for data conversion for obtaining information in the detected area. The core part of the processor unit is an embedded system, which mainly includes a CPU, a memory, and the like, and is mainly responsible for controlling the nodes of the entire sensor and storing the collected data and processing the data obtained by other nodes. The main function of the wireless communication unit is to complete the data transmission without using a wired device. The main part of the energy supply unit is the power supply module, whose main function is to provide energy for the sensor nodes. There are also some other modules, such as positioning systems and mobile systems. Through the cooperation of these units, the wireless sensor network can operate normally.

When using a wireless sensor network to obtain data, in order to make the acquired data more accurate and effective, a large number of sensor nodes need to be set up. Therefore, the number of sensor nodes may be very large. Because the amount of sensor nodes is relatively large and the volume is small, and personnel in some areas cannot arrive accurately in time, the sensors cannot be supplemented by changing the battery. At this point, it makes sense to calculate the energy consumption of the sensor nodes. The main energy-consuming part of the sensor node is very power-consuming when transmitting data on the wireless communication module. The wireless communication module has four states of sending, receiving, idle, and sleeping wireless. The relationship between the energy consumption of wireless communication and the communication distance is shown in Formula 1.

$$ E={kd}^n $$

(1)

Among them, E is the energy consumption of wireless communication, d is the distance, and k is a constant.

As the communication distance increases, the energy consumption will increase dramatically.

3.2 Mathematical model of data mining

Bayesian network is a mathematical model commonly used in data mining process whose expression is shown in Formula 2.

$$ P\left({X}_1,\dots, {X}_N\right)=P\left({X}_1\right)P\left(\left.{X}_2\right|{X}_1\right)P\left(\left.{X}_N\right|{X}_1,\dots, {X}_{N-1}\right) $$

(2)

The formula for the degree of trust of the model is shown in Eq. 3.

$$ p\left(\left.{M}_i\right|D\right)=\frac{p\left(\left.D\right|{M}_i\right)p\left({M}_i\right)}{p(D)} $$

(3)

p is the edge likelihood.

The Bayesian information criterion (BIC) is a large sample approximation of the edge likelihood. By using the Laplacian approximation, a large sample approximation can be performed for P, a BIC scoring function can be derived, the log-likelihood function can be expanded by maximum likelihood estimation, and then the calculation can be converted to a multivariate normal distribution function at the extreme point of neighboring points. First, the Laplacian approximation is used for the posterior probability, as shown in Eq. 4.

$$ p\left(\left.D\right|m\right)=\int p\left(\left.D\right|\theta, m\right)p\left(\left.\theta \right|m\right) d\theta $$

(4)

Through the establishment of data model, it is very convenient to use data mining technology to process data.

3.3 Data cleanup technology based on clustering mode

Current talent management data is faced with the challenge of sudden increase of data. These large databases usually contain data errors or inconsistencies due to some reasons. Causes of errors include incorrect input causes incorrect data values because the input Inconsistent data caused by different formats or using different abbreviations cannot completely collect data information and result in lost data. All of these reasons may cause enterprises and institutions to inevitably make deviations when they make important business decisions, resulting in huge losses. So this situation is tried to avoid, that is, solving the so-called “garbage in, garbage out” problem. The data cleaning process is to solve the common input errors and inconsistencies in large databases, and some simple preprocessing before data cleaning operations can improve the quality of data cleaning. The flow chart in the data clean-up is shown in Fig. 3.

Fig. 3

The main process of data cleaning and preprocessing

The main process of data cleaning preprocessing is shown in Table 1.

Table 1 The main process of data cleaning and preprocessing

3.4 Cluster-based replication record deletion algorithm

Combining large databases often encounters problems such as incorrect data entry, different schemas, or inconsistent abbreviation forms. These problems will cause the merged database to have multiple records that represent the same entity but have slightly different attribute values, which creates an inconsistent data. After cleaning and preprocessing, some simple errors in the database are cleared. However, because the object to be processed is a large database, the amount of data to be faced is very large, so it still contains a lot of errors and inconsistent data. The accuracy metric used in this paper is a pure clustering comparison. The definition of pure clustering refers to that all records contained in a cluster represent the same entity. The experimental method is used to evaluate the data in the large-scale database. The goal of the accuracy in the measurement process is the entire database, not just one data in the database. When the data is recorded using pure clustering, the representation of the records is the same. If the records are in different forms, this clustering form is not a pure clustering, indicating that the clustering method is inaccurate. The use of cluster-based replication record deletion algorithm can largely solve the problem of data inconsistency. This method can reduce the amount of data processing and improve the efficiency of data processing.

4 Result analysis and discussion

In order to make the experiment more accurate, and to be able to effectively verify the accuracy and operational efficiency of the algorithm, known real clustering data will be used to analyze, and the specific values of the data used are determined. The data used in the experiment is talent management record data. The record attributes included seven attributes of talent management. The experiment enters a total of 875 talent management records. Through a certain copy processing and then using a random error handling method, a total of effective talent management records are obtained. The number is 2412. A total of 218 clusters containing more than two records are manually calculated, of which the largest cluster contained a total of 14 records. Canopy clustering detection technology is used to detect duplicate records. There are three main detection parameters: distance thresholds T1 and T2 and constant coefficient k. The choice of T1 and T2 determines the size of Canopy and its degree of overlap, that is, the amount of data that needs to be accurately calculated. The choice of k value determines whether the records can be accurately clustered. At the beginning of data processing, the values of T1 and T2 need to be created. In this paper, the inverted detection method is used to set these two values. In the case of different T1 and T2 (T1≤T2), system clustering needs to calculate the calculated amount of matching pairs to measure the quality of T1 and T2.

The calculation of different T1 and T2 values is shown in Table 2.

Table 2 Calculation of different T1 and T2 values

According to the experimental data in Table 2, when T1 = 0.75 and T2 = 0.75, the data points that need to be accurately calculated are the least. Therefore, T1 = 0.75 and T2 = 0.75 are selected, which means that there is no overlapping Canopy.

According to the ratio of the cluster number obtained and the actual clustering, the value of k is shown in Table 3.

Table 3 Clustering ratio of different k values

Table 3 shows that at k = 3, the clustering ratio is closest to the true clustering, so the experiment selects the distance threshold k = 3. However, the clustering ratio still does not reach 1 because the random error of data makes some data records that should be classified in the same cluster not be correctly classified into one cluster.

The home address of the talent management record table is a compound attribute. The application data conversion method decomposes the city, county, specific street, work unit, and house number of the address into sub-attributes and, at the same time, performs pre-processing before data cleaning based on the use of an external source file. Whether the city corresponding to the home address corresponds to the zip code can clear some dirty data. The external source files are based on postal codes issued by the post office. The scope of experimental data is very small, which only covers Guangxi, so it is relatively easy to establish 100% accuracy when setting up external source files. Figure 4 shows the results of experiments comparing pre-processed and non-preprocessed test copy recording methods, including a sort neighbor method and a test copy recording method using Canopy technology, in which the window size of the sort neighbor method selects two to compare the situation.

Fig. 4

The results of comparing pretreatment and non-preprocessed test

It can be seen from Fig. 4 that the accuracy of the pre-processed duplicated recording detection method is higher than that of the unpreprocessed duplicated recording detection method, but since the experimental data used is not large, the pre-processing cannot be fully displayed. In the two cases of ω=16 and ω=8, ωchooses 16 to be more accurate than 8 because the largest cluster in the experimental data contains 15 records. If ω=8, it will result in some duplicate records, which cannot be detected, although ω= 16 when it is needed to do many unnecessary comparisons, increasing the amount of calculation, but it can guarantee a higher accuracy rate. In this experiment, the pre-processed ω=16 ranking neighbor method is the same as the Canopy cluster’s replication record detection method, but the Canopy method has a higher recall rate, indicating that it can obtain more replication records. The algorithm is more efficient.

5 Conclusion

The development of data mining technology has played a promoting role in many fields. This technology is used to research the method of talent management data cleanup in wireless sensor networks, and certain theoretical results are achieved. The important role of data cleaning in many fields is elaborated, with current research status of data cleaning in the world introduced and the existing deficiencies are pointed out. The data cleansing method is an extremely important research field, which can solve the data inconsistency when identifying the same object and improve the accuracy of recognition. With more and more rapid and convenient access to information at this stage, the amount of data is increasing day by day. When analyzing data and making business decisions, it is necessary to combine some data information to find patterns of interest more easily. The reason will inevitably produce incorrect data or inconsistent data, so that the approximate duplicate records appear in the merge process, which is not allowed in the database and these duplicate records must be deleted. Although a more in-depth study is made, but because data mining technology is not perfect, it will be further studied in the follow-up work.