 Research
 Open Access
 Published:
Measurement and evaluation method of radar antijamming effectiveness based on principal component analysis and machine learning
EURASIP Journal on Wireless Communications and Networking volumeÂ 2023, ArticleÂ number:Â 58 (2023)
Abstract
With the development of modern electronic countermeasure technology, the fight between radar jamming and antijamming has become increasingly fierce. Experts have done a lot of highly effective work on radar antijamming performance. However, the emergence of various new complex interferences has rendered existing methods unable to meet the needs. In this manuscript, we consider the measurement and evaluation method of radar antijamming effectiveness based on principal component analysis and machine learning. Firstly, taking into account the diversity of variables in radar countermeasure experiments and the complexity of constraints between variables, we propose a bipartite covering array for the experimental scheme, which requires that each level combination of any radar parameter and jammer parameter occurs at least once, to ensure the rationality of the experiments. Secondly, according to the characteristics of multiple jammers and the analysis of impacts on radar performances, we combine the existing indicators and use the principal component analysis method to obtain two comprehensive indicators, which better reflect radar performances. Finally, we select the best model as a prediction for radar comprehensive indicators by comparing several machine learning algorithm models, including classification and regression tree, random forest, xgboost, and SVM. Additional experiments verify the effectiveness of the resulted model.
1 Introduction
Geng et al. [1] pointed out that, as radars are widely used in the military field, antiradar electronic jamming technologies have also become important. In recent years, with the rapid development of radar countermeasures, especially antijamming, the application of radars in modern warfare is becoming increasingly strict. In fact, modern warfare has gradually transitioned from mechanization to informatization through the extensive application of electronic information technology in the military field, forming a complex electromagnetic environment in which the two sides confront each other; see Best [2] and Guariglia [3]. Hence, military technologists have been conducting scientific and technological research on this, proposing solutions to maintain radar reconnaissance capabilities. Huang, Zhang, and Xu [4] stated the development status and presumption of the antijamming capability of the existing electronic instruments. Liu et al. [5] modeled and analyzed the interfering signal data in a complex electromagnetic environment. Wang et al. [6] proposed and analyzed radar countermeasures questions in complex electromagnetic environments. In fact, the two sides, the jamming side and the radar side, could be unified on the same issue. The jamming side mainly focuses on whether the jamming can weaken or even fail the radar detection capability; while the radar side is concerned about its working ability under jamming. Both want to know clearly about the antijamming capability of radars in specific confrontation situations to take corresponding measures in the process of technical research and equipment development. Therefore, as argued by Yi and Yuan [7], it is crucial to study the radar antijamming performance, which represents a change in radar performance under the influence of jammers after adopting antijamming technology. Simply put, it is the combined result of the joint effects of the radar side, the jamming side and the countermeasure environment. Researchers are concerned about the relationship between the antijamming effectiveness and the influencing factors, so that the antijamming effectiveness of the radar can be predicted and estimated according to the influencing factors. Nevertheless, the relevant researchers always encounter the practical problems of â€˜impossibility of measurementâ€™ and â€˜inaccuracy of evaluationâ€™. The focus of this manuscript is to obtain a relationship between the antijamming effectiveness and the influencing factors, to accurately predict or estimate the antijamming effectiveness under the presumed influencing factors. The manuscript is mainly divided into the following three parts: the radar jamming experiment, the measurement and the evaluation of radar antijamming effectiveness.
1.1 Radar jamming experiment
The radar jamming experiment can be regarded as the basis for studying the antijamming effectiveness evaluation method. Jia et al. [8] called it the highlight of the evaluation methods research. However, it is very difficult to derive the radar antijamming performance directly from the influencing factor theory. Hence, the study in the evaluation method of antijamming capability is generally based on data collected from various experiments. Radar jamming experiment methods are mainly divided into external field jamming, semiphysical simulation jamming experiments, and fullvirtual simulation jamming experiments on a computer. These three have their own characteristics and advantages. Among them, as Hohlfeld and Cohen [9] pointed out, the fully virtual simulation experiments are favored by researchers owing to their flexibility and ability to demonstrate the diversity of adversarial situations. The fully virtual simulation confrontation experiment system formed by a variety of highfidelity simulation models can provide relatively reliable experimental results. In this manuscript, we use a fully virtual simulation experiment to propose some experimental schemes of the given radar parameters and three kinds of jammer parameters, and give the simulated radar antijamming performance results.
1.2 Measurement of radar antijamming effectiveness
Measurement of radar antijamming effectiveness is the process of establishing the evaluation indicators for radar antijamming effectiveness, and the basis of antijamming effectiveness evaluation. It obtains a specific antijamming effect according to the change in radar performance before and after jamming, also known as the quantification process of antijamming performance; see Kumar et al. [10].
Actually, the antijamming capability is not a real quantity that can be directly measured, so it is generally necessary to select the best indicators reflecting the change in the working capability according to the function of the radar, and then use a comprehensive treatment on these indicators to obtain the antijamming capability of the radar. Experts have done a lot of highly effective work in the evaluation of indicators of antijamming performance. Zhang et al. [11] provided an endtoend antijamming target detection method based on CNN. Aziz, Maud, and Habib [12] proposed a radar antijamming technology based on reinforcement learning, which greatly improved the radar antijamming performance. However, the emergence of various new complex interferences has rendered existing methods unable to meet the needs. Moreover, as new types of radars continue to emerge, radar countermeasures are increasingly diversified and the complex electromagnetic environment where radars are located has also undergone new changes. All these ask for more quantitative and systematic evaluation solutions. Under this background, it is imminent to establish a scientific and reasonable evaluation system to guide radar antijamming performance evaluation.
There are many criteria for measuring the effectiveness of antijamming, which reflect the degree of change in radar performances before and after the jamming from different aspects, including power criterion, information criterion, and efficiency criterion. Through these criteria, many antijamming evaluation indicators can be selected, but selecting a single indicator has many deficiencies in radar performance evaluation. For example, there are repeated evaluations of certain antijamming performance because of too many evaluation methods, and the objectivity and the operability still need to be improved. Given the abovementioned deficiencies and based on the characteristics of various jammers and the impact analysis on radar performance, we use the principal component analysis method and combine the abovementioned existing criteria to extract two comprehensive indicators. This method avoids the onesidedness that the past radar evaluation indicator can evaluate only one antijamming performance.
1.3 Evaluation of radar antijamming effectiveness
Evaluation of radar antijamming effectiveness is a modeling process of comprehensively evaluating the radar antijamming effect according to the real data and obtained indicators. It is a complex area of research. As pointed out by Meng, Zhang and Fan [13], on the one hand, the diversity of radar countermeasures, including the diversity of radars and the jammers, makes the prediction methods of antijamming ability flourish. On the other hand, it leads to the complexity of the evaluating problem that, the subject of the evaluation is the person and the evaluation results are closely related to the purpose of evaluation, the evaluation perspective, and other various benefits. Both lead to the lack of comprehensive and unified practice standards and methods. Moreover, there are many factors that affect the radar antijamming capability, so that the number of countermeasures composed of many influencing factors is huge. In this case, insufficient training samples cause poor prediction results for traditional methods.
Nowadays, the widely used evaluation methods mainly include: the evaluation factor method, the fuzzy comprehensive evaluation method, and machine learning methods based on neural networks. Long, Zhang, and Lin [14] argued a comprehensive evaluation criterion of radar ECCM performance based on an electronic effect matrix. Yang, Li, and Zhang [15] studied the radar countermeasure training and simulation system based on HLA. Lei et al. [16] provided an efficiency evaluation method for radar countermeasure equipment joint netting operation based on the AHPcloud model. Wang, Sun and Wang [17] deeply studied the radar active jamming recognition through the convolutional neural network. Bu et al. [18] provided a radar seeker antijamming performance prediction and evaluation method based on the improved grey wolf optimizer algorithm and support vector machine. Li et al. [19] innovatively provided an experimental evaluation method for interference suppression distance of networked radar. Cheng et al. [20] applied deep learning architectures to 5Â G IoT systems. Raheja et al. [21] used machinelearning diffusion models to predict coronavirus19 outbreaks. This manuscript uses multiple machine learning methods to evaluate the radar antijamming effectiveness. Besides the innovative experiments, this manuscript also focuses on machine learning algorithms and their application in the evaluation of antijamming effectiveness. The former is based on the application requirements to select the appropriate algorithm model, and to solve the problem of model parameter selection and feature selection, while the latter is to find the best model as a prediction for radar comprehensive indicators by comparing several algorithm models.
The remainder of this manuscript is organized as follows: Sect. 2 states the existing analytical methods used in this manuscript. Section 3 proposes a reasonable construction of the experiment scheme. In Sect. 4, the principal component analysis is applied to provide comprehensive radar evaluation indicators. Based on the actual data and the obtained indicators, Sect. 5 provides four machine learning algorithms to establish regression models to achieve the evaluation of the radar antijamming effectiveness. In Sect. 6, the additional experiments verify the reliability of our proposed ideas and methods. Section 7 summarizes the full manuscript.
In this work, the first, fourth, and fifth authors put forward the overall research ideas. The second and third authors completed
the virtual simulation experiments and data collection. The first and fourth authors obtained two comprehensive indicators through data analysis, and completed the measurement of radar antijamming effectiveness. The first and fifth authors constructed models to achieve the evaluation of radar antijamming effectiveness, and wrote and revised the manuscript.
2 Methods
In addition to the three research methods described in the Introduction, â€˜Radar jamming experimentâ€™, â€˜Measurement of radar antijamming effectivenessâ€™ and â€˜Evaluation of radar antijamming effectivenessâ€™, we will also introduce the existing data analysis methods mainly applied in the following sections as follows.
2.1 PCA
Principal component analysis (PCA) is a common data dimensionality reduction method. It transforms a set of potentially correlated variables into a set of linearly uncorrelated variables through orthogonal transformation, and this transformed set is called the principal components.
This manuscript will use this method to obtain two comprehensive indicators for the antijamming effectiveness measurement in Sect. 4.
2.2 Four machine learning algorithms
In the evaluation of radar antijamming effectiveness, machine learning algorithms play a vital role. According to the quantitative characteristics of radar antijamming performance evaluation, it can be regarded as a regression problem. Therefore, here we introduce four different machine learning algorithms used in Sect. 5 to solve the evaluation of radar antijamming performance, namely Classification and regression tree (CART), random forest, Xgboost and support vector machine (SVM), as follows.

(i)
CART
Classification and regression tree (CART) is a widely used machine learning method for both classification and regression, which consists of feature selection, tree generation, and pruning. CART is a learning method to output the conditional probability distribution of the random variable Y under the condition of the given input random variable X. It assumes that the decision tree is a binary tree, and the values of the internal node features are â€™yesâ€™ and â€™noâ€™. The left branch is that with value of â€™yesâ€™, while the right one is that with value of â€™noâ€™. Such a decision tree is equivalent to recursively dicing each feature. It divides the input feature space into finite units, and determines the probability distribution of prediction on these units. The CART algorithm consists of the following two steps:

(1)
Generation of decision tree: Generate the decision tree based on the training data.

(2)
Pruning of decision tree: Prune the generated tree with the test data set and select the optimal subtree. Generally, the minimum loss function is used as the criterion for pruning.

(ii)
Random forest
Random forest is a machine learning algorithm that contains multiple decision trees. Denote the number of samples as N, and that of features as M. Then the random forest builds decision trees according to the following steps:

(1)
Sampling N samples with replacement from the N training samples to train a decision tree (i.e. bootstrap sampling), and use the unselected samples as the test set to evaluate the error of the model.

(2)
When each node of the decision tree needs to be split, randomly select m features from these M ones, where m should be much smaller than M. Then use a certain strategy (such as information gain) on these m features to select one as the split feature of the node.

(3)
In the process of decision tree formation, each node must be split according to step 2 until it can no longer be split. Note that there is no pruning during the entire decision tree formation process.

(4)
Follow steps 1 to 3 to build a large number of decision trees, which constitute a random forest.

(iii)
Xgboost
Xgboost, proposed by Dr. Chen Tianqi, is a massively parallel Boosted tree. It was developed on the basis of Gradient boosting decision tree (GBDT). Since its introduction, Xgboost has received a lot of attention in the Kaggle community competition. In the same situation, its speed is more than 10 times faster than other similar algorithms and it supports parallelization.
GBDT is a combination of decision tree and boosting method. Each decision tree of GBDT trains the errors in the result of the previous one. The training process is linear, and each iteration is the optimization goal. Compared with the traditional GBDT algorithm, Xgboost has a lot of progress. Xgboost uses the second derivative information by performing the secondorder Taylor expansion of the loss function, instead of using only the first derivative information as the traditional GBDT. In addition, Xgboost adds the complexity of the model as a regular term into the objective function to optimize the model, and the pruning into the later stage. A method of shrinkage and column subsampling also used in Xgboost to avoid overfitting easily. Moreover, Xgboost takes the segmentation search algorithm, which automatically uses the sparseness of features to learn the parallelization tree, so it can handle higherdimensional sparse matrices relative to GBDT.

(iv)
SVM
Support vector machine (SVM) is a supervised generalized linear classifier, which can handle both regression and classification problems well. SVM is sought after by many researchers because of its advantages such as the ability to process highdimensional data, to obtain global optimal solutions, and not easy to overfit.
For linear separable samples, SVM commits to find an decision boundary to classify the n input data \(X=\{x_1,\cdots ,x_n\}\) into positive and negative classes. This boundary is also known as the optimal hyperplane \(H: w^\textsf{T}X+b=0\), where w is the normal vector and b is the intercept. It not only accurately separates the samples, but also maximizes the classification interval, which is the distance between the upper and lower interval boundaries \(H_1\) and \(H_2\), as shown in Fig.Â 1. While for nonlinear separable samples, SVM obtains linear separation by applying a kernel function to map the sample space into a higherdimensional feature space.
2.3 Kfold cross validation
Kfold crossvalidation is a common method to prevent overfitting. It splits the initial sampling set into K subsample sets, where a single subsample set is kept as the test set to validate the model, and the other \(K1\) subsample sets are used for training. Crossvalidation is repeated K times, once for each subsample set, and the results are finally averaged over K times to get a relatively accurate estimate. For a Kfold crossvalidation, the rootmeansquare error (RMSE) as the regression model performance measure, generally defined as
and
where \(n_i\) is the number of elements in the ith subsample set, and \(f(x_{j})\) and \(y_{j}\) are, respectively, the prediction result and the true value of the jth sample in the ith subsample set. Note that in the formula (2), \(E_i(f)\) is the RMSE of the ith subsample set, while E(f) in the formula (1) is the RMSE of the total data.
In the evaluation of radar antijamming effectiveness, we will use a 6fold cross validation to prevent overfitting. Then the RMSE of the total data naturally becomes \(E(f)=\frac{1}{6}\sum _{i=1}^{6}E_i(f)\).
3 Construction of experiment scheme
Li et al. [22] argued that, according to the actual combat requirements, it is becoming more and more important to understand the various skills of measuring equipment under different parameters combined through experiments. It is the most authentic and reliable to obtain results by the corresponding data from a large number of confrontation tests, and radar antijamming performance evaluated according to its performance evaluation index. In reality, however, due to the constraints of various factors, it is impossible to exhaust all the combinations that are opposed to each other; see Guariglia and Silvestrov [23]. Therefore, the design of experiment becomes necessary. As Chen and Lei [24] pointed out that, an outstanding construction scheme in the radar countermeasure experiment contributes to the accuracy of the later evaluation of radar antijamming effectiveness. Yilmaz, Cohen and Porter [25] applied the covering arrays for efficient fault characterization in complex configuration spaces. Colbourn et al. [26] gave a construction of mixed covering array with strength two, and emphasized its broad applicability in military data analysis. According to the spacefilling properties of the fractal geometry, Best [27] studied the resonance compression and multiband behavior of fractalshaped wire antennas. MartÃnez et al. [28] used ELAs, coverage arrays, and adaptive testing algorithms to locate errors. Guo and Wang [29] creatively applied unified design methods to the radar antijamming simulation experiment, and achieved ideal analysis results. Taking into account the actual operation and requirements, our study in this manuscript includes the following three types of jammers:

(1)
noise FM jammer,

(2)
noise PM jammer,

(3)
multiple false target jammer.
All these three kinds of jamming are active jammings in intentional jammings. Among them, the first two are covered jammings, while the third one belongs to deceptive jamming. Covered jamming always uses noise or noiselike jamming signals to cover or overwhelm the useful signals and prevents radar from detecting target information. Its principle is to make the strong jamming power into radar receiver, reduce the signaltonoise ratio as much as possible, and make it hard for the radar to detect the target. While the principle of deceptive jamming is that, false targets and information are used in radarâ€™s target detection and tracking system to make the radar cannot correctly detect the real target or measure the parameter information of real targets.
Moreover, both of noise FM jammer and noise PM jammer have five parameters, while the multiple false target jammer has four parameters. And the radar itself has six parameters, which can be set to be different values to observe its performance in the case of carrying three types of jammers. Based on the above analysis, the difficulties of experiment design here under complex electromagnetic environment are mainly the diversity of variable values and various complex constraints among variables. However, the twomap position array and twomap detection array can meet our actual needs of confrontation experiment in a complex electromagnetic environment. They require that each level combination of any radar parameter and jammer parameter occurs at least once, which results in the overall design with less aberration. Here an experiment design with eighteen runs is proposed, shown in Tables 1 and 2.
4 Measurement of antijamming effectiveness
4.1 Single indicator
In the process from detecting and discovering to automatically locking the target, the radar usually has undergone three stages of searching, capturing and tracking in turn. Wherein, the radar scans the airspace of interest over a large area during the search phase, explores and locates possible targets; see Sotiroudis et al. [30]. The tracking phase guides the system to the target and track it. And the capture phase is a process from the search phase to the automatic tracking phase, whose duration is generally short.
In order to achieve quantitative measure of antijamming capability, we are supposed to select metrics reasonably. These metrics are required to reflect the extent of radar jamming in both search phase and tracking phase, and the effect of both covered jamming and deceptive jamming. Based on the above analysis, and combined with principles of power, information, efficiency and time, the following multiple indicators in Table 3 are given as a measure of radar performance. The contents of this table are described as follows. For the target interception time, the distance accuracy, the jamming consumption time resources, the percentage of interference consuming time resources and the number of false tracks, the smaller the value, the worse the jamming effect and the better the radar performance. While for the maximum tracking duration, the tracking points, the track time resources for true goals and the track true target time resource percentage, the conclusion is the opposite, i.e., the larger the value, the worse the jamming effect and the better the radar performance. And when the relative distance accuracy\(>2\), it indicates that the radar performance deteriorates due to the effective jamming.
4.2 Comprehensive indicator
The abovementioned principles of power, information, efficiency and time are usually used to provide a number of indicators for measuring the antijamming performance. However, one single indicator selected by these criteria has many deficiencies in radar performance evaluation; see Guariglia [31]. For example, there may be duplication between the several radar effectiveness evaluations obtained by these single indicators. Therefore, based on the characteristics of multiple jammers and the analysis of their impact on radar performance, this manuscript uses the PCA to obtain two comprehensive indicators that better reflect the radar performance. This method effectively avoids the onesidedness of past performance evaluation indicators. In fact, only according to the extraction and meaning of indicators, all these indicators in Table 3 can be easily simplified into six different indicators, which are:

(1)
target acquisition time, (if the real target is lost, the assignment is 2.4)

(2)
maximum tracking duration,

(3)
relative distance accuracy, (if the real target is lost, the assignment is 18.6)

(4)
time resources for tracking true goals,

(5)
jamming consumption time resources,

(6)
number of false tracks.
For convenience, denote these six radar performance indicators as \(y_1\), \(y_2\), \(y_3\), \(y_4\), \(y_5\) and \(y_6\), respectively. The value of each indicator in the eighteen experiment schemes are shown in Table 4. Then we perform the PCA on the above six radar performance indicators to obtain two comprehensive indicators. These two indicators correctly and comprehensively reflect the radar performance. The analysis results are shown in Table 5.
These analysis results show that the cumulative contribution rate of the first principal component and the second principal component is as high as \(89.1\%\), indicating that the first two principal components have a strong ability to interpret the original indicators. Therefore, these two principal component can replace the original six indicators to achieve the purpose of reducing dimensions with few information loss. For more intuitively illustrating the distribution of radar performance indicators, we consider the load matrix which measures the importance of the indicators to the principal components. Using the biplot() function in R, with the first principal component as the abscissa and the second principal component as the ordinate, a PCA scatter plot can be drawn, as shown in Fig.Â 2. In this way, we are able to intuitively observe the correlation of each indicator and their degree of influence on the principal component.
As shown in Fig.Â 2, the six radar performance indicators are divided into two groups. The first principal component is positively correlated with \(y_1\) and \(y_3\), while negatively correlated with \(y_2\) and \(y_4\). And the second principal component is positively correlated with \(y_5\) and \(y_6\) Therefore, the problem of modeling and analyzing each indicator is successfully transformed into the modeling analysis of these two principal component. According to their reflections of radar performances, we name them as the tracking capability \(z_{1}\) and the antijamming capability \(z_{2}\), respectively. The expressions of \(z_{1}\) and \(z_{2}\) are as follows:
Tracking capability (\(z_{1}\)):
Antijamming capability (\(z_{2}\)):
where \(y_1\), \(y_2\), \(y_3\), \(y_4\), \(y_5\) and \(y_6\) denote the six radar performance indicators as in Table 4.
5 Evaluation of antijamming effectiveness
The evaluation of radar antijamming capability has always been a concern for the researchers of radar antijamming technologies and for manufacturers of jamming equipments. The early evaluation factor method considers only few factors, and the evaluation results are generally not normalized. Obviously, this is not convenient for qualitative evaluation. In recent years, the evaluation method has developed into the fuzzy lake comprehensive evaluation method, and then the machine learning evaluation method. It has been confirmed that the evaluation based on the machine learning for predicting radar performance has certain robustness and generalization capability. By this way, we are able to obtain the influence law of each factor on radar performance. And then the evaluation of radar antijamming capability under specific factors can be realized.
According to the quantitative characteristics of radar antijamming performance evaluation, it can be regarded as a regression problem. Therefore, here we select four popular machine learning algorithms, CART, random forest, Xgboost and SVM, to solve the evaluation of radar antijamming performance. Considering the correctness of each algorithm and their feasibilities in radar antijamming effectiveness evaluation, we also compare the quality of these four machine learning algorithms. In addition, we adopt the crossvalidation method in data analysis for avoiding overfitting. Here we declare that, All specific data in this simulation are derived from the observations in the real radar countermeasure experiments of State Key Laboratory of Complex Electromagnetic Environment Effects on Electronics and Information System (CEMEE) of China. Next we are going to use the experiment data under two comprehensive indicators \((z_{1},\ z_{2})\), to evaluate the generalization ability of the model.
The values of \(z_{1}\) (the tracking capability) and \(z_{2}\) (the antiinterference capability) under the eighteen experiments are shown in Table 6. The smaller the values of \(z_{1}\) and \(z_{2}\), the better the radar performances.
The model evaluation is divided into two steps. The first one is to use the overall sample to train the model, and to predict the value of the target variable for
the same overall sample. That is, the training set and the test set are the same. This approach helps to observe the effect of the model, however there may be a risk of overfitting. Then according to the formulas (1) and (2), the overall RMSE can be obtained, as shown in Table 7. And the second step is to evaluate the generalization ability of each model, and then to conduct crossvalidation. The training error and the verification error under different models are shown in Table 8.
The above results show that Xgboost has the best fitting effect, but the generalization ability is slightly inferior to other models, which is obviously not the best choice. According to further indepth comparison, we find that the average verification error of random forest is the smallest, that is, the generalization ability is the best. Therefore, random forest can be chosen as the final model for predictive analysis. The good generalization ability fully demonstrates the evaluation ability of our model. This is because the two principal components, the tracking capability \(z_1\) and the antijamming capability \(z_2\) obtained by the PCA in Sect. 3, accurately measure the radar antijamming performance. These results and methods complement each other and explain the superiority of each other.
6 Additional verification of models
In order to further verify the accuracy of the obtained model, we supplement some subsequent experiments, and compare and analyse the actual experimental results with the prediction results of the above models.
6.1 Construction of supplementary experiments
Decision trees, random forest, and Xgboost all provide seven important variables that affect antijamming performance. They are CFAR unilateral reference unit number, burn through, front edge tracking, \(SIR_{1}\), \(SIR_{2}\), \(SIR_{3}\) and the number of false targets, which are, respectively, denoted as \(V_{1},\ V_{2},\ V_{3},\ V_{4},\ V_{5},\ V_{6}\) and \(V_{7}\) for convenience.
In order to increase the supplementary experiment, the design can be constructed by using the above variables (the factors that do not contain 4 levels are treated as pseudolevels) with the values of other variables chosen arbitrarily. The construction and the corresponding results of the supplementary experiments are shown in Table 9.
6.2 Comparisons of actuality and prediction
Since that there are two response variables here, \(z_1\) and \(z_2\), the following research is correspondingly divided into two parts, â€™Comparison on \(z_1\)â€™ and â€™Comparison on \(z_2\)â€™. In each part, we make use of existing variables (\(V_1\) to \(V_7\)) to construct several new variables before performing the numerical fitting, in order to reduce the number of variables in the model and avoid making the model too complex.

(i)
Comparison on \(z_{1}\):
Considering that \(V_2\) and \(V_3\) are qualitative variables with two values, we firstly construct a new variable \(V_{8}\) by \(V_{2}\) and \(V_{3}\) as follows to classify the data.
Secondly, within each block, it is noted that the variables \(V_{4},\ V_{5}\) and \(V_{6}\) (\(SIR_1\), \(SIR_2\) and \(SIR_3\)) have some obvious properties. We summarize these properties and construct another new variable \(V_{9}\) with \(V_{4},\ V_{5},\ V_{6}\) and \(V_{8}\) as follows,
In this way, we easily get
Therefore, \(z_{1}\) in Table 9 can be divided into two types according to \(z_{1}>1\) and \(z_{1}\le 1\). Consequently, we fit \(V_{7}\) and \(z_{1}\) with the model \(z_{1}=b_{0}+b_{1}\times V_{7}\) by different classifications of \(z_{1}\), and fit \(V_{4}\), \(V_{6}\) and \(z_{1}\) with the model \(z_{1}=b_{2}+b_{3}\times V_{4}+b_{4}\times V_{6}\). The resulting linear regression equations are as follows
At a 10\(\%\) significance level, the overall and coefficients are both significant. The conclusion unifies the experimental results and the prediction results, which shows the resulted model has excellent prediction effect.

(ii)
Comparison on\(z_{2}\):
Similar to the above, we firstly construct a new variable \(V_{10}\) with \(V_{2}\) and \(V_{3}\) to classify the data,
Secondly another new variable \(V_{11}\) is constructed with \(V_{7}\) (the number of false targets) and \(V_{10}\) as follows to increase the difference between blocks,
Finally we fit the data with the model \(z_{2}=b_{5}+b_{6}\times V_{1}+b_{7}\times V_{3}+b_{8}\times V_{10}\) to get the linear regression equation
At a 5\(\%\) significance level, the overall and coefficients are both significant. The conclusion also unifies the above experimental results and the prediction results, which shows the resulted model has excellent prediction effect.
In addition, we provide the comparison between the true values and the predictive values of \(z_{1}\) and \(z_{2}\) for the above supplementary experiments, as showed in Table 10. As can be intuitively seen from the results in the above table, the prediction effect is quite good. This states that our analysis of radar performances and selection of models are reasonable.
7 Conclusion and discussion
This manuscript is a successful case of applying machine learning and statistical methods to solve military problems, and provides a methodology for data analysis of similar practical projects. Through the analysis of radar antijamming performance and the research on its effectiveness evaluation method, a bipartite covering array is designed for the experimental construction scheme. We combined the existing indicators and used the principal component analysis method to obtain two comprehensive indicators to better reflect the radar performances. Moreover, four machine learning algorithms are used to greatly weaken the influence of subjective factors in the comprehensive evaluation, which has good practical application value. The final additional experiments verify the effectiveness of the final model. This provides a valuable reference for the research of radar antijamming.
However, there may be room for improvement. For example, the manuscript adopts four kinds of machine learning algorithms to evaluate the effectiveness of radar antijamming. Interested readers could use other more suitable algorithms, which may obtain better results. Moreover, in Sect. 6.2, there may be some better methods for the comparisons of actuality and prediction in supplementary experiments. This is also a good topic worth discussing in the future. Whereas, the research methods in this paper set up a framework to solve the analysis problem of real data for common readers. We believe that with the development of theory and technology, the radar antijamming effectiveness evaluation method can be improved, thereby, increasing the reliability of radar antijamming.
Availability of data and materials
All data supporting research in this manuscript are derived from the observations in the real radar countermeasure experiments of State Key Laboratory of Complex Electromagnetic Environment Effects on Electronics and Information System (CEMEE) of China. The results of the data analysis have been displayed in detail in this manuscript. However, due to the confidentiality of military data and other reasons, it may not be possible to obtain specific data sets from its home page at present.
Abbreviations
 Bmn:

The bandwidth of modulation noise
 Bt:

The burn through
 \(Comp_1\) :

The first principal component
 \(Comp_2\) :

The second principal component
 \(Comp_3\) :

The third principal component
 CART:

Classification and regression tree
 Cr:

The chirp rate
 Curun:

CFAR unilateral reference unit number
 Cupun:

CFAR unilateral protection unit number
 Fa:

The frequency agility
 Fc:

FM constant
 Fet:

The front edge tracking
 Ftd:

The false targets distribution
 GBDT:

Gradient boosting decision tree
 Mftjp:

The multiple false targets jamming parameters
 \(n_1\) :

The first quantity
 \(n_2\) :

The second quantity
 \(n_3\) :

The third quantity
 Nfta:

The number of false targets
 Npjp:

The noise PM jamming parameters
 PCA:

Principal component analysis
 Pv:

The predictive value
 RMSE:

The rootmeansquare error
 Sdmn:

The standard deviation of modulation noise
 \(SIR_1\) :

The signal to interference ratio in the noise FM jamming
 \(SIR_2\) :

The signal to interference ratio in the noise PM jamming,
 \(SIR_3\) :

The signal to interference ratio in the multiple false targets jamming
 SVM:

Support vector machine
 Tdr:

The tracking data rate
 Te:

The training error
 Tv:

The true value
 Ve:

The verification error
 \(y_1\) :

The target acquisition time
 \(y_2\) :

The maximum tracking duration
 \(y_3\) :

The relative distance accuracy (compared with No.0)
 \(y_4\) :

The time resources for tracking true goals
 \(y_5\) :

The jamming consumption time resources
 \(y_6\) :

The number of false tracks
References
Z. Geng, H. Yan, J. Zhang, D. Zhu, Deeplearning for radar: a survey. IEEE Access 9, 141800â€“141818 (2021)
S.R. Best, Operating band comparison of the perturbed Sierpinski and modified Parany gasket antennas. IEEE Antennas Wirel. Propag. Lett. 01(01), 35â€“38 (2002)
E. Guariglia, Entropy and fractal antennas. Entropy 18(03), 84 (2016)
H.X. Huang, J.Q. Zhang, H. Xu, The development status and presumption of antijamming ability evaluation. Aerospace Electron. Warfare 30(01), 25â€“28 (2001)
C.T. Liu, R.J. Wu, Z.X. He, X.F. Zhao, H.C. Li, P.Z. Wang, Modeling and analyzing interference signal in a complex electromagnetic environment. EURASIP J. Wirel. Commun. Netw. 01, 01â€“09 (2006)
X.N. Wang, F. Su, W. Yu, J.L. Luo, Radar countermeasure question in complex electromagnetic environment. Modern Radar 04(17), 21â€“25 (2008)
W. Yi, Y. Yuan. Reinforcement learningbased joint adaptive frequency hopping and pulsewidth allocation for radar antijamming, in IEEE Radar Conference (RadarConf20). IEEE, (2020) pp. 1â€“6
L. Jia, Y. Xu, Y. Sun, S. Feng, A. Anpalagan, Stackelberg game approaches for antijamming defence in wireless networks. IEEE Wirel. Commun. 25(06), 120â€“128 (2018)
R.G. Hohlfeld, N. Cohen, Selfsimilarity and the geometric requirements for frequency independence in antennae. Fractals 07(01), 79â€“84 (1999)
S. Kumar, K. Singh, S. Kumar, O. Kaiwartya, Y. Cao, H. Zhou, Delimitated anti jammer scheme for internet of vehicle: machine learning based security approach. IEEE Access 07, 113311â€“113323 (2019)
Y. Zhang, B. Jiu, P. Wang, H. Liu, S. Liang, An endtoend antijamming target detection method based on CNN. IEEE Sens. J. 21(19), 21817â€“21828 (2021)
M.M. Aziz, A.R. Maud, A. Habib. Reinforcement learning based techniques for radar antijamming, in 2021 International Bhurban Conference on Applied Sciences and Technologies (IBCAST), IEEE, (2021), pp. 1021â€“1025
L.J. Meng, S.J. Zhang, S. Fan, Effectiveness evaluation of radar countermeasure equipment based on catastrophe theory. Modern Radar 40(03), 77â€“83 (2018)
F. Long, X.L. Zhang, X.B. Lin, A comprehensive evaluation criterion of radar ECCM performance based on electronic effect matrix. Modern Radar 28(01), 20â€“22 (2006)
W. Yang, J.S. Li, Y.L. Zhang, Study of radar countermeasure training and simulation system based on HLA. Microcomput. Inf. 01, 240â€“242 (2006)
Z.L. Lei, K.B. Qin, M. Xu, H. Shan, Y.C. Bian, Efficiency evaluation of radar countermeasure equipment joint netting operation based on AHPcloud model. Shipboard Electron. Countermeasure 37(06), 77â€“82 (2014)
Y. Wang, B. Sun, N. Wang, Recognition of radar activejamming through convolutional neural networks. J Eng 21(05), 7695â€“7697 (2019)
F. Bu, J. He, H. Li, Q. Fu, Radar seeker antijamming performance prediction and evaluation method based on the improved grey wolf optimizer algorithm and support vector machine, in 2020 IEEE 3rd International Conference on Electronics Technology (ICET), IEEE, (2020), pp. 704â€“710
W.C. Li, H. Li, W.X. Li, F. Ma, B. Tang, Experimental evaluation method for interference suppression distance of networked radar. J. Terahertz Sci. Electron. Inf. 18(03), 385â€“390 (2020)
X.C. Cheng, C.Q. Zhang, Y. Qian, M. Aloqaily, Y. Xiao, Deep learning for 5G IoT systems. Int. J. Mach. Learn. Cybern. 12(11), 3049â€“3051 (2021)
S. Raheja, S. Kasturia, X.C. Cheng, M. Kumar, Machine learningbased diffusion model for prediction of coronavirus19 outbreak, in Neural Computing and Applications, (2021)
K. Li, B. Jiu, P. Wang, H. Liu, Y. Shi, Radar active antagonism through deep reinforcement learning: a way to address the challenge of mainlobe jamming. Signal Process. 186(04), 108130 (2021)
E. Guariglia, S. Silvestrov, Fractionalwavelet analysis of positive definite distributions and wavelets on \(\cal{D} ^{\prime }(\mathbb{C} )\) . Eng. Math. II Springer Proceed. Math. Stat. 179, 337â€“353 (2017)
X.Y. Chen, G.Q. Lei, Integrated experiment design framework of radar countermeasure equipment. Electron. Inf. Warfare Technol. 27(04), 61â€“64 (2012)
C. Yilmaz, M.B. Cohen, A.A. Porter, Covering arrays for efficient fault characterization in complex onfiguration spaces. IEEE Trans. Software Eng. 29(04), 45â€“54 (2004)
C.J. Colbourn, S.S. Martirosyan, G.L. Mullen, D.E. Shasha, G.B. Sherwood, J.L. Yucas, Products of mixed covering arrays of strength two. J. Comb. Des. 14(02), 124â€“138 (2006)
S.R. Best, A discussion on the significance of geometry in determining the resonant behavior of fractal and other noneuclidean wire antennas[J]. IEEE Antennas Propag. Mag. 45(03), 9â€“28 (2003)
Conrado MartÃnez, L. Moura, D. Panario, B. Stevens, Locating errors using ELAs, covering arrays, and adaptive testing algorithms. Siam J. Discrete Math. 23(04), 1776â€“1799 (2010)
X.Y. Guo, J.J. Wang, The application of uniform design methods in radar antijamming simulation experiment. Firepower Command Control 40(08), 160â€“163 (2015)
S.P. Sotiroudis, P. Sarigiannidis, S.K. Goudos, K. Siakavara, Fusing diverse input modalities for path loss prediction: a deep learning approach. IEEE Access 09, 30441â€“30451 (2021)
E. Guariglia, Harmonic Sierpinski gasket and applications. Entropy 20(09), 714 (2018)
Acknowledgements
The authors would like to thank the respected editors and the reviewers for their helpful comments.
Funding
This research was supported by NNSF of China (12071329) and CEMEE (2021G0301 and 2017K0301A1). They are the main source of labor and allowances for the design of the study and collection, analysis, and interpretation of data and writing the manuscript.
Author information
Authors and Affiliations
Contributions
In this work, the first, fourth and fifth authors put forward the overall research ideas. The second and third authors completed the virtual simulation experiments and data collection. The first and fourth authors obtained two comprehensive indicators through data analysis, and completed the measurement of radar antijamming effectiveness. The first and fifth authors constructed models to achieve the evaluation of radar antijamming effectiveness, and wrote and revised the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declared that they have no conflicts of interest to this work, and do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supported by NNSF of China (12071329) and by CEMEE (2021G0301 and 2017K0301A1).
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Qi, L., Zhang, J., Qi, ZF. et al. Measurement and evaluation method of radar antijamming effectiveness based on principal component analysis and machine learning. J Wireless Com Network 2023, 58 (2023). https://doi.org/10.1186/s13638023022623
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13638023022623
Keywords
 Bipartite covering array
 Machine learning
 Measurement and evaluation
 Principal component analysis
 Radar jamming effectiveness