Measurement and evaluation method of radar anti-jamming effectiveness based on principal component analysis and machine learning

Abstract

With the development of modern electronic countermeasure technology, the fight between radar jamming and anti-jamming has become increasingly fierce. Experts have done a lot of highly effective work on radar anti-jamming performance. However, the emergence of various new complex interferences has rendered existing methods unable to meet the needs. In this manuscript, we consider the measurement and evaluation method of radar anti-jamming effectiveness based on principal component analysis and machine learning. Firstly, taking into account the diversity of variables in radar countermeasure experiments and the complexity of constraints between variables, we propose a bipartite covering array for the experimental scheme, which requires that each level combination of any radar parameter and jammer parameter occurs at least once, to ensure the rationality of the experiments. Secondly, according to the characteristics of multiple jammers and the analysis of impacts on radar performances, we combine the existing indicators and use the principal component analysis method to obtain two comprehensive indicators, which better reflect radar performances. Finally, we select the best model as a prediction for radar comprehensive indicators by comparing several machine learning algorithm models, including classification and regression tree, random forest, xgboost, and SVM. Additional experiments verify the effectiveness of the resulted model.

1 Introduction

Geng et al. [1] pointed out that, as radars are widely used in the military field, anti-radar electronic jamming technologies have also become important. In recent years, with the rapid development of radar countermeasures, especially anti-jamming, the application of radars in modern warfare is becoming increasingly strict. In fact, modern warfare has gradually transitioned from mechanization to informatization through the extensive application of electronic information technology in the military field, forming a complex electromagnetic environment in which the two sides confront each other; see Best [2] and Guariglia [3]. Hence, military technologists have been conducting scientific and technological research on this, proposing solutions to maintain radar reconnaissance capabilities. Huang, Zhang, and Xu [4] stated the development status and presumption of the anti-jamming capability of the existing electronic instruments. Liu et al. [5] modeled and analyzed the interfering signal data in a complex electromagnetic environment. Wang et al. [6] proposed and analyzed radar countermeasures questions in complex electromagnetic environments. In fact, the two sides, the jamming side and the radar side, could be unified on the same issue. The jamming side mainly focuses on whether the jamming can weaken or even fail the radar detection capability; while the radar side is concerned about its working ability under jamming. Both want to know clearly about the anti-jamming capability of radars in specific confrontation situations to take corresponding measures in the process of technical research and equipment development. Therefore, as argued by Yi and Yuan [7], it is crucial to study the radar anti-jamming performance, which represents a change in radar performance under the influence of jammers after adopting anti-jamming technology. Simply put, it is the combined result of the joint effects of the radar side, the jamming side and the countermeasure environment. Researchers are concerned about the relationship between the anti-jamming effectiveness and the influencing factors, so that the anti-jamming effectiveness of the radar can be predicted and estimated according to the influencing factors. Nevertheless, the relevant researchers always encounter the practical problems of ‘impossibility of measurement’ and ‘inaccuracy of evaluation’. The focus of this manuscript is to obtain a relationship between the anti-jamming effectiveness and the influencing factors, to accurately predict or estimate the anti-jamming effectiveness under the presumed influencing factors. The manuscript is mainly divided into the following three parts: the radar jamming experiment, the measurement and the evaluation of radar anti-jamming effectiveness.

1.1 Radar jamming experiment

The radar jamming experiment can be regarded as the basis for studying the anti-jamming effectiveness evaluation method. Jia et al. [8] called it the highlight of the evaluation methods research. However, it is very difficult to derive the radar anti-jamming performance directly from the influencing factor theory. Hence, the study in the evaluation method of anti-jamming capability is generally based on data collected from various experiments. Radar jamming experiment methods are mainly divided into external field jamming, semi-physical simulation jamming experiments, and full-virtual simulation jamming experiments on a computer. These three have their own characteristics and advantages. Among them, as Hohlfeld and Cohen [9] pointed out, the fully virtual simulation experiments are favored by researchers owing to their flexibility and ability to demonstrate the diversity of adversarial situations. The fully virtual simulation confrontation experiment system formed by a variety of high-fidelity simulation models can provide relatively reliable experimental results. In this manuscript, we use a fully virtual simulation experiment to propose some experimental schemes of the given radar parameters and three kinds of jammer parameters, and give the simulated radar anti-jamming performance results.

1.2 Measurement of radar anti-jamming effectiveness

Measurement of radar anti-jamming effectiveness is the process of establishing the evaluation indicators for radar anti-jamming effectiveness, and the basis of anti-jamming effectiveness evaluation. It obtains a specific anti-jamming effect according to the change in radar performance before and after jamming, also known as the quantification process of anti-jamming performance; see Kumar et al. [10].

Actually, the anti-jamming capability is not a real quantity that can be directly measured, so it is generally necessary to select the best indicators reflecting the change in the working capability according to the function of the radar, and then use a comprehensive treatment on these indicators to obtain the anti-jamming capability of the radar. Experts have done a lot of highly effective work in the evaluation of indicators of anti-jamming performance. Zhang et al. [11] provided an end-to-end anti-jamming target detection method based on CNN. Aziz, Maud, and Habib [12] proposed a radar anti-jamming technology based on reinforcement learning, which greatly improved the radar anti-jamming performance. However, the emergence of various new complex interferences has rendered existing methods unable to meet the needs. Moreover, as new types of radars continue to emerge, radar countermeasures are increasingly diversified and the complex electromagnetic environment where radars are located has also undergone new changes. All these ask for more quantitative and systematic evaluation solutions. Under this background, it is imminent to establish a scientific and reasonable evaluation system to guide radar anti-jamming performance evaluation.

There are many criteria for measuring the effectiveness of anti-jamming, which reflect the degree of change in radar performances before and after the jamming from different aspects, including power criterion, information criterion, and efficiency criterion. Through these criteria, many anti-jamming evaluation indicators can be selected, but selecting a single indicator has many deficiencies in radar performance evaluation. For example, there are repeated evaluations of certain anti-jamming performance because of too many evaluation methods, and the objectivity and the operability still need to be improved. Given the above-mentioned deficiencies and based on the characteristics of various jammers and the impact analysis on radar performance, we use the principal component analysis method and combine the above-mentioned existing criteria to extract two comprehensive indicators. This method avoids the one-sidedness that the past radar evaluation indicator can evaluate only one anti-jamming performance.

1.3 Evaluation of radar anti-jamming effectiveness

Evaluation of radar anti-jamming effectiveness is a modeling process of comprehensively evaluating the radar anti-jamming effect according to the real data and obtained indicators. It is a complex area of research. As pointed out by Meng, Zhang and Fan [13], on the one hand, the diversity of radar countermeasures, including the diversity of radars and the jammers, makes the prediction methods of anti-jamming ability flourish. On the other hand, it leads to the complexity of the evaluating problem that, the subject of the evaluation is the person and the evaluation results are closely related to the purpose of evaluation, the evaluation perspective, and other various benefits. Both lead to the lack of comprehensive and unified practice standards and methods. Moreover, there are many factors that affect the radar anti-jamming capability, so that the number of countermeasures composed of many influencing factors is huge. In this case, insufficient training samples cause poor prediction results for traditional methods.

Nowadays, the widely used evaluation methods mainly include: the evaluation factor method, the fuzzy comprehensive evaluation method, and machine learning methods based on neural networks. Long, Zhang, and Lin [14] argued a comprehensive evaluation criterion of radar ECCM performance based on an electronic effect matrix. Yang, Li, and Zhang [15] studied the radar countermeasure training and simulation system based on HLA. Lei et al. [16] provided an efficiency evaluation method for radar countermeasure equipment joint netting operation based on the AHP-cloud model. Wang, Sun and Wang [17] deeply studied the radar active jamming recognition through the convolutional neural network. Bu et al. [18] provided a radar seeker anti-jamming performance prediction and evaluation method based on the improved grey wolf optimizer algorithm and support vector machine. Li et al. [19] innovatively provided an experimental evaluation method for interference suppression distance of networked radar. Cheng et al. [20] applied deep learning architectures to 5 G IoT systems. Raheja et al. [21] used machine-learning diffusion models to predict coronavirus-19 outbreaks. This manuscript uses multiple machine learning methods to evaluate the radar anti-jamming effectiveness. Besides the innovative experiments, this manuscript also focuses on machine learning algorithms and their application in the evaluation of anti-jamming effectiveness. The former is based on the application requirements to select the appropriate algorithm model, and to solve the problem of model parameter selection and feature selection, while the latter is to find the best model as a prediction for radar comprehensive indicators by comparing several algorithm models.

The remainder of this manuscript is organized as follows: Sect. 2 states the existing analytical methods used in this manuscript. Section 3 proposes a reasonable construction of the experiment scheme. In Sect. 4, the principal component analysis is applied to provide comprehensive radar evaluation indicators. Based on the actual data and the obtained indicators, Sect. 5 provides four machine learning algorithms to establish regression models to achieve the evaluation of the radar anti-jamming effectiveness. In Sect. 6, the additional experiments verify the reliability of our proposed ideas and methods. Section 7 summarizes the full manuscript.

In this work, the first, fourth, and fifth authors put forward the overall research ideas. The second and third authors completed

the virtual simulation experiments and data collection. The first and fourth authors obtained two comprehensive indicators through data analysis, and completed the measurement of radar anti-jamming effectiveness. The first and fifth authors constructed models to achieve the evaluation of radar anti-jamming effectiveness, and wrote and revised the manuscript.

2 Methods

In addition to the three research methods described in the Introduction, ‘Radar jamming experiment’, ‘Measurement of radar anti-jamming effectiveness’ and ‘Evaluation of radar anti-jamming effectiveness’, we will also introduce the existing data analysis methods mainly applied in the following sections as follows.

2.1 PCA

Principal component analysis (PCA) is a common data dimensionality reduction method. It transforms a set of potentially correlated variables into a set of linearly uncorrelated variables through orthogonal transformation, and this transformed set is called the principal components.

This manuscript will use this method to obtain two comprehensive indicators for the anti-jamming effectiveness measurement in Sect. 4.

2.2 Four machine learning algorithms

In the evaluation of radar anti-jamming effectiveness, machine learning algorithms play a vital role. According to the quantitative characteristics of radar anti-jamming performance evaluation, it can be regarded as a regression problem. Therefore, here we introduce four different machine learning algorithms used in Sect. 5 to solve the evaluation of radar anti-jamming performance, namely Classification and regression tree (CART), random forest, Xgboost and support vector machine (SVM), as follows.

• (i) CART

Classification and regression tree (CART) is a widely used machine learning method for both classification and regression, which consists of feature selection, tree generation, and pruning. CART is a learning method to output the conditional probability distribution of the random variable Y under the condition of the given input random variable X. It assumes that the decision tree is a binary tree, and the values of the internal node features are ’yes’ and ’no’. The left branch is that with value of ’yes’, while the right one is that with value of ’no’. Such a decision tree is equivalent to recursively dicing each feature. It divides the input feature space into finite units, and determines the probability distribution of prediction on these units. The CART algorithm consists of the following two steps:

• (1) Generation of decision tree: Generate the decision tree based on the training data.

• (2) Pruning of decision tree: Prune the generated tree with the test data set and select the optimal subtree. Generally, the minimum loss function is used as the criterion for pruning.

• (ii) Random forest

Random forest is a machine learning algorithm that contains multiple decision trees. Denote the number of samples as N, and that of features as M. Then the random forest builds decision trees according to the following steps:

• (1) Sampling N samples with replacement from the N training samples to train a decision tree (i.e. bootstrap sampling), and use the unselected samples as the test set to evaluate the error of the model.

• (2) When each node of the decision tree needs to be split, randomly select m features from these M ones, where m should be much smaller than M. Then use a certain strategy (such as information gain) on these m features to select one as the split feature of the node.

• (3) In the process of decision tree formation, each node must be split according to step 2 until it can no longer be split. Note that there is no pruning during the entire decision tree formation process.

• (4) Follow steps 1 to 3 to build a large number of decision trees, which constitute a random forest.

• (iii) Xgboost

Xgboost, proposed by Dr. Chen Tianqi, is a massively parallel Boosted tree. It was developed on the basis of Gradient boosting decision tree (GBDT). Since its introduction, Xgboost has received a lot of attention in the Kaggle community competition. In the same situation, its speed is more than 10 times faster than other similar algorithms and it supports parallelization.

GBDT is a combination of decision tree and boosting method. Each decision tree of GBDT trains the errors in the result of the previous one. The training process is linear, and each iteration is the optimization goal. Compared with the traditional GBDT algorithm, Xgboost has a lot of progress. Xgboost uses the second derivative information by performing the second-order Taylor expansion of the loss function, instead of using only the first derivative information as the traditional GBDT. In addition, Xgboost adds the complexity of the model as a regular term into the objective function to optimize the model, and the pruning into the later stage. A method of shrinkage and column sub-sampling also used in Xgboost to avoid overfitting easily. Moreover, Xgboost takes the segmentation search algorithm, which automatically uses the sparseness of features to learn the parallelization tree, so it can handle higher-dimensional sparse matrices relative to GBDT.

• (iv) SVM

Support vector machine (SVM) is a supervised generalized linear classifier, which can handle both regression and classification problems well. SVM is sought after by many researchers because of its advantages such as the ability to process high-dimensional data, to obtain global optimal solutions, and not easy to overfit.

For linear separable samples, SVM commits to find an decision boundary to classify the n input data $X=\{x_1,\cdots ,x_n\}$ into positive and negative classes. This boundary is also known as the optimal hyperplane $H: w^\textsf{T}X+b=0$, where w is the normal vector and b is the intercept. It not only accurately separates the samples, but also maximizes the classification interval, which is the distance between the upper and lower interval boundaries $H_1$ and $H_2$, as shown in Fig. 1. While for nonlinear separable samples, SVM obtains linear separation by applying a kernel function to map the sample space into a higher-dimensional feature space.

Fig. 1

Optimal hyperplane in SVM; The basic purpose of SVM is to find an optimal hyperplane, which not only accurately separates the samples, but also maximizes the classification interval

2.3 K-fold cross validation

K-fold cross-validation is a common method to prevent overfitting. It splits the initial sampling set into K sub-sample sets, where a single sub-sample set is kept as the test set to validate the model, and the other $K-1$ sub-sample sets are used for training. Cross-validation is repeated K times, once for each sub-sample set, and the results are finally averaged over K times to get a relatively accurate estimate. For a K-fold cross-validation, the root-mean-square error (RMSE) as the regression model performance measure, generally defined as

$$\begin{aligned} E(f)=\frac{1}{K}\sum _{i=1}^{K}E_i(f), \end{aligned}$$

(1)

and

$$\begin{aligned} E_i(f)=\sqrt{\frac{1}{n_i} \sum _{j=1}^{n_i}\Big (f(x_{j})-y_{j}\Big )^{2}}, \end{aligned}$$

(2)

where $n_i$ is the number of elements in the i-th sub-sample set, and $f(x_{j})$ and $y_{j}$ are, respectively, the prediction result and the true value of the j-th sample in the i-th sub-sample set. Note that in the formula (2), $E_i(f)$ is the RMSE of the i-th sub-sample set, while E(f) in the formula (1) is the RMSE of the total data.

In the evaluation of radar anti-jamming effectiveness, we will use a 6-fold cross validation to prevent over-fitting. Then the RMSE of the total data naturally becomes $E(f)=\frac{1}{6}\sum _{i=1}^{6}E_i(f)$.

3 Construction of experiment scheme

Li et al. [22] argued that, according to the actual combat requirements, it is becoming more and more important to understand the various skills of measuring equipment under different parameters combined through experiments. It is the most authentic and reliable to obtain results by the corresponding data from a large number of confrontation tests, and radar anti-jamming performance evaluated according to its performance evaluation index. In reality, however, due to the constraints of various factors, it is impossible to exhaust all the combinations that are opposed to each other; see Guariglia and Silvestrov [23]. Therefore, the design of experiment becomes necessary. As Chen and Lei [24] pointed out that, an outstanding construction scheme in the radar countermeasure experiment contributes to the accuracy of the later evaluation of radar anti-jamming effectiveness. Yilmaz, Cohen and Porter [25] applied the covering arrays for efficient fault characterization in complex configuration spaces. Colbourn et al. [26] gave a construction of mixed covering array with strength two, and emphasized its broad applicability in military data analysis. According to the space-filling properties of the fractal geometry, Best [27] studied the resonance compression and multi-band behavior of fractal-shaped wire antennas. Martínez et al. [28] used ELAs, coverage arrays, and adaptive testing algorithms to locate errors. Guo and Wang [29] creatively applied unified design methods to the radar anti-jamming simulation experiment, and achieved ideal analysis results. Taking into account the actual operation and requirements, our study in this manuscript includes the following three types of jammers:

• (1) noise FM jammer,

• (2) noise PM jammer,

• (3) multiple false target jammer.

All these three kinds of jamming are active jammings in intentional jammings. Among them, the first two are covered jammings, while the third one belongs to deceptive jamming. Covered jamming always uses noise or noise-like jamming signals to cover or overwhelm the useful signals and prevents radar from detecting target information. Its principle is to make the strong jamming power into radar receiver, reduce the signal-to-noise ratio as much as possible, and make it hard for the radar to detect the target. While the principle of deceptive jamming is that, false targets and information are used in radar’s target detection and tracking system to make the radar cannot correctly detect the real target or measure the parameter information of real targets.

Moreover, both of noise FM jammer and noise PM jammer have five parameters, while the multiple false target jammer has four parameters. And the radar itself has six parameters, which can be set to be different values to observe its performance in the case of carrying three types of jammers. Based on the above analysis, the difficulties of experiment design here under complex electromagnetic environment are mainly the diversity of variable values and various complex constraints among variables. However, the two-map position array and two-map detection array can meet our actual needs of confrontation experiment in a complex electromagnetic environment. They require that each level combination of any radar parameter and jammer parameter occurs at least once, which results in the overall design with less aberration. Here an experiment design with eighteen runs is proposed, shown in Tables 1 and 2.

Table 1 Radar and noise FM jamming parameters in experimental schemes

Table 2 Noise PM and multiple false targets jamming parameters in experimental schemes

4 Measurement of anti-jamming effectiveness

4.1 Single indicator

In the process from detecting and discovering to automatically locking the target, the radar usually has undergone three stages of searching, capturing and tracking in turn. Wherein, the radar scans the airspace of interest over a large area during the search phase, explores and locates possible targets; see Sotiroudis et al. [30]. The tracking phase guides the system to the target and track it. And the capture phase is a process from the search phase to the automatic tracking phase, whose duration is generally short.

In order to achieve quantitative measure of anti-jamming capability, we are supposed to select metrics reasonably. These metrics are required to reflect the extent of radar jamming in both search phase and tracking phase, and the effect of both covered jamming and deceptive jamming. Based on the above analysis, and combined with principles of power, information, efficiency and time, the following multiple indicators in Table 3 are given as a measure of radar performance. The contents of this table are described as follows. For the target interception time, the distance accuracy, the jamming consumption time resources, the percentage of interference consuming time resources and the number of false tracks, the smaller the value, the worse the jamming effect and the better the radar performance. While for the maximum tracking duration, the tracking points, the track time resources for true goals and the track true target time resource percentage, the conclusion is the opposite, i.e., the larger the value, the worse the jamming effect and the better the radar performance. And when the relative distance accuracy$>2$, it indicates that the radar performance deteriorates due to the effective jamming.

Table 3 Radar performance indicators

4.2 Comprehensive indicator

The above-mentioned principles of power, information, efficiency and time are usually used to provide a number of indicators for measuring the anti-jamming performance. However, one single indicator selected by these criteria has many deficiencies in radar performance evaluation; see Guariglia [31]. For example, there may be duplication between the several radar effectiveness evaluations obtained by these single indicators. Therefore, based on the characteristics of multiple jammers and the analysis of their impact on radar performance, this manuscript uses the PCA to obtain two comprehensive indicators that better reflect the radar performance. This method effectively avoids the one-sidedness of past performance evaluation indicators. In fact, only according to the extraction and meaning of indicators, all these indicators in Table 3 can be easily simplified into six different indicators, which are:

• (1) target acquisition time, (if the real target is lost, the assignment is 2.4)

• (2) maximum tracking duration,

• (3) relative distance accuracy, (if the real target is lost, the assignment is 18.6)

• (4) time resources for tracking true goals,

• (5) jamming consumption time resources,

• (6) number of false tracks.

For convenience, denote these six radar performance indicators as $y_1$, $y_2$, $y_3$, $y_4$, $y_5$ and $y_6$, respectively. The value of each indicator in the eighteen experiment schemes are shown in Table 4. Then we perform the PCA on the above six radar performance indicators to obtain two comprehensive indicators. These two indicators correctly and comprehensively reflect the radar performance. The analysis results are shown in Table 5.

Table 4 Value of each indicator

Table 5 Principal component analysis on radar performance indicators

These analysis results show that the cumulative contribution rate of the first principal component and the second principal component is as high as $89.1\%$, indicating that the first two principal components have a strong ability to interpret the original indicators. Therefore, these two principal component can replace the original six indicators to achieve the purpose of reducing dimensions with few information loss. For more intuitively illustrating the distribution of radar performance indicators, we consider the load matrix which measures the importance of the indicators to the principal components. Using the biplot() function in R, with the first principal component as the abscissa and the second principal component as the ordinate, a PCA scatter plot can be drawn, as shown in Fig. 2. In this way, we are able to intuitively observe the correlation of each indicator and their degree of influence on the principal component.

Fig. 2

Principal component analysis scatter plot; The six radar performance indicators are divided into two groups. The first principal component is positively correlated with $y_1$ and $y_3$, while negatively correlated with $y_2$ and $y_4$. And the second principal component is positively correlated with $y_5$ and $y_6$

As shown in Fig. 2, the six radar performance indicators are divided into two groups. The first principal component is positively correlated with $y_1$ and $y_3$, while negatively correlated with $y_2$ and $y_4$. And the second principal component is positively correlated with $y_5$ and $y_6$ Therefore, the problem of modeling and analyzing each indicator is successfully transformed into the modeling analysis of these two principal component. According to their reflections of radar performances, we name them as the tracking capability $z_{1}$ and the anti-jamming capability $z_{2}$, respectively. The expressions of $z_{1}$ and $z_{2}$ are as follows:

Tracking capability ($z_{1}$):

$$\begin{aligned} z_{1}=0.363+0.743y_{1}-0.036y_{2}+0.078y_{3} -0.179y_{4}-0.012y_{5}-0.045y_{6}, \end{aligned}$$

(3)

Anti-jamming capability ($z_{2}$):

$$\begin{aligned} z_{2}=-0.799+0.012y_{1}-0.004y_{2}+0.003y_{3} -0.157y_{4}-0.060y_{5}-0.156y_{6}, \end{aligned}$$

(4)

where $y_1$, $y_2$, $y_3$, $y_4$, $y_5$ and $y_6$ denote the six radar performance indicators as in Table 4.

5 Evaluation of anti-jamming effectiveness

The evaluation of radar anti-jamming capability has always been a concern for the researchers of radar anti-jamming technologies and for manufacturers of jamming equipments. The early evaluation factor method considers only few factors, and the evaluation results are generally not normalized. Obviously, this is not convenient for qualitative evaluation. In recent years, the evaluation method has developed into the fuzzy lake comprehensive evaluation method, and then the machine learning evaluation method. It has been confirmed that the evaluation based on the machine learning for predicting radar performance has certain robustness and generalization capability. By this way, we are able to obtain the influence law of each factor on radar performance. And then the evaluation of radar anti-jamming capability under specific factors can be realized.

According to the quantitative characteristics of radar anti-jamming performance evaluation, it can be regarded as a regression problem. Therefore, here we select four popular machine learning algorithms, CART, random forest, Xgboost and SVM, to solve the evaluation of radar anti-jamming performance. Considering the correctness of each algorithm and their feasibilities in radar anti-jamming effectiveness evaluation, we also compare the quality of these four machine learning algorithms. In addition, we adopt the cross-validation method in data analysis for avoiding over-fitting. Here we declare that, All specific data in this simulation are derived from the observations in the real radar countermeasure experiments of State Key Laboratory of Complex Electromagnetic Environment Effects on Electronics and Information System (CEMEE) of China. Next we are going to use the experiment data under two comprehensive indicators $(z_{1},\ z_{2})$, to evaluate the generalization ability of the model.

The values of $z_{1}$ (the tracking capability) and $z_{2}$ (the anti-interference capability) under the eighteen experiments are shown in Table 6. The smaller the values of $z_{1}$ and $z_{2}$, the better the radar performances.

Table 6 Results under comprehensive indicators

The model evaluation is divided into two steps. The first one is to use the overall sample to train the model, and to predict the value of the target variable for

the same overall sample. That is, the training set and the test set are the same. This approach helps to observe the effect of the model, however there may be a risk of over-fitting. Then according to the formulas (1) and (2), the overall RMSE can be obtained, as shown in Table 7. And the second step is to evaluate the generalization ability of each model, and then to conduct cross-validation. The training error and the verification error under different models are shown in Table 8.

Table 7 The fitting effect of the four models

Table 8 Generalization capability of four models

The above results show that Xgboost has the best fitting effect, but the generalization ability is slightly inferior to other models, which is obviously not the best choice. According to further in-depth comparison, we find that the average verification error of random forest is the smallest, that is, the generalization ability is the best. Therefore, random forest can be chosen as the final model for predictive analysis. The good generalization ability fully demonstrates the evaluation ability of our model. This is because the two principal components, the tracking capability $z_1$ and the anti-jamming capability $z_2$ obtained by the PCA in Sect. 3, accurately measure the radar anti-jamming performance. These results and methods complement each other and explain the superiority of each other.

6 Additional verification of models

In order to further verify the accuracy of the obtained model, we supplement some subsequent experiments, and compare and analyse the actual experimental results with the prediction results of the above models.

6.1 Construction of supplementary experiments

Decision trees, random forest, and Xgboost all provide seven important variables that affect anti-jamming performance. They are CFAR unilateral reference unit number, burn through, front edge tracking, $SIR_{1}$, $SIR_{2}$, $SIR_{3}$ and the number of false targets, which are, respectively, denoted as $V_{1},\ V_{2},\ V_{3},\ V_{4},\ V_{5},\ V_{6}$ and $V_{7}$ for convenience.

In order to increase the supplementary experiment, the design can be constructed by using the above variables (the factors that do not contain 4 levels are treated as pseudo-levels) with the values of other variables chosen arbitrarily. The construction and the corresponding results of the supplementary experiments are shown in Table 9.

Table 9 Supplementary experiments

6.2 Comparisons of actuality and prediction

Since that there are two response variables here, $z_1$ and $z_2$, the following research is correspondingly divided into two parts, ’Comparison on $z_1$’ and ’Comparison on $z_2$’. In each part, we make use of existing variables ($V_1$ to $V_7$) to construct several new variables before performing the numerical fitting, in order to reduce the number of variables in the model and avoid making the model too complex.

• (i) Comparison on $z_{1}$:

Considering that $V_2$ and $V_3$ are qualitative variables with two values, we firstly construct a new variable $V_{8}$ by $V_{2}$ and $V_{3}$ as follows to classify the data.

$$\begin{aligned} V_{8}= \left\{ \begin{array}{lr} 1,\ \text{ if } \ V_{2}=-1\ \text{ and } \ V_{3}=-1; &{} \\ 2,\ \text{ if } \ V_{2}=-1\ \text{ and } \ V_{3}=1; &{} \\ 3,\ \text{ if } \ V_{2}=1\ \text{ and } \ V_{3}=-1; &{} \\ 4,\ { \text{ if } \ V_{2}=1\ \text{ and } \ V_{3}=1.} &{} \end{array} \right. \end{aligned}$$

(5)

Secondly, within each block, it is noted that the variables $V_{4},\ V_{5}$ and $V_{6}$ ($SIR_1$, $SIR_2$ and $SIR_3$) have some obvious properties. We summarize these properties and construct another new variable $V_{9}$ with $V_{4},\ V_{5},\ V_{6}$ and $V_{8}$ as follows,

$$\begin{aligned} V_{9}= \left\{ \begin{array}{cl} 2\mid {V_{4}}-40\mid +3\mid {V_{5}}-30\mid + \mid {V_{6}}-30\mid ,\ {} &{} \text{ if } \ V_{8}=1; \\ 2\mid {V_{4}}-35\mid +3\mid {V_{5}}-40\mid +\mid {V_{6}}-30\mid ,\ {} &{} \text{ if } \ V_{8}=2; \\ 30,\ &{} \text{ if } \ V_{8}=3; \\ 2\mid {V_{4}}-30\mid +3\mid {V_{5}}-25\mid +\mid {V_{6}}-24\mid ,\ &{} \text{ if } \ V_{8}=4. \end{array} \right. \end{aligned}$$

(6)

In this way, we easily get

$$\begin{aligned} \left\{ \begin{array}{lr} z_{1}>1,\ \text{ if } \ V_{9}\le 20; &{} \\ z_{1}\le 1,\ \text{ if } \ V_{9}>20. &{} \end{array} \right. \end{aligned}$$

(7)

Therefore, $z_{1}$ in Table 9 can be divided into two types according to $z_{1}>1$ and $z_{1}\le 1$. Consequently, we fit $V_{7}$ and $z_{1}$ with the model $z_{1}=b_{0}+b_{1}\times V_{7}$ by different classifications of $z_{1}$, and fit $V_{4}$, $V_{6}$ and $z_{1}$ with the model $z_{1}=b_{2}+b_{3}\times V_{4}+b_{4}\times V_{6}$. The resulting linear regression equations are as follows

$$\begin{aligned} \left\{ \begin{array}{lr} z_{1}=1.444918+0.000229\times V_{7},\ \text{ if } \ z_{1}>1; &{} \\ z_{1}=-2.3437+0.035865\times V_{4}+0.050194\times V_{6},\ \text{ if } \ z_{1}\le 1. &{} \end{array} \right. \end{aligned}$$

(8)

At a 10$\%$ significance level, the overall and coefficients are both significant. The conclusion unifies the experimental results and the prediction results, which shows the resulted model has excellent prediction effect.

• (ii) Comparison on$z_{2}$:

Similar to the above, we firstly construct a new variable $V_{10}$ with $V_{2}$ and $V_{3}$ to classify the data,

$$\begin{aligned} V_{10}= \left\{ \begin{array}{lr} 1,\ \text{ if } \ V_{2}=-1\ \text{ and } \ V_{3}=1; &{} \\ 2,\ \text{ if } \ V_{2}=-1\ \text{ and } \ V_{3}=-1; &{} \\ 3,\ \text{ if } \ V_{2}=1\ \text{ and } \ V_{3}=1; &{} \\ 4,\ \text{ if } \ V_{2}=1\ \text{ and } \ V_{3}=-1. &{} \end{array} \right. \end{aligned}$$

(9)

Secondly another new variable $V_{11}$ is constructed with $V_{7}$ (the number of false targets) and $V_{10}$ as follows to increase the difference between blocks,

$$\begin{aligned} V_{11}=V_{7}\times V_{10}. \end{aligned}$$

(10)

Finally we fit the data with the model $z_{2}=b_{5}+b_{6}\times V_{1}+b_{7}\times V_{3}+b_{8}\times V_{10}$ to get the linear regression equation

$$\begin{aligned} z_{2}=-0.317775-0.011842\times V_{1} +0.216881\times V_{3}+0.049923\times V_{10}. \end{aligned}$$

(11)

At a 5$\%$ significance level, the overall and coefficients are both significant. The conclusion also unifies the above experimental results and the prediction results, which shows the resulted model has excellent prediction effect.

In addition, we provide the comparison between the true values and the predictive values of $z_{1}$ and $z_{2}$ for the above supplementary experiments, as showed in Table 10. As can be intuitively seen from the results in the above table, the prediction effect is quite good. This states that our analysis of radar performances and selection of models are reasonable.

Table 10 Comparison between true values and predictive values

7 Conclusion and discussion

This manuscript is a successful case of applying machine learning and statistical methods to solve military problems, and provides a methodology for data analysis of similar practical projects. Through the analysis of radar anti-jamming performance and the research on its effectiveness evaluation method, a bipartite covering array is designed for the experimental construction scheme. We combined the existing indicators and used the principal component analysis method to obtain two comprehensive indicators to better reflect the radar performances. Moreover, four machine learning algorithms are used to greatly weaken the influence of subjective factors in the comprehensive evaluation, which has good practical application value. The final additional experiments verify the effectiveness of the final model. This provides a valuable reference for the research of radar anti-jamming.

However, there may be room for improvement. For example, the manuscript adopts four kinds of machine learning algorithms to evaluate the effectiveness of radar anti-jamming. Interested readers could use other more suitable algorithms, which may obtain better results. Moreover, in Sect. 6.2, there may be some better methods for the comparisons of actuality and prediction in supplementary experiments. This is also a good topic worth discussing in the future. Whereas, the research methods in this paper set up a framework to solve the analysis problem of real data for common readers. We believe that with the development of theory and technology, the radar anti-jamming effectiveness evaluation method can be improved, thereby, increasing the reliability of radar anti-jamming.