Gesture recognition method based on a single-channel sEMG envelope signal

Abstract

In the past, investigators tend to use multi-channel surface electromyography (sEMG) signal acquisition devices to improve the recognition accuracy for the study of gesture recognition systems based on sEMG. The disadvantages of the method are the increased complexity and the problems such as signal crosstalk. This paper explores a gesture recognition method based on a single-channel sEMG envelope signal feature in the time domain. First, we get the sEMG envelope signal by using a preprocessing circuit. Then, we use the improved method of valid activity segment extraction to find every valid activity segment and extract 15 features from every valid activity segment. Next, we calculate the absolute value of the correlation coefficient between each of the features and target values. After removing the feature with the smaller correlation coefficient, we reserve the 14 features. By the PCA dimensionality reduction algorithm, we transform the 14-dimensional feature into 2-dimensional feature space. Finally, we use the improved KNN algorithm and the soft margin SVM algorithm to complete the classification of five types of gestures. We obtain the gesture recognition rates of 75.8 and 79.4% by using the improved KNN algorithm and the soft margin SVM algorithm.

1 Introduction

Surface electromyography (sEMG) is the temporal and spatial superposition of faint bioelectrical signals generated by the muscle nerve cells during muscle contraction [1]. It is collected and recorded through the skin surface electrodes. Compared to conventional EMG signal acquisition that requires inserting a needle electrode into muscle tissue, the sEMG signal has the advantages of being noninvasive and providing the convenience of collection. The sEMG signal from the arm has been used in human-computer interaction, rehabilitation training, and artificial prosthesis [2,3,4,5]. Because it can indirectly reflect the gestures of the person, it has extensive application and research value [6].

A Canadian company (Thalmic Labs) develops the world’s first Myo armband based on sEMG signal in 2013, which is used in the contactless human-computer interactive device [7,8,9,10]. In 2015, Harbin Institute of Technology in China develops an imitation humanoid “dexterous hand” for the national lunar exploration project. One of the key technologies is gesture recognition based on sEMG [11, 12]. Previously, in the study of gesture recognition based on sEMG signal, investigators tend to increase the number of acquisition channels to improve the recognition rate [2]. Not only does this increase costs, but it also increases the complexity of the acquisition process and the number of data processing. Especially in application for commerce, multi-channel has a high requirement for hardware. So it leads to a high cost and is hard to popularize. Besides, multi-channel is easy to cause signal crosstalk problems [13, 14]. Moreover, previous investigators just study the raw sEMG property. Motivated by the aforementioned discussions, we investigate a new method.

In this paper, we develop and design the sEMG acquisition system and study the envelope property of the signals obtained. We put forward an improved method for extracting valid activity segment, and the method can be used to find with precision the starting point and ending point for each of the valid activity segments, which greatly reduce the noise interference and increase the precision of extraction. We extract 15 common features which include all sEMG envelop properties and analyze the correlation between every feature and five types of gestures. We find that the slope sign change (SSC) is unsuitable for the studying of sEMG envelope signal. However, according to previous research, SSC is an important feature to study of the raw sEMG signal. We also find that sEMG envelop signal can greatly reflect sEMG amplitude property but ignore its frequency property. These are the major differences between raw sEMG signal and sEMG envelop signal. Therefore, we only keep the other 14 features. However, there are still too many data for processing, which increase the burden of classifier. Furthermore, we use the PCA linear dimension reduction algorithm to transform 14-dimensional features into 2-dimensional. Finally, we apply three pattern recognition algorithms to classify the five types of gestures and find that the soft margin SVM is the best classifier with a recognition rate of 79.4%, while the improved KNN algorithm gives a gesture recognition rate of 75.8%. It turns out sEMG envelope property is an important way for the study of sEMG.

2 sEMG signal acquisition

2.1 Introduction of process of this experiment

We design the acquisition system to get the sEMG envelope signal. The sEMG envelope signal is the DC signal after the process of amplification, rectification, integration, and amplification again for the raw sEMG signal by the preprocessing circuit. Then, we use an improved method for extracting valid activity segments. After that, we extract 15 features and analyze the correlation. Next, we reduce the feature dimension by PCA dimension reduction. Finally, we compare the three pattern recognition algorithms. The whole process of the experiment is shown in Fig. 1.

Fig. 1

The whole process of this experiment

2.2 Experimental methods

In this paper, five kinds of gestures are studied: bend the wrist up while flat, bend the wrist down while flat, bend the wrist up while in shake hand position, bend the wrist down while in shake hand position, and make a fist (as shown in Fig. 2). AgCl electrodes are placed in the position of the ulnar flexor carpi ulnaris of the right forearm (as shown in Fig. 3). The subjects are four college students, aged 23–25 years, two men and two women, all right-handed. Before the experiment, the skin of each subject’s right forearm is cleaned with soap [15]. The sampling frequency is set at 2000 Hz. Every gesture is repeated 60 times, and 300 samples are collected from each subject (as shown in Table 1). The typical waveform which corresponded with each of the five types of gestures is shown in Fig. 4.

Fig. 2

Five kinds of gestures in this work

Fig. 3

Electrode’s placement

Table 1 Classification distribution of the sample in training and test sample dataset

Fig. 4

The typical waveform for five types of gestures

3 The experimental algorithm

3.1 An improved method for extracting valid activity segment based on the moving square method

The purpose of valid activity segment extraction is to efficiently identify every valid activity segment to reduce the amount of processing data, to determine the starting and ending sampling points of every activity waveform, to distinguish from noise, and to lower the effect from noise interference. In this paper, an improved method is proposed based on the moving square method, as stated below:

① Assuming x _ki represents the ith sampling point in kth activity segment, and within 32 ms after x _ki there are 64 sampling points at the sampling frequency of f = 2000 Hz, then the average short time energy within 32 ms (window width, w = 64) is:

$$ {E}_k=\frac{1}{64}\sum \limits_{i=0}^{63}{x}_{ki}^2 $$

(1)

② In accord with our experience, we selected the appropriate thresholds as TH₁ and TH₂. TH₁ is the starting point threshold, and TH₂ is the ending point threshold. When satisfying E _k  > TH₁, the first sampling point is recorded x _ks . After x _ks is recorded, the first sampling point is recorded x _ke  (e > s) when satisfying E _k  < TH₂.

③ Activity duration is recorded T, then:

$$ T=\left(e-s\right)\times \frac{1}{f} $$

(2)

④ From sampling point x _ks to x _ke , the average energy is recorded ${\overline{E}}_k$, then

$$ {\overline{E}}_k=\frac{1}{e-s}\sum \limits_{i=s}^e{x}_{ki}^2 $$

(3)

⑤ In accord with our experience, choosing TH₃, TH₄ (TH₃ < TH₄), and TH₅ as the third, fourth, and fifth thresholds, respectively, when satisfying ${\mathrm{TH}}_4>\mathrm{T}>{\mathrm{TH}}_3\kern0.5em \mathrm{and}\kern0.5em {\overline{\mathrm{E}}}_{\mathrm{k}}>{\mathrm{TH}}_5$, the kth activity segment is valid. x _ks is the real starting sampling point, and x _ke is the real ending sampling point during the valid activity segment. Otherwise, the activity segment is regarded as noise and considered invalid.

Figure 5 shows the process of extraction of valid activity segment about the sEMG envelope signal corresponding to the five types of gestures. Figure 5a shows the short time energy (E _k ) distribution of the sEMG envelope signal when the window width w = 64. In order to improve the performance of anti-interference, TH₁ is slightly higher than TH₂. Figure 5b shows the duration (T) of valid activity segment. According to the experiment results, we set TH₃ = 500 ms and TH₄ = 1800 ms. Figure 5c shows the average energy ${\overline{E}}_k$ for five types of gestures. In Fig. 5c, we chose the appropriate threshold as TH₅. Figure 5d shows the extraction of valid activity segment for the sEMG envelope signal.

Fig. 5

The process of extraction of the valid activity segment. a The short-term energy distribution of raw sEMG signal. b Duration (T) of valid activity segment. c The average energy ${\overline{E}}_k$ for five types of gestures. d The extraction of the valid activity segment for sEMG envelope signal. The legends first, second, third, fourth, and fifth represent the five kinds of gestures. Starting point represents the first sampling point in the valid activity segment. Ending point represents the last sampling point in the valid activity segment

3.2 Feature extraction

The sEMG signal is a non-stationary random signal, but it can be regarded as a stationary random signal in short periods of time [2]. At present, the common feature extraction methods applied to raw sEMG signal are feature extraction methods based on time-domain statistical features, frequency-domain statistical features, time-frequency domain statistical features, and parametric model [11]. Time-domain features extract time structures in the sEMG signal [16,17,18,19]. Time-domain statistical features can intuitively reflect the amplitude characteristics of the sEMG signal, and its algorithm is easy to implement. It has the advantage of real-time and speed. In frequency-domain statistical features, investigators first need to take the Fourier transform of the time-domain sEMG signal and then extract power spectrum or spectral characteristics of the sEMG signal. But the method creates difficultly and complexity. In this experiment, because the sEMG envelope signal loses its frequency property, we only extract the features from time-domain and parametric model. In the experiment, we find that the trend lines of the envelopes corresponding to the five gestures are obviously different. Therefore, we extract 15 features of the sEMG envelope signal in the time-domain and the parametric model.

3.2.1 Feature extraction based on time-domain statistical features

Time-domain statistical characteristics mainly extracted the sEMG signal’s duration T, integral IEMG, mean absolute value (MAV), variance (VAR), standard deviation (s), average energy (E), maximum (MAX), slope sign change (SSC), skewness (Sk), and kurtosis (Ku). Assuming that {x _ki | i = 1, 2……n} represent all the sampling points in the kth valid activity segment, where n represents the total number of sampling points in the valid activity segment, x_k1 represents the starting sampling point of this valid activity segment, and x _kn represents the ending sampling point of this valid activity segment. In time-domain statistical features, the formulas are as follows:

① Time of duration for each valid activity segment T:

$$ T=\frac{n-1}{f} $$

(4)

② Integral, IEMG:

$$ \mathrm{IEMG}=\sum \limits_{i=1}^n\left|{x}_{ki}\right| $$

(5)

③ Mean absolute value, MAV:

$$ \mathrm{MAV}=\frac{1}{n-1}\sum \limits_{i=1}^n\mid {x}_{ki}\mid $$

(6)

④ Variance, VAR:

$$ \mathrm{VAR}=\frac{1}{n-1}\sum \limits_{i=1}^n{\left({x}_{ki}-\mu \right)}^2 $$

(7)

⑤ Standard deviation, s:

$$ s=\sqrt{\frac{1}{n-1}\sum \limits_{i=1}^n{\left({x}_{ki}-\mu \right)}^2} $$

(8)

⑥ Average energy, E:

$$ E=\frac{1}{n-1}\sum \limits_{i=1}^n{x}_{ki}^2 $$

(9)

⑦ Maximum, MAX:

$$ \operatorname{MAX}=\max \left({x}_{k1},{x}_{k2},{x}_{k3}\cdots {x}_{kn}\right) $$

(10)

⑧ Slope sign change, SSC:

$$ \mathrm{SSC}=\frac{1}{n-2}\sum \limits_{i=2}^{n-1}f(i) $$

(11)

$$ f(i)=\left\{\begin{array}{c}0,\left({x}_{ki}-{x}_{k\left(i-1\right)}\right)\left({x}_{k\left(i+1\right)}-{x}_{ki}\right)>0\\ {}1,\left({x}_{ki}-{x}_{k\left(i-1\right)}\right)\left({x}_{k\left(i+1\right)}-{x}_{ki}\right)<0\end{array}\right. $$

(12)

⑨ Skewness as the index of data symmetry, Sk:

$$ \mathrm{Sk}=\frac{n}{\left(n-1\right)\left(n-2\right){s}^3}\sum \limits_{i=1}^n{\left({x}_{ki}-\mu \right)}^3 $$

(13)

⑩ Kurtosis (Ku) describes the steep degree of data state, the formula is as follows:

$$ \mathrm{Ku}=\frac{n\left(n+1\right)}{\left(n-1\right)\left(n-2\right)\left(n-3\right){s}^4}\sum \limits_{i=1}^n{\left({x}_{ki}-\mu \right)}^4-\frac{3{\left(n-1\right)}^2}{\left(n-2\right)\left(n-3\right)} $$

(14)

where f is the frequency of sampling, s is the standard deviation of this valid activity segment, and μ is the mean.

3.2.2 Feature extraction based on parameter model

Although the sEMG signal is a nonstationary random signal, it has good stability in short time intervals and can be regarded as a piecewise stationary quasi stationary signal [11, 12]. Therefore, a mathematical model can be established for this property. According to the AR model, the jth sampling point in the kth valid activity segment is

$$ {x}_{kj}=\sum \limits_{i=1}^p{\alpha}_i{x}_{k\left(j-i\right)}+\beta (j) $$

(15)

where α _i is the model coefficient, β(j) is the white noise, p is the order of the model, and p = 4 is appropriate according to the experimental results.

In this experiment, the AR model intercept (IT) and the fourth-order (p = 4) model coefficients α₁、α₂, α₃, and α₄ of the AR model are chosen as the features. Finally, we extract 15 features in total.

3.3 The correlation analysis between features and five types of gestures

We calculate absolute value of the correlation coefficient between each of the 15 features and five types of gestures, respectively [16]. Calculating it as follows:

$$ r\left(t,{y}_j\right)=\frac{\sum \limits_{i=1}^n\left({t}_i-\overline{t}\right)\left({y}_{ij}-{\overline{y}}_j\right)}{\sqrt{\left[\sum \limits_{i=1}^n{\left({t}_i-\overline{t}\right)}^2\sum \limits_{i=1}^n{\left({y}_{ij}-{\overline{y}}_j\right)}^2\right]}} $$

(16)

where n is the number of all samples, y _ij represents the jth (1 ≤ j ≤ 15) feature of the ith sample, ${\overline{y}}_j$ represents the mean of the jth feature in all samples, t represents the target value, and $\overline{t}$ is the mean of the target value. Define the target value of the first gesture is 1, the target value of the second gesture is 2, the target value of the third gesture is 3, the target value of the fourth gesture is 4, and the target value of the fifth gesture is 5. Each final correlation coefficient is the mean absolute value of all experimenters’ correlation coefficient. Then, we illustrate with a bar graph (as shown in Fig. 6). In this paper, we obtain the sEMG envelope signal, and this is the DC signal after the raw sEMG signal is amplified, rectified, integrated, and amplified again. As a result, the sEMG envelope signal is different from the raw sEMG signal. So, we make the correlation analysis about the suitability of the classification for the five types of gestures. It can be seen from Fig. 6 that the absolute value of the correlation coefficient between SSC and the target value is the smallest and is approximately equal to zero. It is irrelevant. So SSC almost has no effect for classification of five types of gestures. We remove the SSC from all features. Lastly, we get the 14 features.

Fig. 6

The absolute value of correlation coefficient

3.4 PCA linear dimensionality reduction algorithm

High-dimensional spatial data cannot be visually expressed, so we cannot choose the appropriate classifier subjectively. In addition, the higher the dimension means the greater the amount of data, the more complex the data for handling, and the higher the hardware requirements. In order to observe distribution of the sample data and simplify the task of classifier, we use the dimensionality reduction algorithm. PCA (principal component analysis) is the most commonly used linear dimensionality reduction algorithm. Through a kind of linear projection, this algorithm can make the data be mapped to low-dimensional space from high-dimensional space. We assume that n dimensional vector w is the mapping vector, then make the mapping data variance maximum:

$$ \underset{\boldsymbol{W}}{\max}\frac{1}{m-1}\sum \limits_{i=1}^m{\left({\boldsymbol{w}}^T\left({\boldsymbol{X}}_i-\overline{\boldsymbol{X}}\right)\right)}^2 $$

(17)

where m is the number of all samples, X _i is the eigenvector which consists of all the features of every sample, and $\overline{\boldsymbol{X}}$ is the average vector of all the features of all samples. W is a matrix containing all the mapping vectors as column vectors. After transformation, the following optimal projection matrix can be obtained:

$$ \left\{\begin{array}{c}\underset{\boldsymbol{W}}{\min}\mathrm{tr}\left({\boldsymbol{W}}^T\boldsymbol{AW}\right),\kern0.5em S.T.\kern0.5em {\boldsymbol{W}}^T\boldsymbol{W}=\boldsymbol{I}\\ {}\boldsymbol{A}=\frac{1}{m-1}\sum \limits_{i=1}^m\left({\boldsymbol{X}}_i-\overline{\boldsymbol{X}}\right){\left({\boldsymbol{X}}_i-\overline{\boldsymbol{X}}\right)}^T\end{array}\kern0.75em \right. $$

(18)

where tr is the trace of the matrix, I is the unit matrix, and A is the covariance matrix of all the sample data. The optimal projection vector W^′ is composed of eigenvectors that are column vectors which are determined by the front k largest eigenvalues of the sample data covariance matrix A, so that the transformed dimensionality reduction matrix can be obtained as follows:

$$ {\boldsymbol{Y}}_{m\times k}={\boldsymbol{X}}_{m\times n}{\boldsymbol{W}}_{n\times k}^{\prime } $$

(19)

where n (n > k) is the number of all original features. Finally, we get the k-dimensional features. Experiments show that when feature matrix is reduced to 2-dimensional (k = 2), the proximate recognition effect as the original data can be achieved with the least amount of data. After the 14-dimensional feature matrix is reduced to 2-dimensional feature matrix, the data distribution of five types of gestures is shown in Fig. 7. In Fig. 7, the horizontal axis x represents the first dimensional value and the vertical axis y represents the second dimensional value in the 2-dimensional feature matrix. The legends first, second, third, fourth, and fifth represent the five kinds of gestures. It can be seen from Fig. 7 that every type within the five types of gestures is grouped together. According to the figure, we can select an optimal pattern recognition algorithm to achieve the optimal classification effect.

Fig. 7

The distribution of sample data point for five types of gestures. The horizontal axis x represents the first dimensional value, and the vertical axis y represents the second dimensional value in the 2-dimensional feature matrix. The legends first, second, third, fourth, and fifth represent the five kinds of gestures

3.5 Pattern recognition algorithm

3.5.1 KNN algorithm based on weight optimization

In the traditional k-nearest neighbor (KNN) algorithm, in order to judge the category of unknown instances, all known instances are taken as references. The distance between each unknown instance and all known instances is calculated, and the k-nearest neighbor in known instance sets are selected. We obey the rules that the minority obeys the majority. And lastly, we make the unknown instances be classified into the category of the largest number of k-nearest neighbor samples. This traditional algorithm has obvious shortcomings. When the sample distribution is unbalanced, such as one of the samples is too large or dominant, the new unknown sample can easily be classified as the dominant or scattered category. In view of such a problem, this paper proposes an improved KNN algorithm, taking the reciprocal of distance as the weight. Assuming that the known set of instances of known categories is Ω = {X₁, X₂, X₃⋯⋯X _n } and the set of instances of unknown categories to be classified is Ψ = {Y₁, Y₂, Y₃⋯⋯Y _m }. The process is as follows:

① We calculate all distance d _ij between each unknown instance Y _j to be classified and all X _i in Ω, respectively.

$$ {d}_{ij}=\parallel {\boldsymbol{X}}_i-{\boldsymbol{Y}}_j\parallel $$

(20)

② In descending order of d _ij , we select k-nearest neighbor known instances from Ω.

③ We calculate the k reciprocal of distances d _ij , respectively, and make the reciprocal $\frac{1}{d_{ij}}$ as the weight.

④ In these k known instances, we calculate the number of instances of each category as α and sum of the weights of each category as $\beta =\sum \limits_{i=1}^{\alpha}\frac{1}{d_{ij}}$.

⑤ Next, we calculate the value of α × β as the weight of this category and name the value of α × β as θ.

⑥ Lastly, we classify the unknown instance into the category that has the highest weight θ.

3.5.2 Soft margin support vector machine algorithm

Support vector machine (SVM) is a supervised algorithm [20]. Its principle is to maximize the margin between a data set and the hyper plane separating two data sets. The soft margin SVM algorithm is to solve the linear nonseparable problem. Its idea is to find an n − 1 dimensional hyperplane in n-dimensional space as a decision function and divide the data into two parts, each of which belongs to a category. Its model is as follows:

$$ \underset{\boldsymbol{w}}{\min}\frac{\parallel \boldsymbol{w}{\parallel}^2}{2}+C\sum \limits_{i=1}^m{\xi}_i $$

(21)

$$ \mathrm{S}.\mathrm{T}.\kern0.75em {y}_i\left({\boldsymbol{w}}^{\boldsymbol{T}}{\boldsymbol{x}}_i+b\right)\ge 1-{\xi}_i,\kern0.5em {\xi}_i\ge 0,\kern0.5em i=1,2\dots m $$

where w is the vector representing adaptive model parameters, C is the weight of the outlier, that is, the penalty factor. The larger the C means the greater the impact of the outlier for the target. ξ _i is a slack variable, and m is the number of sample data points. y _i is the label associated with a data point x _i , and y _i is − 1 or 1. x _i is the vector representing a data point. b is the intercept of the hyper plane. After computing for formula (21), we get the hyper plane of classification for the two categories as follows:

$$ {\boldsymbol{w}}^{\ast}\boldsymbol{x}+{b}^{\ast }=0 $$

(22)

In addition, it has been successfully applied to classification for multiclass.

3.5.3 Comparison of recognition results

The experiment finds that waveform features from the same gesture differ from one person to another [21], so we only need to train the sample data collected from different people in order to recognize the gestures from different people. Table 2 and Fig. 8 show the average recognition accuracy of the four subjects by three recognition algorithms. In order to verify the effect of PCA algorithm, we also get the recognition accuracy in the 14-dimensional feature space. In Fig. 8, the legends first, second, third, fourth, fifth, and mean, respectively, represent the first gesture, the second gesture, the third gesture, the fourth gesture, the fifth gesture, and mean of five types of gestures. T-KNN represents the traditional KNN algorithm, and I-KNN represents the improved KNN algorithm. S-SVM represents the soft margin SVM algorithm. Figure 8a shows the accuracy in 14-dimensional feature space. Figure 8b shows the accuracy in 2-dimensional feature space. It can be seen that the overall recognition accuracy in 14-dimensional feature space and in 2-dimensional feature space belong to the same order of magnitude, which illustrates the reliability of the PCA dimensionality reduction algorithm in this experiment. It can also be seen from Fig. 8 that the KNN algorithm based on weight optimization is better than the traditional KNN algorithm and that the soft margin algorithm is the best in this experiment.

Table 2 The accuracy comparison of KNN and improved KNN and soft margin SVM for 14-D and 2-D gesture recognition

Fig. 8

Comparison of accuracy of recognition for three algorithms. a Accuracy in 14-dimensional feature space. b Accuracy in 2-dimensional feature space. The legends first, second, third, fourth, fifth, and mean, respectively, represent the first gesture, the second gesture, the third gesture, the fourth gesture, the fifth gesture, and mean of five types of gestures. T-KNN represents the traditional KNN algorithm, and I-KNN represents the improved KNN algorithm. S-SVM represents the soft margin SVM algorithm

4 Conclusions

This paper explores a gesture recognition method based on a single-channel sEMG envelope signal. In the experiment, after the raw sEMG signal passed the preprocessing circuit, we obtain its envelope signal. We just use one channel to extract features from the sEMG envelope signal and get the about 80% average accuracy. It shows that the study of gesture recognition based on the sEMG signal can also be researched from the sEMG envelope signal. The property of sEMG envelope signal contains great information of gesture. If we ignore the problems caused by multi-channel, we will get the higher recognition accuracy by using multi-channel acquisition. So how to deal with the contradiction between these problems caused by multi-channel acquisition and high-accuracy will be our next task in the future. This paper provides a reference value for future research, especially applications for prosthesis and biomimetic robot.