Research on robot boxing movement simulation based on kinect sensor

Abstract

In the process of continuous development, robot technology has been widely used in aerospace, medical, education, and service industries, and the relationship between robots and people is getting closer and closer. The improvement of robot intelligence is a process of continuous learning of external knowledge. Since the introduction of human-computer interaction technology, it has provided more and more technical support for human research robots. This paper firstly takes Tai Chi as an example, analyzes and deconstructs its standard Tai Chi putter movements, and obtains the corresponding movement parameter information. Then, the human body depth image and bone structure are obtained through the kinect sensor, and the human joint position data is recorded, and then the coordinate data is sent to the brain robot, thereby realizing the imitation learning of the robot’s push movements in Tai Chi. The experimental results show that the proposed method can simulate the practice process of inversion of Tai Chi, can well analyze the shortcomings in the practice process, and can then promote the development of the sport.

1 Introduction

Robotics has been widely used in aerospace, medical, education, and service industries in the continuous development process. In this process, the robot is accessible to ordinary people. The robot is the product of computer automatic control, new materials, new energy, artificial intelligence, and other high-tech cross-fusion. At present, the research on robot technology has made great progress, and some artificial intelligence robots have a considerable degree of independent thinking and learning ability. However, for the industrialization of robots, there are still many technical problems that need to be solved, such as how to improve the learning ability of robots and how to better realize the interaction between humans and robots. The convenience and effectiveness of human-computer interaction has a very large space for improvement in robot applications [1]. A good human-computer interaction method will make the robot have high practical value. Human-computer interaction has always been the focus of robotics research. A successful interaction often relies on more factors, such as whether the interaction can be accepted by others; interaction in a restricted environment cannot work well. Therefore, in order to improve the recognition of human-computer interaction by the society, the primary problem to be solved is the adaptation of the robot to the characteristics of the human body. The movements and postures of some humanoid robots are important manifestations of the robot itself, and these also have a profound impact on the user. Therefore, controlling the robot’s limb movement is a more challenging task, and it is also a relatively important topic in robot research.

The specific contributions of this paper include the following:

• (1) Taking Tai Chi as an example, mechanical analysis and deconstruction of standard Tai Chi putter movements.

• (2) Obtain the human body depth image and bone structure through the kinect sensor, record the human joint position data, and send the coordinate conversion to the brain robot to realize the robot’s imitation learning of Tai Chi push movements

• (3) The brain robot attitude conversion method based on kinect human attitude analysis method is designed and implemented.

The rest of this paper is organized as follows. Section 2 discusses Tai Chi pusher action analysis, followed by the methods in Section 3. Robotic Tai Chi pushing action imitation based on somatosensory Human-computer interaction is discussed in Section 4. Section 5 shows the results, and Section 6 is the discussion.

2 Related work

In order to study the human-computer interaction problem of the supple mechanical arm in the inclusive environment, this topic selects the Tai Chi pusher with the supple interaction feature to study the representative of human-computer interaction [2].

2.1 Tai chi pusher action analysis

Tai Chi pusher is also called pusher and beater. It is a freehand exercise against Tai Chi. It is a confrontational part of the upper limbs and the trunk as the attacking part. It uses a technique such as “shed, push, squeeze, press, pick, butterfly, elbow, and lean” to achieve the advantage of borrowing power and causing the other body to lose balance. As a branch of traditional Chinese martial arts, Tai Chi pusher has Chinese traditional characteristics. Tai Chi pusher has been widely loved for its unique techniques and its fitness, entertainment, and enjoyment. It has become one of the most widely spread items of Chinese martial arts [3].

At the beginning of the pacing push, the two sides stand opposite each other, with their knees slightly bent, the right foot in the right front of the body, and the left foot in the left rear. The standing position of the two feet is basically fixed during the pushing process. The two hands are in contact with each other. This is the “right hand” in the Tai Chi term. The right hand trajectory of both sides of the grading and flat pushing process basically constitutes a “circle” on a horizontal plane. (This circle mainly highlights the repeatability and regularity of the movement, not a circular shape in the strict sense.) In the hands of the vertical circle, the right hand trajectory of the two sides roughly constitutes a circle on the vertical plane of the middle position between the two bodies.

Combined with the main and passive characteristics of both sides of the pusher process, the single-step pusher movement can be divided into four stages. The characteristics of each stage are shown in Fig. 1.

Fig. 1

Division of the stage of the pusher

At the beginning of the push, stage 1 is the attacker’s offensive stage, also called the active stage. The pusher uses the right palm and pushes forward, and pushes it toward the chest of the pusher. In this stage, the pusher’s arm actively pushes the other side to be the “press” in the Tai Chi terminology, and the pusher needs to maintain the resistance so as to avoid being pushed against his chest. This follows the opponent’s movement but maintains a certain resistance [4]. The “shed” as the saying goes. Stage 2 is the conversion phase. At the beginning, the pusher is about to be pushed to his chest, and the pusher uses the waist to turn to the back of the body. At the same time, the palm of the pusher is gradually guided from the chest of the pusher, and the pusher is pushed forward by the pusher and pushed to the right side of the chest. The guiding force adopted by the pusher in this process is the rolling back called in the Tai Chi terminology. The pusher’s hand conforms to the “hold,” and the push is the pressing of the Tai Chi term. Phase 3 is the offensive/active phase of the pusher, which is the passive phase of the pusher. At this time, the pusher actively pushes the hand of the pusher forward to the front of the chest of the pusher. In this stage, the pusher uses the “press” force, while the pusher uses the shed to avoid being pushed to his chest. Phase 4 is the conversion phase of the promoter. At the beginning of this phase, the pusher is about to be pushed to the chest of the pusher [5]. At this time, the pusher will guide the hand of the pusher to his right side with the palm of his hand. In stage 4, the pusher adopts the “will” and is pushed by the pusher. Squeeze “strength.” At the end of this phase, both parties enter the next push cycle.

2.2 Biomechanical measurement of tai chi pusher motion

In order to deeply study the supple interaction process of Tai Chi push hands, it is necessary to study the characteristics of human body’s movement, muscle strength, and joints during the process of Tai Chi pushing. This requires a biomechanical measurement of the human body’s Tai Chi pusher. The step-by-step flat pusher is the most basic and most important form of Tai Chi. In this paper, the movement of the singularity and the mechanism of interaction between people and human being sports biomechanical measurement techniques are divided into: kinematic parameter measurement techniques such as measurement time, displacement, and derived kinematic parameters; kinetic parameter measurement techniques such as measurement time, mass, and derived kinetic parameters; biological measurement techniques such as human morphology measurement methods, human body weight measurement methods, and myoelectric measurement methods.

In this paper, the biomechanical measurement of Tai Chi push hands is divided into two parts:

• (1) Kinematics data

The first part is the kinematic data of the human body during the measurement of the Tai Chi pusher. In order to measure the motion information of various parts of the whole body during the process of pushing the hand of the testee, it is necessary to use a motion capture system. Through research, the motion capture system is divided into three types: optical, inertial, and mechanical. After the investigation, the Canadian BIOSYN company’s inertial motion capture system was used to measure the motion information of the human body during the Tai Chi pusher.

• (2) Surface electromyography

At present, most of the sports biomechanics studies surface EMG signals to study muscle activity during human exercise. In the Tai Chi pacing and round single pusher movement, the two sides reached the right hand and pushed the hand. Therefore, this paper uses surface electromyography equipment to measure the surface myoelectric signal of the right forearm of the human body during the pusher process. Through research, the surface electromyography equipment uses the MYO armband launched by the Canadian company ALMIC LABS [6]. The measurement test schematic is shown in the Fig. 2:

Fig. 2

Measurement process

2.3 Measurement data and analysis

The data measured by the FAB motion capture system includes the kinematic data of the whole body and the pressure data of the soles of the feet. As mentioned above, the Tai Chi pacing and flat single pusher movement is 1 cycle after another, and the measurement results are 4 cycles in which the data is relatively stable and the time is relatively advanced. This selection is to avoid the muscle fatigue of the test subject for a long time to ensure the reliability and accuracy of the data. The data of the joint motion data of the 4 cycles of the FAB measured by the FAB, that is, the joint motion angle data and the foot sole pressure measured by the foot insoles, were processed and edited, and the motion curve of the upper limb shoulder is shown in Fig. 3 [7]. All solid lines in the figure are the joint pitch angle, and the uplift is positive. The dotted line is the angle of the joint roll turning left to positive.

Fig. 3

Curve of shoulder movement

The total time of 4 cycles of exercise is 14582.5 ms, which is 13.08 s, and the average push cycle time is 3.27 s. In 1 cycle, the stages of 1, 2, 3, and 4 of the step-by-step flat pusher are 0.5 s, 0.68 s, 0.99 s, and 1.1 s. The curve of the sole is shown in Fig. 4. The left and right foot pressures are expressed as a percentage of the total body weight of the subject.

Fig. 4

Curve of plantar pressure

From the motion capture data and the plantar pressure curve, it can be seen that the three-dimensional kinematic angle and the plantar pressure of each joint show good periodicity. It can be seen from Fig. 6 that in the first three stages of the Tai Chi pusher pacing and flat pusher, the right foot pressure is greater than the left foot pressure. In summary, the upper right limb and the right lower limb of the whole exercise process have a larger amplitude [8]. The movement of the right lower limb and the lower back (pelvis) causes the upper limbs of the human body to move, causing heavy sputum and changes to be the main factors causing changes in plantar pressure.

3 Methods

Robots usually have multiple joints. Different joint variables cause the robot to assume different states. The state of the robot’s end depends on the value of each joint variable from the link chain. How to solve the robot end state according to each joint variable is the concern of the robot positive kinematics.

3.1 Robot position and attitude description

Analyzing the position and attitude of the robot in space can facilitate the establishment of a mathematical model of the robot to represent the state of the robot. Each link of the robot is considered a rigid body in a Cartesian coordinate system. To analyze the pose information of a rigid body in the reference coordinate system, you need to establish the coordinate system of the rigid body. The homogeneous matrix is used to describe the pose relationship between the rigid body coordinate system and the reference coordinate system [9]. The homogeneous matrix T is expressed as follows:

$$ T=\left(\begin{array}{cccc}n& o& a& p\\ {}0& 0& 0& 1\end{array}\right)=\left(\begin{array}{cccc}{n}_x& {o}_x& {a}_x& {p}_x\\ {}{n}_y& {o}_y& {a}_y& {p}_y\\ {}{n}_z& {o}_z& {a}_y& {p}_y\\ {}0& 0& 0& 1\end{array}\right) $$

(1)

In which, p represents the positional information of the origin of the rigid body coordinate system relative to the reference coordinate system and n, o, a, respectively, represent the coordinates of the unit vector on the three axes of the rigid body coordinate system in the reference coordinate system [10]. Together they form the rotation matrix R of the rigid body relative to the reference coordinate system:

$$ R=\left(\begin{array}{ccc}{n}_x& {o}_x& a{}_x\\ {}{n}_y& {o}_y& {a}_y\\ {}{n}_z& {o}_z& {a}_z\end{array}\right) $$

(2)

The rotation matrix R describes the attitude information of the rigid body in the reference coordinate system. The position vector p and the rotation matrix R together form a homogeneous pose matrix T of the rigid body with respect to the reference coordinate system.

3.2 Robot positive kinematics solution

In order to establish a positive kinematics model of the robot, it is necessary to use the pose relationship of the adjacent links of the robot. First, analyze the case where the common reference coordinate system number is 0, the joint number is i, and the next joint number adjacent to the joint is i + 1. The pose matrix of the joint of the i joint in the reference coordinate system and the pose matrix of the joint i + 1 in the coordinate system of the i joint are known [11]. Then, the homogeneous pose matrix of joint i+1 in the common reference coordinate system can be obtained by the following formula:

$$ {{}^0T}_{i+1}={{}^0T}_i\cdot {{}^iT}_{i+1} $$

(3)

Therefore, we can call ⁱT_i + 1 as the homogeneous transformation matrix of the joint coordinate system of the ith joint to the coordinate system of the i + 1th joint. Similarly, the pose matrix T can be understood as a homogeneous transformation matrix from the joint coordinate system to the rigid body coordinate system. For a joint chain of a robot, the reference coordinate system number is 0, and each rigid body number is 1, 2, 3, ⋯n in sequence. The pose transformation matrix of the adjacent two parts is denoted as ⁱT_i + 1 [12]. Therefore, the homogeneous pose matrix ⁰T_n of the robot end relative to the reference coordinate system can be calculated by the following formula:

$$ {{}^0T}_n={{}^0T}_1\cdot {{}^1T}_2\cdot {{}^2T}_3\cdots {{}^{n-1}T}_n $$

(4)

This is the equation of robot kinematics. The idea of solving the positive motion of the robot is to obtain each pose transformation matrix according to each joint variable, and then use the robot kinematics equation to calculate the pose matrix of the robot end in the reference coordinates. The key to the forward kinematics of robots is the establishment of a joint coordinate system. In 1955, Denavit and Hatenberg proposed a matrix method for establishing a coordinate system for each member of the joint chain, referred to as the D-H method. It has been widely used in robot kinematics modeling and has become the standard method for establishing robot linkage coordinate system, as shown in Fig. 5 [13].

Fig. 5

DH method linkage coordinates

Therefore, the four parameters of θ_i, d_i, l_i, α_i describe the transformation process of adjacent link coordinate systems, which are called D-H parameters. The homogeneous transformation matrix of the adjacent link coordinate system is called the D-H transformation matrix, which can be calculated by these four parameters:

$$ {}^{i-1}T=\left(\begin{array}{llll}\cos \theta & -\sin {\theta}_i\cos {\alpha}_i& \sin {\theta}_i\sin {\alpha}_i& {l}_i\cos {\theta}_i\\ {}\sin {\theta}_i& \cos {\theta}_i\cos {\alpha}_i& -\cos {\theta}_i\sin {\alpha}_i& {l}_i\sin {\theta}_i\\ {}0& \sin {\alpha}_i& \cos {\alpha}_i& {d}_i\\ {}0& 0& 0& 1\end{array}\right) $$

(5)

For a rotating joint, the joint angle determines i, and the remaining three parameters are constant, so the homogeneous transformation matrix is determined by the joint angle. Generally, according to the joint link model of the robot, the coordinate system is established by the D-H method, and four parameters are determined, thereby obtaining a homogeneous transformation matrix of each group of adjacent joints [14]. Then the robot kinematics equation can be used to obtain the robot's end pose matrix.

3.3 Brain robot motion control system

Brain is a humanoid robot developed by Aldebaran of France. He is 58 cm tall and has a very good design. He is widely used in robotics labs around the world. As shown in Fig. 6, the brain robot has 25 degrees of freedom and is flexible. The brain robot maintains excellent balance and precision during walking and movement. In addition, he is equipped with a variety of sensors including ultrasound, pressure, infrared, camera, touch, and so on, which can be used not only for scientific research, but also an excellent entertainment and service robot [15].

Fig. 6

Brain robot structure

The brain robot has excellent motion control performance, and the ALMotion module is responsible for the motion control module of the robot. The ALMotion module primarily controls the joint strength, joint position, travel, and Cartesian coordinate space control of the robot. ALMotion works at 50 Hz. Whenever a user calls the API to request a motion, the corresponding “motion task” is created to respond to the user’s request. The coordinate space of the brain robot is defined as the following: the X-axis is facing the front of brain, the Y-axis is from right to left, and the Z-axis is the vertical positive direction.

DCM (Devices Communication Manager) is a software module of the brain robot system. It controls the data communication (main board, sensors, actuators, etc.) of all robotic devices and connects the robot’s superstructure (CPU layer) and the underlying structure (device layer). When any module wants to send a command to the executor, the module first sends the updated command to DCM using the timed command. Because other modules cannot directly change the DCM data stored in ALMemory, they must first send a request to DCM. The DCM then modifies the actuator data stored in ALMemory [10].

The brain robot operating system defines timed command = {target value, time}. Each time a timed command is sent to a child device, DCM stores all commands sent. These commands are then processed periodically. The difference algorithm is used when DCM processes commands. So each time a motion control command is sent to the joint. It is first sent to the DCM, and then the control of the joint is obtained by the difference algorithm. The joint strength value of the brain robot is defined as 0 to 1. When the value is 0.0, the joint is in the free state, and when the value is 1.0, the joint moves to the specified position with the maximum torque.

4 Robotic tai chi pushing action imitation based on somatosensory human-computer interaction

This section builds a robotic attitude simulation system using Microsoft’s somatosensory device kinect and the biped humanoid robot developed by Aldebaran Robotics of France. Analyze the characteristics of kinect bone information and brain robot joint model, and use the analytical method to transform the human body posture to realize the imitation of the Tai Chi pusher action by the brain robot.

4.1 Extracting human skeleton posture data based on kinect

Kinect bone tracking technology can distinguish the various parts of the human skeleton, and at the same time, can process the coordinate information of each joint of the human body by processing the depth data. The robot motion simulation first needs to acquire the human body posture and acquire the three-dimensional information of each joint for coordinate conversion.

Bone is a hard organ that makes up vertebrate. Its function is to exercise, support, and protect the body. The body has 206 bones. Skeletal tracking is the core technology of kinect, as shown in Fig. 7. A small black dot represents a joint name of the human body, and the human skeleton can be represented by 20 joint structures. Kinect can represent the human skeleton structure by capturing 20 joints, and the position of each joint point is represented by the (x, y, z) coordinate values [16]. The kinect sensor is at the origin of the coordinate system, and the entire coordinate space conforms to the right-handed spiral rule. The y coordinate axis is parallel to the direction of the kinect placement, the x coordinate axis is perpendicular to the orientation of the kinect, and the z coordinate axis is parallel to the direction of the kinect sensor [17, 18]. Therefore, when human bones appear in the field of kinect’s field of view, the kinect sensor can calculate the coordinate space of each joint.

Fig. 7

Human skeletal architecture

4.2 Action simulation control process

This paper builds a motion simulation system to complete the brain robot to imitate human motion in real time. Firstly, the human skeleton information is extracted by the kinect sensory sensor, and the human body posture represented by the joint position is converted into the robot posture represented by the joint angle by the analytic method, thereby obtaining the posture sequence. The pose sequence is then processed to eliminate disturbances and limit motion speed. In order to make the robot maintain its stability while imitating the movement, it is necessary to determine the support of the feet and adjust the direction and height of the soles. The stability of the robot is determined according to the COM projection method, and the unstable attitude is corrected to control the joint angle of the brain robot. The general control flow of the system is shown in Fig. 8.

Fig. 8

Brain robot action simulation process

4.3 Imitation experiment of tai chi pusher action

The computer is connected to kinect via USB and uses the depth information it collects to detect the instructor and extract the skeleton information composed of the joint position of the teaching staff. Then, the computer processes the human body posture according to the joint model of brain robot and obtains the control amount of each joint of the robot. The robot’s motion is monitored and controlled by Wi-Fi, so that the brain robot can imitate the human body posture online.

Consider the above proposed pose transformation algorithm on a system consisting of brain robot, kinect, and PC. From the perspective of analytic geometry, the algorithm transforms the target pose of the brain robot according to the human body posture. Due to the various differences between the robot and the human body, the robot’s whole body mimicking movement will inevitably involve the maintenance of the stability of the robot. The above algorithm is verified to avoid robot stability problems. The experiment is divided into two parts: the upper body posture imitation and the leg posture imitation.

The pose in the figure relates to the rotation of all the controlled joints of the robot hand. For each human posture in the figure, the robot can correctly imitate. The experimental results show that the attitude transformation algorithm in this paper can accurately calculate the angle of all controlled joints and control the brain robot to obtain a robot posture similar to the human body, that is, the robot can accurately imitate the human body’s posture, thus verifying the effectiveness of the human body posture simulation algorithm.

5 Results

To verify the performance of the DV-Hop localization algorithm, computer-aided platform was adopted to do simulation test. The computer operating system here is Windows 7, and the simulation software is MATLAB -R2015b. The area range of the wireless sensor network is set to 100*100*100 m, 600 nodes are arranged randomly, which includes anchor nodes and unknown nodes. Each experiment is carried out in a single random manner, and the simulation calculation is carried out 100 times; the final mean value of these data is retained as an evaluation structure. The experimental results adopt the average positioning error and the positioning accuracy index to do comparison test. When the communication radius is R = 20 m and 40 m in the experiment, when the number of anchor nodes changes, the experimental results of the traditional DV-Hop location algorithm and the optimized DV-Hop positioning algorithm are compared. Figure 9 is the positioning error comparison under the number of different anchor nodes. It can be seen from the diagram that when the communication radius is fixed and the number of anchor nodes rises, the average positioning errors of the two algorithms are all in a reduced state. When the number of anchor nodes is around 40, the average positioning error fluctuates, because with the increase of the number of anchor nodes, nodes are in the state of random distribution, there will be a certain state of freedom and there will be instability. However, when the anchor nodes are less, the error of the average location will be larger. This is because when the number of anchor nodes is insufficient, the reference index is too little and the positioning accuracy is poor. It can be seen from the graph that the positioning error of optimized DV-Hop location algorithm is smaller than the traditional algorithm in the number of different anchor nodes.

Fig. 9

Comparison of positioning error under the number of different anchor nodes

When the communication radius is R = 20 m and 40 m, when the number of anchor nodes is changed, the research of relative error of the two algorithms was carried out, and the experimental results are shown in Fig. 10. It can be seen from the figure that, in case of the communication radius is fixed, the performance of the optimized algorithm improves around 10% in the communication radius of 20 m, and in the communication radius of 40 m, the performance improves around 15%, the overall positioning effect increase steadily with the increase of the number of anchor nodes, and the performance of optimized algorithm is better.

Fig. 10

Comparison of positioning error under the number of different anchor nodes

Figure 10 is the comparison of the location error of the unknown nodes in the two algorithms when the communication radius is 30 m and the number of anchor nodes is 30. As you can see from the diagram, the location of the two algorithms is very different, the traditional algorithm fluctuates greatly, and the optimized algorithm has less fluctuation. This is due to the random distribution of the nodes in the traditional algorithm, and under the situation of the information is inaccurate, the error increased. After the optimization algorithm improving the traditional algorithm, the error of the algorithm has been improved significantly.

6 Discussion

Under the great attention and vigorous promotion at home and abroad, robot technology is developing rapidly. Human body mimicry uses natural human-computer interaction to allow robots to quickly learn human action behaviors, which is conducive to robotic operations and better serve humans. This paper first takes Tai Chi as an example and analyzes and deconstructs its standard Tai Chi pusher action, and obtains corresponding action parameterization information. Then, using the Microsoft Somatosensory Sensor Kinect, the humanoid robot brain and the PC, which can simultaneously acquire color images and depth information, a humanoid robot attitude motion simulation system is constructed. Then, based on the kinect human skeleton information and the characteristics of the brain robot joint model, a transformation method based on the analytical method of kinect human posture to brain robot attitude is designed and implemented. Under the condition of not considering the stability of humanoid robot, the imitation experiment of human body training posture is carried out. The experimental results show that the attitude transformation algorithm of this paper can make a similar posture to the human body.