A hybrid approach to inferring the Internet of Things for complex activity recognition

With the rapid development and large-scale uptake of the Internet of Things, smart home is evolving from a vision towards a realistically viable solution for assisted living. Activity recognition is one of the fundamental tasks in order to provide accurate and timely assistance and service. As daily living scenarios are full of similar activities, missing data, and noise, inferring complex activities using knowledge-driven reasoning algorithms suffers from several drawbacks, e.g., real-time raw sensor data segmentation, poor generalization, higher computational complexity, and scalability. To address these problems, this paper proposes a hybrid approach to complex daily activity recognition by merging the first-order logic and probability graphic modeling. Specifically, we develop a novel “Markov logic network” combining data-driven multi-feature and simplified rule-based modeling and inference, thus enabling and supporting the applicability and robustness of daily activity recognition. To evaluate the approach and associated methods, we design a testing scenario with a number of similar activity groups, missing data, or disturbance test datasets in a multi-modeling sensor scene. Initial results show our approach outperforms the traditional approach with a better accuracy in the situations of similar activities with missing data and noise disturbance. Experiments are also conducted to compare the Gibbs sampling and MC-SAT sampling algorithms for Markov logic network, and the results show that the Gibbs is better in our experimental settings.


Introduction
Activity Recognition is a fundamental task for Pervasive Computing, Context-aware Computing, Smart Activity, Healthy Ageing, Wise Information Technology and etc. [1].In the literature of activity recognition, most of the works build activity model based on the temporal spatio characters [2,3], which faces the miscalculation with the similar activities because of the few features.Another research bottleneck is the poor robustness of the activity model which is difficult to handle the data missing, noisy data, habit changing and so on [4,5].In addition, the complex multi-activity are more popular which require an active partition method for the continuous time sequence sensor data [6,7,8].Therefore, designing a comprehensive solution for the daily activity recognition is the main work of this paper.
Recognition of Activity of Daily living (ADL) aims to recognize the residents' action intent or intention (activity), living habits, health status to detect the abnormal behavior and provide the personalized service.There are three types of devices to collect data, vision equipment, wearable devices [9], and ambient sensors.The wearable devices need to be worn all time and charged in time, the wearable devices are excluded in our research which sometimes cause interference.Vision and ambient sensors have own merits and demerits respectively.The vision sensors have been widely adopted directly [10,11] which is obvious to record in video.While, the privacy requirements have been paid attention to gradually, ambient sensors are easier to be accepted by residents.Therefore, the non-vision ambient sensors have become the research hotspot [12].That method also has another advantage that the multi-modeling sensing data has more proofs for the inference to make the more accurate result.Like human beings, feeling of touching, smelling, tasting, hearing and their attributes can be sensed and visible by sensors.For many home activities, ambient sensors like infrared, touch, pressure, light are more intuitive to represent their related features.A typical indoor scene layout of multi-modeling sensors has been shown in Figure 1 (Kitchen), Figure 2 (Living Room), Figure 3 (Water Machine), Figure 4 (Cup Model), Figure 5 (Deployed Cup Model).
Raw data collection is the first step, the second step is establishing the inference model.The usual inference methods adopted can be divided two categories, data-driven and knowledge-driven measures.Data-driven method trains the activity model which represent the implicit general rules by huge numbers of labeled data.Actually, the earliest studies have adopted the data-driven method mostly [13,14,15].This method is a typical probability statistical model which has good performance on single resident's single activity recognition.For example, the well known key technologies includes Hidden Markov Models, Dynamic Bayesian Network.It belongs to the series probability graphical model which has the high accuracy and the ability of mining complex correlation for complex activities.But the model usually trained by the experimenters who is not the users.There has an essence problem for the different distribution between test data and train data which means the model is not impracticable and inapplicable to recognize the personalized activities.Another method, knowledge-driven, is an expert system relying on the background and domain knowledge.These knowledge builds mapping from activity entity, appliance entity and context (sensor entity, sensor value entity), temporalspatial traits, hierarchical methodology, rules (formulas) [16,17].This method has shown the good performance to use directly who overcomes "cold start" problem [18].General rules of the model represent most residents' activities habit which have the high reusability.While, the preference, details, sufficient description of one activity cannot be taken into full consideration.For better performance, the activity model is dynamic and uncertainty which should be updated with the habits changing.How to match best results?Trying to combine the two method to learn from other's strong points to offset one's weakness which is called hybrid model [19].Gabriele proposed a novel hybrid approach with probabilistic and knowledgebased reasoning, adopted the unsupervised feedback to label the rules which has the good performance in the real-world dataset [20].Shin, Hyo-Sang presented the regular expressions-based string matching algorithm, the pattern model is trained by labeled data based on the experts knowledge base [21].These hybrid methods are good performance for complement and correcting each other.Expert knowledge is more consistent which is utilized at the first step to model the basics formula [22].Data modification is more flexible which revises the formula.In this paper, we adopt one hybrid model, Markov Logic Network (MLN), which use both of the first-order logic (FOL) and statistical probability.Establishing the rules' model by expert knowledge, then giving the satisfiability probability to every rule by learning from series of real raw sensor data sequences.Main advantages are the the strong ability for complex activities, having the good performance to handle the dynamic and the data missing.The details have shown in Section 3.
The main contributions of our paper are as follows: • Improving the Markov Logic Network by combining the simple words which can express the action, entity, time, time period, place, which covers all attributes of sensor data and reduces the decription complexity.• In order to improve the robutness of activity model, we adopt the sub rules to express the activity which just has one evidence for the result.Inference is a combination of the rules which is related to the activity.• Proving the Gibbs sampling algorithm has the better performance than traditional adopted method, MC-SAT algorithm in these complex activities.In this paper, the Section 2 presents the background, Markov Logic Network which merges the first-order logic and Markov Network.A novel method based on Markov Logic Network has been shown in Section 3, includes data expression, entity expression, activity expression with time series and periods and simplifying rules expression.The Section 4 describes the experiment design and settings.The Section 5 shows the results and discussion of different complex activities situations, and contrasts the different algorithms in these special situations.The Section 6 gives the conclusion and perspective.

Graphy Model of First-Order Logic
The first-order logic formula also calls as first order predicate calculus, consists of assertion and quantify.Constant symbols (e.g., time: 20181223, 20190107, etc.) represent the real instance.Variable symbols (e.g., x, y, etc.) are the abstraction of constant symbols which range over the real instance.Predicate symbols (e.g.m UseBootle, UseTeabag) represent the relations and attributes which mapping into a sets of category events, including sensor events, entity events, activity events.In addition, the object has its own attributes like location and usage which is one feature value to be added into layers.Term is any expression of object (constant, variable) (e.g., 20181223, x), like sensor or entity events, with the time character, for example day-hour-minute-second.Atom is a predicate symbol applied to a tuple of terms or one term (e.g., UseTeabag(b)).Formulas are recursively constructed from atom using logical connectives and quantifiers.For example, negation (¬), conjunction (∧), disjunction (∨), implication (→), equivalence (↔) [23].It is convenient to convert formulas to a more regular form, called clausal form which is defined by disjunctive normal form (DNF) [24].Knowledge base is a conjunction of clauses, every clause is a disjunction of literals.
Graph G is composed by nodes and edges, nodes represent the terms, edges represent the relations, clique of G represents first-order logic formulas (atoms).The formula is non-negative and real-valued.According to the graphical methodology, there is an edge between two nodes who has the appear in at least one formula.

Learning Weight of First-Order Logic by Probability
First-order logic formula weights can be learned generatively by maximizing the likelihood of a relational training database.Automatically giving weight by data is much better than manually refine with less work.The learning algorithm is based on convex optimization.Gradient descent algorithm is one of the optimization method for searching in first-order class which use the gradient g, with learning rate η to update the weight ω, the formula has been shown in (1) [25].The formula of g has been shown in (2), where E ω,y is the expectation over the non-evidence atoms.For second-order class, the optimization method aims at searching direction from function as a quadratic surface [26].

A Typical Method based First-Order Logic and Probability (MLN)
Markov Logic Network (MLN) is a combination of first-order logic and Markov Networks [27].Its knowledge model is first-order formulas with weights.Markov network is one of the models for the joint distribution of a set of variables which is an undirected graph G and a set of potential function φ k .Each state of M L,C (all nodes have values) represents a possible world.The conception of MLN, L is a set of pairs (F i , ω i ), with a finite set of constants , the nodes are binary value, 1 if the ground predicate is true, 0 otherwise.In M L,C , each possible grounding of each formula has been represented.The fewer formulas a world violates, the more probable it is.The associated weight is bigger reflecting the constraint is stronger.
The probability distribution over possible world x specified by the ground Markov network M L,C is given by formula (3) [27], where n i (x) is the number of the true groundings of F i in x, x {i} is the state (truth values) of the predicates appearing in F i , and φ i (x {i} ) = e ωi [28].The second right part is the most convenient approach with a mixture of hard and soft constraints.Because for some formulas with 0 potential, which means the probability is 0. (3) Reasoning probabilities for complex relationships is the key step of this paper.Due to the size and complexity of the grounding Markov network, it is infeasible to inference many scenarios [29].MLN has two inference types, one is the most likely state, another is the conditional probabilities.We adopts the second inference to calculate the every ground query atoms conditional probabilities.Gibbs sampling is a typical algorithm of Markov Chain Monte Carlo algorithm.MC-SAT algorithm is a slice sampling Markov Chain Monte Carlo algorithm which combining the satisfiability testing and simulated annealing.Satisfiability solver (WalkSAT) is the efficiently finding isolated modes in the distribution [30].
All relevant atoms have been retrieved by the Markov blanket concept which has been mentioned before.The probability of a ground predicated X l when its Markov blanket B l is in state b l is in (4).F l is the set of ground formulas that X l appears in, and MLN has many outstanding advantages in followings: • Merging multiple expert knowledge even though these formulas are exclusive and inconsistent.• Learning and inferencing by ground atoms, even the not independent and identically distributed atoms.• Reducing the reliance on expert knowledge.

Method
This paper offers a solution method based on MLN to build a recognition model from the buttom up, including the raw data pre-processing, expert system for knowledge rules, rules' weight learning by labeled data, inference.The flow chart of the novel method has been shown in Figure 6.
In Section 1, the processing approach for sensor data has been mentioned.In this section, the hierarchical concept graph will be explained in details in Figure 7.
There are three layers, the first layer is converting the raw sensor data to entity event.Sensors have been attached in physical objects and reflects an attribute of this object [31].Entity event layer is established according to the activity concept.Activity is a series of entity sequences by single or multiple residents.Entity event is one action which has been triggered by a series sensor because the action must have closely connection with one physical entity.

Expression of sensor data
The collecting data of sensors has been stored as the following format.A data record has the sensor name, sensor id, sensor value, sensor triggered time, sensor place.
T ime, P lace, SensorN ame, SensorID, V alue (5) In order to recognize the similar activities, adding the time period feature and time series to the data.Combining the all attributes in one compact expression like: Sensor(ID) (T ime, P lace, T imeP eriod), M eetsCondition ¬Sensor(ID) (T ime, P lace, T imeP eriod), otherwise (6) The specific definition of sensor, place, time period has been shown in table 1, time inlcuding the year, month, day, hour, minute and second.

Simplifying the activity rules by entity expression
The typical entity event is action events which consists of several continuous sensor data, multiple sensor events are triggered in a period of time.The related sensors can be combined together to generate the entity event which can be divided to three categories.
• Motion, Touch, Tilt, Pressure,Light Action is one kind of entity event, which lead to the position change or the relative position change of entity.Using the multiple kinds sensors, including the human-sensor and object-sensor, closing to the object combining with the object self-change is the entity event which triggered by resident.The comprehensive rules of entity event just concern a very short time just several seconds, and requir the small range space avoid the misunderstanding of the multi-resident.

• Pressure
Weight change is also one kind of entity event, which has the significant show for the dirnk, eat and other weight transfer events obviously.

• Gas, Temperature, Humidity
Enviornment change is key of the context awareness which is one kind of entity event, when the enviornment parameter cross the threshold value, the sensing data has been recorded which can provide the assistance for other change situations.Note, time span of entity event is just seconds which can be ignored, that reduce the demand of storage and simplify the inference process with the simple expression of evidence.

Activity expression
Activity events have more attributes than sensor and entity events, which add time period, change the triggered time to begin time and end time which is accurate to express the activities.The time period is assisting to distinguish the similar activities which has the similar entity events series.The evaluation and experiment have been shown in section 4.

Time series expression
The time relationship has Bef ore, Af ter, Equal three kinds.There are some definitions of these relations.
• Bef ore(x, y) ↔ Af ter(y, x) • Bef ore(x, y) ∨ Equal(x, y) ↔ ¬Bef ore(y, x) • Af ter(x, y) ∨ Equal(x, y) ↔ ¬Af ter(y, x) • Equal(x, y) ↔ Equal(y, x) The time relationship between the entity event (entity time) and the activity event (activity begintime, activity endtime).We difine the activity begintime and endtime has the relation as Bef ore(activity begintime, activity endtime).There has the five kinds which are incompatible between each other, Before, After, Between, Begin, End.The specific definition of them has shown in following: • Bef ore(entity time, activity begintime) ↔ Bef ore(entity, activity)

Simplifying rules expression (sub rules)
Reference [32] and [33] have adopted MLN to recognize the activities, but they works mainly for sequence activities without the probability learning which just a knowledge-driven method.Gabriele Civitarese has presented the duration concept by calculating the difference between the beginning time and ending time [20].These papers design the multi restricted conditions in one rules, but the satifiability probability is for the whole rule which let the rejection for the rule just by a small dissatisfaction condition.But the dissatisfaction condition is generated by many situations, including the data missing, dynamic habits, noisy data.In order to improve the flexible of the activity model, the clausal form just includes the two event atoms which is a group sub rules of the traditional rules, one is an entity event, one is an activity event.While the time series of entity events also is one of the key evidence for inference, the time relationship between entity event and activity event is also contained into the clausal.
For example, before we difine the Drink activity rules as U seCup(a) (x, z, p) ∧ DecreaseCup(a) (y, z, p) → Drink (z, p, x, y), now we divide the rule to sub rules as U seCup(a) (x, z, p) → Drink (z, p, x, y) and U seCup(a) (x, z, p) → Drink (z, p, x, y).The two evidenc entity event has been consisted into different rules which can has improved the flexiblity of the inference.

Experiments
We designed 3 similar groups (Drink Tea, Drink Coffee, Have Meal, Do Dishes, Sweep, Wipe) activities in kitchen and living room by a top-down approach in Table 2.In our experiment, two volunteers have living in the enviornment 2 weeks (as the small training data set to improve model), and label all the training data.Then, we have collected another 1 week data to test the model.The following data is based on the test data.From bottom to top, connecting these sensors by Raspberry Pi, Arduino Mega 2560, Arduino Nano, communicating with PC by Serial Port Communication, Bluetooth HC-05, WIFI module, cellular network, storaging the raw data in ".json" file.Segmenting the sensor data by the time-window (width = 10 seconds), then obtaining the entity event which has been stored into the ".db" file.Alchemy 2.0 is one of a engine to inference by MLN which we have used in this experiment.Our work is discussing the performance (average precision rate) of the time period for the similar activities, like Coffee and Drink Tea.Another work is improveing the robutness of the activity model with the missing data and error data.The last work is to compare the Gibbs sampling with MC-SAT, find the best method in the complex activity situations.

Similar activities by multi-attributes
For example, the term definition and first-order logic of DrinkT ea and DrinkCof f ee have been shown in following:
Definition 2 The predicate or atom names begin with uppercase letters (e.g., Drink, Work).
Definition 3 The evidence term and query term need to be noted before the inference.
Definition 4 The rules define by the temporal-spatial characteristic and the basic traits, the sequence has been realized by like Bef ore predicate.
Description 1 In first-order logic, the negation by ¬ symbol, in MLN, that is same the !symbol.
In the experiment, the two activities has more attributes of time series and time period, the accuracy of the similar activities has been improved which shown in the Table 3. Contrasting with the time series and time period rules and traditional rules, the accuracy of the similar activities has a significant improvement.

Missing data and disturbance situations
In this part, simplifying the rules by dividing the rules to the common rules and special rules two part.Because of the vital role of the special rules, initial weight value is bigger than common rules, the connection weight value between common and special rules is same with the common rules.We have tested the missing data for different activities, the accuracy has a significant improvement which has been in Table 4.For example, the simplifying rules of DrinkT ea and DrinkCof f ee has been shown in following:

Comprision the Gibbs sampling with MC-SAT algorithms
In order to infer the similar activities we have contrasted the two methods to find the applicable one.In the similar triggerd activities situation, the traditional MCMC-Gibbs sampling has the good performance to recognize the two activities simultaneously.In another hand, improving the accurancy of the missing data or disturbance situations, Gibbs has the better performance than MC-SAT.Therefore, for the complex activities, especially for the similar activities and missing data or disturbance situations, the Gibbs sampling algorithm has the better performance.Gibbs sampling algorithm has the strong robustness which can be adopted by the future.

Discussion
We can find out from the above experiments, as the two vital features of activity, time series and time periods express the different time dimensions by operations for the triggered time of entity events.We can easily find the simplifying rules has extrmely good performance of inference by the missing data of one activity, while, it has the fluctuations in performance, even has the degradation in the disturbing situation.By comprehensive consideration, simplifying rules has better performance than the original rules.
We can find the Gibbs sampling algorithm has the better performance than MC-SAT.Because the two similar activities has the many similar rules from entity events, always adopting the MC-SAT to handling the independence situations which means the two interleaved activities are independent.That generates the conflicting results for the complex activities.
The better result relies on the rules and MLN features.Using the time, time series, periods, location attributes to represent activity and the simplifying rules, the complex similar and data missing or disturbance situations' activities can be easily recognition.MLN is the typical hybrid method of the data-driven and knowledgedriven which draws the undirected graph structure, that will make all rules more clearly and make the results more accurately.

Conclusion
In this paper, the main work is realizing the complex activities recognition, including the similar activities, data missing or disturbance situations by Markov Logic Network.In another important work, we have contrasted the typical methods, Gibbs sampling and MC-SAT algorithm, finding the best option in the complex situations.First of all, deploying the multi-modeling sensor in our daily life which can collect many and varied data to improve the inference accuracy.For data processing, the hierarchical methodology is more efficient for MLN.The structure reduces many repetitive works.Especially for the computational of inference iteration, the exponential grow of the inference prefers the small size of data.In addition, MLN can establish the rule formulas for daily activity based on the time, time series, location, period features of entity events.The simplifying rules has been adopted to improve the robustness and handle the uncertain situations.Then, MLN is the combination of the first-order logic and probability, construct Markov network, which has been given weight by the determinative learning method, including the Gibbs sampling and MC-SAT algorithms.Afterall, the complex activities have been recognized.We can find that method has a typical advantage of the soften rules which means even though the personalization habit and data-missing all can be accepted and be solved.
This method is a typical hybrid inference which has the obvious improvement of the complex activities.For inhabitants, the MLN can reflect the preference and habits, except inferencing the activities, mining the relation between these activities is another important work.Also, the next work can go ahead to design the user portrait, especially for the multi-resident scenes.From another angle, recognizing the executor of one activity is a point which has the research value.We can do many extend work which will start and prepare based this work.
Recognition of the daily activity is the fundamental work of many smart devices which is the basics and support for better service for human beings.We will continue to do the related work.

Figure 1 AFigure 2 A
Figure 1 A kitchen layout of multi-modeling sensors

Figure 3 AFigure 4 A
Figure 3 A water machine model of multi-modeling sensors

Figure 5 A
Figure 5 A deployed cup model of multi-modeling sensors

Table 1
Definition of the sensors, place and time period

Table 2
Activity-Entity-Sensor Design for 5 Activities

Table 3
Two similar activities probability

Table 4
The missing data or wrong data disturbance situation's inference precision of different activities Activity Preci.(no)Preci.(withseries and periods)

Table 5
The inference probability of the two similar interleaved activities

Table 6
The missing data or wrong data disturbance situation's inference precision of different