Titan: An Enabling Framework for Activity-Aware "Pervasive Apps" in Opportunistic Personal Area Networks

3.2.1. Services and Service Pool

Titan nodes provide a set of services stored within a service pool. Services can implement signal processing function, classification tasks, sensor readout, or other kinds of processing. Not all Titan nodes implement the same kinds of services. For instance, nodes that do not contain sensors would not offer sensor readout services, while nodes with higher computational capability may offer more computationally intensive services. Services are flashed into the Titan nodes at design time.

Services have a set of input ports, from which they read data, process it, and deliver it to a set of output ports. Connections deliver data from a service output port to a service input port and store the data as packets in FIFO queues.

The services go through the following phases when they are used.

(1) Configuration. At this point, the service manager instantiates a service. To each service, it passes configuration data, which adapts the service to application needs. Configuration data may include, for example, sampling frequency and window size in signal processing services. The service can allocate dynamic memory to store state information.

(2) Runtime. Every time a service receives a packet, a callback function is executed to process the data. Titan provides the service with the state information it has set up during the configuration time. Services are executed in the sequence they receive a packet, and each service runs to completion before the next service can start.

(3) Shutdown. This phase is executed when the service subgraph is terminated on the node. All services have to free the resources they have reserved.

3.2.2. Service Manager

The service manager is the system allowing to reconfigure a Titan node. It instantiates the executed services according to the network manager's requests (see Section 3.3). The service manager is responsible for reorganizing the service subgraph executed on the local sensor node during a reconfiguration.

3.2.3. Dynamic Memory

The dynamic memory module allows services to be instantiated multiple times, and reduces static memory requirements of the implementation. The services can allocate memory in this space for their individual state information. This module is needed as TinyOS does not have an own dynamic memory management.

3.2.4. Packet Memory

The Packet Memory module stores the packets used by the services to communicate with each other. The packets are organized in FIFO queues, from which services can allocate packets before sending them. This data space is shared among the services.

3.2.5. Connections

Packets exchanged between the services carry a timestamp and information of the data length and type they contain. Services reading the packets can decide on what to do with different data types. If unknown data types are received, they may issue an error to the service manager, which may forward it to the network manager to take appropriate actions.

To send a packet from one Titan Node to another, Titan provides a communication service, which can be instantiated on both network nodes to transmit packets over a wireless link protocol as shown in Figure 4. During configuration time, the communication service is told which one of its input ports is connected to which output port of the receiving service on the other node. The two communication services ensure a reliable transmission of the packet data. The communication service is automatically instantiated by the network manager to distribute a service graph over multiple sensor nodes. Thus, for the programmer, there is no distinction when a service graph is mapped on one or more Titan nodes.

The recommended maximum size of a packet for Titan Nodes is 24 bytes, as it can easily be fitted with 5 bytes header into a TinyOS active message. The active message is used to transmit data over wireless links and offers 29 bytes of payload.

3.2.6. Service Manager and Service Discovery

A programmer designs his application by interconnecting services in the form a service graph. Service parameters as well as location constraints can also be defined.

The mapping of a service graph into executed services is controlled by the network manager. In order to support the network manager, the Titan nodes answer to broadcast service discovery messages originating from the network manager by providing a list of matching services available in the service pool and by providing status information about the node.

The network manager then decides on a partitioning of the full service graph realizing the application and provides the service manager of the Titan nodes with the specific subsets of the service graph to instantiate.

When data needs to be exchanged across nodes, communication services (see Section 3.2.5) are automatically inserted. The resulting service subgraphs containing the services to be executed on every sensor node are then send to each participating node's service manager, which takes care of the local instantiation as shown in Figure 4. After the configuration has been issued, the network manager keeps polling the Service managers about their state and changes the network configuration if needed. On node failures, the network manager recomputes a working configuration and updates the subgraphs on individual sensor nodes where changes need to be made, resulting in a dynamic reorganization of the network as a whole.

3.2.7. Synchronization

When sensors are sampled at two sensor nodes and their data is delivered to a third node for processing, the data streams may not be synchronized due to differing processing and communication delays in the data path. As a consequence, a single event measured at the two nodes can be mistaken for two.

If the two sensor nodes are synchronized by a timing synchronization protocol, a timestamp can be added to the data packet when it is measured. The data streams can then be synchronized by matching incoming packets with corresponding timestamps. Timing protocols have been implemented on TinyOS with an accuracy of a few 10  [38, 39].

If the two sensor nodes are not synchronized, the sensor data can be examined as in [40]. The idea is to wait until an event occurs that all sensors can measure, for example, a jump for accelerometers on the body. Subsequent packets reference their timestamp to the last occurrence of the event. This functionality is provided in the Synchronizer service.

3.3.1. Mapping Services to Network Nodes

When the execution of a specific service graph is requested, the network manager first inspects the capabilities of the sensor nodes in the environment by broadcasting a service discovery message containing a list of services to be found. Every node within a certain hop-count responds with the matching services it has in its service pool. From this information, the network manager builds the service directory.

The network manager then optimizes service allocation such that the overall energy consumption is minimized. For this purpose, it uses a metric summing up the main energy consumers, namely wireless communication, sensors and actuators, and the processing resources needed. The result of this allocation is communicated to the service manager of the concerned Titan nodes in the form of service subgraphs. Each node typically receives a subset of the overall service graph, thereby leading to a distributed execution of the entire service graph on multiple Titan nodes.

The Service Manager on the Titan Nodes then takes care of the service instantiation and that the data generated by one service is delivered to the next service according to the specification of the service graph. This occurs transparently, such that individual services are not aware of whether the next service is executed locally or whether the data first has to be transmitted to another sensor node.

Titan nodes can also invoke at run time the network manager to ask for reconfiguration (e.g., if battery runs low). During the execution of the service graph, the network manager monitors the network via the service manager on the Titan nodes to determine whether problems occur. In case a node fails, a new mapping of the service graph can be issued.

The task of the network manager is formally described as to map a service graph , where is the set of services, and ,   is the interconnections between them, onto a network graph . The network graph is described by a set of nodes and communication links , . The network manager's goal is to find a mapping , such that a given cost function is minimized.

Various cost functions targeting different tradeoffs have been proposed for such a task, such as the minimization of transmission cost, total energy consumed, or the maximization network lifetime [41]. We use here a metric targeting minimization of the total energy used in the network. The cost function makes use of a model of the sensor node using values stemming from benchmarking the Titan implementation on real sensor nodes (see Section 4 and [21]) with a TI MSP430 microcontroller and a CC2420 transceiver. The metric used for the evaluation relies on three main cost functions.

(i) Processing Cost . The cost of processing service on node . This cost results into a measure for whether enough CPU cycles are available to execute all services of the subset assigned to the given node. To achieve an energy value, the time for processing on the nodes' microcontroller is determined and multiplied by the power consumption difference from active to standby mode.

(ii) Sensor Cost . The cost of using sensor required by service on node to collect data for the algorithm. As sensors can usually be turned off when not sampling, this cost value describes the additional energy dissipated on the node while sampling and includes possible duty cycling.

(iii) Communication Cost . The cost of communicating data from one service to another for the node . The communication cost is zero for two services communicating within the same node. For external communication, it prioritizes intracluster communication and introduces penalties for cross-cluster communication. The cost is determined per message and includes energy dissipated at the sending and receiving part.

The mapping is constrained by the maximum processing power and communication rate a node can support. These limits ensure the executability of the tasks on the nodes and guarantee that the maximum transmission capacity is not exceeded without modeling node load and scheduling overhead explicitly. Consequently, there is no guarantee on whether latency requirements on the algorithm can be met. The constraints are given for the service graph subset assigned to a node :

(1)

Each interconnection is mapped to an edge and added to two sets as outgoing and incoming connections. Failure in meeting the constraints results in the service graph not being implementable. In such a case, the execution cost will be set to infinity.

The total execution cost of the network is achieved by summing up all costs incurring at nodes participating in the execution:

(2)

The costs introduced above depend on the device type to which they apply. The parameters for the device model and service models are sent to the service directory along with the node address upon service discovery. The service model in particular includes a mapping to determine the output data rate given a certain input data rate and the service parameters in the service graph description. When determining execution cost, the network manager first derives an estimation of the data communicated from service to service by propagating the data rates generated from each service to each successor. The individual cost functions make use of the service models and device models to produce the total mapping cost.

The contributions of the individual cost components vary with the application that is executed and the network it is running on. Typically, communication costs dominate, as for the energy of sending 1 bit over the air, a microcontroller can perform roughly 1000 instructions [42] for the same energy. Sensor costs on the other hand are usually constant as long as the actually used sensors have similar energy consumption per sample. The mapping thus tries to keep communication intensive connections between services on a single node. In most application, this means to draw as much processing as possible to the data source, as processing in most cases reduces the communication rate. In the case of activity recognition algorithms, this means that the processing such as data filtering and feature extraction is preferably run on the Titan node containing the sensors.

An exhaustive search of the best mapping is intractable for service graphs and networks of moderate size, as the search space grows with (see [43]). Therefore, we use a genetic algorithm (GA) to optimize the mapping, as GAs are known to provide robust optimization tools for complex search spaces [44]. The GA parameters are selected in order to favor convergence to the global maximum by selecting a large population size, avoiding premature convergence, and by performing several runs. The resulting performance is the maximum of the performance obtained in each runs.

The service graph is encoded for the GA as chromosome with genes, one for every service in the service graph. Each gene contains the set of nodes in the network providing the corresponding service. Mutations are applied by moving services from one node to another. Crossovers arbitrarily select two chromosomes, randomly pick a gene, and swap the gene and all its successors between the two chromosomes, which are then added to the population. The fitness of the chromosomes is evaluated using the cost metric given above.

Once the implementation of the service graph with the lowest cost has been found, the service graph subsets are sent to the individual network manager of the Titan nodes for execution. Additional aspects related to modeling and convergence speed are discussed in [43].

3.3.2. Application Logic as Services

The logic of Pervasive Apps is likely unique to each application. Thus, it does not lend itself to be realized by generic services, such as the ones provided by Titan nodes. In order to enable for a large variety of Pervasive Apps, Titan allows for application repositories to download application-specific services to the mobile device, in the form of Java code (this is the "control service" in Figure 1).

This Java code can access to all the features of the mobile device (usually a mobile phone), such as screen, touch input, audio output.

In other respects, the downloaded Java services follow the same service model as the Titan nodes and can interact with them. Thus, the service running on the mobile device forms part of the service graph describing the application, exactly like any other sensor node. In particular, the Java services have access to packet communication methods to exchange data with the other services running on the Titan nodes. Since the Titan nodes use an 802.15.4 radio, we have built a custom Bluetooth to 802.15.4 gateway to allow communication between the mobile device and the Titan nodes. The Java service thus communicates over Bluetooth to the gateway, and the gateway relays the data to the 802.15.4 interface.

The Titan network manager additionally provides a Java API that can be used by the Pervasive App to dynamically reconfigure the network with new service graphs. This allows tailoring the processing to the current Pervasive App state and turning unneeded sensors to low-power states.

4.1.1. Internet Application Repositories

The dice game service graph is composed at runtime by the application server and involves only the dice that are available on the schoolyard at the moment (see Figure 6).

The activity recognition part of the game is represented by service graphs which are executed in a distributed manner in the network. Within the same game, four unique service graphs are designed to recognize one of the game states: dice pickup, shaking, throwing, and scoring. While this could be realized by a single service graph, by using multiple service graphs we capitalize on the dynamic reconfiguration capability of Titan to minimize the number of resources used at any time point in the activity recognition process. Reducing the number of resources used for activity recognition is a key to enhance the sensor network operating time [46]. The core logic of the Farkle game is a downloadable service in Java that is run on the mobile phone (Farkle game service). It receives the output of the service graphs, keeps the game scores and player sequences, and instructs the network manager which service graph to load next (i.e., when a dice is thrown, the next instantiated service graph is the one doing scoring).

4.1.2. The Mobile Phone

When the game is started by the players, the network manager on the mobile phone starts the Farkle game control service, which in turn instructs the network manager to map and execute the first service graph on the smart objects according to the mechanisms described before. The Titan framework then takes care of service instantiation and that data generated by one service is delivered to the next service according to the specification of the service graph. This occurs transparently, such that individual services are not aware of whether the next service is executed locally or the data first has to be transmitted to another smart object. The Farkle control service receives the results of the service graphs running on the smart objects and decides when the game progresses from one state to the next. When this occurs, it instructs the network manager to exchange the current service graph for a new one. During the execution of the service graph, the network manager monitors the network to determine whether problems occur. In case a node fails, it may issue a new mapping of the service graph and update the participating smart objects' configurations.

4.1.3. Titan Nodes

Ideally, the Titan nodes recognize individually what is happening to them. Thus, data acquisition, segmentation, feature extraction, and classification are completed on the smart objects and wristband for the local sensing modalities. By only communicating their perceived context, the communicated data volume is reduced. A network classifier can fuse those reports to get a global view of the situation (see, e.g., [46]).

The Titan nodes in the Farkle game feature three axis acceleration sensors integrated into the dice and in the children's wristbands. Those acceleration sensors are used to determine the four states of a player's throw: picking up the dice, shaking them, rolling, and determining their score. For each of the states, the Farkle game control service adapts the executed service graphs, such that only the wrist sensors and the dice that have been picked up are used. All nodes sample the data at 20 Hz and process the data locally as explained below.

The game is decomposed in four different stages. In each stage, service graphs on the Titan nodes perform local sensor data processing and notify the Farkle game service of relevant events, such as the completion of a stage. The Farkle game service then reconfigures the service graphs on the Titan nodes to enter the next stage. Here, each stage corresponds to a different activity recognition task. Figure 6 illustrates the game stages, the service graphs mapped on the sensor nodes when a player chooses to throw two dice, and the corresponding service graphs executed on the smart objects and on the wristband.

Stage 1.

The first game state configures the wrist sensor of the current player to determine whether the player reaches down to pick up a dice. This event is broadcasted to the smart dice. The smart dice periodically sample their acceleration sensors using an acceleration service to detect whether they are moved by a variance and threshold service. The decision tree service shown in Figure 6 runs on the smart dice and reports to the mobile phone when the pickup and moving events coincide. Correlation between the pickup movement and the movement of the dice indicates which player has picked up the dice. The indication of who has picked up the dice is sent to the Farkle game service. This information is used to monitor that the game rules are appropriately followed. If the wrong player picks up the dice, a message is displayed on the screen and the application asks the players to restart the turn by throwing the dice anew. If the right player has picked up the dice, the Farkle game service reconfigures the Titan nodes with the service graphs to recognize the next activities.

Stage 2.

The second state of the game determines whether the dice are jointly shaken by the same player, allowing multiple people to play simultaneously. Only the dice shaken together with a player's wrist are used to follow that player's score. This detection is realized by computing variance and zero crossing rate of the acceleration on the dice and player's wrist, and classifying this into a binary decision indicating whether a specific dice is shaken by a specific person. This is notified to the Farkle game service.

Stage 3.

The third state waits for the end of the rolling motion of the dice. This is done by extracting the variance of the acceleration signal within a window (a measure of the energy of the acceleration related to the movement of the dice). A comparison to a trained threshold indicates whether the dice is moving or has stopped. Once the dice reaches a standstill, the Farkle game service is notified.

Stage 4.

The final state determines the eye counts of the dice from the measured gravity vector. The corresponding eye count is forwarded to the Farkle game service, which determines the throw's best scoring combination and identifies which player's throw is next. The state sequence then starts anew. The recognition of the eye count uses a decision tree classifier as well. It classifies the static acceleration sensed by the Titan node in the dice into a set of 6 output classes corresponding to the eye count. The classification result is sent to the Farkle game service.

4.2.1. Application Instantiation Results

Downloading the Farkle game description including the Farkle game control service and the service graphs from the application server via HTTP required 38.9 kB of data transfer. In our office, it took on average 17 seconds to obtain it using an HSDPA connection. The reconfiguration time of a single node has been measured using a minimal service graph involving a counter and an LED display service. This service graph is encoded in 20 bytes and fits into a single configuration message. The resulting reconfiguration time averages at 106 ms and involves the transfer of a configuration message from the mobile phone over the gateway to the smart object, reconfiguration, and a confirmation message in return. Reconfiguring the node itself takes 650  . The execution times of individual services on smart objects range between a few microseconds and several milliseconds (see Section 4.3 for details for individual services). The most complex service graph of the Farkle game involves 41 services to recognize "shaking" for all 6 dice. Mapping it to the BAN takes 3.5 seconds while the complete switching time including wireless reconfiguration amounts to 4.4 seconds. Titan's reconfiguration time is short enough to be useful for user interaction applications. The recognition algorithms can focus on the few activities of interest in the current state and reconfigure for other activities when the state changes. The activities that should be recognizable after reconfiguration need to be detectable for longer than the reconfiguration time of the new service graph, which is in Farkle a maximum of 4.4 seconds.

4.2.2. Activity Recognition Performance

We characterize the most difficult activity recognition state of the game: "shake dice," (performance of Stages 3 and 4 is close to perfect as they consist of significantly simpler classification services).

In the "shake dice" the wrist sensor and all dice picked up in the previous state sample their accelerometer at 20 Hz and detect correlated shaking. The data is segmented in 3-second windows with an overlap of 50 samples. Three features, mean, variance, and zero crossing rate are computed on the magnitude of the 3-axis acceleration. One dice sends its three features once per window to all other participating dice. Each dice then combines the received data to their own locally computed features into a feature vector that is classified with a decision tree to determine common motion. Executing the service graph on the MSP430 takes 4.88 ms after each sample, and 7.44 ms for feature calculation and classification when a window is complete. Using a dataset of 99 minutes of correlated and uncorrelated shaking of different frequency and amplitudes by five different subjects and performing a 5-fold crossvalidation, we reached an average accuracy of 83.8%. Better results were obtained using signal correlation as feature, which achieved an accuracy of 91.3. However, using correlation requires transmitting the complete 20 Hz magnitude data from the wrist to the dice instead of just transmitting three feature values every 2.5 seconds. It would thus increase the communicated data volume by a factor of about 10. A network classification using window features has also the advantage of needing less accurate synchronization between the sensors. In our tests, the shaking detection accuracy decreased by 12.8% when using correlation as feature and signals were misaligned by as few as 5 ms. For a similar misalignment, the window-based feature classification on the other hand decreased by only 0.1%.

4.2.3. Data Volume

In a centralized solution, all nodes constantly transmit their 20 Hz samples to a central node where all processing is done. In the distributed solution on the other hand, the wrist broadcasts its features once per window period directly to the dice, which then report their classification result to the central node. This reduces the transmitted data volume for dice by a factor of

(3)

For dices, this amounts to a very significant reduction in data transfer.

This significantly reduced bandwidth need enables the smart objects to use low-power communication with lower bandwidth as they only transmit events instead of continuous data streams. An important observation is here that using direct communication from the wrist to the dice reduces the required bandwidth by a factor of 2.3. This shows that a mesh topology here has an advantage over the usually chosen star topology for BAN.

5.1.1. Dynamic Mapping

A first mechanism lies in the network manager which maps the service graph it has been assigned to the available resources while minimizing a cost function. Since the communication between services is handled transparently, there is great flexibility for the network manager to deploy the service graph on the available nodes. In particular, services providing computation can be very easily and transparently moved across nodes.

When processing nodes disappear (e.g., if the energy runs out, or the user moves away from the resource), Titan can seamlessly map the service graph to the remaining available resources. When new processing node appears, Titan can remap the service graph to minimize the cost of the implementation. Beside saving costs, a fast reconfiguration can also be exploited to virtually extend available resources by, for example, always only recognizing activities that are possible at the moment and thus increasing the total number of activities that can be recognized [50].

5.1.2. Service Composition

Another mechanism lies in the Internet application repositories. The application servers contain application templates. They also hold sets of replacement services. When one service is not available, it can be replaced by one or more other services to reach the same desired function. For instance, a service based on an FFT to compute the dominant frequency in a signal (e.g., to detect walking) could be substituted by a lower-complexity mean crossing rate service. Foreseeing replacement services is currently left to the application developer which does this at design time.

5.2.1. Small Sensor Displacement

We developed an unsupervised classifier self-calibration technique which we showed can improve the resilience to small changes in on-body sensor placement of activity recognition techniques [52]. This is applicable to any sensor modality. It only assumes that when a sensor is displaced the structure of the activities in the feature space retains a similar relative topological organization and that the class displacement in the feature space remains comparable to the interclass separation. As such, this approach is suited to small displacements on body segments (e.g., arms or legs). This approach is also suited to cope with changing sensor characteristic (e.g., sensitivity or offset, typical in sensors included in textile fibers), or to changing user action-motor patterns (e.g., due to aging).

Within Titan, the application service repositories may automatically instantiate an unsupervised classifier self-calibration service when the application designer indicates that a requested sensor may be subject to displacement.

Kunze and Lukowicz have shown that features that are robust to displacement can be designed using body models and combining accelerometer and gyroscope modalities [53].

Within Titan, the application service repositories may automatically instantiate the computation of robust features when the service directory reports the availability of the necessary modalities. This may happen transparently, even though the application developer only specifies the use of an acceleration sensor in his application template.

5.2.2. Major Sensor Displacement

Kunze et al. have developed a method that allows for acceleration sensor to self-characterize their on-body placement [54] and orientation [55] using machine learning techniques. They also show that symbolic location in the environment can be obtained by similar self-characterization [56].

Within Titan, we envision that Titan nodes geared for on-body use (e.g., mobile phone that can be in various pockets or belt clip, watch which may be on the left or right arm or in a pocket) autonomously determine their on-body location and orientation using these methods and report this upon service discovery queries as a self-characterization parameter. Thus, the service directory receives the notification of a presence of a sensor and additional self-characterization. The Internet application repositories may then instantiate an appropriate activity recognition service graphs according to the availability and placement of the sensors. The application designer would design various service graphs for the possible foreseen on-body placement of the sensors. For instance, walking can be detected from sensors placed at the hip or at the ankle with a similar recognition algorithm. The difference between the sensor placement translates, however, in different threshold parameters of the classification algorithm [57] which can be offered as different services.

5.2.3. Modality Replacement

Kunze et al. have shown that a sensor modality (magnetic field sensor) can be replaced by another modality (gyroscope) [12]. Calatroni et al. argue that there is a large number of modality replacements that can be envisioned in ambient intelligence environments [8]. They suggest that many sensors can be repurposed for activity recognition even though they were initially provided for other uses. They characterize, for instance, how reed switches placed in windows for security purposes can be used to infer standing or walking, by means of assumptions about human behavior when interacting with these objects. They further propose a method of transfer learning to capitalize on such sporadic information gathered when the user interacts with the environment to acquire ground truth labels that can be used to train existing on-body sensors into recognizing these activities [58]. Thus, while initially walking or standing could only be inferred during a short period of time around the interaction with the environment, after transfer learning to an on-body sensor, the capability to infer walking or standing is continuously available. Such an approach is especially interesting when the modality, placement, and orientation of the sensor on-body is unknown to the system. Transfer learning allows to attribute meaning to the signals of that sensor at run time. Thus, this method can in principle be applied to any known and yet-to-be-developed sensor modalities.

Within Titan, the application server can autonomously propose alternative service graphs and the corresponding supplementary processing to exploit alternative sensor modalities (e.g., replace magnetic field sensor by a gyroscope). When Pervasive Apps operate over longer period of time, transfer learning can be exploited to attribute meaning to signals originating from unknown new sensors. This can be realized by means of a developer-generated database on the Internet repository side containing the behavioral assumptions, and the corresponding methods to operate transfer learning to a new sensor.

5.2.4. Dynamic Sensor Ensembles

Modern activity recognition systems are usually multimodal and capitalize on a wide availability of resources [4]. When a large number of multimodal sensors are available in the environment, a common way to perform activity recognition is to rely on the global decision fusion of individual classification performed on the sensor nodes (ensemble classifiers [59]). We showed that enlarging the number of sensors contributing to the global decision rapidly enhances the activity recognition accuracy of the system [60]. However, it also increases overall energy use. Since wireless sensor nodes are generally battery operated and rely on energy scavenging, a better approach consists of managing a power-performance tradeoff of the system. We have showed empirical heuristics to dynamically shape ensemble of nodes to reach a dynamic power-performance tradeoff, specified either in terms of target activity recognition accuracy, or network lifetime [46]. The key of this approach is the heuristic that allows to find which sensors (one or more) should be included in the dynamic sensor ensemble to replace the loss of another one.

Within Titan, we envision that the application developer characterizes the contributions of the various sensors foreseen for activity recognition, as described in [46]. Based on this, the Internet application server may instantiate the Pervasive App template with various subsets of the available sensing services, in order to realize power-performance management. This also can lead to improved robustness to faults, as the loss of a sensor leads then to the use of a sufficient number of other additional sensing services to replace its contribution and maintain the desired target performance.