Modern smart device-based concept of sensoric networks

Abstract

The modern society evolves into a sensorial network environment where individual sensor data can be transformed into cumulative and comprehensive representation for human. In a real time, it is independent of current location or behavior. The only limits to increase productivity and to create a smarter surrounding environment are personal habits and technology progress. The paper describes sensorial framework, which is dealing with the current aspects of technology, sociology, and usability in daily life usage of ubiquitous mobile devices with sensors, and arising computed and energy power. Nowadays, most of the common tasks of our lives are mainly influenced by network and social connectivity where infrastructural speed limits are provided by the information flow.

1. Introduction

In the last few years, we have been witnessing the market explosion with the end-user mobile devices that are enhanced with many types of sensors. The time of plain non-sensorial smart phones is more or less over, and we are facing the future environment where built-in sensors as ambient temperature, magnetic field, accelerometer, gravity, light, humidity, and others are common equipment of current mobile devices [1, 2]. All of these sensors are supportive to the provided inner content. For instance, the user interface has been enhanced using gyroscope, and therefore, the visualization can be controlled by rotation or current position of a device [3]. However, we might also start seeing sensors as receptors of global human body where, if we gather more sensorial data, we are being transformed into a smarter and more enhanced society. We recognized the most significant influence of sensors in computing estimation capability of surrounding environment and user behavior recognition where identification of human stereotypes is leading to the increase of productivity in terms of association corresponding processes with partial or full automation. Today’s low-cost sensorial technologies’ doors are opening up for daily life improvements and are shorting distances between content from producer and to consumer. Typically, the content could be served depending on current environment or user’s behavior. For instance, the user is located at a home environment, and therefore, the applications associated with the mobile device, such as home controlling or leisure, have higher priority. Under specific circumstances, we assume that the applications and content are so predictable that we could talk about Artificial User Interface as an improvement of device usability. Devices could offer predefined or the most preferred content launched automatically or initiate routines in specific circumstances.

Sensors within network-connected devices increase the capabilities of networks, and we assume that the internet is stepping into the sensorial network era where the key benefits are based in higher human productivity and openness of new sensorial informational channels. Those are personally related to behavior or environment, which leads to more open modern society. Although the openness could be noted as a negative factor due to the behavior, location, or environmental-based personal privacy information, that is, supposed not to be publishable, we consider that as a new opening stream of interpersonal communication which is from the view of historical context, more and more wanted to be open and available. That leads to the reduction of the privacy borders [4, 5]. The question is - are we prepared for a society based on free informational resources where any knowledge is reachable, and is it considerable to view it as a positive factor of human evolution?

The following sections describe the problem from a technical point of view of sensorial networks as well as the conceptual and architectural details of designed framework Senzoric.com. The benefit of sensorial framework research is aimed at effective distribution of information to corresponding interpersonal channels with consideration of current technological limits [6]. Nowadays, the challenge is about providing important social relative information without explicit request definition.

2. Problem of sensory networks

All possible sensors of mobile devices are producing a large amount of data in total which are not necessarily to be transferred over network and reasonably shared in real time. That is why we could consider most of them in raw form as redundant or without meaningful informational value. Also, processing of sensorial values with maximal refresh data rate would be unreasonable in terms of battery management or use of non-real-time system bases.

The key to sensorial networks, where mobile devices are an essential part, is in the balanced design of outlined high-level architecture (Figure 1). There, the device is producing a meaningful amount and type of sensor’s data, which depend on current network connectivity and are required to measure representational device states. For better overview, we will divide the problem into several parts (sensors and device states, connectivity and data, battery management) and describe them in more details.

Figure 1

Sensor informational flow.

2.1. Problem of sensors and their states

We will presume that all the sensors are possible providers of raw informative data, where the main ones are outlined in Table 1. These are commonly embedded in mobile devices nowadays, for instance, temperature, light, gyroscope, etc.; we also expect to find complementary sensors there, such as microphone, camera, or battery indicator described in Table 2. The way of gathering sensorial data, we identified in supported system calls dedicated to specific mobile device platform, was based on the study of literature [7–12] and mainly on articles [13–17].

Table 1 Native sensors embedded in current smart devices[16, 17]

Table 2 Complementary sensors derived from smart devices

We divided the sensors into two groups where the first one is with short-term and second one is with long-term data change characteristics. According to the data amount, the first group required more frequent measurement to gather the correct sample of data for better pattern recognition, while the second group is resistant to infrequent precise measurement. Further in Table 2, we highlighted complementary sensors that are more related to pattern environment recognition than behavior.

From all available defined sensors, we announced which of them are contributory to specific behavior and environment pattern recognition with brief description.

Behavior patterns are the bases of motion effects of device sensors. That means recognition whether the device is worn and somehow influenced by a human body. The running motion is significant with fast location movements and periodical short shocks, while walking is distinguishable by lower shocks and slower location changes. By standing, we assume a static body position with small movement interferences, while sitting is more stable and longer lasting [19]. Sleeping behavior is recognizable in deferred device position where surrounding specific pattern noise occurs. That was a brief description of basic motions, and more will be described in further discovery, as dancing, watching, gym, fun, etc. All of them are based on accelerometer, gyroscope, rotation, or magnetic field.

Another point of view is environmental-based resolution which is more limited and relies on external resource and social or network collected group knowledge. We defined the basic environment as home, work, transportation, or others which are less statistically probable. The home environment is recognizable as a place located overnight staying with the most count of occurrences in time. Work environment is a place located over recognizable specific equal time duration consumption over an awaken user state and most likely during a day in a different location than home. The transportation by car, plane, boat, train, bike, etc., is recognizable by the speed of a device, respectively, measured by location differences. Therefore, environment recognitions are essential location sensors and supportive sensors as humidity, temperature, pressure, light, and charging indicator.

2.2. Connectivity and data management

The connectivity is a basic factor which influences the necessity of transmitted data where the aggregation ratio between raw sensorial data and representational device state information implicates a reasonable data stream, which competes to be the most effective informational dataflow in time, energy consumption, and expected system functionality of delivery. We consider the minimal granularity of sensor’s transmitted information as bounded by maximal network throughput dependable on mobile device connectivity over well-known standards General Packet Radio Service, Enhance Data rates for GSM Evolution (EDGE), Universal Mobile Telecommunication System (UTMS), High Speed Packet Access, Wireless Local Area Network (WLAN), or Worldwide Interoperability for Microwave Access (WiMax). We also see considerable differences between Quality of Service according to a network type, current distance between mobile device and network access point, and other environmental circumstances.

2.2.1. Networks

We focus on wireless networks where data transfer rate fluctuates more than wired networks; therefore, there is an issue with data optimization that has to be considered in application development depending on location and provided technology. Nowadays, wireless networks are being challenged by increasing amount of users and data application dependency. That is why we build applications for mobile devices that would test the performance of current wireless connectivity [20]. The measurements are provided over transport control protocol (TCP). Backend server is located depending on the tested technology that means using virtual private server (VPS) with backbone connectivity for mobile networks, and using local network server for wireless fidelity (WiFi). The results are outlined in Table 3. There are theoretical and practical capabilities according to the types of networks and serve for the smarter definition of a development data management concept. Bandwidth emphasizes the possibilities of data amount stream, the distance range recalls the possibility of locational change possibility during transmitting, and measurement overview real data statistics of latency; upload and download provided by Android mobile devices ZT3-Blade by application available on Google Play [21, 22].

Table 3 Wireless network limits for mobile devices

The data acquired depend also on the mobile network provider and its implementation of network standard. Therefore, the test results could be different. In addition, the environmental and distance factors influence the network connectivity. For further discovery, we expect more date from crowd with more statistical results according to countries. Figure 2 outlines the test application, where latency, the download and upload measurements, is provided in basic testing flow triggered by a user on the start button. The principals of testing are to disable all remaining network traffic and to locate the server to the closest location to test the device. We provide only one server in Prague, so the latency could be influenced by the maximal round trip around the world, which is in the worst case around 200 ms and around the Europe 30 ms. The testing process starts from a client who is sending Transport Protocol Packets with minimal size 1B containing urgent processing flags and waits until the response is received from the server, where the time dilation is a real Round Time Trip. The download is requested from the client after latency timeout. The server produces the maximal buffer size output stream which client receives in specific amount sizes, estimates speed as time dilation rate, and receives sizes supported by the average value for fluent result flow. The upload is the same on the server side just when the server is sending back to client a 4B packet with the speed encoded from integer value. For better overview, the following figure represents the application.

Figure 2

Developed speed test network application (for smart devices based on Android platform).

The results (Table 3) were gathered from approximately 400 mobile devices by downloading applications from market. We expect to elaborate the statistics over time and cover all available network types in further discovery.

2.2.2. Information flow

To provide the maximal efficient informational stream between the producer and consumer, we have to consider data granularity and timing of transmitting values [23]. The reason is to minimalize energy consumption of mobile devices and maximize provided informational value. We assume the maximal granularity of sensor’s information; therefore, the behavior and environmental patterns that we defined are over time period recognized by mobile device and transmit only a minimal amount of data. We outlined defined patterns in Table 4. They are identified on the producer side as well as on the consumer side.

Table 4 Behavior and environment information

The minimal transition bundle of information has to consist of pattern identification (4B), timestamps (8B*2) of starting and ending time of recognition, location (8B*2) of latitude and longitude and the last device id or temporary token (16B). Therefore, a real-time data transmission has a minimal size (52B) in online mode delivery, which is not energy efficient. We also consider an offline mode for data flow, where the lists of recognition states are transferred over network that is demanded by remote request or periodically uploaded to data distribution server by time period or buffer size limit. The data flow type of service is according to battery management limited in real-time processing; therefore, the historical and current data are flushed to server only if required, with possibilities of in-time authorization on the producer’s side. Otherwise, in case of real-time processing requirements where the predefined authorization variant exists, the data flow depends on the point of interest and on the status change events with maximal battery effectiveness of sensor data gathering.

Other aspect, which would be taken in consideration, is network availability where the online mode is in fact commonly expected during the day, but it may also occur on specific occasions when the mobile device is temporarily or in the long-term without network connection. In that case, we expect to continue saving locally the desired sets of sensorial information for correct results in the objective of large scale. Therefore, the data could be stored on the device locally with optional network batch upload later on.

2.3. Battery management

According to the tested results from discovery [7], the energy consumption of localization sensors is the most effective in Global System for Mobile communication (GSM) cell bases evaluation where energy costs lowest than <20 mJ followed by WLAN around 545 mJ while Global Positioning System (GPS) sensor is consuming much more >1,424 mJ. Therefore, the priority in location estimation is about to be used as available capability as ordered by energy efficient factor. Other facts, which are suppressing energy sustainability, we consider to be redundant sensors where key efficient way is lower than information provided according to frequent measurements. Therefore, the principal use of the battery efficiency, we concluded, would be as linear acceleration in combination to cell location bases where uploading time in real-time behavior is about 1 s. Other sensors are beneficial only if battery is in the charging mode. In another words, the consumption nowadays is essential for usability of mobile devices where there is also required the implementation of such benefits as a battery recharge reminder, which in home environment reminds you to plug in your mobile device to power supply.

3. Related works

Activity recognition system [15] was developed as a real-time monitoring system for mobile devices that embrace neuronal network motion pattern recognition by body accessories, wirelessly connected over Bluetooth to provide sensorial data. The proposed solution requires additional components outside of the mobile device and additional energy consumption for local communication [24].

Another related work [14] considers providing sensorial information by context-aware web browsers which are able to merge web application tags with current mobile device sensorial data but only as a foreground process invoked by user actions, and therefore, for smart environmental-behaving solutions, it is insufficient.

On the other hand, InContexto [17] is a solid work that represents background service with still, walk, and run recognition with 97% accuracy, but the only sensor used for gathering this data is accelerometer which is not enough for larger scale of behavior and environment recognition.

For environment recognition and connected activities, where data is being minded in a recorded log, the case is to predict or analyze future patterns or to recognize the environment as a great work [16] where the point of view of soft sensors is taken into account, but the hard sensor are considered as a future possible improvement.

Another interesting point of view, from activity recognition algorithm classification [10], provides the accuracy comparison and results with 97.7% measured by a mobile device in a pocket but with the fact that the tested data are gathered with high sampling frequency which leads to higher battery consumption. Related works are outlined in Table 5, created to compare the needed solution for the cross features and also to define a more precise goal of developing the application and to define specific requirements.

Table 5 Related work functions summary

According to the demonstrated summary in Table 5, it is hard to find a complex solution around all the mentioned solutions. This fact leads us to design and develop a new architecture as well as to implement some parts to provide a basic evaluation of the proposed solution. This will be outlined in the next few sections.

4. Proposal concept

Based on the mentioned problems (Table 5), the architecture that was designed was developed according to producer consumer type (Figure 3). The core concept consists of producer, consumer, and middleware. The producer is gathering sensorial raw data which are computed on client side where they are generalized. Then, in portions, they are transmitted to the middleware connection server over TCP, User Datagram Protocol (UDP), or HyperText Transfer Protocol (HTTP), where the content is wrapped into JavaScript Object Notation (JSON), eXtensive Markup Language, or native byte code data format, and they securely persist in the database.

Figure 3

Concept of framework architecture.

The consumers are accessing the gathered data by web interface with aggregating functions by web console upon request, or the data are pushed down to subscribed device by cloud messaging, email, Short Message Service, and others. Overview of prototype side ‘Producer’ is described in more detail in the following section.

4.1. Producer

All high-level information about the environment, motion, or behavior of users are estimated from sensorial raw data from a device where a high amount of real-time data is produced and is unreasonable to transmit over network. Therefore, raw data are being processed to high-level valued data in client application by environment and motion resolving unit (Figure 4). All sensor listeners are included in one background service which gathers raw data for processing [25]. The refresh rate of sensorial data is customized for efficiency of tracked information and type of sensors, where motion sensors are the needs in principal higher frequency of measurement for correct resolving. The processed data are stored in a local cache and flushed periodically to a connection server with minimal data amount, if the network is provided.

Figure 4

Mobile device concept of producer.

The sensorial raw data are expressed in float array values dependent on the sensor type where one, two, or three dimensions are possible. Gathering data on device requires at least separate thread to perform more precise measurements independent of user interface thread. Behavior and environment resolution is connected to changing events within minimal energy consumption, for instance, accelerometer is much more efficient than other motion sensors [10]. Therefore, motion recognition relays on accelerometer raw data which is compared in between the neighbor’s values and counted into intensity. The filtered data with rational thresholds, related to the type of sensor, estimates the motion type over intervals or patter recognition. In the next chapter, we describe server architecture and used technology with communication protocol (Figure 5).

Figure 5

Mobile device concept of producer prototype (activity diagram in Unified Markup Language (UML)).

4.2. Middle ware

The concept of server side architecture is designed over Java technology. The reason is hidden in effectiveness of productivity, scalability, and reusability of available open source components. Figure 6 highlights the important parts of server architecture. The server side consists of three main parts which are connected by socket communication.

Figure 6

Middle ware architecture concept (component model in UML).

The first connection part is responsible for correct data delivery from producers with storage management. It consists of socket listener and message resolver (communicator (Figure 7)). Those are the main functionalities where the socket listener listens on the predefined port and accepts connection if it occurs. If the network connection is successful, it dispatches to it, the message resolver is called, and the transmitted data are resolved and saved into object database. Objects are transported over Java serialization technology where content is about to be either quickly serialized or de-serialized on binary code and sent over network. The null message called Heartbeat protocol, which serves to keep the connection live, is resent every 10 s. Next, core system part and data interface part basically handle core features of a system from sensors monitoring up to system external data storing, and system authorized access ability for all kinds of requests from front or background processing. The last part of the system is focused on data collection interface where the data are persisting to the object database. The next chapter overviews a consumer of real-time data or historical records retrieved from the database.

Figure 7

Class diagram of server communicator.

4.3. Consumer

The application of consumer receives the data stream of information after a successful registration to an authorized channel. The real-time communication is based on a data stream realized by long time polling for HTTP where messages are pushed down in timeout sequences required by client. In cases of UDP, they are enabled by mobile network provider. The messages are sent directly by an opened socket-type communication from the server to a client. Dataset consists of timestamp, identification of motion, or environment and location. In further discovery, we plan to transmit also audio and video stream which requires higher bandwidth capabilities. The prototype has to implement a visualization of received data on a map with recognized activity related to a specific user and also convenient, supposing the notification mechanism for received new data that is defined by user settings. Core process of the consumer application is defined in the following figure in the activity diagram (Figure 8).

Figure 8

Activity diagram of consumer prototype.

5. Implementation

Development tools are open source based and preferred by Java programming language whenever it is possible. The development environment Eclipse provides beneficial features as a remote system-controlling Unified Markup Language diagram and model generation from Java code or additional program and code measurements and metrics evaluation. We describe the following parts from implementation point of view and describe key factors for success software prototype implementation.

5.1. Producer

The producer prototype application is developed on Android platform [26] and available to download for free on Google Play [27]. Application is maintained by four tab views that include embedded widgets of content. The first one, user information, is only an overview of implemented recognition patterns and their corresponding visualization. The second one is a graph representation of sensorial raw data that are gathered in real time, and calculates statistical overview in numbers and lines outlined in the graph. The visualization of data representation is implemented as the rotation buffers. The values are expressed in a graph by lines or stock piles quoted in a resizable grid with marked values (Figures 9 and 10). The third tab provides logging information about communication with server and other notifications from the application, and the last one - Options tab, is providing settings of the application such as gathering sensorial data and preferred runtime modes. The results of the measurement are saved locally due to the historical overview purposes and also posted to the middle ware, depending on current user settings and network connectivity.

Figure 9

Prototype of producer mobile device application. Info screen (left, a) and graph representation (right, b).

Figure 10

Prototype of producer mobile device application. Logger screen (left, a) and options tab with multiple switchers (right, b). Screen was cut.

5.2. Middle ware

The middle ware is implemented in Java and running on remote VPS on the Czech backbone network datacenter as Linux service used CentOS 6. Development corresponds with Rapid Application Development; therefore, Object Relation Mapping suites Java Persistent Api technology, which is provided by EclipseLink 2.3.2. as a data mapper and allows fast and comprehensive data management. The data objects are defined in Java classes, and relations between entities are expressed as members of a concrete class with specific annotation, specific cardinality and type of relation. The objects are transformed to the database through persistent commands, and after a commitment, they are saved to a hard file on a disk. Database engine is used in object database ObjectDB for its excellent performance which is running on the same server as the service. Tomcat is used as a web server with dynamic web module that allows Servlet 3.0 annotations directly into classes instead of configuration file web.xml. Consumer requests are served over HTTP request/response by JSON due to the simple and fast remote data access, which is used by a web client or Java console. For web service, [28] used a Representational State Transfer technology, which provides fast, comprehensive, and well-maintainable implementation, and allows multidata-type output assembled by simple annotation.

5.3. Consumer

The consumer is developed as an Android client, who is accessing resources by HTTP requests or socket connection depending on user datagram connectivity. Implementation uses MapView widget from Google as a map layer where other components are viewed. The location is translated to map points and pictures with names available from social networks connectors, such as, Google+, Facebook, or Twitter. The activity is viewed as a small icon on the bottom of user’s picture depending on the data from producer (Figure 11).

Figure 11

Application prototype of consumer.

6. Discussion

The prototype application that we developed consists of three components: producer, consumer, and middleware base on proposal concept. The minimal sampling rate for recognition activities and environment is gathered from measured data and corresponds with the table above (Table 4). Currently, we are gathering data from a crowd by free distribution application on the market and by testing user suitability and battery consumption. Furthermore, we enable connecting tasks based on activity and environment recognition. The solution overviewed (Table 5) is in parts as a motivation of improvements. On the other hand, from the usability point of view, the external sensors are out of scope for daily life routines and other systems are overheated by data without reasonable user conclusions or influence. The key to providing smart environment based on a mobile device application is in definition of goals and establishing an effective balance of used sensors data providers, where the minimization of power consumption is essential.

6.1. Privacy assumptions

The privacy is important issue for current social network services. Therefore, we assume that there are three security levels which fulfill the individual’s privacy requirements regarding the usage of Senzoric framework.

The first one can be seen from the network payload perspective where sensitive data could be captured by ‘man in the middle.’ We provide Secured Socket Layer communication between consumer and producer over all middleware. Therefore, only the knowledge of 4,096 bits of RSA key would be able to breakdown ciphered payload transmitted over the network.

The second perspective is from the database point of view. We consider most data provided from sensors and stored in database as anonymous data collections where no passwords or user details are stored. The whole process of association anonymous data collection to specific user is based on a token which is generated on user’s device by BCrypt [29] function from password or identity of third parties. Therefore, full database access rights are not eligible to associate user’s identity to concrete datasets. The only knowledge which could be data minded from database are statistical overviews and anonymous data interpretations.

The last perspective is done from application point of view where the access token is stored on each device in memory cache for correct association of device within the whole system. That could be captured on rooted android mobile devices, but we are not responsible for malicious software and user decisions.

6.2. Efficiency of assumptions

The key factor considerable in terms of system effectiveness is energy consumption on client side where mobile device users are sensitive about the battery usage for installed third-party applications. Another key factor is data amount transit for getting or setting information within the system which results in the usability being improved by data traffic efficiency. The last mandatory factor is the extraction ratio between sensor data and full comprehensive information. Extraction ratio is dependable on granularity of provided information and sampling frequency of sensorial captured data (Table 6).

Table 6 Related work effectiveness of sensor networks

We use, as future discoveries are made, this ratio of informational extraction effectiveness and employing of neural networks for pattern recognition instead of simple intensity computation model.

7. Conclusions

The problem of behavior and environmental application development for mobile devices was presented. The effective concept for distributive sensorial information creates a challenge concerning the battery consumption, communication, and effective recognition aspect. All of these aspects are evaluated equally in order to help to increase the usability. We proposed application concept that is available for testing with real users from whom the data are gathered. The key benefit of the proposed architecture is in scalability and applicability for further location, motion, and environmentally based real-time solutions.

Future trends of sensorial networks and smart solutions based on them are founded in global knowledge of motion and environmental recognition. It is also built as automated assigned appropriate process that recreates a new society where one is driven by smart solutions and decision makers [12]. For instance, one will never wait in a situation where traffic periodical mechanism is provided without global sensorial network knowledge, which can have an impact on various kinds of day life activities [1, 30, 31].

The work proposes effective information distribution within minimal usability impacts on device, user, and networks. In the future, the social networks and sensor networks will merge. We stand at the beginning of a new era of sensorial social networks where we will perceive only information which is relevant to us. In other words, we will create, through our activities, our own artificial informational shield or receiver, and through our life style, we will obtain corresponding informational channels.