We conduct extensive evaluations of SmartLoc in different scenarios, both in downtown Chicago and on nearby highways. In our evaluations, a Samsung Galaxy S3 with latest Android 4.1.2 is mounted to the windshield. We drive for over 100 different road segments in downtown Chicago ranging from 1 to 10 km and over 30 km on nearby highways. Since the motion sensor will provide the orientation of driving, knowing the driving distance from the starting point and the driving orientation, one can compute the real-time location easily. Therefore, the key problem becomes estimating the driving distance. The evaluation metrics include the traveling distance, road infrastructure recognition, accuracy, and energy consumption.
Traveling distance estimation
First of all, the typical frequency of updating position through GPS in a smartphone is much lower than that of motion sensors: it takes around 2 s on average to update GPS location. We denote the driving distance in a time slot as a traveling segment. We focus on the evaluation of traveling distance estimation in two aspects: (1) the accuracy in distance estimating in traveling segments and (2) the accuracy in final distance estimation of long road segments.
Evaluation on highway
We first evaluate SmartLoc when driving on the highway, where the driving velocity is relatively stable and there are no stops most of the time. We drive over ten different highway segments, where total distance is over 60 km (with driving speed in the range of about 100 km/hr to about 120 km/hr). The smartphone has access to the precise location information from the GPS, which is updated every 2 s and could be considered as the ground truth. Meanwhile, we collect the readings from the motion sensor and calculate the parameters in linear regression for 3 km. Then, we predict the traveling distance for the next 2 km and compare the distance among ground truth, SmartLoc, and pure motion sensor.
Figure 6
a illustrates the comparisons of driving distance estimation using SmartLoc (with motion sensors) and GPS. The ground truth (GPS readings) is plotted by the green curve. It is obvious that the errors between the distance from motion sensors and the ground truth are becoming larger along the time. The reason is that the errors will be accumulated without any calibration. Through learning techniques, SmartLoc calculates suitable parameters and apply them into the prediction, so that the estimation errors are much smaller. Figure 6
b indicates that the largest error is about 12 m among the ten different highway segments (each of length 2 km), and over 80 % errors are less than 5 m. Compared to the actual distance extracted from the ground truth, as shown in Fig. 6
c, over 95 % errors (among all locations where GPS location can be extracted) are less than 1 % of the actual driving distance, and the largest error is less than 2 % of the actual driving distance. We also notice that the accuracy of predicting the traveling distance will decrease with the increase of driving distance. We predict the driving distance for both 1 and 2 km after taking the data of the first 3 km to build a dead reckoning model. In our experiments, 80 % of the errors in the first group of the cases (predicting 1 km) are less than 10 m, but they become 15 m in the cases of predicting 2 km. The largest error is around 19.8 and 23 m, respectively. We plot the results in CDF as shown in Fig. 6
d.
However, based on the evaluation, we discover that the estimation results cannot maintain high accuracy for a long distance. The main reasons come from the user dependent driving behavior and unpredictable road conditions. We find that SmartLoc has better estimation accuracy when the driving speed remains stable. When the driving speed fluctuates frequently, the predicted results by SmartLoc divert from the actual value. Calibrating the location periodically is a feasible way to improve the location accuracy in real applications.
Evaluation in city without using landmarks
We then test SmartLoc in a city scenario but without using landmarks. We drive in downtown Chicago for over 30 different roads, where some roads have reliable GPS signals and some do not. We separate these roads into road segments (more than 100 road segments are produced) whose sizes are determined by our evaluations presented in the rest of this section. Similarly, we analyze the performance of SmartLoc in two phases as aforementioned.
Note that GPS signals in downtown Chicago are relatively poor and also time dependent. It is difficult to get the ground truth of all locations through smartphones. To address this, we evaluate SmartLoc in the peripheral area of downtown Chicago, where the buildings are not so high as in the center of downtown, and GPS signal is relatively good. It is much easier to get the ground truth through GPS. In our evaluations, we assume that some of the road segments have poor GPS (by removing the GPS values) so that we can apply SmartLoc to calculate the location in these areas and compare our results with the ground truth.
Obviously, the driving habit and road conditions in a city are different from those on the highway, and slight decelerating will lead to the deviation between the estimated results and the ground truth. We first evaluate the reliability of the SmartLoc system when different driving distances are used to train the system, ranging from 0.5 to 3.5 km. Generally speaking, the accuracy will increase when the learning distance increases, as illustrated in Fig. 7
a. In this figure, the X axis indicates the driving distance used for training our dead-reckoning model in SmartLoc, and the Y axis represents the mean distance (between the actual location reported by GPS and the location estimated by SmartLoc) in every time slot when we sample GPS locations (i.e., every 2 s, or about every 22 m when driving at speed about 40 km/hr). This experiment measures the accuracy for the prediction when we drive for over four different road segments with length from 0.5 to 2 km (24 different cases in total). Due to the unstable driving behavior, short road segments for training of SmartLoc may lead to a large estimation error in each time slot. When SmartLoc only learns with the trace of 1 km, the mean error in every time slot in different scenarios is around 15 m (the largest one is nearly 30 m). When SmartLoc trains the dead-reckoning model using longer trace, the mean estimation error decreases in all testing cases. The smallest error is less than 6 m, which is less than half of the error when the training trace is 1 km. We also observe that the error grows with the increase of the length of the testing road segment in most scenarios. For example, by training SmartLoc with a trace of 3.5 km, the mean error of estimating all locations in a 2-km road segment is nearly twice of that when testing a 0.5-km road segment.
We then evaluate the error on estimating the overall driving distance, as shown in Fig. 7
b. We test SmartLoc for all the road segments and measure the error between the predicted driving distance and the actual driving distance for each segment (of all segments with distance from 0.5 to 2 km) under different training traces. If SmartLoc only learns with 1 km trace, the parameters in Eq. (9) cannot be computed accurately enough. Thus, the estimation error increases to 180 m in all our tests. When SmartLoc learns enough samples, the parameters are much more reliable, and the average accumulated error is below 30 m, which is significantly better than the GPS in Chicago downtown.
Evaluation in city with landmarks
SmartLoc could calibrate the location as soon as it detects specific patterns for landmarks, especially traffic lights and turnings. Since, in the city, the probability of facing green lights in all the crossroads or always driving on a straight street is extremely low, we can take advantage of these driving features and landmarks to improve the estimation accuracy so that we do not have to estimate for a long distance. We then extensively evaluate the performance of SmartLoc with the calibration using such landmarks.
We compare the performances of three different methods: using SmartLoc without noise canceling and landmark calibration (called motion sensor), using SmartLoc calibration by traffic lights (called motion sensor and traffic light), and SmartLoc using calibration from all detected landmarks. In this experiment, we assume the first 3400 m are with reliable GPS signals and the precise locations are accessible. The estimation starts from 3400 m, and the first three subfigures in Fig. 8 indicate the driving distance from the starting point versus time dimension.
In Fig. 8
a, for the method using mobile sensors but without any calibration or noise canceling, the double integration on acceleration leads to the final deviation of over 400 m after driving about 1200 m. When the road pattern detection is introduced, the location will be calibrated when SmartLoc senses the road infrastructure patterns. During the same experiment, our vehicle crosses five traffic lights in total, and successfully detected all five traffic lights. The estimated locations are all then adjusted accordingly. From Fig. 8
b, the errors are still high, especially in the crossroads. Surprisingly, after integrating the machine learning and noise canceling, SmartLoc’s result is much better, as shown in Fig. 8
c. For about the first 900 m, the curve of SmartLoc nearly overlaps with the curve of the ground truth. For the first 450 m, the vehicle passes three crossroads with all green lights, and the error is less than 20 m most of the time. After the final traffic light, the vehicle has to drive in a relatively low speed because of the road construction. The predicted distance, consequently, deviates from the ground truth slightly, but at the ending of the road, the errors remain small. We plot all the estimated distances by three methods in Fig. 8
d, with the X axis being the ground truth distance and the Y axis being the predicted distance. SmartLoc results are distributed around the diagonal, and pure motion sensor approach deviates greatly.
The deviation of the results from the ground truth comes from the accumulated errors from all time slots. For the previous experiments, we also plot the errors in every time slot in Fig. 9
a. SmartLoc system using all detected landmarks for the calibration has the smallest mean error of estimated location for all time slots: 90 % of them are lower than 20 m from the CDF in Fig. 9
b. The other two approaches have larger errors. Figure 9
c describes the CDF of the total driving distance errors.
Evaluation analysis
Based on the evaluation results presented in this section, an obvious conclusion is that SmartLoc could provide precise driving distance estimation in certain scenarios. In every time slot, the driving distance is estimated through current sensory data and parameters from linear regression. Suppose the error (denoted as D
i
) in the estimation of each time slot i follows normal distribution: D
i
∼N(μ,σ
2), with mean μ and variance σ
2. Then, the estimation of total traveling distance S
t
in a period of t timeslots is bounded by the summation of traveling distance in all time slots: \(S_{t} = {\sum \limits _{i=1}^{t} D_{i}}\). In this case, the error, from a long-term perspective, will be accumulated. Obviously, S
t
∼N(t
μ,t
σ
2). The variance of the variable S
t
will be t
σ
2. Thus, the mean error will often increase along the time, which leads to the conclusion that it is difficult to predict the traveling distance precisely in the long term, although sometimes the deviation in some continuous timeslots may be neutralized. For a given error bound δ, P
r(S
t
≥δ) is higher when t is larger.
Localization in the city
We then present the localization results of SmartLoc in Chicago downtown. As aforementioned, it is difficult to get the ground truth for the majority of the sampling locations. We set the goal of localization is to find the final location. Since, Section 3 has demonstrated that there are nine bad road segments with lengths over 400 m, which is less than three blocks in downtown Chicago. The goal of SmartLoc is then to obtain a relatively accurate location estimation within three blocks. We randomly select 100 points as destinations in the experiment, and a destination could be one block, two blocks, or three blocks away from the starting point. We drive to these destination points to evaluate if the location of the destination can be precisely calculated by SmartLoc. We assume that the GPS signals are good before the starting point, and SmartLoc trains the dead-reckoning model during the driving. In this experiment, we test the accuracy of estimating the location of each sampling point in every time slot and of estimating the location of the final destination, as shown in Fig. 10
a, b, respectively. When SmartLoc only navigates to the destination within one block, with probability 70 %, the error of estimating the location for each sampling slot is less than 10 m, and with probability 85 %, the mean error is less than 30 m. When the destination is two blocks away, about 75 % of the errors are less than 30 m; when the destination is three blocks away, about 80 % errors are less than 50 m. From these results, the error of destination locating within a few blocks is acceptable. We also plot the localization results for one road segment with length over 6400 m in Fig. 11. In this figure, the red spots denote the ground truth generated from GPS, and the blue spots represent the localization calculated by SmartLoc, where the green line between them is the localization error for every location. It is clear that with longer segment the accuracy of localization is pretty good.
Energy consumption analysis
SmartLoc provides the location through both GPS and inertial sensors. Empirically, the profiles of energy consumption for GPS module and inertial sensors in smartphones are different. The average electric current of GPS module in Android smartphone is approximately 135 mA, which is larger than that of motion sensors [54]. In addition, Android API provides four rate modes of delivering sensory data, which are Fastest, Game, UI and Normal. The Fastest mode delivers the sensory data without any delay; thus, the energy consumption is the largest (95 mA [54]), while the Normal is the slowest, and also the most energy efficient one (15 mA [54]). Clearly, SmartLoc can reduce the energy consumption by leveraging inertial sensors instead of GPS compared with the approach using only GPS.
Suppose the energy cost of a GPS module for every time slot is E
GPS
, and the energy cost of inertial sensors are E
AMG
. Obviously, E
AMG
≤E
GPS
. Generally, the main purpose of the GPS is calibration and training, as shown in Fig. 12, where GPS indicates the localization through the GPS, and AMG denotes the location estimation through motion sensors. Initially, during T
1, the system will train the model and calculate suitable parameters through the linear regression, and such parameters will be employed in estimating the trajectory in T
2. Periodically, SmartLoc will calibrate the location from dead-reckoning when GPS signal becomes good again in T
3, and the parameters will be updated. Then the ratio of saved energy will be \(\frac {(E_{GPS} - E_{AMG})\cdot {T_{2}}}{E_{GPS}\cdot {(T_{2} + T_{3})}}\). The durations of T
1, T
2, and T
3 will affect the acceptable accuracy and energy consumption of localization of SmartLoc. Our first experiment is taken under the circumstance of activating all the motion sensors and GPS modules, the driving duration lasts for 30 min, and the battery of smartphone discharges from 84 to 70 %. Similarly, GPS localization for half an hour will lead to the drop of battery from 70 to 61 %. In most of our cases, in order to get higher accuracy, the ratio between training duration and estimating duration is ranging from 6:3 to 8:3. For example, in the experiment shown in Fig. 8
c, the training distance is about 3400 m and SmartLoc estimates the following 1200 m (with GPS off), which reduces the energy by approximately 10 % without affecting the localization accuracy. Note that as long as the GPS signal is strong during T
1 and T
3, we can turn off GPS in T
2. If GPS signal is weak in T
1 or T
3, we can keep the GPS alive for longer periods. Another possible working pattern is turning the GPS on/off according to the condition of the GPS signal. When GPS signal is good, SmartLoc turns the GPS to calibrate the system parameters, turns off the GPS when signal is weak, and then use the newly updated parameters to calculate the driving speed and distance.