A privacy-preserving exception handling approach for dynamic mobile crowdsourcing applications

Abstract

The ever-increasing popularity of mobile devices (e.g., mobile phones and smart watches) has created a variety of crowdsourcing applications by employing the massive and distributed mobile computing resources. Typically, a task requester sends his/her task request and constraint conditions to a crowdsourcing platform, and then the crowdsourcing platform is responsible for finding a set of appropriate workers (e.g., mobile users) from massive candidates to satisfy the task request. However, for a mobile crowdsourcing task being executed by a set of workers, a pre-selected worker may become unavailable due to various exceptions. In this situation, it is significant for the crowdsourcing platform to quickly find another similar worker to replace the unavailable worker so as to smooth the crowdsourcing process. However, the above exception handling process is often challenging as candidate workers are often not willing to release their sensitive information to the platform due to privacy concerns. In view of this challenge, in this paper, a novel privacy-preserving exception handling approach, named ExH_Simhash, is put forward based on Simhash technique. Finally, through a set of simulated experiments, we validate the feasibility of ExH_Simhash in terms of substitution equivalence and computational time.

1 Introduction

With the ever-increasing popularity of mobile computing techniques in daily life, people are apt to execute their business applications or complete their computing tasks through various mobile devices, such as mobile phones, smart watches, and laptops [1,2,3,4,5,6,7,8]. By utilizing the lightweight and smart mobile computing capabilities, a user can complete his/her complex computing tasks that cannot be executed traditionally by a single device or a single person, which gives birth to the mobile crowdsourcing technique. For example, if a user hopes to know about the real-time traffic condition of a routine that he/she is interested in, he/she can employ a set of mobile users to take some real-time pictures of the places near the routine. This way, on the one hand, the user can get his/her needed traffic data or information quickly; on the other hand, the mobile users that are employed for task execution can get a certain reward from the employer.

Generally, as Fig. 1 shows, the mobile crowdsourcing model consists of three roles: task requesters, candidate workers, and crowdsourcing platform. Typically, a task requester sends his/her outsourcing task to the crowdsourcing platform stating the task content (e.g., “take a set of pictures about the traffic condition of a route”) and constraint conditions (e.g., deadline, fees); candidate workers are the users who are willing to utilize their mobile devices to satisfy the outsourcing task from the task requester; the crowdsourcing platform receives the outsourcing tasks from task requesters and publish them to the candidate workers. If a worker meets the functional and non-functional requirements of the task simultaneously, then the crowdsourcing platform binds the requester and the worker; namely, an outsourcing contract is created between the two parties.

Fig. 1

Mobile crowdsourcing model

However, in the dynamic crowdsourcing environment, a previously selected worker may become unavailable for the crowdsourcing task due to various exceptions. For example, a worker who agreed to take some photos for the traffic conditions required by a task requester may become unavailable due to the low quantity of electric charge of his/her mobile phone. In this situation, for the task requester, it is necessary to quickly find another qualified worker who is similar with the unavailable worker to replace him/her, so as to smooth the normal execution of the mobile crowdsourcing task. However, the above similar worker search process is often a challenging problem, because the candidate workers are often not willing to release or publish their personal information to the crowdsourcing platform due to privacy concerns. For example, a worker is probably reluctant to tell the crowdsourcing platform that he/she has ever undertaken an outsourcing task “take some photos for a beautiful girl without her permission”.

In view of the abovementioned challenge, a time-efficient and privacy-preserving data protection technique, i.e., Simhash [9], is introduced into the exception handling process of mobile crowdsourcing applications incurred by the unavailable workers. Furthermore, we put forward a novel exception handling approach based on Simhash, named ExH_Simhash to enhance the robustness of traditional mobile crowdsourcing techniques.

Concretely, the major contributions of this paper are summarized as follows:

• 1. We recognize the probable mobile crowdsourcing exceptions caused by the unavailability of previously selected workers. The exceptions decrease the robustness of the crowdsourcing approaches.

• 2. We introduce the time-efficient and privacy-preserving data protection technique, i.e., Simhash into the mobile crowdsourcing process and further put forward a novel approach named ExH_Simhash, so as to handle the exceptions incurred by the unavailable workers.

• 3. We conduct a set of simulated experiments based on the well-known WS-DREAM dataset. Experiment results show that our suggested ExH_Simhash approach performs better than other competitive approaches, especially in terms of substitution equivalence and computational time.

The reminder of this paper is structured as below. Related work is briefly introduced in Section 2. In Section 3, we formulate the privacy-preserving exception handling problems in mobile crowdsourcing applications. The suggested exception handling approach, i.e., ExH_Simhash, is specified in Section 4. In Section 5, we designed, deployed, and tested a set of experiments to prove the effectiveness and efficiency of ExH_Simhash. At last, in Section 6, we summarize the whole paper and discuss the potential improvement directions in the future research.

2 Related work

In recent years, mobile crowdsourcing technique has been introduced to satisfy the complex requirements that cannot be solved by the traditional techniques. During the mobile crowdsourcing process, selecting proper workers for the crowdsourcing task is crucial for providing better services. Many researchers have investigated this hot topic and provided their resolutions. In this section, we introduce the related work from the following two aspects.

2.1 Ability-based worker selection in crowdsourcing

The ability values of candidate workers play an important role in worker selection for mobile crowdsourcing applications. In the outsourcing platform Amazon Mechanical Turk, the candidate workers are evaluated, compared, and selected for an outsourcing task based on their ability values, e.g., worker reputation and worker skills. In [10], a self-organized outsourcing toolkit, i.e., Crowdlet is proposed to employ a set of workers to execute the crowdsourcing task from the requester. In Crowdlet, a candidate worker is selected or not depends on the worker’s service quality levels which are influenced by the skills, arrival time, and rewards of the worker. While the reputation values and mobile device performance are often neglected. In [11], the authors suggest an online crowdsourcing framework which is response for allocating appropriate workers to crowdsourcing tasks based on the important information from both workers and requesters. The final goal is to maximize the utility function defined in the crowdsourcing framework. While the utility function only depends on the state of network. In [12], a behavior-aware worker selection strategy is suggested to improve crowdsourcing performances, which is often self-adaptive. Typically, the strategy calculates the reputation value of a candidate worker based on the worker’s historical service quality data. However, the reputation model only considers the skills of workers without considering other elements such as device capability. In [13], a budget-aware crowdsourcing model evaluates and generates the final crowdsourcing results based on the workers’ information, such as his/her willingness, his/her reliability in executing the task. Therefore, high-quality crowdsourcing results can be expected within a certain budget. However, the decision-making basis of crowdsourcing is still not comprehensive enough.

2.2 Dynamic mobile crowdsourcing selection

An online auction mechanism is put forward in [14] to satisfy the dynamic crowdsourcing scenarios where each candidate worker can arrive at or leave the crowdsourcing platform at will. The auction mechanism can be utilized to accommodate the dynamic arrivals of both candidate workers and crowdsourcing tasks. However, it is often hard to get the optimal crowdsourcing performance through auction. In [15], an incentive mechanism is brought forth to evaluate and rank the candidate workers for a certain crowdsourcing task based on the workers’ context information, e.g., location and time. However, in this work, the proposed mechanism does not consider the ability values of workers. In [16], the authors put forward an online double auction strategy between candidate workers and task requesters. The proposal can guarantee to achieve better crowdsourcing results than the simple one-side interaction between workers and requesters. An online learning method is brought forth in [17] to produce the optimal pricing strategy and incentive mechanism, whose final goal is regret minimization. In [18], two online auction mechanisms are suggested to accommodate the dynamic arrival of candidate workers; the final optimization goal is to recruit enough workers to maximize the defined utility value before deadline constraint. In [19], the candidate workers are encouraged to publish their configuration files to the crowdsourcing platform; this way, the platform can reduce the budget while selecting enough workers to execute a crowdsourcing task. However, the abovementioned auction strategies in [17,18,19] are often not efficient enough to satisfy the quick response requirements from the task requesters in mobile crowdsourcing applications. Moreover, additional auction burdens are possible for both the candidate workers and the task requesters.

Through the above analyses, a conclusion can be drawn that existing mobile crowdsourcing approaches seldom consider the probable exceptions (as well as the resulted privacy leakage in similar worker search and replacement) caused by worker unavailability in the dynamic crowdsourcing environment. Considering this drawback, a privacy-preserving exception handling approach for dynamic mobile crowdsourcing is proposed in this paper, to smooth the normal execution of crowdsourcing process and improve the crowdsourcing robustness. The details of our exception handling approach will be introduced in Section 4.

3 Formulation

In this section, we formulate the mobile crowdsourcing problems and introduce the symbols to be used in the following discussions of this paper.

• 1. Requester is a user who outsources his/her tasks to a crowdsourcing platform.

• 2. WORKER = {worker₁, …, worker_m} denotes the candidate worker set.

• 3. CP is a crowdsourcing platform that receives tasks from Requester and publish or advertise them to the candidate workers in set WORKER.

• 4. TASK = {task₁, …, task_n} denotes the task set in historical crowdsourcing applications.

• 5. For worker_i (i = 1, …, m), his/her ever-executed crowdsourcing task set can be specified with an n-dimensional vector V_i = (v_i,1, …, v_i,n), where v_i,j denotes whether worker_i has ever executed task t_j: if the answer is yes, then v_i,j = 1; otherwise, v_i,j = 0.

With the above definitions, we can further specify the dynamic mobile crowdsourcing problem formally: the crowdsourcing platform CP selects a set of qualified workers from candidate set WORKER to satisfy the requirements of a task (from Requester) in set TASK. Furthermore, the privacy-preserving exception handling problem can be specified as below: if a previously selected worker worker_excep becomes unavailable due to some exceptions, the crowdsourcing platform CP should quickly find another candidate worker worker_x who is similar with worker_excep, and meanwhile guarantee that the sensitive information contained in vectors V_i (1 ≤ i ≤ m) is still secure.

4 Exception handling for mobile crowdsourcing based on Simhash

In this section, we introduce the details of our suggested exception handling approach for mobile crowdsourcing, i.e., ExH_Simhash. The main idea of ExH_Simhash is: when the previously selected worker_excep turns unavailable, the crowdsourcing platform CP utilizes Simhash technique to search for the workers (denoted by set WORKER_X) similar with worker_excep in an efficient and privacy-preserving manner, and utilizes the optimal worker_optimal (∈WORKER_X) to replace the unavailable worker_excep.

Concretely, ExH_Simhash consists of the following three steps.

• Step 1: Building worker indices offline based on Simhash. According to the historical crowdsourcing tasks ever-executed by workers, build worker indices offline based on Simhash.

• Step 2: Search for the similar workers of unavailable worker. For the unavailable worker worker_excep, search for his/her similar workers worker_x based on the indices built in step 1.

• Step 3: Optimal worker selection and replacement. From all the similar worker_x obtained in step 2, we select an optimal one based on the service quality and utilize him/her to replace the unavailable worker_excep.

4.1 Step 1: building worker indices offline based on Simhash

As Section 3 introduces, if worker_i (∈WORKER) has ever executed the crowdsourcing task task_j (∈TASK), then the value (indicating whether a worker has ever executed a task) for worker_i and task_j (denoted by v_i,j in this paper) is equal to 1; else, v_i,j = 0. Afterwards, an m*n matrix is obtained as in Eq. (1), where each entry is a Boolean value. Furthermore, the ith row of the matrix, i.e., V_i = (v_i,1, …, v_i,n) represents the historical task execution records of worker_i. As we introduce in Section 1, V_i should be protected by the crowdsourcing platform CP as V_i often contains certain sensitive information of worker_i.

$$ \left[\begin{array}{ccc}{v}_{1,1}& \dots & {v}_{1,n}\\ {}\vdots & \ddots & \vdots \\ {}{v}_{m,1}& \dots & {v}_{m,n}\end{array}\right] $$

(1)

Next, we introduce how to transform the sensitive V_i (1 ≤ i ≤ m) data into the less sensitive hash value H(V_i) through the Simhash technique. Here, we set parameter r = $\left\lceil {\mathit{\log}}_2^n\right\rceil$. Thus each crowdsourcing task can be depicted by an r-dimensional vector. We demonstrate the calculation process with the following example. Assume there are 100 tasks and worker_i has ever executed ten tasks (i.e., task₁, …, task₁₀) in the past, then the ten tasks can be denoted by the following seven-dimensional vectors, respectively.

$$ {\displaystyle \begin{array}{l}{task}_1=\left(0,0,0,0,0,0,1\right)\\ {}{task}_2=\left(0,0,0,0,0,1,0\right)\\ {}\dots \\ {}{task}_{10}=\left(0,0,0,1,0,1,0\right)\end{array}} $$

Afterwards, for the above ten seven-dimensional vectors, we replace the element “0” by “− 1” (for example, task₁₀ is transformed from (0, 0, 0, 1, 0, 1, 0) to (− 1, − 1, − 1, 1, − 1, 1, − 1)) and then count the sum of each column of the matrix constituted by the above ten seven-dimensional vectors. Afterwards, we derive a seven-dimensional sum vector W as presented in (2). In (2), the negative values are substituted by “0”, the positive values are substituted by “1”. Afterwards, we obtain the new vector W′ in (3). In our suggested ExH_Simhash approach, W′ is regarded as the index value of worker_i, i.e., H(V_i) = W′ holds. Thus through the above process, we transform the original 100-dimensional vector for worker_i, i.e., V_i into a seven-dimensional worker index H(V_i). Therefore, the data scale is reduced significantly. Besides, the index values H(V_i) employed for similar worker search are less sensitive; therefore, the workers’ private information is secure.

$$ W=\left(-10,-10,-10,-4,-2,0,0\right) $$

(2)

$$ {W}^{\prime }=\left(0,0,0,0,0,0,0\right) $$

(3)

As the historical worker-task execution records are already recorded in the crowdsourcing platform CP, the worker indices building job can be finished offline. Therefore, the efficiency of subsequent exception handling process is improved significantly.

4.2 Step 2: search for the similar workers of unavailable worker

Next, we utilize the worker indices derived in step 1 to search for the similar workers of the unavailable worker worker_excep. Concretely, as formulated in step 1, H(V_excep) = (p₁, …, p_r) and H(V_i) = (q₁, …, q_r). Thus their Hamming distance d(H(V_excep), H(V_i)) can be calculated by (4), where “x” means XOR. According to the nature of Simhash [9], if condition d(H(V_excep), H(V_i)) < 3, we can conclude that the tasks executed by worker_i are approximately equal to the tasks executed by worker_excep; namely, worker_i and worker_excep are similar with high probability. Here, we utilize WORKER_X to denote the similar worker set of the unavailable worker_excep; namely, the equation in (5) holds.

$$ d\left(H\left({V}_{excep}\right),H\left({V}_i\right)\right)=\left({p}_1\mathrm{x}{q}_1,{p}_2\mathrm{x}{q}_2,\dots, {p}_r\mathrm{x}{q}_r\right) $$

(4)

$$ {WORKER}_X=\left\{{worker}_i\ |\ {worker}_i\ \mathrm{is}\ \mathrm{similar}\ \mathrm{with}\ {worker}_{excep}\right\} $$

(5)

4.3 Step 3: optimal worker selection and replacement

In step 2, we have obtained the similar workers of the unavailable worker_excep, i.e., WORKER_X. The set WORKER_X may contain multiple elements and each element is a qualified worker who can replace the unavailable worker_excep. Next, we evaluate and select an optimal candidate worker from set WORKER_X.

Concretely, for each worker_x in set WORKER_X, we calculate his/her Hamming distance with worker_excep, i.e., d(H(V_excep), H(V_x)). According to the Simhash theory, a smaller Hamming distance d(H(V_excep), H(V_x)) often means a larger similarity between worker_x and worker_excep. Therefore, we select the optimal (i.e., the most similar) worker, denoted by worker_optimal, from set WORKER_X according to the Hamming distance; in other words, the equation in (6) holds. Specifically, if multiple workers have the minimal Hamming distance, then the crowdsourcing platform CP selects a worker randomly and utilizes the worker to replace the unavailable worker_excep. This way, the exception caused by the unavailable worker is handled successfully.

$$ {worker}_{optimal}=\left\{{worker}_x\ |\ {worker}_x\in {WORKER}_X\ \mathrm{and}\ d\left(H\left({V}_{excep}\right),H\left({V}_x\right)\right)\ \mathrm{is}\ \mathrm{the}\ \mathrm{minimal}\right\} $$

(6)

5 Experiments

5.1 Experiment dataset and environment

We conduct a set of experiments based on the WS-DREAM [20] dataset to test the performances of ExH_Simhash. We compare the suggested ExH_Simhash approach with three approaches, i.e., UCF (user-based CF), ICF (item-based CF), and P-UIPCC [21]. To evaluate the performances of different exception handling approaches, time cost and substitution equivalence are compared, respectively. If the ever-executed task sets of workers u and v are similar, then we can conclude that v is an ideal alternative for u when u becomes unavailable. Motivated by this hypothesis, a new criterion “substitution equivalence” (∈[0, 1], the larger the better) is defined as in (7), where A and B denote the set of tasks executed by worker_optimal and worker_excep, respectively (Fig. 2).

$$ \mathrm{substitution}\ \mathrm{equivalence}\kern0.5em =\kern0.5em \frac{\mid A\cap B\mid }{\mid A\cup B\mid}\ast 100\% $$

(7)

Fig. 2

Three steps of our exception handling approach ExH_Simhash

Our experiments are deployed on a PC with 2.40 GHz processor and 4.0 GB RAM. The operation system is Windows 7 and the programming language is JAVA. Experiments are executed ten times repeated and their average values are registered.

5.2 Experiment results

Next, three experiment profiles are designed, deployed, and tested, respectively. Please note that m and n represent the size of worker set and the size of crowdsourcing task set, respectively.

5.2.1 Profile 1: efficiency of different approaches

Here, we test the exception handling efficiency of different mobile crowdsourcing approaches. The following are the parameter settings: m = {50, 100, 150, 200, 250, 300}, n = {1000, 2000, 3000, 4000, 5000}. Concrete experiment results are presented in Fig. 3. As Fig. 3a (n = 5000 holds) and Fig. 3b (m = 300 holds) shows, the exception handling efficiencies of UCF, ICF, and P-UIPCC all decrease when m or n grows; this is because all the candidate workers and crowdsourcing tasks are recruited in the three exception handling approaches. While in ExH_Simhash, most jobs (e.g., worker indices building) can be finished offline before the crowdsourcing platform begins to handle the exception. Therefore, the exception handling efficiency of our approach performs better than the other three approaches, which means that ExH_Simhash is suitable for the mobile crowdsourcing situations when a quick exception handling response is required by the task requester.

Fig. 3

Exception handling efficiency of different approaches w.r.t. m and n. a n = 5000. b m = 300

5.2.2 Profile 2: substitution equivalence of different approaches

Substitution equivalence is an important criterion to evaluate the performance of an exception handling approach. In this profile, we measure and compare the substitution equivalence values of different exception handling approaches. The following are the parameter settings: m = {50, 100, 150, 200, 250, 300}, n = {1000, 2000, 3000, 4000, 5000}. Experiment results are demonstrated in Fig. 4.

Fig. 4

Substitution equivalence of different approaches w.r.t. m and n. a n = 5000. b m = 300

As can be seen from Fig. 4a (n = 5000 holds) and Fig. 4b (m = 300 holds), the substitution equivalence of P-UIPCC is not as high as expected, because a set of approximate strategies are adopted to achieve the privacy-preservation goal. Due to the inherent tradeoff between data privacy and availability, an increment of privacy-preservation capability often leads to a drop of substitution equivalence. While in our ExH_Simhash approach, the nature of Simhash technique can guarantee to find the workers who are similar with the unavailable worker and recruit them to replace the unavailable worker. Therefore, the substitution equivalence of the ExH_Simhash approach is improved accordingly and outperforms that of the two benchmark approaches, i.e., UCF and ICF.

5.2.3 Profile 3: successful rate of ExH _Simhash with respect to m and n

Simhash is essentially a probability-based search technique. So our suggested exception handling approach ExH_Simhash cannot guarantee a 100% successful rate. Considering this, it is necessary to measure the successful rate of ExH_Simhash. Here, the successful rate is equal to the ratio between the successful replacement times and total replacement times. The following are the concrete parameter settings: m = {50, 100, 150, 200, 250, 300}, n = {1000, 2000, 3000, 4000, 5000}. Experiment results are demonstrated in Fig. 5.

Fig. 5

Successful rate of ExH_Simhash w.r.t. m and n. a n = 5000. b m = 300

As Fig. 5 indicates, there is an approximately linear relationship between the successful rate and the parameters m and n. This is due to the fact that when m or n increases, more workers who are similar with the unavailable worker can be returned to the crowdsourcing platform as the replacement candidates; accordingly, the successful rate of exception handling approach ExH_Simhash is enhanced. Therefore, through tuning the parameters m and n, we can guarantee to maintain a relatively high successful rate in handling the exception.

5.3 Discussions

However, there are still several potential shortcomings in the experiments.

• 1. First, in the mobile crowdsourcing process, we only consider one evaluation criterion or dimension in similar worker search, i.e., whether a worker has executed a task. While multi-dimensional application scenarios are more common in practical applications [22,23,24,25,26,27,28,29,30,31,32,33,34] where each dimension is assigned a weight value to indicate the significance [35,36,37,38,39,40,41].

• 2. Second, only the dimension with Boolean values is tested without considering the diversity of data types, such as the discrete data [42,43,44,45,46,47,48,49,50,51,52], fuzzy data [53,54,55], and continuous data [56,57,58,59,60,61,62,63], as well as their integration issues.

• 3. For simplicity, crowdsourcing task execution context is assumed to be fixed in this paper; in the future, we will improve our exception handling method to be more comprehensive and reasonable by considering more context factors, e.g., time factor that is crucial in many domains [64, 65].

• 4. Finally, due to the inherent shortcoming of Simhash technique, it is really hard to measure the privacy-preservation capability of the suggested exception handling method ExH_Simhash in this paper; we will investigate this complex scientific problem in the future work.

6 Conclusions and future work

The ever-increasing popularity of mobile devices has created a variety of crowdsourcing applications by employing the massive and distributed mobile computing resources. However, in the mobile crowdsourcing process, a previously selected worker may become unavailable due to various exceptions. In this situation, it is significant for the crowdsourcing platform to quickly find another similar worker to replace the unavailable worker, in an efficient and privacy-preserving way. In view of this challenge, a novel privacy-preserving exception handling approach, named ExH_Simhash, is put forward in this paper. Finally, through a set of simulated experiments, we validate the feasibility of ExH_Simhash in terms of substitution equivalence and computational time.

In the future, we will refine our exception handling approach by considering more context factors and multiple data types. Additionally, how to enhance the successful rate of the proposed exception handling method is still another research challenge that calls for intensive study.