Conventional incentive mechanisms were designed for business environments involving static business processes and a limited number of actors. They are not easily applicable to crowdsourcing and other social computing platforms, characterized by dynamic collaboration patterns and high numbers of actors, because the effects of incentives in these environments are often unforeseen and more costly than in a well-controlled environment of a traditional company. In this paper we investigate how to design and calibrate incentive schemes for crowdsourcing processes by simulating joint effects of a combination of different participation and incentive mechanisms applied to a working crowd. More specifically, we present a simulation model of incentive schemes and evaluate it on a relevant real-world scenario. We show how the model is used to simulate different compositions of incentive mechanisms and model parameters, and how these choices influence the costs on the system provider side and the number of malicious workers.

1. 21st International Conference on Cooperative Information Systems (CoopIS’13)
September 11-13, 2013, Graz, Austria
Ognjen Scekic, Christoph Dorn, Schahram Dustdar
Distributed Systems Group
Vienna University of Technology
http://dsg.tuwien.ac.at
Simulation-Based Modeling and Evaluation
of Incentive Schemes in Crowdsourcing

2. 2 CoopIS’13
Outline
 Incentives in Crowdsourcing today and tomorrow
 Problems with evaluating incentives
 Our approach
– Simulation model
– Simulation methodology & tools
 Real-world scenario example
 Conclusion and Outlook

3. 3 CoopIS’13
Incentives & Rewards
• Incentives
Stimulate (motivate) or discourage
certain worker activities before the
actual execution of those activities.
• Rewards
Any kind of recompense for worthy
services rendered or retribution for
wrongdoing exerted upon workers
after the completion of activity.
• Incentive Mechanism
A plan (rule) for assigning rewards.

4. 4 CoopIS’13
Evolution of Crowdsourcing
Conventional workflows
• formal description
• structured execution
• predefined roles and activities
• complex tasks
Crowdsourcing
• simple tasks
• anonymous replaceable actors
• short, unstructured interactions
• No interaction/collaboration
among actors
+
=
Socio-technical Collective Adaptive Systems
• ad-hoc assembled teams
• complex tasks
• social orchestration
• indirect adaptation

5. 5 CoopIS’13
 Incentive schemes can be built
by composing and customizing
well-known incentive elements.
 Programmable incentive management
 Portable, reusable, scalable incentives.
 Problem:
Composition  evaluation complexity
– How to prevent malicious workers?
– How to anticipate free riding, multitasking,
tragedy of the commons?
– How to assess appropriate reward amounts
Modeling Incentives – Problems

6. 6 CoopIS’13
 Need a systematic approach in designing and evaluating
incentive schemes before deployment on real systems.
 How to select, customize and evaluate appropriate atomic
incentive mechanisms and how to compose them for a given
crowdsourcing scenario?
 We present:
– Simulation model of incentive mechanism
– Modeling and simulation methodology for approximate
estimation of the composition of incentive mechanisms.
Contributions

7. 7 CoopIS’13
1) Mathematical incentive models
(e.g., principal-agent theory, game theory)
2) Empirical evaluation
Existing Evaluation Approaches
PRO CON
precise and reliable related to particular collaboration
patterns, cannot handle unforeseen
runtime changes
PRO CON
good for evaluating simple existing
incentives and behavioral responses
impossibility to isolate particular
mechanisms and their effects, or
causes of behavior in complex cases;
platform limitations (e.g.,
communication channels, predefined
incentives and metrics);

8. 8 CoopIS’13
3) Experimental evaluation
(e.g., on micro-task platforms, with students, volunteers)
Existing Evaluation Approaches
PRO CON
controlled environment and
reproducible setups
platform limitations (e.g.,
communication channels, predefined
incentives and metrics);
limited monetary funds may derive
skewed results;
working with people inherently willing
or forced to perform work may derive
skewed results

9. 9 CoopIS’13
 Offer methodology for quickly selecting, composing and
customizing existing incentive mechanisms.
 Roughly predicting effects of composition in dynamic
crowdsourcing environments.
 Model and simulation parameters can be changed dynamically,
allowing quick testing of different incentive scheme setups and
behavioral responses at low cost.
 Modeling of incentives and responses of arbitrary complexity.
 We do not devise novel nor optimal incentive mechanisms!
Our approach

10. 10 CoopIS’13
Simulation Model of Incentive Mechanism
Decision-making function fa considers:
1) the statistically or intentionally determined
personality of the worker St
2) historical records of past actions {S0, … , St-1}
3) authority’s view of worker’s performance Mj
4) performance of other workers {Mk}, k ≠ j
5) promised rewards R
Incentive mechanism IM considers:
1) current state of artifact Ki
2) the current performance metrics of
worker Mj
3) output from another incentive mechanism
returning the same type of reward R′ak

11. 11 CoopIS’13
 True power of incentives  composition of incentive mechanisms
 Two basic operators on incentive mechanisms:
– addition (+) and functional composition ()
– operate on common metrics
– final metrics’ values advertised to workers represent the promised reward
 Major difficulty in designing successful incentive strategies is to
properly choose performance metrics, basic incentive
mechanisms and the proper composition.
Simulating Complex Incentive Strategies

12. 12 CoopIS’13
Simulation Methodology
1) Defining domain-specific meta-model by extending core meta-model
2) Capturing worker behavioral patterns and reward calculation into executable model
3) Defining scenarios, assumptions, and configurations for individual simulation runs
4) Evaluating and interpreting simulation results
1 2 3 4

13. 13 CoopIS’13
Simulation Methodology
1) Defining domain-specific meta-model by extending core meta-model
2) Capturing worker behavioral patterns and reward calculation into executable model
3) Defining scenarios, assumptions, and configurations for individual simulation runs
4) Evaluating and interpreting simulation results
1 2
DomainPro
Designer
*www.quandarypeak.com

14. 14 CoopIS’13
Simulation Methodology
1) Defining domain-specific meta-model by extending core meta-model
2) Capturing worker behavioral patterns and reward calculation into executable model
3) Defining scenarios, assumptions, and configurations for individual simulation runs
4) Evaluating and interpreting simulation results
3
DomainPro Analyst
4
*www.quandarypeak.com

15. 15 CoopIS’13
 Simulation model of a realistic scenario, inspired by
– Citizen-driven traffic reporting (SmartJourney – Aberdeen)
– Crowdsourced software testing
 Generalized scenario:
– Entities: Authority, Workers, Situations, Reports
– Activities: Submit, Improve, Rate, Report duplicates
– Metrics: Reputation (for trustworthiness), Points (for productivity)
– Incentives: Three incentive mechanisms:
 IM1 – fixed amounts of points per activity
 IM2 – points related with report quality
 IM3 – users are assigned reputation based on past activities
Evaluation

16. 16 CoopIS’13
 Composite Incentive Schemes (CIS) evaluated:
 3 Experiments:
– Exp1: Compare impact of CIS1, CIS2, and CIS3 on authority cost.
– Exp2: Analyze effects of having too few or too many
workers per situation
– Exp3: Evaluate effects of malicious workers (0-50%) on cost.
Evaluation

17. 17 CoopIS’13
 CIS3 most reasonable to use. Can cope well with up to
20% of malicious workers.
Evaluation Results – Example

18. 18 CoopIS’13
 Presented a methodology for modeling and simulating
incentives in crowdsourcing environments.
 Advantages:
– Useful for quick, runtime, approximate evaluations
of different compositions of incentive mechanisms.
 Drawbacks:
– Inconclusive results. See: Advantages
 Future Work:
– Devise suitable models for more complex socio-technical systems.
Conclusion & Outlook

19. 21st International Conference on Cooperative Information Systems (CoopIS’13)
September 11-13, 2013, Graz, Austria
Ognjen Scekic, Christoph Dorn, Schahram Dustdar
Distributed Systems Group
Vienna University of Technology
http://dsg.tuwien.ac.at
Thank you!
Questions?
21st International Conference on Cooperative Information Systems (CoopIS’13)
September 11-13, 2013, Graz, Austria