ISWC 2010: Tutoria: ten ways to make your semantic app addictive

1. Methods and techniques to
analyze and design incentivized
semantic applications
www.insemtives.eu
Roberta Cuel, University of Trento, IT and
Markus Rohde, University of Siegen, DE
ISWC 2010

2. 1/29/2015 www.insemtives.eu 2
Why is my app not as successful as
Facebook…
1/29/2015 www.insemtives.eu 2
• Motivation and incentives
– Reciprocity
– Reputation
– Competition
– Altruism
– Self-esteem
– Fun
– Money
• This session is about the methods
and techniques you can use
– To study your scenario prior to
application design
– To evaluate your incentives strategy
and adjust your application

3. Overview
• Basic notions and definitions
• Methods and techniques for analysis, design
and evaluation
• Case studies
1/29/2015 www.insemtives.eu 3

4. BASIC NOTIONS AND DEFINITIONS
Basic notions and definitions
Methods and techniques for analysis, design and evaluation
Case studies
1/29/2015 www.insemtives.eu 4

5. Starting with motivation…
• Basic tenets of organizational behavior
– Performance : f(ability*motivation)
1/29/2015 www.insemtives.eu 5

6. Motivation and incentives
• Intrinsic vs extrinsic motivations
• Incentives are ‘rewards’ assigned to performer
to make him/her commit
– Can be totally uncorrelated to the nature of the
task
– Need to be compliant with the values and beliefs
of the acting person
1/29/2015 www.insemtives.eu 61/29/2015 www.insemtives.eu 6

7. Example: DBPedia
The DBpedia community collaborates using
the following tools
– Mailing lists, Facebook group
– Blog, quality assurance and bug tracking
– Development
• “The framework is easily extended and we offer the possibility to do this
yourself, increasing the likelihood of your desired data being included
in the next DBpedia release.”
– DBpedia consulting (money)
1/29/2015 www.insemtives.eu 7

8. 1/29/2015 www.insemtives.eu 8
Typology of motivations
Motivations Internal
(embedded in structure,
e.g., task, tools)
External
(additional to structure,
external re-inforcements)
Intrinsic
(predispositioned in
person, e.g., drives,
needs, desires )
Fun, joy, gaming,
interest, satisfaction,
self-actualization, self-re-
inforcement
Social appreciation,
reputation, love, trust,
social capital, community
support
Extrinsic
(additional to personal
predispositions, extern re-
inforcements )
Usability, sociability,
Design-for-fun, curiosity,
community-building
support
Material/financial capital,
money, rewards, prices,
medals, credit points
Structure
Person
Example: FLOSS software (Ghosh & Prakash, in Lerner & Tirole, 2005)

9. METHODS AND TECHNIQUES FOR
ANALYSIS, DESIGN AND EVALUATION
Basic notions and definitions
Methods and techniques for analysis, design and evaluation
Case studies
1/29/2015 www.insemtives.eu 9

10. Overview of methods and
techniques
• To analyze your scenario and design your
application
– Game theory, mechanism design
– Participatory design and end-user development
• To evaluate an incentives strategy in its context
– Interviews and questionnaires and field experiments
– Heuristic evaluation, guideline reviews
– Cognitive and pluralistic walk-throughs
– Participatory/User evaluation
1/29/2015 www.insemtives.eu 101/29/2015 www.insemtives.eu 10

11. Game theory
• Game theory is a formal way to analyze interaction among a
number of rational agents who behave strategically
– The rational agents: players involved in the situation (best choice)
– A number of players: more than one
– Rationality/payoffs: what are the players’ preferences over the
outcomes of the game
– The interactions: one player’s behavior affects another
– The rules: who moves when, what do they know, what can they do
– The outcomes: what is the outcome of the game (for each move)
1/29/2015 www.insemtives.eu 11

12. Mechanism design
• Mechanism design is about how to translate game
theory in effective behavior
– To design rules such that a desired set of outcomes
happens
– Alignment of interests between parties and production of
maximum social welfare
• Relevant variables
– Structural and personal motivations
– Goal and nature of good produced
– Tasks and skills required
– Social structure
1/29/2015 www.insemtives.eu 12

13. Analysis matrix
• A multidimensional tool that allows for an
analysis of a particular task with respect to the
variables relevant for mechanism design
1/29/2015 www.insemtives.eu 13
Goal Tasks Social structure Nature of good
produced
Skill
variety/level
Communication
level
Variety of Hierarchy-
neutral
Private good Skilled ability
Participation
level
Specificity of Hierarchical Public good Competence
Clearness level Identification
with
Common
resource
Club good

14. 1/29/2015 www.insemtives.eu 14
The matrix in practice: semantic
annotation
• Goal of the annotation or ontology population exercise
• Task, or more typically, an ordered collection of tasks into
which the annotation exercise can be broken down
• Social structure, a stylized and simplified set of social
relationships among the subjects participating in the
exercise
• Nature of good, a stylized description, in game-theoretical
terms, of the relationship between what good is produced
and who consumes it
• Required skills of the agents to complete the annotation
task
1/29/2015 www.insemtives.eu 14

15. The matrix in practice: crowd-sourcing
of ontology evolution
• Goal: communication and participation level
– Coordination (free-open)
• Interests are already aligned (free riding)
• Private and public benefit coincide
• Task: variety vs specificity
• Social structure: hierarchy neutral (large vs small
groups)
• Nature of good: public good non-rival (use does not
limit use of others), non-exclusive (open access)
• Skill variety/level: Skilled ability
1/29/2015 www.insemtives.eu 151/29/2015 www.insemtives.eu 15

16. Participatory Design (PD)
• To develop software for the participation of end-users, we
propose a participatory way of designing these software
tools, integrating potential users by participatory design
methods
• Participatory Design
– Improvement of the participation of workers in software development
processes
– The cooperation between software developers and end-users
– Participate in IT development projects as experts of their own work
1/29/2015 www.insemtives.eu 16

17. 1/29/2015 www.insemtives.eu 17
Methods for PD
• Participatory-design projects combine
– Design-by-doing methods
– PD workshops
– Scenarios
– Different forms of prototyping (mockups, rapid
prototypes)
– Work organization games
– Ethnographic methods

18. 1/29/2015 www.insemtives.eu 18
End-User Development (EUD)
• EUD is about designing highly flexible systems that enable
users to participate during the use of the system by
adapting and modifying the tools according to their
needs/preferences
• Goal: Empower end-users to develop and adapt systems
themselves by designing these systems to be easy to
– Understand
– Learn
– Use
– Teach

19. 1/29/2015 www.insemtives.eu 19
How to design an incentivized
application
• Ideally: field  desk  lab  field
• Analyze the domain and find yourselves in the matrixes
– Find the relevant point of that situation (goal and tasks)
– Focus on a small group of individuals (social structure)
– Analyze their motivation (internal/external intrinsic/extrinsic)
– Analyze the other relevant variables (nature of good being
produced, kill variety/level)
• Design a simplest possible model that can effectively
support contributors
• Test and get feedback
• Fine-tune the experiment and add other elements

20. 1/29/2015 www.insemtives.eu 20
Methods
• Domain observations (second-hand data)
• Ethnography or qualitative face-to-face interviews
• Questionnaires
• Observations with selected individuals
• Quantitative analysis (data collections)
• Usability-design methods
1/29/2015 www.insemtives.eu 20

21. CASE STUDIES
Basic notions and definitions
Methods and techniques for analysis, design and evaluation
Case studies
1/29/2015 www.insemtives.eu 21

22. Examples from our project
• Telefónica Investigación y Desarrollo (TID - Spain): Knowledge
Intranet Platform
• Pepper’s Ghost (PGP-UK): Virtual Games/Worlds
• Seekda! (Austria): Web Services Search Portal
1/29/2015 www.insemtives.eu 22

23. 1/29/2015 www.insemtives.eu 23
Field work: TID
• Domain analysis
– Site visit, semi-structured, qualitative interviews
• Communication processes
• Existing usage practices and problems
• Existing tools/solutions
• Semantic annotation solutions
– Tape recording, transcription
– Data analysis per ex-post categorization
• Focus group discussion
– Usability lab tests
– Expert walkthroughs

24. 1/29/2015 www.insemtives.eu 24
Field work: TID (2)
1/29/2015 www.insemtives.eu 24
• Find TID in the matrix
– Goal: communication and participation (contribute to the
knowledge portal)
– Social structure (various structures co-exist)
• Strongly hierarchical organization (control is an issue)
• working groups and community of experts
– Nature of good: public good vs. private, club goods
– Skill variety/level: Skilled ability (knowledge workers)
– Motivations: fun, visibility, reputation, promotion, money

25. 1/29/2015 www.insemtives.eu 25
Field work: PGP
• Domain analysis
– Data analysis (Benchmark: Galaxyzoo, Moonzoo)
– Usability lab tests and expert walkthroughs
– Collect data on users such as logs files
• Find PGP in the matrix
– Goal: participation level
– Task: specific
– Social structure: hierarchy neutral
– Nature of good: public good non-rival (use does not limit
use of others), non-exclusive (open access)
– Skill variety/level: non skilled ability

26. 1/29/2015 www.insemtives.eu 26
Field work: SEEKDA
• Domain analysis
– Interviews and workshops
– Replicating Telefonica model
• Find SEEKDA in the matrix:
– Goal: communication and participation level
– Task: variety vs. specificity
– Social structure: hierarchy neutral (mainly)
– Nature of good: public good non-rival (use does not limit
use of others), exclusive, or club good
– Skill variety/level: Skilled ability

27. Thank you
Any question?
1/29/2015 www.insemtives.eu 27