Mobile Survey Formats Usability

Self-Administered
Mobile Surveys
MRC 2011 Workshop (Part 1)

London (UK)
April 18th, 2011

Michael Bosnjak, PhD, Assoc. Prof.
Free University of Bozen-Bolzano, School of Economics and Management
1

2

Self-Administered Mobile Surveys?
Mobile Surveys?

• Definitions of mobile surveys
– Interviewer-administered surveys
• Interviews among mobile phone users
• Interactive voice response surveys among mobile phone
users
– Self-administered surveys
• SMS (text messaging) surveys
• Browser-based surveys on mobile devices (e.g., mobile
phones having mobile Internet-access, Smartphones,
etc.)
• Our focus:
– Self-administered surveys AND
– using a mobile phone AND
– browser-based.

3

Selected Applications

Directly at point of sale:
in shopping malls and At trade fairs
at points of service

At public
Insights In training
from difficult to seminars
venues, such
as concerts reach target
groups, event/
At schoolyards, in incident-based
surveys, In workplaces
universities &
without internet
recreational immediacy access
facilities

En-route En-route
with bus or
train & at B2C B2B with bus or
train & at
the airport the airport

Overall Goal

• Providing a very brief introduction into the
methodological foundations of self-administered
mobile surveys (esp. sources of biases known from
survey methodology)

• Summarizing key findings of an own methodological
study series conducted between 2008-2011

• Discussing practical, evidence-based
recommendations (esp. on measurement and
nonresponse issues)

Agenda

• Background
– Survey research: Overall aims and scope
– The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)

• Measurement issues
– What can be presented/assessed?
– How usable are mobile question formats?
– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?

• Nonresponse issues
– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?
– Speed of participation?
– Optimal length of mobile surveys?

• Take-home messages and discussion

6

Background: Overall Aim of Surveys
• Measuring ´true scores´, i.e. yielding unbiased
estimates for facts and/or latent variables.
– Examples of factual questions to measure facts:
• Household-level income/expense estimates > Disposable
income
• Behavioral frequency estimates > Behavior
– Examples of indicators supposed to measure latent
variables:
• Evaluative judgments > Attitudes
• Behavioral likelihood scales > Intentions
• Brand/product related attributes > Image
• Sources of errors in surveys:
• Representation-related biases: Coverage, Sampling,
Nonresponse
• Measurement-related biases/errors

7

Background: Total Survey Error

Measurement Representation
Construct Population
Measurement Coverage

Measurement
Inappropriate
operationalization
(range restriction,
Sampling Frame
reliability, validity)
Sampling
Measurement Sample
Inappropriate
implementation into Nonresponse
a speciﬁc mode:
Undesired design-
related effects Response Respondents

Representative (valid)
Representative for the
for the construct in Survey estimate population in question?
question?

8

Background: Survey Errors/Biases

• Coverage Error
Members of the target population have no chance of being
selected in the sample (e.g., no access to the Internet,
incomplete lists etc.). Error due to the fact that not every unit
in the population is represented on the frame.
• Sampling Error
... arises from the fact that not all members of the frame
population are measured.
• Nonresponse Error
The responses of people who have not been surveyed are
different from those who actually have participated in a survey.
• Measurement Error
Deviation of the answers of respondents from their true values
on the measure, e.g. due to inappropriate operationalizations
of (latent) constructs, design features and context effects.

9

Mobile Survey Methodology: Study Series
1. Web: Item development: Determinants of the willingess to
Mobile Study I
(1.7.-2.9.08) participate in mobile surveys (Sozioland Web-Panel)
2. Pre-Testing: Expert usability assessment at YOC
3. Web: Determinants of the willingess to participate S4
(YOC Mobile-Panel; 979 panelists, 272 participants)
4. Olympic Games 2008 Mobile Survey
(YOC Mobile-Panel; 979 panelists, 413 participants)
5. Web: Usability of S4 from participants´ perspective
(YOC Mobile-Panel; 413 panelists from S4, 187 completes)

Mobile Study II 6. Mobile survey: Evaluation of last vacation
(29.9.-18.10.09) (Respondi Web-Panel; 3270 panalists, 540 completes)
7. Web: Usability of S6 from participants´ perspective
(Respondi Web-Panel; 540 panelists from S6, 318
completes)

Mobile Study III 8. Usability of voice capturing/recognition technology
(March/April 2011) (presentation of results at tomorrow at MRC 2011, April 19,
2011)

10

www.mobileresearchconference.com

12

What can be presented/assessed? (I)

Single Multiple Drop-Down
choice choice menu

13

What can be presented/assessed? (II)

Matrix /
Textfield Polarity Voice / image /video
profile capturing

14

How ´usable´ are standard formats?
Subjective Usability Assessment
Observed
Post-hoc survey (Web) one week after mobile survey completion
Indicators for usability score: fluency, simplicity, ease of use
Item- Drop-
NR Out

Single choice
Einfachauswahl untereinander 89,2

Multiple choice 9%
Mehrfachauswahl untereinander 87,3
Fragetyp

Drop-Down menu
Geschlossene Auswahlliste 82,7

Textfield
Textfeld einzeilig 74,7
Voice recognition / capturing ?
45% 9%

Image mit Bild
Fragetyp map 87,9
23%
65,00 73,75 82,50 91,25 100,00
Usability score (Range: 0-100 Punkte)

Sources: MS I and MS II combined

15

Technical Implementation: iPhone App

16

Technical Implementation: Android

17

GPS positioning: Privacy concerns?

Acceptance of 9%
GPS-Location
among
participants with
an iPhone
(MS II; n=45)

91%

Yes (willing to disclose)
No (not willing to disclose)

19

Nonparticipation: Industry perceptions?
Mobile Research Barometer 2/2011

• Survey among 327 market researchers about
acceptance and use of mobile surveys in D/A/CH
• Top 3 advantages of mobile surveys:
– 51%: Independence of time/location
– 49%: Context-sensitive, fast surveys
Mobile Research
– 43%: Reachability of hard-to-reach, mobile target groups
Barometer
• Top 3 barriers for mobile surveys:
– 35%: Costs incurred to survey participants (data traffic)
– 35%: Difficulties entering information (esp. open-ended
questions)
– 33%: Software/platform heterogeneity
Februar 2011

20

Nonparticipation: Self-Reports?

Post survey in Wave 3, open responses, N= 63

21

´True´ Reasons for (Non)Participation I
• What is the influence of the following potential
determinants of the willingness to participate?
1. Attitude towards participating
2. Hedonic aspects (perceived enjoyment)
3. Social aspects (subjective norm)
4. Image and perceived self-congruity
5. Perceived benefits and costs
• Hypothetical model
Extended technology acceptance model
(Venkatesh et al., 2003)
• Prospective study design (MS I)
– S1: Developing and optimizing measurement models
– S3: Assessing all above mentioned determinants
– S4: Olympic games mobile survey (non)participation

22

Bosnjak et al. 357
´True´ Reasons for (Non)Participation II

Highest influences:

> Hedonic aspects
> Self-congruity

Not relevant:
Fit Indices (robust)
SB-Χ²=407; df=296 > Expected costs (!)
p<.05, Χ²/df= 1.37 > Opinions of others
NNFI=.98
RMSEA=.04
(.03-.05)
*std.β, sig. at α=.
05, N= 272

23

´True´ Reasons for (Non)Participation III
• If hedonic factors outperform cost/benefit-related,
then
– ´exciting´ incentives (lottery drawing) should
increase participation rates
– compensation for incurred costs should undermine
hedonic motivation (salience of costs is increased)

• MS I experiment, manipulating basic compensation
(1 EUR, yes/no) and announced prize draw (100 EUR
voucher, yes/no)

• Results confirmed our expectations (see Appendix):
– highest access and participation ratesfor „lottery &
no incentive condition“

24

Speed of participation? (MS I)
Geschwindigkeit des Zugriffs auf Welle 1 und 2

For about 4.5 hours, Mobile
response rates are higher
compared to Web
Kumuliertert prozentualer Anteil

Faster responses for Mobile
compared to Web:

approx. 35% Mobile versus
aaprox. 10 % Web

Stunden seit Einladung

25

Speed of participation? (MS II)

6

5 MS I

3

2

0
8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00

Mean response speed in hours for different contact/invitation time points
(sent out via SMS)

26

Current context of participation? (MS I)
At home 63,79%

In the office / at work 17,24%
„Were have you been
taking part in the In a car 6,90%

survey?“ At the bus or train station 4,31%
(Wave 3; N=116) Using public transport 2,59%

On the move (other reasons) 5,17%

At home busy with my PC 17,35%

Watched TV 14,29%

Worked at home 10,20% "Which activity did you have to
Read 8,16% disrupt to take part in the mobile
In the office / at work 11,22%
survey?/ What have you done in
Preparing / eating a meal 7,14%
that very situation?“
(Wave 3; N=98; open responses)
On the move 8,16%

Nothing disrupted 23,47%

27

Optimal length of mobile surveys? (MS II)
„Do you want to continue answering the
survey mobile or online (in this case you
will get a link via email)?“
Participants: n= 540

100 % 30,0
10,3 %

75 % 22,5 19,8 20,1
19,3
68,9 %

Minnutes
50 % 15,0
89,7 %

25 % 7,5 5,2 5,1 5,2
31,1 %

0% 0
non iPhone iPhone Total iPhone Non iPhone
mobile Part 1: Mobile- Initial Survey
online Part 1 & 2: Mobile Survey

29

Take-Home Messages & Discussion
• ... on mobile survey measurement:
– Most ´classical´ closed-ended item formats can be included and
are in many cases sufficiently usable
– Various measurement options ´beyond´ the usual self-
administered formats do exist (e.g. GPS positioning, multimedia
upload)
– Open-ended text may need to be replaced by voice capturing/
recognition (to be discussed tomorrow)
• ... on mobile survey (non)response:
– Industry perceptions and self-perception of potential mobile survey
participants on the reasons for nonresponse may be misleading
– Most probable motivators: anticipated enjoyment, image
– Boomerang effects for (over-)compensation
– Fast responses, given in various contexts
– ´Optimal length´ may not exist, various factors appear to
influence the willingness to spend time on (mobile) surveys

Thank you!

michael.bosnjak@unibz.it
http://www.bosnjak.eu

Michael Bosnjak, PhD, Assoc. Prof.
Free University of Bozen-Bolzano, School of Economics and Management
30

Acceptance and users‘ behavior
Influencing participants‘ behavior: design

Basic compensation (1 €): participation in mobile survey

yes no

Prize draw Prize draw
(100 € Amazon voucher) (100 € Amazon voucher)

yes no yes no
Incentive information (timing)

in the
Group 1 Group 2 Group 3 Group 4
SMS

on the
survey
landing
page

Influencing participants‘ behavior

Landing
yes no
page
Prize draw Prize draw
access (100 € Amazon voucher) (100 € Amazon voucher)

yes no yes no
Incentive Information (Timing)

in the
SMS 8,9% 17,3% 21,2% 12,3%

on the
survey Groups not relevant, first contact on7landing page 8
Group 5 Group 6 Group Group
landing
page


Response rates in wave 2 against time

Response rates maximized
with price draw (group 3),
additional compensation
Cumulated response rate

undermines motivation
(see group 1: 1€ and price
draw).

SMS information
Group 1 (1 EUR + price draw)
Group 2 (1 EUR)
Group 3 (prize draw)
Reminder
Group 4 (no incentive information)

Hours since SMS invitation



All yes no
questions Prize draw Prize draw
answered (100 € Amazon voucher) (100 € Amazon voucher)

yes no yes no
Incentive Information (Timing)

in the
SMS 5,9% 12,8% 14,4% 9,2%

on the Group 5 Group 6 Group 7 Group 8
survey
landing
page
9,8% 9,6% 9,1% 10,5%

Nonresponse issues: Background
Why increasing response rates to surveys?

nonresponse
rate

nonresponse true difference
error
Black Box
yr ! = statistic of interest for respondents
yt ! = statistic of the total sample
ynr ! = statistic of interest for nonrespondents 36

Nonresponse: Background:
Types of nonresponse

Source: Bosnjak (2001) 37

Nonresponse: Background:
Generic reasons for nonresponse

• Failure to deliver the survey request
• Spam guards
• Unused or infrequently checked e-mail addresses
• Non-availability during ﬁelding period

• Inability to provide the requested data
• Lack of knowledge
• Insufﬁcient information readily available

• Noncompliance: Refusals to survey requests

38

Nonresponse: Theory:
Why do people (not) respond to surveys?

• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory

39

Rationale:
• Economic exchange view Respondents are motivated by the
monetary beneﬁts promised/
• Human needs and values expected.

Actionable recommendations:
• Compliance heuristics „Pay respondents“ according to the
time/effort invested
Caveats:
• Planned behavior approach • Peoples´ price points vary greatly
and are unknown a-priori
• May largely increase non-
• Leverage-salience theory response bias
•Undermines intrinsic motivation
• Social exchange theory and may increase measurement
error (low survey involvement)
• Promised monetary incentives
NOT consistently effective (!)

40


Rationale:
• Economic exchange view
Some values are
• Human needs and values systematically related to
• Compliance heuristics the propensity to
respond (higher order
• Transactional view needs, civit duty
• Planned behavior approach orientation, etc.)

• Leverage-salience theory Caveats:
• Social exchange theory • Effects small (if any)
• Actionable
recommendations?

41

Rationale:
• Economic exchange view Certain aspects of the survey
announcement and survey
• Human needs and values implementation do induce
compliant behavior:
1. Reciprocity
• Compliance heuristics 2. Scarcity
3. Authority
• Transactional view 4. Consistency
5. Consensus
• Planned behavior approach 6. Liking

• Leverage-salience theory Can be derived from persuasion
literatures, but speciﬁc prescriptive
• Social exchange theory models on how to tailor them
toward survey situations are rare.

Groves, Cialdini & Couper (1992); Cialdini (2008);
http://www.inﬂuenceatwork.com/ 42


Rationale:
• Economic exchange view Larger response propensity
if communication style
• Human needs and values reﬂects positive regard and
• Compliance heuristics avoids adult-to-child
communication styles.
Caveats:
• Planned behavior approach • Limited scope
• Leverage-salience theory • Empirical evidence scarce
• Covered by other theories
• Social exchange theory (compliance heuristics,
social exchange)

Comley (2006)
43

Rationale:
• Economic exchange view The propensity to respond to
surveys is primarily a function of
• Human needs and values three factors:
• Attitude to participate
• Subjective norms
• Compliance heuristics • Perceived behavioral control
• Moral obligation
• Planned behavior approach If weights are known for a speciﬁc
population/sample: Enables the
researcher to design survey
• Leverage-salience theory participation requests

• Social exchange theory Caveats:
Restricted to optimize survey
announcements

Bosnjak (2002); Bosnjak, Tuten & Wittmann (2005)
44

Rationale:
• Economic exchange view Respondents are differentially
motivated by
• Human needs and values • different aspects of the survey
(leverage, e.g. type of incentives)
and by
• Compliance heuristics • how much emphasis is put on
each aspect by the surveyor
• Transactional view (salience, e.g. preference for certain
incentives )
• Planned behavior approach Actionable recommendations:
Because of the interaction between
• Leverage-salience theory leverage*salience, improving
response rates is not always
• Social exchange theory desirable!
Nonresponse bias may be
inﬂuenced by leverage*salience
interaction.

Groves, Singer & Corning (2000)
45

Rationale:
• Economic exchange view Survey participation as social
exchange: The likelihood of
• Human needs and values responding is greater when the
respondent trusts that the expected
rewards will outweigh the
• Compliance heuristics anticipated costs of responding.

• Transactional view Actionable recommendations:
Tailored Design Method, a well-
• Planned behavior approach developed set of practical
recommendations on all aspects of
survey design/implementation,
• Leverage-salience theory aimed at:
•establishing trust
• Social exchange theory •increasing participation beneﬁts
•decreasing participation costs

Dillman, Smyth & Christian (2009)
46

TDM-based recommendations (selection)

To increase beneﬁts To decrease costs of
To establish trust
of participation participation
•Obtain sponsorship by •Provide information about •Make it convenient to
legitimate authority the survey respond
•Provide a token of •Ask for help or advice •Avoid subordinate language
appreciation in advance •Show positive regard •Make the questionnaire short
•Make the task appear •Say thank you and easy to complete
important •Support group values •Minimize requests to obtain
•Ensure conﬁdentiality and •Give tangible rewards personal or sensitive
security of information •Make the questionnaire information
interesting •Emphasize similarity to other
•Provide social validation requests or tasks to which a
•Inform people that person has responded
opportunities to respond are
limited

Dillman, Smyth & Christian (2009, p. 38) 47

Nonresponse: Evidence: Mail surveys:
Effective methods & procedures I
• Most effective factors in mail surveys
(only factors under the researchers full control listed):
• Personalization of requests to participate
(Dillman, 1978, 2000; Edwards et al., 2007; Fox, Crask, & Kim,
1988; Heberlein & Baumgartner, 1978; Yammarino et al.,1991;
Yu & Cooper, 1983)
• Prepaid monetary incentives
(Church, 1993)
• Number of contacts made (esp. if prenotiﬁer is included)
(Armstrong & Lusk, 1987; Edwards et al., 2007; Fox et al., 1988;
Heberlein & Baumgartner, 1978; Yammarino et al.,1991;
Yu & Cooper, 1983)
➡ Integrated and reﬁned within the Total-Design-Method
(Dillman, Smyth & Christian, 2009)
48

Nonresponse: Evidence: Mail surveys:
Effective methods & procedures II
• Effective, but not covered because of limited control:
• Survey topic / topic involvement
• Length
• Sponsorship (University / commercial)
• Factors reducing response rates
(1, 2: Edwards et al., 2007; 3: Singer, Hippler & Schwarz, 1992):
1. Starting with the most general question (e.g. demographics)
2. Opportunity to opt-out of the study
3. Over-emphasizing data protection/conﬁdentiality
• Partly covered later for Web surveys:
• Questionnaire design effects on nonresponse

49

Nonresponse: Evidence: Web surveys:
Effective methods & procedures III
• Personalization:
• Personal salutation (name) is effective (esp. for powerful sender)
(e.g., Heerwegh, et al., 2005; Joinson & Reips, 2007)
• (Monetary) Incentives:
• In general effective but small overall effect (Göritz, 2006)
• Pre-paid monetary incentives need to be tangible to be effective
(Birnholtz et al., 2004; Bosnjak & Tuten, 2003)
• Lotteries esp. effective, timing important (immediate notiﬁcation)
(Bosnjak & Tuten, 2003; Tuten, Galesic & Bosnjak, 2004)
• Contact features:
• No of contacts very effective (Cook, Heath & Thompson, 2000)
• SMS prenotiﬁer very effective (Bosnjak et al., 2008)
50

Mobile Survey Formats Usability

Recommandé

Recommandé

Contenu connexe

Similaire à Mobile Survey Formats Usability

Similaire à Mobile Survey Formats Usability (20)

Plus de QuestBack AG

Plus de QuestBack AG (20)

Dernier

Dernier (20)

Mobile Survey Formats Usability