Measurement options, measurement error, and usability in mobile surveys. Presentation held at General Online Research 2009 (GOR09) in Vienna

1. Measurement options,
measurement error, and usability
in mobile surveys
In cooperation with:
Dr. Tanja Pferdekämper, Globalpark AG
Prof. Dr. Michael Bosnjak, Free University of Bozen-Bolzabo 1
Gottfried Metzger, University of Mannheim

2. Agenda
1 Current Situation and Objectives
2 Subject of Research Study
3 Fact Sheet of Research Study
4 Findings
5 Summary & Outlook
2

3. Agenda
1 Current Situation and Objectives
2 Subject of Research Study
3 Fact Sheet of Research Study
4 Findings
5 Summary & Outlook
3

4. Current Situation and Objectives
Percentage of mobile users in county
4

5. Current Situation and Objectives
Market Research Challenge
Mobility & Flexibility
- Instant Feedback at the Point of Sale
- Reaching high mobility target group
- Location independent surveys
Speed
- Contemporary, event triggered surveys
- Fast responses
Reporting
- Results available in real-time
- Faster decision making/evaluation
5

6. Agenda
1 Current Situation and Objectives
2 Subject of Research Study
3 Fact Sheet of Research Study
4 Findings
5 Summary & Outlook
6

7. Research Questions:
How is Content
?
? displayed?
?
?
How is the usability of
?
different question types ?
? rated?
?
?
?
? ?
? 7

8. Study Focus
Mobile Surveys here are defined as:
Self-administered surveys with mobile devices
Not topic of the study:
- Types of interviewer-administered
surveys
- PDA-based and similar surveys with
Interviewer
- CATI-Surveys conducted with cell
phone users

9. Agenda
1 Current Situation and Objectives
2 Subject of Research Study
3 Fact Sheet of Research Study
4 Findings
5 Summary & Outlook
9

10. Research Summary:
Joint project by Globalpark AG, FU Bozen & YOC AG
Two survey waves to analyse respondent intention ,
behavior, usability and the acceptance of mobile
Aim
surveys
2 surveys from 18.08. – 01.09.2008
Field work
1. Mobile survey Olympia 2008
(YOC Mobile-Panel, N = 171)
Sample source and
2. Post web based survey to evaluate usability from a
content
user perspective (YOC Mobile-Panel, N = 413)
Had not taken part in mobile survey before; 33.9% had
Sample used mobile internet over a year; 49.7% had never
accessed the internet with their mobile phone
10

11. Display of different question types
We used 5 prototypical mobile question types in the survey:
How interested are
Did you watch the
Where do you look
you in the Olympic
2008 Olympic
up information &
Summer Games of
Opening Ceremony
results?
2008?
on TV?
Single response vertical Multi response vertical
Question with image
11

12. Display of different question types
We used 5 prototypical mobile question types in the survey:
Which of the 28
How many hours
sporting disciplines
will you watch the
interests you
Olympic
most?
competitions?
Single line text input Closed response list
12

13. Agenda
1 Current Situation and Objectives
2 Subject of Research Study
3 Fact Sheet of Research Study
4 Findings
5 Summary & Outlook
13

14. Technical Requirements and Restrictions
Input devices and types of connections (Own Study: Germany)
The Connection is
Although Data
mostly GPRS (30%),
Tarifs are often
UMTS (11%) and GSM
calculated based on
(10%). 40% do not know
transfer volume (29%);
their type of
36% of respondents do
connection.
not know how much it
costs.
Most cell phones were
manufactured by Nokia
(39%) and Sony Ericsson
(25%).
(Own Study in 2008, N = 170)
14

15. Technical Requirements and Restrictions
Traffic, Manufacturers and Devices (Benchmark: Western Europe)
Apple leads with
31% share of overall
requests in Western
Europe in the AdMob
Network.
( AdMob Mobile Metrics Report, January 2009)
15

16. Display of different question types
Drop-out Rates – objective Measure
Question with
Multi response Single text
During the survey, 23% of
image
vertical input
respondents cancelled: Mostly
at Question with Image (5%),
followed by Multi response
Cancellations
and Text input (each with
(N=150)
2% 2% 2%), all other pages without
5%
questions (14%).
Single text Single
input response
vertical
Item-Nonresponse is high
Item - Non with Text input (26%) and
response
low with Single response
(N=115)
26% 2% (2%).
16

17. Scrolling 1 (2)
“While answering, I
Significant Difference between
had to move the
Mean Values on 5%-Level
content around
(‘scrolling‘).”
Text input Closed Question Single response Multi response
single line response list with picture vertical vertical
2,65 2,86 3,42 3,52 3,93
Mean Values from a scale of ‘never’ [1] to ‘often’ [7]
17

18. Scrolling 2 (2)
“While answering, I
Significant Difference between
had to move the
Mean Values on 5%-Level
content around
(‘scrolling‘).”
Text input Closed Question Single response Multi response
single line response list with picture vertical vertical
2,65 2,86 3,42 3,52 3,93
Mean Values from a scale of ‘never’ [1] to ‘often’ [7]
18

19. Ease of answering
“With this question
Significant Difference between
type, I found it easy
Mean Values on 5%-Level
to select an
answer.”
Question Single response Multi response Closed Text input
with picture vertical vertical response list single line
6,39 6,36 6,10 5,82 5,17
Mean Values from a scale of ‘Strongly disagree’ [1] to ‘Strongly agree’ [7]
19

20. Unintentional input
Significant Difference between “During completion,
Mean Values on 5%-Level I made unintended
selections.”
Single response Question Text input
Multi response Closed
vertical with picture single line
vertical response list
1,54 1,67 1,67 2,01 2,24
Mean Values from a Scale of ‘never’ [1] to ‘extremely often’ [7]
20

21. Fluent answering
“I could use this
Significant Difference between
question type
Mean Values on 5%-Level
fluently while
answering.”
Question Multi response Single response Closed Text input
with picture vertical vertical response list single line
6,35 6,28 6,26 6.03 5,55
Mean Values from a scale of ‘Strongly disagree’ [1] to ‘Strongly agree’ [7]
21

22. Ease of Use
“This question type
Significant Difference between
was ease to answer
Mean Values on 5%-Level
on my mobile
phone.”
Question Single response Multi response Closed Text input
with picture vertical vertical response list single line
6,46 6,31 6,25 6.00 5,44
Mean Values from a scale of ‘Strongly disagree’ [1] to ‘Strongly agree’ [7]
22

23. Display of different question types
During the post-survey every respondent was asked about the 5
question types.
Additionally, we also looked for non-obtrusive indicators for
measuring usability
Highly Scores given by Respondents
Single response
correlated 89.2
vertical
attributes of Multi response vertical 87.3
Question type
the five 82.7
Closed list
question types 74.7
Single line text input
were 87.9
Question with Image
summarised
0 25 50 75 100
into an Index: 'Fit for Purpose' (0-100 Points)
23

24. Display of different question types:
Fit for Purpose - Summary
Overall, the difference in the subjective user evaluation
+ are small
The disadvantages of Text input fields are notable
-
Still, overall, respondents rated the question types
+ positively
Limitations
- Retrospective Survey
- General assumptions regarding question content and
questionnaire length should be examined further
24

25. Summary
Technical Requirements & Restrictions
- Efficiency of Design- and Implementation Technology for Mobile
surveys
- Diversity on the part of network providers and handset manufacturers
(Data transfer rates, Display of content): Pre tests required!
Display of content
- All common question types: Positive ratings by respondents
Usability Rating
- Different perception with regard to scrolling effort, ease of use,
unintended answers and fluency
- Relative disadvantage of text input questions and partly with Drop-
Down-Menus
- Variation of respondents subjective perceptions and objective
usability properties particular with regard to question type with pictures
25

26. Outlook
Future research topics:
1 Robustness of results for a variety of topics?
Influence of participation context on mobile survey
2 error dimensions largely unexplored.
Development of perceived barriers in light of
3 technological developments (more advanced
devices, such as smartphones; flatrates): Cost
concerns may disappear, usability ratings may go up,
broader applicability possible.
26

27. Thank you!
Visit our mobile survey with your cell phone at:
Mobile.opst.de/uc/main/157f
There you can directly participate in the mobile survey.
Tanja Pferdekämper Michael Bosnjak Gottfried Metzger
tanja.pferdekaemper@globalpark.com http://contact.bosnjak.eu MetzgerGottfried@aol.com
27

28. Globalpark mit Sitz bei Köln, in London, New York und Wien gehört zu den weltweit führenden Anbietern von
Online-Feedback-Software für Marktforschung, Personalwesen, Marketing und
Kundenbeziehungsmanagement. Im deutschsprachigen Raum ist Globalpark in diesen Segmenten Marktführer.
Über Globalpark
Mehr als 1.000 Kunden arbeiten weltweit erfolgreich mit Globalpark-Software. Über 350 davon sind
internationale Konzerne, führende Marktforschungsinstitute und Beratungsunternehmen. Zu den Kunden von
Globalpark zählen namhafte Unternehmen wie Continental, Daimler, die Deutsche Lufthansa, die Deutsche
Telekom, die GfK Gruppe, Siemens, Warner Music und Wrigley.
Globalpark USA
Globalpark AG
Globalpark UK Ltd.
405 Lexington Ave.
Kalscheurener Str. 19a
5 Archie Street
New York, NY 10174
50354 Hürth
London SE1 3JT
Vereinigte Staaten von Amerika
Deutschland (Hauptsitz)
Großbritannien
Tel.: +1 888 2999422
Tel.: +49 2233 7933 6
Tel.: +44 207 4033900
Standorte
Globalpark Österreich GmbH
Wassergasse 25
1030 Wien
Österreich
Tel.: +43 1715 028911
Dr. Tanja Pferdekämper
Head of Consulting
Kontakt
Globalpark AG
Phone: +49 2233 7933 721
E-Mail: tanja.pferdekaemper@globalpark.com

29. Nutzungshinweise
© 2009 – Globalpark AG, Globalpark UK Ltd., Globalpark USA, Globalpark Österreich GmbH
Die in dieser Publikation enthaltene Information ist Eigentum der Globalpark AG und ihrer Unternehmenstöchter.
Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus ist nur mit ausdrücklicher schriftlicher
Genehmigung durch die Globalpark AG oder ihre Unternehmenstöchter gestattet.
Diese Publikation wird ohne jegliche Gewähr bereitgestellt. Enthaltene Informationen können ohne vorherige Ankündigung
geändert werden. Zudem übernimmt Globalpark keine Garantie für die Exaktheit oder Vollständigkeit der Informationen,
Texte, Grafiken, Links und sonstigen in dieser Publikation enthaltenen Elemente. Bei dieser Publikation handelt es sich um
eine vorläufige Version, die gültigen Lizenzverträgen oder anderen Vereinbarungen mit Globalpark nicht unterliegt. Die von
der Globalpark AG, ihren Unternehmenstöchtern und/oder ihren Vertriebsfirmen angebotenen Softwareprodukte können
Softwarekomponenten anderer Softwarehersteller enthalten. Alle Produkte können länderspezifische Unterschiede
aufweisen.
Globalpark, Enterprise Feedback Suite, EFS, EFS Survey, EFS Panel, EFS Employee und EFS Leadership, weitere im Text
erwähnte Globalpark-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken und können
eingetragene Marken der Globalpark AG und/oder ihrer Unternehmenstöchter in Deutschland und anderen Ländern
weltweit sein. Alle anderen Namen von Produkten und Dienstleistungen sind Marken der jeweiligen Firmen.