This document summarizes a webinar presented by UserZoom on mobile banking benchmarking research. The webinar covered: - An overview of UserZoom's platform and mobile research capabilities. - A competitive benchmark study of four banks' mobile sites which found CIBC had the best experience overall, while Wells Fargo had the lowest ratings. - Key findings from tasks of researching accounts, finding customer service numbers, and locating branches on each site. CIBC performed best across tasks while Wells Fargo had most issues. - Overall, CIBC improved participants' brand perception the most while Wells Fargo improved the least after using the mobile sites.

1. Webinar:
Mobile Banking Benchmark
Ann Rochanayon, Sr. Director CX/ UX Research at UserZoom
Sneha Kanneganti, UX Researcher at UserZoom
Alfonso de la Nuez, Co-Founder and Co-CEO at UserZoom
#uzwebinar

2. Ann Rochanayon
Sr. Director CX/UX Research
UserZoom
Alfonso de la Nuez
Co-Founder and Co-CEO
UserZoom
Speakers:
Sneha Kanneganti
UX Researcher
UserZoom

3. www.userzoom.com
• Chat box is available if you have any questions
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of the
slides/recording
• Twitter hashtag: #uzwebinar
Quick Housekeeping

4. www.userzoom.com
• About UserZoom / Mobile Research
• About UX Benchmarking
• Mobile Banking Competitive Benchmark Study
• Methodology
• Overall Findings
• Detailed Task Findings
• Tips on Mobile Research
• Q&A
Agenda

5. www.userzoom.com
About UserZoom
 UX Consultants since ‟01, SaaS since „09
 In US, UK, Germany and Spain
 UserZoom offers an all-in-one, multi channel and agile approach to
managing digital Customer Experiences by enabling brands to cost-
effectively plan, research, design, and measure CX and UX on a
unified software platform.

6. Platform Overview
www.userzoom.com
Mobile
Video Recording
Remote Usability Testing
VOC Solutions UX & Usability Solutions Results
Select Product Features
Intercept
Surveys
Feedback
Tabs
Mobile App
VOC
Remote
Usability
Testing
Mobile
Usability
Testing
Card Sorting
Tree
TestingScreenshot
Click
Testing
Screenshot
Timeout
Testing
Online
Surveys
Usability
Data
Survey
Responses
IA Data
Behavioral
Data
Video
Recording

7. www.userzoom.com
Highlighted Customers

8. www.userzoom.com
What is UX Benchmarking
• The process of comparing a website‟s (or any digital
product) performance, specifically in relation to UX
metrics, including effectiveness, efficiency, overall
satisfaction and quality perception, to another that is
widely considered to be an industry standard
benchmark or best practice
• UX Benchmarking provides a snapshot of the
performance of your product and helps you
understand where you are in relation to a particular
standard.

9. www.userzoom.com
Why is UX Benchmarking important
• Measuring the quality of UX is more important than ever
• Consumers have gained a lot of power in recent years
• More competition, more maturity in the online marketplace
• You can‟t really manage what you can‟t measure. UX Pros
must deliver more measured value. Mgmt Teams LOVE
metrics…
UX Benchmarking allows you to MEASURE performance in two
ways:
1. Baseline studies: Measure UX over time, compare
evolution of your own product and/or service
2. Against competitors within your industry

10. www.userzoom.com
4 Must-dos with UX Benchmarking
1. Plan ahead:
• Define measurable & comparable UX-related KPIs (will show them in the case study
ahead)
• Select competitors to benchmark against
• Define target audience
• Identify common site activities/tasks (recommended between-subjects design)
2. Conduct research using task-based testing
3. Gather both quantitative & qualitative data
• Use the quantitative data to test for statistically significant differences between means or
proportions
• Use qualitative data to support your findings and understand the why
4. Research at least twice a year (or even quarterly) to measure design changes and trends in
expectations

11. www.userzoom.com
Agenda
Mobile Banking Competitive Benchmark

12. www.userzoom.com
Methodology

13. www.userzoom.com
• Goals
• Determine which bank offers the best mobile experience for
prospective customers
• Assess the impact of mobile banking experience on brand
perception, Net Promoter Score, and conversion
• Sites evaluated
• Participants
• Tested from February 24 – March 5, 2014
• Between-subjects design
• N=200, 50 participants per site
• Ages 26+
• Not a current customer of the site evaluated
• Have prior experience with mobile banking
• Participated via. SmartPhone
Methodology: Study Background

14. www.userzoom.com
Methodology: Recruitment
SSI is the premier global provider of sampling, data
collection and data analytic solutions for survey
research, reaching respondents in 86 countries via
Internet, telephone, mobile/wireless and mixed-access
offerings. SSI staff operates from 25 offices in 18
countries, offering sample across every mode, online and
CATI data collection, questionnaire design
consultation, programming and hosting, online custom
reporting and data processing. SSI‟s 3,300 employees
serve more than 3,000 clients worldwide. For additional
information, please visit www.surveysampling.com.

15. www.userzoom.com
Methodology: Study Flow
Tasks
Research Account: Research the savings accounts available at [Bank] in the [ZIP
/ POSTAL CODE]. Find the minimum balance for the savings account that will earn
you the most interest.
Customer Service: Find a phone number you can call to ask questions about
banking with [BANK]. What are the last four digits of the number?
Find a Branch: Find a [BANK] branch in [CITY, ZIP / POSTAL CODE] to open a
savings account in person. Determine what information the site provides about the
branch (e.g., services available, phone number, address)
Introduction
Screener
Wrap-up

16. www.userzoom.com
Methodology: Participant Experience

17. www.userzoom.com
Methodology: Participant Background
90%
87%
76%
74%
72%
71%
60%
43%
Check account balance
View transactions
Find an ATM / branch
View / pay bills
View account statement
Transfer Money
Research banking services
(e.g., accounts, credit cards, loans)
Deposit money
Banking Activities Conducted via
Smartphone
N=198

18. www.userzoom.com
Summary

19. www.userzoom.com
Summary (1of 2)
• CIBC had the highest overall ratings.
– Participants found it easiest to research accounts on CIBC compared to the
other sites and encountered the least number of issues when performing the
three tasks.
• KeyBank tied with CIBC for the highest overall satisfaction
ratings and the highest Net Promoter Score.
– Although task success was very low for researching an account, ease of use
ratings and frustration ratings were not significantly different from the top site
(CIBC).
– Participants found KeyBank easiest to use when finding customer service
information and also rated KeyBank highest when finding a branch
• USAA performed third-best in the study.
– Participants found the site harder to use when researching accounts and
finding customer service information and generally found that too many clicks
were required to find information.

20. www.userzoom.com
Summary (2 of 2)
• Wells Fargo had the lowest overall ratings for ease of
use, organization and layout and satisfaction primarily due
difficulties in navigation and participants having to use the full site
to perform the tasks
– Although Wells Fargo has a mobile optimized web site, participants were
forced to navigate to the full site to research an account
– For task 2, locating a customer service phone number 22% reported frustration
in using the full site, even though this information was available on the mobile
site
– Clicking on the incorrect link was among the top problems
• Participants brand perception improved significantly from pre to
post mobile site evaluation
– However, Wells Fargo had the smallest improvement in brand perception, while
the brand perception improvements in the other site were comparable
• Net promoter scores for Wells Fargo was lowest overall (-
44%), while KeyBank had the highest Net Promoter Score (8%)
Bank Delta
Wells Fargo + 13%
KeyBank + 31%
USAA + 32%
CIBC + 27%

21. www.userzoom.com
Overall Findings

22. www.userzoom.com
Overall Findings – Key Performance Metrics
KeyBank USAA CIBC
Ease of use 30%* 56% 46% 58%
Organization and Layout 36% 46% 40% 50%
Satisfaction 34% 50% 42% 50%
Likelihood to open an account 16% 16% 14% 22%
Site with the highest rating Site with the lowest rating*Significantly lower than site with highest rating at 90% CI
Ease of use, Organization and Layout, and Satisfaction: Responded 6-7 on a 7-point scale when asked, “Please rate your experience with the
following aspects of the mobile site.” where 1 = Negative and 7 = Positive
Likelihood to open an account: Responded 6-7 on a 7-point scale when asked, “Based only on your experience today, how likely would you be to
open a bank account with [BANK]?” where 1 = Not at all likely and 7=Extremely likely

23. www.userzoom.com
Overall Findings – CIBC Video Question
Compared to the other
sites, CIBC rated highest across
all key performance measures:
• Ease of use
• Organization and layout
• Overall satisfaction
• Likelihood to open an account

24. www.userzoom.com
Overall Findings – Privacy and Trust
The bank seems to handle
privacy and security well
32% 32% 28% 36%
The mobile online bank
seems trustworthy
34% 44% 32% 44%
I would trust this bank with my
personal information
32% 38% 34% 42%
Site with the highest rating Site with the lowest rating*Significantly lower than site with highest rating at 90% CI
Based on your experience with the [BANK] site on your mobile device, please rate your level
of agreement with the following statements.
Responses of 6-7 shown from a 7-point scale where 1 = Strongly disagree and 7=Strongly agree

25. www.userzoom.com
Overall Findings – Brand Perception
13%
11% 12%
17%
26%
42%
44% 44%
Wells Fargo KeyBank USAA CIBC
Pre Brand Perception Post Brand Perception
N=48 N=50 N=18 N=50 N=25 N=50 N=46 N=50
Bank Delta
Wells Fargo + 13%
KeyBank + 31%
USAA + 32%
CIBC + 27%
Pre vs. Post Brand Perception
Based on what you know / your experience
today, what is your perception of [BANK]?
Responses of 6-7 shown from a 7-point scale where 1 = Negative and 7=Positive

26. www.userzoom.com
Overall Findings – Brand Attributes
11%
-9%
10%
-17%
-5%
1%
5%
-8%
-13%
-5%
-3%
16%
25%
15%
14%
4%
-10%
-12%
43%
-16%
-5%
2%
0%
-4%
20%
4%
8%
10%
4%
-6%
42%
10%
2%
0%
-2%
4%
11%
2%
-3%
6%
2%
3%
11%
-4%
-5%
1%
-1%
-1%
Wells Fargo
KeyBank
USAA
CIBC
Reliable
Convenient
Competitive
Easy access
Comprehensive
Trustworthy
Secure
Exclusive
Expensive
Untrustworthy
Unreliable
Difficult
Change in Associated Brand Attributes
Pre vs. Post Mobile Site Evaluation
Positive
Negative
Which of the following words or phrases would you use to describe [BANK]?

27. www.userzoom.com
Overall Findings – Wells Fargo Video Question
Although fewer participants
associated Wells Fargo with the
“Expensive” after evaluating the
mobile site, 10% still considered
Wells Fargo to be “Expensive”.

28. www.userzoom.com
Overall Findings – Net Promoter Score
60%
26%
52%
38%
24%
40%
30%
46%
16%
34%
18%
16%
Wells Fargo
KeyBank
USAA
CIBC
Detractor (0-6) Passives (7-8) Promoters (9-10)
- 44%
8 %
- 34%
- 22%
NPS
How likely is it that you would recommend this website to a friend or a
colleague?

29. www.userzoom.com
Task – 1 Research Account
Description: Research the savings accounts available at [Bank] in the [ZIP /
POSTAL CODE]. Find the minimum balance for the savings account that will
earn you the most interest.

30. www.userzoom.com
Task 1- Research Account
Task success 34% 4%* 20% 28%
Ease of use 24% 22% 20%* 36%
Did not encounter problems /
frustrations
8%* 22% 28% 32%
Time on task (min) 3.07 2.21 1.35 1.52
Site with the highest rating / fastest time on task Site with the lowest rating / slowest time on task
*Significantly lower than site with highest rating at 90% CI
Task Success: Selected the correct response when asked, “What is the minimum balance for a savings account that will earn you the most interest?”
Ease of use: Responded 6-7 on a 7-point scale when asked, “How easy or difficult was it to find the minimum balance for the savings account that will
earn you the most interest?” where 1 = Difficult and 7=Easy.
Did not encounter problems / frustrations: Selected “I did not encounter problems and frustrations” when asked, “Which of the following problems or
frustrations, if any, did you encounter while researching savings accounts at [BANK] from your mobile device?”

31. www.userzoom.com
Task 1 – Amount of Information
42%
54%
44%
26%
40%
38%
50%
70%
18%
8% 6% 4%
Wells Fargo Key Bank USAA CIBC
Far Too Much
Just Right
Too Little
“Information
comparing each
account with the other”
“I would have liked to
see more information
on how long before
you accumulate
interest”
“Ability to search
that would then
connect to full site”
“Clearer info about
options on mobile
site”
“Information about their
savings account. There
should be a little more
information on every
title/name of the
accounts.”
“A comparison chart for
accounts”
“Key details like
minimum amounts
and interest”
“Details like fees for if
your balance drops
below the minimum”
How would you rate the amount of information available on the site?
What information was lacking?

32. www.userzoom.com
Task 1 – Problems and Frustrations
46%
18%
22%
14%
28%
20%
16%
28%
18%
10%
24%
2%
2%
8%
12%
6%
12%
4%
8%
12%
12%
40%
32%
8%
24%
12%
8%
22%
12%
6%
4%
0%
8%
4%
6%
20%
16%
14%
6%
4%
4%
28%
6%
12%
14%
2%
8%
10%
4%
18%
38%
16%
8%
2%
4%
32%
Had to use the full site
Full site did not fit onto my mobile screen
Had to readjust screen frequently
Links were too close together
Text was too small / difficult to read
Clicked on the wrong link
Account information was scattered
Too many clicks required to find information
Difficult to compare accounts
Not enough account information available
Information was confusing / unclear
Site was slow
Encountered errors / technical issues
I did not encounter any problems or…
Wells Fargo
Key Bank
USAA
CIBC
Content
Navigation
Site
Performance
Site
Interaction
Which of the following problems or frustrations, if any, did you encounter while researching
savings accounts at [BANK] from your mobile device? [Check all that apply]

33. www.userzoom.com
Task 1 – Wells Fargo First clicks
Sign On
17%
ATMs/Location
s
15%
CEO Mobile
4%
Quick Guide
45%
Full Site
19%
Most efficient path

34. www.userzoom.com
Task 1– Wells Fargo Video Session Replay

35. www.userzoom.com
Task – 2 Customer Service
Description: Find a phone number you can call to ask questions about
banking with [BANK]. What are the last four digits of the number?

36. www.userzoom.com
Task 2 – Customer Service
Task success 66%* 68%* 94% 72%*
Ease of use 74% 84% 72% 78%
Did not encounter problems /
frustrations
58%* 82% 76% 86%
Time on task (min) 0.43 0.28 0.39 0.30
Task Success: Selected the correct response when asked, “What are the last four digits of the phone number you would call for questions about
banking?”
Ease of use: Responded 6-7 on a 7-point scale when asked, “How easy or difficult was it to find a phone number?” where 1 = Difficult and 7=Easy.
Did not encounter problems / frustrations: Selected “I did not encounter problems and frustrations” when asked, “Which of the following problems or
frustrations, if any, did you encounter while finding a phone number?”
Site with the highest rating / fastest time on task Site with the lowest rating / slowest time on task
*Significantly lower than site with highest rating at 90% CI

37. www.userzoom.com
Task 2 – USAA Video Session Replay

38. www.userzoom.com
Task 2 – Customer Service
22%
2%
2%
2%
2%
8%
6%
2%
4%
4%
58%
0%
2%
0%
0%
4%
6%
2%
2%
2%
4%
82%
2%
0%
4%
0%
0%
2%
14%
4%
0%
0%
76%
0%
4%
4%
4%
2%
4%
6%
2%
2%
0%
86%
Had to use the full site
Site did not fit onto my mobile screen
Had to readjust screen frequently
Links were too close together
Text was too small / difficult to read
Clicked on the wrong link
Too many clicks required to find information
Not enough information provided
Site was slow
Encountered errors / technical issues
I did not encounter any problems or
frustrations
Wells Fargo
Key Bank
USAA
CIBC
Site
Interaction
Navigation
Content
Technical Issues
Which of the following problems or frustrations, if any, did you encounter while finding a phone
number? [Check all that apply]

39. www.userzoom.com
Task – 3 Find a Branch
Description: Find a [BANK] branch in [CITY, ZIP / POSTAL CODE] to open a
savings account in person. Determine what information the site provides about
the branch (e.g., services available, phone number, address)

40. www.userzoom.com
Task 3 - Find a Branch
Task success 72%* 86% 84% 76%
Ease of use 58%* 78% 62% 64%
Did not encounter problems /
frustrations
46%* 74% 56% 62%
Time on task (min) 1.07 1.03 1.18 1.14
Task Success: Selected “Yes” when asked, “Were you able to find a bank branch in the 90015 zip code?”
Ease of use: Responded 6-7 on a 7-point scale when asked, “How easy or difficult was it to find a bank branch on your mobile device?” where 1 =
Difficult and 7=Easy.
Did not encounter problems / frustrations: Selected “I did not encounter problems and frustrations” when asked, “Which of the following problems or
frustrations, if any, did you encounter while finding a bank branch on your mobile device?”
Site with the highest rating / fastest time on task Site with the lowest rating / slowest time on task
*Significantly lower than site with highest rating at 90% CI

41. www.userzoom.com
Task 3 – CIBC Video Session Replay

42. www.userzoom.com
Task 3 – Problems and Frustrations
12%
2%
2%
0%
6%
12%
10%
10%
10%
4%
8%
46%
0%
4%
0%
2%
4%
2%
10%
2%
2%
0%
4%
74%
8%
4%
0%
6%
0%
10%
8%
22%
6%
4%
0%
56%
0%
6%
2%
4%
4%
2%
6%
12%
4%
4%
0%
62%
Had to use the full site
Site did not fit onto my mobile screen
Links were too close together
Had to readjust screen frequently
Text was too small / difficult to read
Information was confusing / unclear
Not enough information provided
Too many clicks required to find information
Clicked on the wrong link
Site was slow
Encountered errors / technical issues
I did not encounter any problems or frustrations
Wells Fargo
Key Bank
USAA
CIBC
Site
Interaction
Content
Navigation
Site
Performance
Which of the following problems or frustrations, if any, did you encounter while finding a bank
branch on your mobile device? [Check all that apply]

43. www.userzoom.com
Mobile Best Practices

44. www.userzoom.com
Best Practices – Mobile Research (1of 5)
1. Duration
• Restrict mobile study length to 15 minutes or 2-3
tasks and 15 questions.
• After this threshold, the drop-off rate doubles.
• As the duration increases, incentives must increase
commensurately.
2. Expectations
• Provide clear descriptions about the task. E.g. How many tasks? Websites?
• Do users have to visit a location? E.g. A store, at home, outside, etc.
Goal: Improve completion rates & improve quality of response

45. www.userzoom.com
Best Practices – Mobile Research (2 of 5)
3. Study Requirements
• State the mobile software requirements.
E.g., iOS 4+ or Android 4+
• Notify participants if there is a minimum
battery charge. E.g. 30% min battery to
record video session replay
• Inform participants if Wifi should be
enabled

46. www.userzoom.com
Best Practices – Mobile Research (3 of 5)
4. Question types
• Use rating scales, radio or multiple choice
responses and drag and drop features.
• Restrict open ended question type to the
minimum. Use video question type to minimize
using tiny keyboard.
• Minimize matrix questions which can be
difficult for users to respond on small screens.
• Keep list choices to a minimum. More than 10
will require scrolling

47. www.userzoom.com
Best Practices – Mobile Research (4 of 5)
5. Instructions
• Provide concise and easy-to-read task
descriptions
• Highlight key parts through the use of
bolding, a different color, underlining, or italics
to facilitate scanning.
• Update participants of their progress through
the study (E.g., Task 2 of 3)

48. www.userzoom.com
Best Practices – Mobile Research (5 of 5)
6. Recruitment
• Use panel vendor that specializes in
mobile recruitment
• Explore methods for recruitment, email
invitation, text message or QR code
7. Incentives
• Higher incentives may be required for
mobile studies
• ~ $ 12 – 25 depending on the complexity
and duration of the study

49. www.userzoom.com
Q&A
Follow us:
Twitter: @userzoom
Linkedin: https://www.linkedin.com/company/userzoom