Improving Website User Experience with A/B Testing

1. ALATechSourceWorkshop
June24,2015
Improving Website User Experience
with A/ BTesting
ScottW.H.Young
@hei_scott

2. What is this workshop about?

3. What is this workshop about?
A research and design methodology that centers the user

4. What is this workshop about?
A research and design methodology that centers the user
How to make better decisions with better data

5. What is this workshop about?
A research and design methodology that centers the user
How to make better decisionswith better data
How to build a website that generates trust and satisfaction

6. User Experience

7. ux

8. ux
Aflexible framework of analysis
with wide-ranging applications

9. usef,ul
usable desirable
valuable
fiindable
credible

10. Content
Reactions
User
Goals *
Actions Transactions
Interaction
Business
Goals

11. INFORMATION
ARCHITECTURE
ARCHITECTURE

12. UX: Primary Purpose

13. UX: Primary Purpose
Understand the user

14. UX: Primary Purpose
Understand the user
Design for the user's expectations,
behaviors, motivations, and goals

15. UX: Primary Outcome

16. UX: Primary Outcome
Increased user satisfaction and trust

17. ux

18. ux
Trust +Satisfaction*

19. ux
Trust +Satisfaction*
*Casal6,LuisV.(2010)."GeneratingTrustandSatisfactionin E-Services:The Impactof Usability onConsumer
Behavior".Journalof RelationshipMarketing 9 (4),p.247.

21. What do users think they will get?

22. What do users think they will get?
What do users actually get?

23. What do users think they will get?
What do users actually get?
How do they feel about that?

24. UX + Kanye

25. [t iFollowKANYE WEST O
.. @kanyewest
if you put a link to my site people will be able to chop
& screw the video. this is a step towards the
possibilities of internet based video
10:46 AM - 22 Jul 2013
2,734 RETWEETS 1,604 FAVORITES

26. KANYE WEST O
@kanyewest
[ Follow
please don't diminish the user experience byjust
posting it as a normal video ...
my site
10:46 AM - 22 Jul 2013
please link them to
3,278 RETWEETS 2,165 FAVORITES

27. UX + Healthcare.gov

28. HealthCare.gov Get Insurance Log in

29. ux

30. ux
Establishing trust by meeting expectations

31. ux
Establishing trust by meeting expectations
Long-term positive effects of informed* design

32. ux
Establishing trust by meeting expectations
Long-term positive effects of informed* design
*by user data

33. A LANDSCAPE OF USER RESEARCH METHODS
BEHAVIORAL
ATIITUDINAL
© 2014
Christian Rohrer
I Eyetracking Clickstream Analysis
9 A/B Testing
I I Usability Benchmarking (in lab)
Usability Lab Studies Moderated Remote Usability Studies
t
..• Ethnographic Field Studies
i
Unmoderated Remote Panel Studies Unmoderated UX Studies
.• True Intent Studies ........ ..1
• e • • • I e I I I I • I I • • I I11I I I I IIII • I I 0 I• II I IIIII
+ Concept Testing :>
9 Diary/Camera Studies
9 Customer Feedback
+
6.
Desirability Studies )
Card Sorting - - -
+Participatory Design
Focus Groups
8 Interviews
9 Intercept Surveys
6.Email Surveys
QUALITATIVE {DIRECT) QUANTITATIVE {INDIRECT)
KEY FOR CONTEXT OF PRODUCT USE DURING DATA COLLECTION
Natural use of product
Scripted (often lab-based) use of product
De-contextualized I not using product
+ Combination I hybrid

34. Enter:

35. Enter:
A/ BTesting

36. What is A/ BTesting?

37. What is A/ BTesting?
A listening technique

38. What is A/ BTesting?
A listening technique
Real-time experiments on a site's live traffic

39. What is A/ BTesting?

40. What is A/ BTesting?
A quantitative behavioral UX research method
for asking and answering questions about
products and services

41. A = Original Version
B = Variation 1
n =Additional Variations

42. A = Original Version
B = Variation 1
n =Additional Variations
Isolate a design variable on the page, then
serve different variations randomly to a
portion of users.

43. Wait.

44. Wait.
That's crazy

45. Who else tests their live site with live users?

46. Who else tests their live site with live users?
Etsy

47. Who else tests their live site with live users?
Etsy
Google

48. Who else tests their live site with live users?
Etsy
Google
Twitter

49. Who else tests their live site with live users?
Etsy
Google
Twitter
and many others

52. Tons of active A/B tests and rampups.
www.e ty.com/llhng/99045805/ttddlt· alltn-h;at'ref• r
e o e 1:ll(dd·tAlltl'I Hat brbl• IOf> x
J: >
-User 5158774
rey Vour Aoooml Help SignOu1
I I
Etsy
"
roddrt Cort
Reddlt Allen Hat
26
I M l l

53. Matt Cutts O
@mattcutts
[W'Follow!
A/B testing can be really helpful. A classic blog post
from 2 0 0 9 : goo.gl/r45Xj Lesson: buttons get clicked
more.
4:45 PM - 15 Apr 2012
40 RETWEETS 63 FAVORITES

54. Experiments @ Twitter
Thursday,September 12, 2013 I ByAlex Roetter (@aroetter),VP, Engineering [19:14 UTC]
As we've talked about before, we're constantly working to make
it even easier to follow what you care about, connect with
people and discover something new on Twitter.
- -
1·
Tweet
To that end, we are constantly evolving the product. Some changes are visible -
they may help you protect your Twitter account or make it easier to share photos;
others are under-the-hood changes that help us suggest relevant content in real time
and make Twitter more engaging.
A common thread across recent releases has been experimentation. We've tested
various features with small groups of our 200 million users before determining what
we'll release. These tests are essential to delivering the best possible user
experience.
It's rare for a day to go by when we're not
releasing at least one experiment.
Tweet
We also experiment with features that may never be released to everyone who uses
Twitter. Those experiments are perhaps even more valuable because they help us
decide what not to do - which is important as we work to keep Twitter simple while

55. Key Quotes

56. Key Quotes
Etsy
---------
"We love experiments."

57. Key Quotes
Etsy---------
"We love experiments."
Google
--------------
"A/ Btesting can be really helpful."

58. Key Quotes
Etsy
---------
"We love experiments."
Google
--------------
"A/ Btesting can be really helpful."
Twitter
''It's rare for a day to go by when we're not
releasing at least one experiment."

59. A/ BTesting

60. A/ BTesting
Beloved

61. A/ BTesting
Beloved
Time-honored

62. GROUP I
-rnoo[!]
@) [§]
co[ID @)
[Q]
GROUP I l
- OJ(ID @)
GROUP m [fil CIDm
- [ii 00[Q]
GROUP N
OJ[ID [ID
(ID[!]
moooo
(Q1
(Q)[)
[ID{ID
rn
(ID [fil @

63. R.L. Deininger, "Human Factors Engineering
Studies of the Design and Use of Pushbutton
Telephone Sets," Bell Systems Technical
Journal (July 1960): 995-1012.
http://archive.org/details/bstj39-4-995

64. Key Quotes

65. Key Quotes
"Experimental Approach"

66. Key Quotes
"Experimental Approach"
"Design Variables"

67. Key Quotes
"Experimental Approach"
"Design Variables"
"Perhaps the most important factor in the
information processing is the individual himself."

68. A/ Btesting allows designers*
to be guided by users

69. A/ Btesting allows designers*
to be guided by users
*theweb committee

70. What does anA/ Btest
look like for libraries?

71. The A/ B Process

72. TheA/ B Process
Ask a question about your website

73. The A/ B Process
Ask a question about your website
Research the question with user interviews

74. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis

75. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
Define & run the experiment

76. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
Define & run the experiment
Collect data & analyze results

77. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
Define & run the experiment
Collect data & analyze results
Report to web committee & make decision

78. ..A!la... MONTANA I LIBRARY Ask A Librarian '
STATE UNIVERSITY
Inspiration, Discovery,Knowledge
CatSearch rSearch for articles, books, and more
Advanced Search
JI
News Hours Twitter
FIND REQUEST INTERACT Tip of the Week: ScanPro 2000
Find research
materials, including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Learn about the
library and meet with
us for research
assistance, writing
help, and tech
support
6/24 Library Workshop: Social
Media
6/18 Excel 2010: Formulas and
Functions [webinar]
Whois on First?: A Brief Bozeman
History
I: f
© MSU About MSU Library O. Accessibility Contact Us Privacy Policy o Mobile Site Help Site Index & Site Search

79. Find Request Interact

80. Find Request Interact
We thought we were so brilliant.

81. Find Request Interact
We thought we were so brilliant.
Guess who we didn't ask?

82. Find Request Interact
We thought we were so brilliant.
Guess who we didn't ask?
Our users.

84. Interact

85. Interact
About, Reference Services, FAQ, Staff Directory

86. Interact
About, Reference Services, FAQ, Staff Directory
2%of total homepage clicks

87. Ask a question about your website

88. Ask a question about your website
Why are Interact click actions so low?

89. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?

90. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?

91. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
Connect.

92. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
Connect. Learn.

93. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
Connect. Learn. Help.

94. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
Connect. Learn. Help. Services.

95. Research the question with user interviews

96. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."

97. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HIdidn't know that 'About' was under Interact."

98. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HIdidn't know that 'About' was under Interact."
''What am I interacting with?"

99. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HIdidn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."

100. Researchthe questionwith user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HI didn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
"Connect is better than Interact, but neither are very good."

101. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HIdidn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
"Connect is better than Interact, but neither are very good."
Hearn doesn't work."

102. Researchthe questionwith user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HI didn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
"Connect is better than Interact, but neither are very good."
Hearn doesn't work."
"Learn doesn't really work. Ijust think, what am I learning?

103. Researchthe questionwith user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HI didn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
"Connect is better than Interact, but neither are very good."
Hearn doesn't work."
"Learn doesn't really work. Ijust think, what am I learning?
I think of reading abook or something."

104. Research the question with user interviews

105. Research the question with user interviews
''Services is more accurate. Help is stronger."

106. Researchthe questionwith user interviews
''Services is more accurate. Help is stronger."
"I am not an English speaker, so I look for strong words.
I look for help, so Help is the best, then Services too."

107. Researchthe questionwith user interviews
''Services is more accurate. Help is stronger."
"I am not an English speaker, so I look for strong words.
I look for help, so Help is the best, then Services too."
"Help makes sense. When I'm in the library, and I think I need
help, it would at least get me to click there to find out what sort
of help there is."

108. Formulate a hypothesis

109. Formulate a hypothesis
Help or Services will generate more clicks and engagement
than Learn, Connect, and Interact.

110. Define & run the experiment

111. Define & run the experiment
Success metrics:
click-through rate, drop-off rate, homepage return rate

112. Define & runthe experiment
Success metrics:
click-through rate, drop-off rate, homepage return rate
Google Analytics ''Experiments" for the mechanics of A/ B

113. Define & runthe experiment
Success metrics:
click-through rate, drop-off rate, homepage return rate
GoogleAnalytics ''Experiments" for the mechanics of A/ B
CrazyEgg for click data & vizualization

114. • MONTANA I LIBRARY Ask A Librarian " '
_A!&.. SfATE UNIVERSITY
Inspiration, Discovery, Knowledge
- ...... ;.r;-
- - - i :
CatSearch iSearch for articles, books, and more
Advanced Search
)I
News Hours Twitter
FIND REQUEST INTERACT Tip of the Week: ScanPro 2000
Find research
materials, including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Learn about the
library and meet with
us for research
assistance, writing
help, and tech
support
6/24 Library Workshop: Social
Media
6/18 Excel 2010: Formulas and
Functions [webinar]
Whois on First?: A Brief Bozeman
History
t: f
© MSU About MSU Library b Accessibility Contact Us Privacy Policy o Mobile Site Help Site Index & Site Search

115. ...A!&... MONTANA I LIBRARY Ask A Librarian 9t,
>lATE UNIVERSITY
Inspiration, Discovery, Knowledge
CatSearch ISearch for articles, books, and more JI SEARCH )
Advanced Search
News Hours Twitter
FIND REQUEST CONNECT l ip of the Week: ScanPro 2000
Find research
materials, including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Learn about the
library and meet with
us for research
assistance, writing
help, and tech
support
6/24 Library Workshop: Social
Media
6/18 Excel 2010: Formulas and
Functions [webinar]
Who is on First?: A Brief Bozeman
History
t: f dm
© MSU About MSU Library b Accessibility Contact Us Privacy Policy o Mobile Site Help Site Index & Site Search

116. ..A!&... MONTANA I LIBRARY Ask A Librarian 9 'STATE UN IVERSITY
Inspiration, Discovery,Knowledge
CatSearch ISearch for articles, books, and more
Advanced Search
_ _ _ )ISEARCH
News Hours Twitter
FIND , REQUEST LEARN Tip of the Week: ScanPro 2000
Find research
materials, including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Learn about the
library and meet with
us for research
assistance, writing
help, and tech
support
6/24 Library Workshop: Social
Media
6/18 Excel 201O: Formulas and
Functions [webinar]
Whois on First?: A Brief Bozeman
History
t: f 1£1!.
Im
© MSU About MSU Library O. Accessibility Contact Us Privacy Policy o Mobile Site Help Site Index & Site Search

117. • MONTANA I LIBRARY Ask A Librarian 9e,
_A!.ti.._ SfATE UNIVERSITY
Inspiration ,Discovery, K nowledge
CatSearch rSearchfor articles, books, and more ) ( SEARCH
)
Advanced Search
News Hours Twitter
FIND
REQUEST HELP Tip of the Week: ScanPro 2000
Find research
materials, including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Learn about the
library and meet with
us for research
assistance, writing
help, and tech
support
6/24 Library Workshop: Social
Media
6/18 Excel 2010: Formulas and
Functions [webinar]
Who is on First?:A Brief Bozeman
History
t: f ii
© MSU About MSU Library b Accessibility Contact Us Privacy Policy o Mobile Site Help Site Index & Site Search

118. • MONTANA I LIBRARY Ask A Librarian 91?.,A!cl._ STATE UNIVERSITY
Inspiration, Discovery, Knowledge
CatSearch {Search for articles, books, and more
Advanced Search
)ISEARCH )
News Hours Twitter
FIND REQUEST SERVICES l ip of the Week: ScanPro 2000
Find research
materials, including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Learn about the
library and meet with
us for research
assistance, writing
help, and tech
support
6/24 Library Workshop: Social
Media
6/18 Excel 201O: Formulas and
Functions [webinar]
Who is on First?:A Brief Bozeman
History
t: f
© MSU About MSU Library b Accessibility Contact Us Privacy Policy a Mobile Site Help Site Index & Site Search

119. STANDARD REPORTS
Real-11me
•• Audience
-,. Aoquisttion
• Behavior
Overview
Behavior Flow
• Site Content
• Site Speed
t Sita Search
• Events
• AdSense
Experiments
In-Page AnalyUcs
I'll Conversions
• Goals
• Eoommerce
• Multi-Channel Funnels
Reporting
Experiment stopped before winner was found
Convonslon Raio .,,. vs . Select a melrio D• lf Week Month 9,612 Experiment Visits
21 days of data
100°/ovisitors Included
• Original - Interact
Variation 4 - Services
100%
Variation 1- Connect • Variation 2 - Learn • Variation 3 - Help
0% • • • • • • • • • • • • • • • • • • •
Status:
Notenough data to identify
winner
Jun 2 Jun 6 Jun 10 Jun 14
Primaryrnmonsion: Var at on
Variation Experiment Vlalta + PagH / Vialt Avg. Vlait Duration % New Vlalte Bounce Rat.
.,/ • Original - Interact @) 1,910 3.74 00:02:42 35.23% 39.35%
"' • Variation 3 - Help &9 1;983 3.03 00:02:42 29.80% 49.31%
./ • Variation 2 - earn @) 1,980 3.03 00:02:11 30.94% 45.79%
.,/ Variation 4 Services 1,94.t 2.95 00:02:10 32.96% 48.44%
1./ Variat on 1- Connect 1,795 3.90 00:02:18 36.19% 42.59%

120. Collect data & analyze results

121. Click-through rates - Homepage, Interact
I I Find - 4 0 %
11Search - 25o/o
Other - 24.5%
11Request - 7.7°/o
I I Interact - 2.8%

122. Click-through rates - Homepage, Learn
• Find- 44.2%
• Search - 25.2%
Other- 23.9%
• Request - 5%
• Learn- 1 .7%
Click-through rates - Homepage, Connect
• Find- 42.4%
• Search- 27%
• Other - 20.9%
• Request - 5.7%
• Connect - 4 %
Click-through rates - Homepage, Help
• Find - 44.8%
• Search - 26.3%
Other - 21.3%
• Request - 5%
• Help - 2 . 6 %
Click-through rates - Homepage, Services
• Find- 37.5%
• Search - 32.7%
Other - 1 9.3%
• Request - 5.8%
• Services - 4 .7%

123. Click-through rate - Homepage
5
1
Services Connect Interact Help Learn
Experiment Variables

124. Drop-off Rate
10
7.5 18
..Q
...)
.
a<t
:i
Q)
a16
: ......Q)
::t::
5
a:
Q)
12
0I
g.
a E0
l :
6
0
Interact Connect Learn Services
Experiment Variables
Homepage Return Rate
24
0
Connect Interact Learn
Experiment Variables
Services

125. Experiment Results

126. Experiment Results
More users are clicking through
and following through with Services.

127. Report to web committee & make decision

128. yfw MONTANA
STATE UNIVERSITY
I LIBRARY Ask A Librarian
_.A!&...
Inspiration, Discovery, Knowledge
CatSearch fSearch for articles, books, and more
Advanced Search
) AR
News Hours Twitter
FIND REQUEST SERVICES Tip of the Week: ScanPro 2000
Find research
materials, including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Learn about the
library and meet with
us for research
assistance, writing
help, and tech
support
6/24 Library Workshop: Social
Media
6/18 Excel 2010: Formulas and
Functions [webinar]
Whois on First?: A Brief Bozeman
History
t: f ti
© MSU About MSU Library b Accessibility Contact Us Privacy Policy o Mobile Site Help Site Index & Site Search

129. • MONTANA I L IBRARY As k th e Library 91!,
,...A!.I..._ STATE UNIVERSITY
CatSearch for articles,journ als, books and more
Advanced Search
Hours News Chat
FIND REQUEST SERVICES
Find research
materials, including
articles, books,
databases, journals ,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Meet w ith us for
research assistance,
writing help, and
tech support
Today
7 am - 9 pm
Tomorrow
7 am - 9 pm
More Hours

130. y(w MONTANA I LIBRARY Ask A Librarian . . .
..A!.&.. STATE UNIVERSITY
lrupiration,Discovery, Knowledge
CatSearch [ Search for articles, books, and more
Advanoed Search
SEARCH
News Hours Twitter
FIND REQUEST SERVICES l ip of the Week: ScanPro 2000
6124 library Workshop: Social
Media
Find research
materials, including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Learn about the
library and meet with
us for research
assistance, writing
help, and tech
support
6/18 Excel 2010: Formulas and
Functions [webinar)
Who is on First?: A Brief Bozeman
History
© MSU About MSU Library 0 Accessibility Contact Us Privacy Policy o Mobile Site Help Site Index & Site Search
Summer 2013
• MONTANA I LIBRARY Ask the Library
STATE UNIVERSITY
--CatSearch for articles,journals. books and more . .
-' -Advanced Search
FIND
Hours News Chat
Today
7am - 9 pm
Tomorrow
7am - 9 pm
More Hours
REQUEST SERVICES
Find research
materials,including
articles, books,
databases, journals,
and course reserves
Request resources
and services,
includinggroup
study rooms,
laptops.documents.
and books
Meet with us for
research assistance,
writing help,and
techsupport
Summer 2015

131. Let's see another example.

132. MSU Library Digital Initiatives and Digital Collections
Digital Home Photos and Papers Citation Databases
Glacier National Park, ca. 1920
Video and Audio
Search our digital collections...
About MSU Library Digital Initiatives
Educational Outreach
Montana State University Library Digital Collections has
historical photos, original rangelands science research
materials, academic peer reviewed bibliographies and
proceedings, and research databases of MSU Library's
Special Collections and Archives.
Research and Development
Services and Projects
Mission and Purpose
Contact Us
MSU Library Digital Collections cover a wide variety of
academic studies including:
• Rangelands management, ecosystem sciences, and snow
science
• Indigenous cultures and native art of the western plains
• Prominent Montana and regional historical figures and
events
• Montana State University, Bozeman, and Gallatin Valley
history
• Yellowstone Park ecosystem and history
© MSU About MSU Library b Accessibility Contact Us Privacy Policy CJMobile Site Help Site Index & Site Search

133. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
Define and run the experiment
Collect data & analyze results
Report to web committee & make decision

134. Ask a question about your website

135. Ask a question about your website
Users want easy access to discovery tools.
How can we best enable that?

136. MSU Library Digital Initiatives and Digital Collections
mgital Home Photos end Papers Citation Databases
Glacier National Park, ca. 1920
V deo and Audio
Search our digital collections... 0..
About MSU Library Digital Initiatives
Montana State University ibrary Digital Collections has
hlstorical photos, original rangelands science research
materials, academic peer reviewed bibliographies and
proceedings, and research databases of MSU Library's
Speclal Collections and Archives.
MSU l ibrary Dlgftal Collections cover a wide variety of
academic studies Including:
• Rangelands management, erosystem sciences, and snow
science
• Indigenous cultures and native art of the we.stem plains
• Prominent Montana and regional historical figures and
events
• Montana State University, Bozeman, and Gallatin Valley
history
• Yellowstone Park ecosystem and history
EducatlonalOuueach
Resea11chand Development
Services and Pro.!ects
Mission and Purpose
Contact Us
© MSU About MSU Library 6. Acces.sibillty Contact Us Privacy PoJicy D Mobile Site Help Site Index & Site Search

137. Ask a question about your website

138. Ask a question about your website
What are the primary actions on this page?

139. Ask a question about your website
What are the primary actions on this page?
Are users misled by the "Special Collections" link?

140. Ask a question about your website
What are the primary actions on this page?
Are users misled by the "Special Collections" link?
Is our landing page wording not concise?

141. Ask a question about your website
What are the primary actions on this page?
Are users misled by the "Special Collections" link?
Is our landing page wording not concise?
What if we streamlined our language?

142. [JiMSU Library Digital Initiatives and Digital Collections
Digital Home Photos and Papers Citation Databases
Glacier National Park, ca. 1920
Video and Audio
Search our digital collections...
About MSU Library Digital Initiatives
Educational Outreach
We offer access to rare and unique materials
for scholars, researchers, and the public. Research and Development
• Rangelands management, ecosystem sciences, and snow
science
• Indigenous cultures and native art of the western plains
• Prominent Montana and regional historical figures and
events
• Montana State University, Bozeman, and Gallatin Valley
history
• Yellowstone Park ecosystem and history
Services and Projects
Mission and Purpose
Special Collections & Archives
Contact Us
© MSU About MSU Library b Accessibility Contact Us Privacy Policy O Mobile Site Help Site Index & Site Search

143. 1IJMSULibraryDigital InitiativesandDigitalCollections
Digital Home PhotosandPapers CitationDatabases Video andAudio
Glacier NationalPark,ca.1920
Searchour digital collections...
Montana StateUniversity Library Digital Collectionshas
historical photos,original rangelands scienceresearch
materials,academic peerreviewed bibliographiesand
proceedings,and researchdatabasesofMSULibrary's
Special CollectionsandArchives.
MSULibrary DigitalCollectionscoverawidevarietyof
academic studies including:
• Rangelands management, ecosystem sciences,andsnow
science
• Indigenous culturesandnativeart ofthewesternplains
• Prominent Montanaandregional historicalfiguresand
events
• MontanaStateUniversity,Bozeman, andGallatinValley
history
• YellowstoneParkecosystem andhistory
About MSULibrary Digital Initiatives
EducationalOutreach
ResearchandDevelopment
ServicesandProjects
MissionandPurpose
Contact Us
©MSU AboutMSULibrary b Accessibility ContactUs PrivacyPolicy O MobileSite Help SiteIndex&SiteSearch
[!]MSULibrary Digital Initiatives andDigital Collections
Digital Home PhotosandPapers Citation Databases Video andAudio
GlacierNational Park,ca.1920
Searchour digital collections...
About MSULibrary DigitalInitiatives
Weoffer accessto rareanduniquematerials
for scholars,researchers, andthepublic.
EducationalOutreach
• Rangelands management, ecosystem sciences,and snow
science
• Indigenous culturesandnative artof thewesternplains
• ProminentMontanaandregional historicalfiguresand
events
• MontanaStateUniversity, Bozeman, andGallatinValley
history
• YellowstoneParkecosystem andhistory
ResearchandDevelopment
ServicesandProjects
MissionandPurpose
Special Collections&Archives
Contact Us
©MSU AboutMSULibrary b Accessibility Contact Us PrivacyPolicy O MobileSite Help SiteIndex&SiteSearch

144. Research the question with user interviews

145. Research the question with user interviews
''What's a digital collection?"

146. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."

147. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
"Maybe I can go to Special Collections to find things that I want?"

148. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
"Maybe I can go to Special Collections to find things that I want?"
"Honestly I don't read the page. My eyes just go past all that text."

149. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
"Maybe I can go to Special Collections to find things that I want?"
"Honestly I don't read the page. My eyes just go past all that text."
HI like search."

150. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
"Maybe I can go to Special Collections to find things that I want?"
"Honestly I don't read the page. My eyes just go past all that text."
HI like search."
New version: "It's clearer. There's just...less"

151. Formulate a hypothesis

152. Formulate a hypothesis
Astreamlined design will generate more searches.

153. Define & runthe experiment
Google Analytics ''Experiments" for the mechanics of A/ B
CrazyEgg for click data & vizualization

154. Define & runthe experiment
Success metrics:
total searches, pages per visit, average visit duration
Google Analytics ''Experiments" for the mechanics of A/ B
CrazyEgg for click data & vizualization

155. Digital Homepage - Special Collections II
v Stopped I Filters applied I View settings
Conversions I Page Metrics ....
Aug 7, 2013 -0 ct 10, 2013
• c• cD
• c
D
•a
c • c o
v
0
All Visits
100.00%
I Explorer
Conversions Site Usage Goal Set 1 Goal Set 2 Goal Set 3 Ecommerce
Experiment stopped before winner was found
Conversion Rate .... vs. Select a metric Day Week Month 1,395 Experiment Visits
65 days of data
100% visitors included
• Original
40%
• Variation
Status: 1
Not enough data to identify
winner
20%
Au... Aug 22 Sep 5 Sep 19 Oct 3
Primary Dimension: Variation
Variation Experiment Visits ..j, Pages I Visit Avg. Visit Duration % New Visits Bounce Rate
• Original 702 6.76 00:03:37 6.88% 40.97%
• Variation 693 11.38 00:04:30 11.14% 29.66%

157. Collect data & analyze results

158. Digital Collections Landing Page - Original
• Search
• Photos/Papers
Citations
• AudioN ideo
• Special
• Other
Digital Collections Landing Page - Variation
• Search
• Photos/Papers
Citations
• AudioN ideo
• Special
• Other

159. Original A vs. Variation B

160. Original A vs. Variation B
30%increase in Search

161. Original A vs. Variation B
30%increase in Search
68% Increase in Pages per Visit

162. Original A vs. Variation B
30%increase in Search
68% Increase in Pages per Visit
27%Increase in Average Visit Duration

163. Experiment Results

164. Experiment Results
Users are staying longer and viewing
more pages with the variation

165. Report to web committee & make decision

166. [JiMSU Library Digital Initiatives and Digital Collections
Digital Home Photos and Papers Citation Databases
Glacier National Park, ca. 1920
Video and Audio
Search our digital collections...
About MSU Library Digital Initiatives
Educational Outreach
We offer access to rare and unique materials
for scholars, researchers, and the public. Research and Development
• Rangelands management, ecosystem sciences, and snow
science
• Indigenous cultures and native art of the western plains
• Prominent Montana and regional historical figures and
events
• Montana State University, Bozeman, and Gallatin Valley
history
• Yellowstone Park ecosystem and history
Services and Projects
Mission and Purpose
Special Collections & Archives
Contact Us
© MSU About MSU Library b Accessibility Contact Us Privacy Policy O Mobile Site Help Site Index & Site Search

167. UX Lessons fromA/ BTesting

168. UX Lessons fromA/ BTesting
Be open to surprises.

169. UX Lessons fromA/ BTesting
Be open to surprises.
User behavior insights can be unexpected.

170. UX Lessons fromA/ BTesting
Be open to surprises.
User behavior insights can be unexpected.
We don't have all the answers.

171. UX Lessons fromA/ BTesting
Be open to surprises.
User behavior insights can be unexpected.
We don't have all the answers.
Experimentation helps inform decisions.

172. UX Lessons fromA/ BTesting

173. UX Lessons fromA/ BTesting
Good UX is built on good user data.

174. Other A/ BApplications

175. Amanda L. Goodman
@godaisies
[ W Follow ]
Just sent out my first A/B testing email newsletter campaign.
Let's see what happens...
4:18 PM - 31 Jul 2014
t.• * 1

176. http://www.slideshare.net/amandagoodman/ab-testing-in-library-emails

177. "At Darien Library, we are using A/B testing to discover how to
design email newsletters that get opened and acted upon. Our
most recent test of changing a subject line generated 10%
additional opens.Then by adding a digital service to an email,we
increased usage of the service by 15 1%"

178. GroupA· Winner! Group B
Mistakes were made. You Are What You Read!
abuse complaints O
total reclp ents 942
successful dellveries
bounces
91S(97 1 )
27 ( 9 )
recipients who opened
average times ,opened
total times opened 18
last n date
recipients who clicked
unique URUredpient dicks 22
s.
22
305(33.3Y:)
1 4 um
*
0 abuse compl
nts
942 total redp ems
(97 7'6}920 successful deliveries
(2 ) 22 bounces
(17 3?.>) , 59
recipients who opened
average times opened
total tlmes op ned
' 3
213
/27115 1:16PM last open date
(2 ) 20 recipients Who clicked
28 un que URUrecipient chdcs
i., clicks/unique o n
31 total clicks
1n1ns 1:1sPM
(0.0 *) 0
last click da te
times forwarded
0 forwarded
opens
unsubscribes
0
*
*
*1127/t S8 1SPM
clicks/unlque open
total dicks
****
16(2. )
last click date
times forwarded O(0
121611.!S'.!8?11.1
forwarded opens O
unsubscribes 0
Winner And Open Rate Champ on
Sent 1215114 S:OOPM
ce TryGroup B(loser)

179. - t
- Bohyun Kim
@bohyunkim
[ Follow ]
@godaisies is now talking about doing an A/B testing using
MailChimp to improve the reach of their library newsletter.
#alamw15 #uxig
10:07 AM - 1 Feb 2015
t.• *2

180. Other A/ BTesting Applications

181. Other A/ BTesting Applications
Digital Signage

182. Other A/ BTesting Applications
Digital Signage
Physical Signage

183. Other A/ BTesting Applications
Digital Signage
Physical Signage
MobileApp Notifications

184. Other A/ BTesting Applications
Digital Signage
Physical Signage
MobileApp Notifications
The Library building itself

185. Open Hardware &Ubraries
Imagine having a Google- Analytics- style dashboard for your Library
bui ding: number of visits, what patrons browsed, what parts of the Library
were busy during which parts of the day, and more. Measure the Fut ure is
going to make that happen by using simp le and inexpensive sensors that can
col ect data about bui ding usage that is now invisib le. Making these
invisib le occurrences explicit wi l al ow Librarians to make strategic decisions
that create more efficient and effective experiences for their patrons.

186. Building Analytics

187. BuildingAnalytics
Which stacks get the most traffic

188. BuildingAnalytics
Which stacks get the most traffic
Which stacks have high traffic but low circ rates

189. BuildingAnalytics
Which stacks get the most traffic
Which stacks have high traffic but low circ rates
What is the most popular piece of furniture in the library

190. BuildingAnalytics
Which stacks get the most traffic
Which stacks have high traffic but low circ rates
What is the most popular piece of furniture in the library
How many people look at a book display

191. BuildingAnalytics
Which stacks get the most traffic
Which stacks have high traffic but low circ rates
What is the most popular piece of furniture in the library
How many people look at a book display
"Once a library gather s data, it can conduct A/ B tests by creating
different arrangements and measuring the impact. I haven't been
this excited by a library project in quite some time."
- Aaron Schmidt

192. Other A/ BTools

193. Over 6,000 Happy Customers
FICO
I ..OXIINTERACTIVE MED IA
t l guardian CBS®
The
r fSNrpCompany

196. Experimentation in Libraries

197. Experimentation in Libraries
Conceptual
UX Improvements

198. Experimentation in Libraries
Conceptual
UX Improvements
Technical
Google Analytics

199. Experimentation in Libraries
Conceptual
UX Improvements
Technical
Google Analytics
Organizational

200. Experimentation in Libraries
Conceptual
UX Improvements
Technical
Google Analytics
Organizational
Open Communication

201. Experimentation in Libraries

202. Experimentation in Libraries
Constant beta is real.

203. - Jeremy G Snell
@jgsnell
[" # Follow
.@benbizzle "Websites have about a 4 year life cycle."
#il2013
12:37 PM - 28 Oct 2013
2 RETWEETS 2 FAVORITES
• t.+ *

204. [" # Follow- Jeremy G Snell
@jgsnell
.@benbizzle "Websites have about a 4 year life cycle."
#il2013
12:37 PM - 28 Oct 2013
2 RETWEETS 2 FAVORITES
• t.+ *
FollowBen Bizzle
@benbizzle
@jaclark @pcsweeney @jgsne ll @hello_scott_ I was
suggestion 4 years for total redesign. Small
enhancements during that time are a must.
2:52 PM - 28 Oct 2013
1 FAVORITE
• t.+ *

205. Experimentation in Libraries
Communicate
Communicate
Communicate

206. Experimentation in Libraries
Communicate
with public services
Communicate
Communicate

207. Experimentation in Libraries
Communicate
with public services
Communicate
with admistrators
Communicate

208. Experimentation in Libraries
Communicate
with public services
Communicate
with admistrators
Communicate
with users

209. Experimentation in Libraries

210. Experimentation in Libraries
There will be disruption
on the road to improvement.

211. Experimentation in Libraries

212. Experimentation in Libraries
lt1
s worth it because it works.

213. The A/ B testing process
provides the structure
to ask and answer questions
about our services and our users.

214. Caveats and Limitations

215. A/ Btesting is only one UX method
within a broader UX methodology
that balances various approaches

216. A/ B testing can answer what,
but it cannot answer why.

217. A LANDSCAPE OF USER RESEARCH METHODS
BEHAVIORAL
ATIITUDINAL
© 2014
Christian Rohrer
I Eyetracking Clickstream Analysis
9 A/B Testing
I I Usability Benchmarking (in lab)
Usability Lab Studies Moderated Remote Usability Studies
t
..• Ethnographic Field Studies
i
Unmoderated Remote Panel Studies Unmoderated UX Studies
.• True Intent Studies ........ ..1
• e • • • I e I I I I • I I • • I I11I I I I IIII • I I 0 I• II I IIIII
+ Concept Testing :>
9 Diary/Camera Studies
9 Customer Feedback
+
6.
Desirability Studies )
Card Sorting - - -
+Participatory Design
Focus Groups
8 Interviews
9 Intercept Surveys
6.Email Surveys
QUALITATIVE {DIRECT) QUANTITATIVE {INDIRECT)
KEY FOR CONTEXT OF PRODUCT USE DURING DATA COLLECTION
Natural use of product
Scripted (often lab-based) use of product
De-contextualized I not using product
+ Combination I hybrid

218. For Librarians new to UX

219. For Librarians new to UX
UX is a natural fit into existing
patron-focused library services

220. For Librarians experienced in UX

221. For Librarians experienced in UX
A/ Btesting is a valuable method that
can integrate into existing UX programs

222. - Followlindsey mclean
@lmclean001
Better user experience means I spend less time
teaching students how to use a website, eresource,
catalog, etc. #il2013 #internetlibrarian
3:18 PM - 29 Oct 2013
5 RETWEETS 3 FAVORITES
• t+ *

223. UX in Libraries

224. UX in Libraries
UX design can lead to better libraries,
andA/ Btesting is a valuable part
of that design process.

225. Other Resources

226. Other Resources
Aaron Schmidt
The User Experience LJ Column

227. Other Resources
Aaron Schmidt
The User Experience LJ Column
LibUX.co

228. Other Resources
Aaron Schmidt
The User Experience LJ Column
LibUX.co
Erin White

229. Other Resources
Aaron Schmidt
The User Experience LJ Column
LibUX.co
Erin White
Weaveux.org

230. •
--------------------------------------

231. uestions for you
-------------------------------

232. uestions for you
-------------------------------
What questions can you ask about your services?

233. uestions for you
-------------------------------
What questions can you ask about your services?
What A/ Btests can you conduct?

234. uestions for you
-------------------------------
What questions can you ask about your services?
What A/ Btests can you conduct?
How can UX grow at your library?

235. •
-------------------------------
ScottYoung
@hei_scott