Slides used by Martin Bazley as part of UX Oxford day 18 March 2013 organised by the Bodleian Libraries

1. Understanding online
audiences
Planning and implementing
research into online audiences
UX day Oxford 18 March 2013
Martin Bazley
Online experience consultant
Martin Bazley & Associates

2. Martin Bazley
Previously
• Teaching (7 yrs)
• Science Museum, London,
Internet Projects (7yrs)
• E-Learning Officer, MLA South East (3yrs)
• Founder: Digital Learning Network DLNET

3. Martin Bazley
• Current
• Developing online resources, websites,
user testing, evaluation, training,
consultancy…
Martin Bazley & Associates
www.martinbazley.com
Slides and notes available afterwards

4. Note to self:
check stats
tomorrow to see
if anyone looked
up the website
www.martinbazley.com

5. How can we get a sense of who our online
visitors are and what they do with our online
content?
How do we gather data to help us improve
what we do?
How do we measure success from the
user's point of view, and / or against our own
objectives and constraints?
For example, how justify investment (or lack
of it) in social networks etc?

6. Reasons for doing audience research:
Evaluation
• Did your project/product/service do what
you wanted it to do?
• Provide information for stakeholders
• Gauge audience satisfaction

7. Reasons for doing audience research:
Promotion
• Improve your offer for your target
audiences
• Increase usage
• Widen access

8. Reasons for doing audience research:
Planning
• Inform development of a new
product/service
• Inform business planning
• Prove interest in a related activity

9. Data gathering tools
• Qualitative: focus groups, “free text”
questions in surveys, interviews
• Quantitative: web statistics, “multiple
choice” questions in surveys, visitor
tracking
• Observational: user testing, ethnographic

10. Define audience
Plan methodology
research goal
Use results to guide Collect data
changes
Analyse data

11. Define audience research
Plan methodology
goal
Use results to guide Collect data
changes
Analyse data

12. Define audience research
Plan methodology
goal
Use results to guide Collect data
changes
Analyse data

13. Define audience research
Plan methodology
goal
Use results to guide Collect data
changes
Analyse data

14. Define audience research
Plan methodology
goal
Use results to guide Collect data
changes
Analyse data

15. Define audience
Plan methodology
research goal
Use results to guide Collect data
changes
Analyse data

16. Strengths and weaknesses of different
data gathering techniques

17. Data gathering techniques
User testing
- early in development and again near end
Online questionnaires
– emailed to people or linked from website
Focus groups
- best near beginning of project, or at
redevelopment stage
Visitor surveys
- link online and real visits
Web stats
- useful for long term trends /events etc

18. Need to distinguish between:
Diagnostics
– making a project or service better
Reporting
– to funders, or for advocacy

19. Online questionnaires
(+) once set up they gather numerical and
qualitative data with no further effort –
given time can build up large datasets
(+) the datasets can be easily exported and
manipulated, can be sampled at various times,
and structured queries can yield useful results
(–) respondents are self-selected and this will skew
results – best to compare with similar data from
other sources, like visitor surveys
(–) the number and nature of responses may
depend on how the online questionnaire is
displayed and promoted on the website

20. Focus groups
(+) can explore specific issues in more
depth, yielding rich feedback
(+) possible to control participant
composition to ensure representative
(–) comparatively time-consuming
(expensive) to organise and analyse
(–) yield qualitative data only - small
numbers mean numerical comparisons
are unreliable

21. Visitor surveys
(+) possible to control participant
composition to ensure representative
(–) comparatively time-consuming
(expensive) to organise and analyse
(–) responses can be affected by various
factors including interviewer, weather on
the day, day of the week, etc, reducing
validity of numerical comparisons between
museums

22. Web stats
(+) Easy to gather data – can decide what to
do with it later
(+) Person-independent data generated - it
is the interpretation, rather than the data
themselves, which is subjective. This
means others can review the same data
and verify or amend initial conclusions
reached

23. Web stats
(–) Different systems generate different data
for the same web activity – for example no
of unique visits measured via Google
Analytics is generally lower than that
derived via server log files
(–) Metrics are complicated and require
specialist knowledge to appreciate them
fully

24. Web stats
(–) As the amount of off-website web activity
increases (e.g. Web 2.0 style interactions)
the validity of website stats decreases,
especially for reporting purposes, but also
for diagnostics
(–) Agreeing a common format for
presentation of data and analysis requires
collaborative working to be meaningful

25. Online surveys
SurveyMonkey
www.surveymonkey.com

26. Web stats
Google Analytics

27. Learn GA: short intro videos etc
https://www.google.com/analytics/iq.html

28. The best way to learn GA
is to use it:
www.google.com/analytics/

29. Web stats: Focus on trends
rather than absolute values

30. The ‘long tail’
An example of a power law graph showing popularity
ranking. To the right is the long tail; to the left are the
few that dominate. Notice that the areas of both
regions match. [Wikipedia: Long Tail]

31. The ‘long tail’
The tail becomes bigger and longer in new markets (depicted in
red). In other words, whereas traditional retailers have focused
on the area to the left of the chart, online bookstores derive
more sales from the area to the right.[Wikipedia: Long Tail]

32. SCA guidance
http://sca.jiscinvolve.org/wp/audience-publications/
Good overview
Step by step approach
Culture 24 Let’s Get Real
http://weareculture24.org.uk/projects/action-research/

33. More information / advice /
ideas
Happy to help - phone number on site:
Martin Bazley
0780 3580 737
www.martinbazley.com

34. Extra slides
not used in session
Some of these may be useful -
please feel free to call for
clarification or more info

35. When to evaluate or test and why
• Before funding approval – project planning
• Post-funding - project development
• Post-project – summative evaluation

36. Testing is an iterative process
Testing isn’t something you do once
Make something
=> test it
=> refine it
=> test it again

37. Before funding – project planning
• *Evaluation of other websites
– Who for? What for? How use it? etc
– awareness raising: issues, opportunities
– contributes to market research Research
– possible elements, graphic feel etc
• *Concept testing
– check idea makes sense with audience
– reshape project based on user feedback
Focus group

39. Post-funding - project development
• *Concept testing
– refine project outcomes based on
feedback from intended users
Focus group
• Refine website structure
– does it work for users?
One-to-one tasks
• *Evaluate initial look and feel
– graphics,navigation etc
Focus group

42. Card sorting - get various people to try out the
website structure before you build it

44. Post-funding - project development 2
• *Full evaluation of a draft working
version
– usability AND content: do activities work, how
engaging is it, what else could be offered, etc
Observation of actual use of website
by intended users,
using it for intended purpose,
in intended context – workplace, classroom, library, home, etc

56. Post-funding - project development 3
• Acceptance testing of ‘finished’
website
– last minute check, minor corrections only
– often offered by web developers
• Summative evaluation
– report for funders, etc
– learn lessons at project level for next time

57. Website evaluation and testing
Need to think ahead a bit:
– what are you trying to find out?
– how do you intend to test it?
– why? what will do you do as a result?
The Why? should drive this process

58. Evaluating online learning
resources in the classroom
Martin Bazley
Online experience consultant

59. Key point:
for a site designed for schools,
the most effective user testing observations
will be made in a real classroom situation

60. National Archives Moving Here project
For teachers of 8 – 14 yr olds
History Geography and Citizenship
Features: Interactives, activity sheets, audio and video clips

61. Moving Here Schools:
For 8 – 14 yr olds studying:
History Geography and Citizenship
Features:
Interactives, activity sheets, audio and video
clips

62. 1. preliminary testing
sessions –
conventional user-testing
with teachers (at TNA)

63. 2. in-class testing –
teachers used the Moving
Here Schools site with pupils
in their own classrooms
This meant sitting at the back of the classroom
observing and taking notes…

64. Evaluation: 2-phase approach

65. Site ready in parts – but not too ready:

66. The environment had a significant
impact on how the site was used.
The class dynamic within the
different groups contributed to
how much the students learned.

67. The environment and social dynamics
The environment had a significant
impact on how the site was used.
The class dynamic within the
different groups contributed to
how much the students learned.

68. in-class testing picked up elements
not there in conventional user
testing.
teachers in preliminary user testing
did not spot some problems until
actually in the classroom. For
example…

69. interactive activities:
looked big enough when
viewed on a screen
nearby…

71. … but text/images too
small for some children to
see from the back of the
class…

73. …so interactives needed
to be viewable full-screen

74. Only spotted during in-class testing:
…so interactives needed
to be viewable full-screen

75. content:
when students tried to read
text out loud, teachers
realised some text was too
difficult or complex

76. activity sheets:
some sheets did not have
spaces for students to put
their names - caused
confusion when printing 30
at same time…

77. Manchester Art Gallery art interactive
For teachers of 8 – 11 yr olds, and for pupils
History Art and Citizenship
Features: interactive with built in video, quiz, etc,
plus activity sheets and background info

79. Martin Bazley

80. Martin Bazley

81. Martin Bazley

82. Martin Bazley

83. 'This classroom user testing is all very well, but...'
How can you see everything in a class of 30
children – don't you miss things?
You see things in a classroom that
don't arise in one-to-one testing
They are the real issues

84. 'This classroom user testing is all very well, but...'
How can you see everything in a class of 30
children – don't you miss things?
You see things in a classroom that
don't arise in one-to-one testing
They are the real issues

85. 'This classroom user testing is all very well, but...' but...'
'This classroom user testing is all very well,
Doesn't using a specific class with particular
needs skew the results?
» For example, low ability, poor English, equipment not working,
behaviour issues, etc - are results as reliable as those in a
'neutral' environment?
» ‘neutral environment’ ? – no such thing - any test will be
subjective, and in any case:
» Testing is to make website work well in classroom, - need to
see effects of factors like those.

86. 'This classroomclassroom user testing is very well,but...'
'This user testing is all all very well, but...'
Doesn't using a specific class with particular
needs skew the results?
» For example, low ability, poor English, equipment not working,
behaviour issues, etc - are results as reliable as those in a
'neutral' environment?
» ‘neutral environment’ ? – no such thing - any test will be
subjective, and in any case:
» Testing is to make website work well in classroom, - need to
see effects of factors like those.

87. 'This classroomclassroom user testing is very well,but...'
'This user testing is all all very well, but...'
Can't my Web developer do the testing for us?
» best not to use external developer to do user
testing - conflict of interest
» also likely to focus more on the technical
aspects of the site than on effect on the
teacher and pupils.
» observe classes yourself but use an
independent evaluator for key decision
points

88. 'This classroomclassroom user testing is very well,but...'
'This user testing is all all very well, but...'
Can't my Web developer do the testing for us?
» best not to use external developer to do user
testing - conflict of interest
» also likely to focus more on the technical
aspects of the site than on effect on the
teacher and pupils.
» visit a classroom yourself but use an
independent evaluator for key decision
points

89. 'This classroom user testing is all very well, but...' but...'
'This classroom user testing is all very well,
I don't have the time or budget to do this!
» need cost no more than conventional user testing. one person
could attend a one-hour class session in a school, giving the
teacher the same small token payment
» This programme had evaluation built into project: 6.7% of
total Schools site budget.
» Allow 5 -10% of total project budget for user testing
=> videos

90. Video clips
• Moving Here key ideas not lesson plans etc
http://www.vimeo.com/18888798
• http://www.vimeo.com/18892401 Lesson
starter
• Time saver http://www.vimeo.com/18867252
S

91. User test early
Testing one user early on in the project…
…is better than testing 50 near the end

92. Two usability testing techniques
“Get it” testing
- do they understand the purpose, how it
works, etc
Key task testing
- ask the user to do something, watch how
well they do
Ideally, do a bit of each, in that order

94. User testing – who should do it?
• The worst person to conduct (or interpret)
user testing of your own site is…
– you!
• Beware of hearing what you want to
hear…
• Useful to have an external viewpoint
• First 5mins in a genuine setting tells you
80% of what’s wrong with the site

95. Strengths and weaknesses of different
data gathering techniques

96. Data gathering techniques
User testing
- early in development and again near end
Online questionnaires
– emailed to people or linked from website
Focus groups
- best near beginning of project, or at
redevelopment stage
Visitor surveys
- link online and real visits
Web stats
- useful for long term trends /events etc

97. Need to distinguish between:
Diagnostics
– making a project or service better
Reporting
– to funders, or for advocacy

98. Online questionnaires
(+) once set up they gather numerical and
qualitative data with no further effort –
given time can build up large datasets
(+) the datasets can be easily exported and
manipulated, can be sampled at various times,
and structured queries can yield useful results
(–) respondents are self-selected and this will skew
results – best to compare with similar data from
other sources, like visitor surveys
(–) the number and nature of responses may
depend on how the online questionnaire is
displayed and promoted on the website

99. Focus groups
(+) can explore specific issues in more
depth, yielding rich feedback
(+) possible to control participant
composition to ensure representative
(–) comparatively time-consuming
(expensive) to organise and analyse
(–) yield qualitative data only - small
numbers mean numerical comparisons
are unreliable

100. Visitor surveys
(+) possible to control participant
composition to ensure representative
(–) comparatively time-consuming
(expensive) to organise and analyse
(–) responses can be affected by various
factors including interviewer, weather on
the day, day of the week, etc, reducing
validity of numerical comparisons between
museums

101. Web stats
(+) Easy to gather data – can decide what to
do with it later
(+) Person-independent data generated - it
is the interpretation, rather than the data
themselves, which is subjective. This
means others can review the same data
and verify or amend initial conclusions
reached

102. Web stats
(–) Different systems generate different data
for the same web activity – for example no
of unique visits measured via Google
Analytics is generally lower than that
derived via server log files
(–) Metrics are complicated and require
specialist knowledge to appreciate them
fully

103. Web stats
(–) As the amount of off-website web activity
increases (e.g. Web 2.0 style interactions)
the validity of website stats decreases,
especially for reporting purposes, but also
for diagnostics
(–) Agreeing a common format for
presentation of data and analysis requires
collaborative working to be meaningful

104. More information / advice /
ideas
Martin Bazley
0780 3580 737
www.martinbazley.com

105. SCA guidance
http://sca.jiscinvolve.org/wp/audience-publications/
Good overview
Step by step approach
Culture 24 Let’s Get Real
http://weareculture24.org.uk/projects/action-research/

106. Crit room
‘simulated user testing’

107. Crit room protocol
Simulating user testing – usually one-to-one
in quiet room
No one (especially site stakeholders) other
than tester say anything for first part of
session
In this simulation we will focus on
- Look and feel of site
- Usability
- Content

108. More information / advice /
ideas
Happy to help - phone number on site:
Martin Bazley
0780 3580 737
www.martinbazley.com