FIndings from a Loop11 remote testing phase

1. Findings: Loop 11
Remote User testing report
1

2. Contents
Methodology
3
Summary
4
Detailed analysis
6
Task 1
6
Task 2
10
Task 3
13
Task 4
16
Task 5
21
2

3. Methodology
Remote user testing was conducted via an online testing toolkit to cast a wider net across users of the
current site within NSW health. Five tasks were devised to target key user activities as well as explore
problem areas identified in the emerging research from the Heuristic evaluation, Google analytics data
and user interviews. These tasks were worded using a combination of natural language and terms that
match exactly the content.
A total of 43 staff from across the state participated in the test. Participants were given the choice to
abandon or complete each task. Users who spend less than 10 seconds on any task were filtered out
from that task only.
The data available from the online testing toolkit* allowed us to view total time taken by each participant
as well as a screen-by-screen breakdown of their user journey through each task. Findings focus on
the following attributes from each participants’ journey as a way of breaking down the experience:
• Initial search behaviour (Eg, Quick search Vs All Indicators)
• Lateral search behaviour (related indicators or change view)
• Total time taken
• Total pages viewed
• Total indicators viewed
The following report delivers a summary of the data gathered, and identifies a set of opportunities that
present themselves as solutions to problem areas.
* Limitations in the data available include an ability to see which tabs were opened or closed on a page and data relating
to the Quick Search. The My Report feature, which is php driven was also difficult to track using remote testing.
3

4. Summary:
The majority of tasks had around a 50% success rate. The ‘success’ group was more likely to access
more pages and more indicators in less time.
The ‘grey zone’ lies between the success and fail-abandon groups and tell us about particularly
interesting patterns where users are almost able to successfully complete the task. The ‘grey zone’
group typically spends the longest on tasks, yet arrives at the least pages and indicators.
The ‘fail/abandon’ group on average spent less time and accessed less pages than the ‘success’
group, suggesting that they came across barriers to content or had difficulty accessing the right
content.
Success
Grey zone
Fail/Abandon
2.9 mins
Task 1
9 pages
5 indicators
Task 2
Task 3
5 pages
2 indicators
4 mins
Task 4
Task 5
0
25
50
75
100
2 mins
6 pages
3 indicators
4

5. 9 pages
2.9 mins
5 indicators
5 pages
4 mins
2 indicators
6 pages
2 mins
3 indicators
5

6. Themes for failure, abandon and
time-consuming user journeys:
Opportunities for an improved user
experience:
High bounce rate at the home page - users
not knowing how or where to begin task.
Improving pathways to content from the home
page
High bounce rate at Indicator pages - users
not knowing how or where to begin lateral
searching or data slicing.
Users leaving Indicator pages and searching/
browsing then repeating the process to arrive
at data slices or related content.
Users spending the overwhelming
percentage of their task navigating lists of
links in ‘Browse’ methods such as All
Indicators, Topics etc.
Users arriving at many indicators that
• were related to the indicator that they
were after, but didn’t contain the precise
information they needed
• included the precise information they
needed among other superfluous data
Creating a stronger design focus for lateral
searching techniques (related indicators and
data slices) on Indicator pages.
Create more efficient navigation for information
categories in pages that use lists of links.
Increasing visibility of filter and other subcategories of data slice tools at a higher
navigation level
Improve match between data slice titles and
Indicator content
6

7. Task 1: Find specific Indicator
“You are researching Antenatal care across NSW.
Download a graph that shows the percentage of mothers who made an
antenatal visit before 14 weeks in 2010.”
AIM:
•to observe participant methods when conducting a targeted search.
•determine barriers and accelerators to arriving at specific content.
•NB. due to the limitations of the data available it was not possible to see when the download tab
was opened on the site. ‘Success’ was therefore defined as having arrived at the correct Indicator.
20 Participants
62% Successfuly found the Indicator
9.5% Found the Indicator by LGA (‘Grey’ succcess area)
28.5% Abandoned the task
7

8. The ‘Grey zone’
The ‘Grey zone’ of success in this task denotes participants who arrived at the indicators to suit their
purpose, but not precisely. This shows barriers to navigating related content. ‘Grey zone’ participants
spent longer on their task and viewed more pages and indicators than other participants, yet were not
able to find the precise indicator.
80 %
60 %
40 %
20 %
0
All
Quick
None
Mothers and BabiesMothers Topics
Quick Search Topics All Indicators - Home Page
report
None
Indicators
Search
and
Babies
How they got there
•
•
•
Both browse and search are highly represented among the ‘successful’ and ‘grey zone’ participants.
Evidence suggests that browsing participants were using the browsers’ find-on-page function to arrive
at indicators instead of the website’s navigation. This explains why related indicators such as ‘by LGA’
and ‘by Aboriginality’ were accessed from the Mothers and Babies report.
Those that abandoned the task largely did so on the home page, however, journeys of up to 13 minutes
were taken by those who browsed All Indicators.
8

9. 5 pages viewed
1.8 mins
1 indicator viewed
6 pages viewed
5.1 mins
2 indicators viewed
2 pages viewed
2.5 mins
0 indicators viewed
Main reasons for longer journeys
•
•
•
Successful participants - navigating home page and navigating topics
Grey zone participants - navigating home page and navigating mothers and babies report
Abandon participants - navigating all indicators and navigating mothers and babies report
9

10. Opportunities for improvement
•
•
•
More transparent pathways to related content on Indicator pages
More efficient navigation for information categories in on pages with lists of links (E.g, html reports,
All Indicators and Topics).
Stronger design focus on pathways to content from homepage
10

11. Task 2: Finding related content
“You are researching Antenatal care over NSW, and you have found this
indicator trend shown below. Now download related indicators that
might be useful to your research.”
AIM:
•determine the availability of related content.
•determine the usability of the download function.
•NB. due to the limitations of the data available it was not possible to see when the download tab
was opened on the site. ‘Success’ was therefore defined as having arrived at the correct Indicator.
20 Participants
45% Successfuly found content at Related Indicator tab
55% Failed to find content and Related Indicator tab
11

12. 5 pages viewed
1 min
3 pages viewed
1.6 mins
The majority of participants failed to find content through the Related Indicator tab. Although less
pages were accessed by this group, they typically spent longer on the task, indicating an issue with
the visibility of related content at Indicator pages.
100%
How they got there
•
75%
50%
•
25%
0
Change
Related Topics Quick Search
related indicator
topics Quick change view None
nothing
Indicator
Tab
Search
View
Although all successful participants used the
related indicator tab, a few continued to
search for related content elsewhere.
The majority of those that failed the task took
no action at all. Others looked to Change View
or went out of the Indicator page, back to the
home screen to conduct a Quick Search showing a lack of awareness for lateral or
context-driven search methods.
12

13. Opportunities for improvement
•
•
Increase visibility of related content and lateral search techniques at indicator views
Improve taxonomy relating to ‘Change View’ and ‘Related Indicators’ so that it is clear that
‘Change View’ does not list all indicators relating to the one visible.
13

14. Task 3: Data slicing with Change View
“You are researching Antenatal care over NSW, and you have found this
indicator trend shown below. Now you want to compare antenatal trends
of ‘Major Cities’ against the rest of NSW. Find the graph that shows
this.”
AIM:
•observe how users are investigating specific findings through data slicing
•determine the usability of ‘change view’ drop down menu
20 Participants
58% Successfuly used Change View to find the Indicator
21% Used Change View to find an incorrect Indicator
21% Abandoned the task
14

15. The ‘Grey zone’
There ‘Grey zone’ of success in this story explains the proportion of participants who went directly to
Change View as a way of accessing different data slices, but who did not arrive at the correct
Indicator. This tells us that users were either not able to match the natural wording of the question to
the drop down titles, and/or that the drop down titles do not effectively match the content shown in the
Indicator slices that they point to.
100 %
75 %
50 %
25 %
0
How they got there
•
•
•
via a
directly
Topics
Quick
No
correct slice
incorrect slice search Topics
quick
None - Action Page
Home
number
to correct
Search
of slices
slice
58% of the Success group went directly to the correct Indicator via the Change View menu, followed by
33% viewing a number of indicators in the Change View menu first.
The ‘Grey’ success group are characterized by looking only at incorrect indicators in the Change View
menu
The fail group are halved between not acting at all, and navigating to ‘topics’ in order to find the
Indicator described in the question.
15

16. 4 pages viewed
1.4 mins
2 indicators viewed
3 pages viewed
2 mins
2 indicators viewed
3 pages viewed
1 min
0 indicators viewed
Main reasons for longer journeys
•
•
•
Successful participants - difficulty perusing Change View drop downs and filters.
Grey zone participants - difficulty perusing Change View drop downs and filters.
Abandon participants - not aware of Change View functionality at all. Time spent on Topics page.
Opportunities for improvement
•
•
•
•
Improved match of taxonomy to Indicator content
Improved match of taxonomy to users language
Visibility of filter and other sub-categories of data slice tools at a higher navigation level Improved
match of taxonomy to Indicator content
Stronger design focus on data slice tools at Indicator page
16

17. Task 4: Using ‘My Report’
“You are giving a presentation on Smoking in NSW. Tailor and download
a PDF report that contains all content from the following 3 indicators:
‘Smoking Status, trends’, ‘Smoking Status, area trends’, and ‘Current
Smoking by Age, trends.’”
AIM:
•To determine usability of ‘My Report’ feature
•To observe how users search for a number of related Indicators
NB. due to the technology behind the ‘My Report’ feature, results did not contain the data necessary
to collect data on participants success in using this feature. Success was therefore redefined as
having access all 3 of the said Indicators as well as ‘My Report’ page at some point in their journey.
16 Participants
53% Accessed all 3 indicators and My Report
47% Abandoned the task
17

18. All of the 53% that successfully accessed all indicators were also able to access My Report. Only 1
participant from the Abandoned-task group had accessed My Report in their task journey.
100 %
100 %
75 %
75 %
50 %
50 %
25
%
25
0
Found
found all 3
all 3
indicators
Found
Found
foundFound found 1
2
found none
only 1
no
only 2
indicator
indicators
indicators
0
%
Accessed My
accessed my report
Report
18

19. Initial Search Methods
Lateral Search Methods
100 %
80 %
75 %
60 %
50 %
40 %
25 %
20 %
0
0
Quick
Search
Topics
All
Indicators
Used
Change
View
Used
Related
Indicator
tab
How they got there
•
•
•
Majority of both groups used Quick Search as the initial method
Success group was more likely to search laterally from the Indicator page using both Change View
and Related Indicator tab.
Abandon group was more likely to browse through Topics and All Indicators, and less likely to
search laterally from the Indicator page using Change View and Related Indicator tab.
19

20. 18 pages viewed
11 indicators viewed
6.7 mins
14 pages viewed
4.5 mins
21%
21% of participants accessed
‘Using This Report’ during this task.
6 indicators viewed
The minimum time taken amongst Success-group was 1.5
minutes, suggesting that task-completion can be improved by the
opportunities listed above.
There is a level of ambiguity in this data relating to how much time
was spent on the My Report section. However, looking at the
Abandon-group alone, who did not access My Report - we can
see that there is a barrier to arriving at content easily for those
users who don’t use lateral-search methods. This high ratio
between pages viewed and indicators viewed could be
responsible for abandon rate at an overall shorter time threshold.
Interestingly, 21% of all participants accessed ‘Using This Report’
during this task.
20

21. Opportunities for improvement
•
•
Improved category navigation for browse-like searches such as Topics and All Indicators
Stronger design focus on lateral searching for related content from Indicator pages.
21

22. Task 5: Drilling down to specific data
“You are investigating data relating to Smoking in NSW and it’s impact
on the hospital system. Find out the total hospitalizations in 2011-12
caused by smoking in: 1. all of NSW, and 2. just the Sydney Local
Government Area. Download the 2 graphs.”
AIM:
•observe methods of arriving at 2 related Indicators based on location.
•determine barriers and accelerators to arriving at specific content.
•NB. due to the limitations of the data available it was not possible to see when the download tab
was opened on the site. ‘Success’ was therefore defined as having arrived at the correct Indicator.
13 Participants
61% Successfully accessed both indicators
39% Abandoned the task
22

23. 100 %
75%
50 %
25%
0
Found
found all 3
both
indicators
Found
Found
indicator
indicators
found 1 1 found none
no
only
How they got there
•
•
•
Majority of both groups used Quick Search as the initial method, closely followed by Topics page
in both groups.
Success group was only slightly more likely to search laterally from the Indicator page.
Abandon group was more likely to browse All Indicators page.
Initial Search Methods
Lateral Search Methods
70 %
80%
52.5 %
60%
35 %
40%
17.5 %
20%
0
0
Quick
Search
Topics
All
LHD
Indicators Indicators
No
Action
Used
Change View
23

24. 3.6 mins
14 pages viewed
7 Indicators viewed
9 pages viewed
2.3 mins
5 indicators viewed
Main reasons for longer journeys
•
All participants - difficulty finding specific indicators in Topics, All Indicators and Indicators by LHD
pages. Barriers to content are evident in 2:1 ratio above of pages to Indicator content
Opportunities for improvement
•
•
Improved match of taxonomy to Indicator content
Simplified navigation through information categories on pages with lists of links (E.g, html reports,
All Indicators and Topics)
24