Brian explains and presents case study material on how to measure intranet performance by observing people trying to carry out Top Tasks.
- Usability practices involved
- Clear reporting using simple measures
- Continuous improvement approach
5. The Kheel Center for Labor-Management Documentation and
Archives
https://creativecommons.org/licenses/by/2.0/
Observe users
trying to
complete
#TopTasks
Note
Success/Failure
and time taken
Use the Master Apprentice model
where users are the masters
and the observers are the apprentices
8. Example Performance Dashboard
Metric Current Task
Performance
Target
Performance
Current improvement
actions
Task: find a
person
45% 75% Implementation of new
SharePoint solution
News 37% 75% Improving archive function
Improving links guidelines
Benefits 45% 75%
Leave 27% 75%
Search Not yet
measured
Progress against plan
Common
architecture
9.
10. Step 2: Recruit Participants
Recruit 15-18
representative
people
11. Remote Measurement Sessions
1. Record session
2. “We are not measuring you”
3. Measure time for each task.
4. Observe problems and successes.
5. No “think aloud”, no discussions
12. Test representative tasks
Tasks Total Vote
% of Total
Vote 8730
Cumulative
Vote
Cumulative
Carewords
1
Find people (phone directory/book,
expert, employee photos) 915 10% 10% 1%
2
Pay / salary details, view your
paycheck, tax witholdings 818 9% 20% 3%
3 Board news (internal for employees) 815 9% 29% 4%
4
Benefits (enrollments, insurance
coverage, workers compensation) 453 5% 34% 5%
5
Leave and attendance (holidays,
maternity, sick, annual, overtime) 395 5% 39% 7%
6
Online applications (Oracle,
PeopleSoft, Discoverer, JP Morgan
Chase, PaymentNet, Travel Manager) 372 4% 43% 8%
7 NewsClips (from newspapers) 339 4% 47% 9%
8 Credit Union 282 3% 50% 11%
9 Shuttle schedule 282 3% 54% 12%
10 Policies and procedures 240 3% 56% 14%
11
Parking, carpools/vanpools, lottery,
passes 231 3% 59% 15%
12 Cafeteria menus 226 3% 61% 16%
13 Division news 221 3% 64% 18%
14
Calendars (events, meetings,
holidays, pay, training) 203 2% 66% 19%
15 Retirement (Thrift Plan, gifts, parties) 201 2% 69% 20%
16 Organization charts 183 2% 71% 22%
17 Forms, memo templates 154 2% 73% 23%
18 Visitors (tours, registration) 153 2% 74% 24%
Task ItB Find People poll result*
Find out which immediate section/unit XX works
in
1 – Employee’s immediate
unit/section
Find out which immediate section/unit XX works
in
1 – Employee’s immediate
unit/section
Find out who the Chief Financial Officer reports
to
3 – Place in organization chart
Find out who the Director of the Corporate
Affaires is
3 – Place in organization chart
XXis out of the office but you need to speak to
someone else on his team. Find another person
who works in the same immediate section/unit
4 – Who to contact if this person is
not around
Find the mobile phone number for XX 5 – Employee’s mobile phone
number
Find an expert in Web Usability 6 – Expertise
*Note: Poll result #2 was photo/picture
13. Find out which immediate section/unit
XX works in (top task #1)
2010
Sharepoint 6/2013
Sharepoint 9/2013
(after changes)
80%
20%
Succes
s
Failure
Median time on
task = 44
seconds
63%12%
25%
Succes
s
Failure
Median time on
task = 57
seconds
100
%
Succes
s
Failure Median time on
task = 32
seconds
14. Find out which immediate section/unit
XX works in (top task #1)
2010
Sharepoint 6/2013
Sharepoint 9/2013
(after changes)
Median time on
task = 35
seconds
100
%
Succes
s
Failure
Median time on
task = 14
seconds
100
%
Succes
s
Failure
68%6%
25%
Succes
s
Failure
Median time on
task = 38
seconds
15. Find out who the Chief Financial Officer
reports to (top task #3)
Old ItB (2010)*
Sharepoint 6/2013
Sharepoint 9/2013
(after changes)
40%27%
Success
Failure
Disaster33%
Median time on
task = 73
seconds
38%
50%
12%
Success
Failure
Disaster
Median time on
task = 135
seconds
100%
Success
Failure
Disaster
Median time on
task = 120
seconds**
(45 seconds)
**”All sites” link causes
problems
16. XX is out of the office but you need to speak to someone else on
his team. Find another person who works in the same immediate
section/unit (top task #4)
Old ItB (2010)*
Sharepoint 6/2013
60%
7%
Success
Failure
Disaster
88%
12%
Success
Failure
Disaster
33
%
Median time on task =
45 seconds
Median time on task =
57 seconds
17. Find out who the Director of the Office of
Inspector General is (top task #3)
Sharepoint 6/2013
Median time on
task = 122
seconds
38%
62%
Success
Failure
Disaster
Problems:
• Searching by division acronym “OIG” doesn’t help
• Mr. XXs title “Inspector General” led to confusion
• Section XX org chart in Sharepoint is useless
18. Find the mobile phone number
for XX (top task #5)
Sharepoint 6/2013
100
%
Succes
s
Failure
Median time on task = 14
seconds
19. Find an expert in web usability (top task
#6)
Sharepoint 6/2013 63%
12%
Success
Failure
Disaster
25%
Median time on task = 52 seconds
Problems:
• What does “Ask Me About” really mean?
Expertise?
• “All sites” tab causing problems
• Just not intuitive you can do this
20. Aggregate results
Old ItB (2010)
Sharepoint 6/2013
Sharepoint 9/2013
(after changes)
63%
24%
Succes
s
Failure13%
67%
12%
Succes
s
Failure
21%
Median time on task = 52
seconds
Median time on task = 76
seconds
100
%
Succes
s
Failure
Median time on task = 55
seconds**
(25
seconds)
**”All sites” link causes
problems
21. What does this mean?
What does this mean?
– People get slightly more accurate answers with Sharepoint,
but it takes them longer to find those answers. To vastly
improve the usefulness of Sharepoint and to make it easier
to Find People, we need to improve Sharepoint.
So, how do we improve?
22. How to improve 1 – the easy stuff
Simple, high impact improvements (what
worked in the September retests?)
– Explicitly labeling fields (Division, Section, Job Title,
etc)
Mysite pages
Search results pages
– Making it easy for people to search by both acronym
and full name
Divisions
Job titles
– Allow search by Division (full name and acronym) from
the main search box
23. How to improve 2
Other suggestions
– Get rid of “All sites” button
– Improve trust in the org chart viewer
Accuracy
Design (who reports to whom?)
– Taxonomy must work well or people will
quickly lose trust in it
– Improve uniformity (job titles, dealing with
misspellings, section names, division names,
etc)
24. MEASURE
• Find out who the Chief Financial Officer
reports to (top task #3)
brian@customercareowrds.com
Image from: Ari Bakker http://www.sharepointconfig.com/2011/05/how-to-create-a-simple-
sharepoint-2010-people-directory/
25. How to improve 1 – the easy stuff
Simple, high impact improvements (what
worked in the September retests?)
– Explicitly labeling fields (Division, Section, Job Title,
etc)
Mysite pages
Search results pages
– Making it easy for people to search by both acronym
and full name
Divisions
Job titles
– Allow search by Division (full name and acronym) from
the main search box