Dwayne Pearce INN530 Assignment 2

1. Conducted by Dwayne Pearce
INN530 – Online Information Services
Assignment 2
Usability and its use for Academic
Libraries: A case study of an online
learning tool

2. Presentation Agenda
Background to Project
Overview Key Concepts
Research Question, Objectives, and Outcomes
Data Collection and Analysis
Findings
Discussion - What can we learn from this?
Discussion – Limitations of Research
Recommendations

3. Background to Project
• This project is a part of a greater work that has been
commissioned by South Bank Institute of Technology to
investigate the perceived effectiveness of SAIL.
• The research used to help improve the product and provide users
with a more interactive and dynamic learning tool.
• The importance of usability testing to the overall redesign and
maintenance of an online based resource.
• The ways in which usability testing can improve the user
experience of an online resource.

4. Key Concepts
Usability
Iterative Design

5. Research Question, Objectives, and
Outcomes
Research question –
•How usable is the online tutorial tool SAIL from a student’s perspective?
Research Objectives
•The research project focuses on the following objectives:
•To explore the student user experience of SAIL through conducting usability testing.
•To examine the value of usability testing in libraries.
•To develop an understanding of the ways usability testing can be used by libraries to improve the
online information literacy tools they deliver.
•Research Outcomes
•The researcher expects the following outcomes from the project:
•To provide insight into the usability of the SAIL tutorial to inform further development of the site.
•To raise awareness of the value of usability testing in libraries.
•To contribute to further understanding about usability considerations for online information literacy
tools to the library and information studies research space.

6. Data Collection
Participants were selected based on the following criteria which have been outlined by Southbank Institute of Technology:
•The students must be a student of Southbank Institute of Technology.
•The student must be studying a course that is certified as Certification III and above.
A usability test was then conducted in which 5 participants will be observed using SAIL with a small list of tasks they must perform. This
tests all consisted of 3 main components:
•Task based usability testing – The participant must complete a series of tasks, and data will collected concerning completion time,
completion rate, and ease of use.
•Think aloud protocol - The test users are asked to verbalise their thoughts, feelings, and opinions while interacting with the system
(Nielsen, 1993).
•Post Session Questionnaire – The test user will be asked a series of questions based on their interaction with the SAIL tool.
The data was collected in two forms:
•Quantitative data: Success rates, task time, error rates, and satisfaction questionnaire ratings.
•Qualitative data: Observations about pathways participants took, problems experienced, comments, and answers to open-ended
questions.
(U.S. Department of Health & Human Resources, n.d., para. 2)

7. Data Analysis
Following the usability test, an analysis of the results was presented. This section contained an analysis of:
•The percentage of participants who completed each task, and all tasks.
•The average time taken to complete each task for those who completed the task.
•The satisfaction results.
•The tasks that had the highest and lowest completion rates.
•A summary of the successful task completion rates by participant, task, and average success rate by task and show the data in a table.
•An examination of the answers provided to open-ended questions.
•A summary of the post session questionnaire results
•A description of the data recorded during the usability test.
•An analysis of the problems encountered during the usability test.
(U.S. Department of Health & Human Resources, n.d., para. 2)
The data analysis was presented in the following forms:
•Severity ratings, which are comprised of:
•The frequency with which the problem occurs.
•The impact of the problem if it occurs.
•The persistence of the problem.
•(Nielsen, n.d., para. 2)
•Descriptive statistics that identify user performance issues such as errors and difficulties, and tasks that do not meet acceptable criteria e.g. the number of
errors that occur on tasks and the number of users who successfully perform tasks (University of Texas, 2011, para. 2).
•Thematic coding, identifying general trends or categories of users’ behaviour such as how well users can complete a task and where they are encountering
problems, users’ general perceptions and opinions of the technology’s usability (University of Texas, 2011, para. 3).
•Observation notes that describe and identify themes and trends in the data collected which will then be presented in a table format (University of Texas,
2011, para. 4).

8. Findings – Quantitative Results
Task Number and Task Description Percentage of participants who completed each
task
Task 1: Use SAIL to reference a journal article
using APA Referencing conventions.
100%
Task 2: Use SAIL to find an example of an
instruction word.
0%
Task 3: Use SAIL to find the section “How do you
get copyright permission?”
100%
Task 4: Use SAIL to find the section “The Rough
Draft.”
100%
Task 5: Use SAIL to find the section “Locating
information: Using databases.”
60%
All tasks 0%
Table 1:
The percentage of participants who
completed each task, and all tasks.

9. Findings – Quantitative Results
Task Number and Task Description Average time taken to complete task
Task 1: Use SAIL to reference a journal article
using APA Referencing conventions.
30 seconds
Task 2: Use SAIL to find an example of an
instruction word.
1:30:00 (100% unsuccessful)
Task 3: Use SAIL to find the section “How do
you get copyright permission?”
15 seconds
Task 4: Use SAIL to find the section “The Rough
Draft.”
25 seconds
Task 5: Use SAIL to find the section “Locating
information: Using databases.”
45 seconds
Table 2:
The average time taken to complete each task
for those who completed the task.

10. Findings – Quantitative Results
Task Number and Task Description Average Satisfaction result
Task 1: Use SAIL to reference a journal article
using APA Referencing conventions.
0
Task 2: Use SAIL to find an example of an
instruction word.
3 (100% unsuccessful)
Task 3: Use SAIL to find the section “How do
you get copyright permission?”
0
Task 4: Use SAIL to find the section “The Rough
Draft.”
2
Task 5: Use SAIL to find the section “Locating
information: Using databases.”
2
Table 3:
Satisfaction results
Scale used –
0 = User completed task with zero difficulty. (Zero
Frustration)
1 = User completed task with only minor problem(s).
(Little Frustration)
2 = User completed task, but it required more
effort/time/dead-ends than the user expected.
(Medium/High Frustration)
3 = User did not complete task. (Point of Failure)

11. Findings – Qualitative Results
Task Number and Task Description Recurring words, phrases, and expressions
expressed throughout the Think Aloud protocol
Task 1: Use SAIL to reference a journal article
using APA Referencing conventions.
“Where is it?”, “Easy to find”, “Should be
separated from SAIL”, “Why is referencing part
SAIL?”
Task 2: Use SAIL to find an example of an
instruction word.
“I have no idea where to look”, “It isn’t obvious”,
“Do I click the boxes?” “I really have no clue” “It
seems important, so shouldn’t it be easier to
find?”
Task 3: Use SAIL to find the section “How do you
get copyright permission?”
“Easy to find”, “This was a lot easier to find”, “A
pop menu on the side would be nice, on the
copyright section”, “Found it easily”
Task 4: Use SAIL to find the section “The Rough
Draft.”
“Again, this was easy to find”, “Pretty easy to
find”, “Is it near Assignment Calculator? It should
be”, “Again, pop out menus would work better”
Task 5: Use SAIL to find the section “Locating
information: Using databases.”
“Can I click on the pictures?”, “What do the
pictures do?”, “Is it one of the pictures?” “It
should be at the beginning, as one of the
pictures”
Table 4:
Recurring words, phrases, and expressions
expressed throughout the Think Aloud protocol
for those who completed the task.

12. Findings – Qualitative Results
Task Number and Task Description Recurring words, phrases, and expressions
expressed throughout the Post Session
Questionnaire
What is your overall impression of SAIL? Words:
Clearer, bigger presence, hidden, reference tool, more
obvious, more accessible
Phrases:
“An effective tool to help students reference”, “It could
be more detailed”, “It is important to use, but it needs to
be a little clear” “Needs to be more obvious on the library
homepage, I didn’t know it was there”
From your experience, what did you like best
about SAIL?
Words:
Current, referencing help, assignment help, simple to use
Phrases:
“It provides a lot of information”, “Very simple, and
really helps provide an outline for assignments”, “ The
mind maps and other activities help get your mind
going”, “It is really good to return to freshen up on
assignment writing if you haven’t studied for a long time”
Table 5:
Recurring answers provided during the Post Session
Questionnaire

13. Findings – Qualitative Results
From your experience, what did you like least
about SAIL?
Words:
Lost, awkward, clumsy, hard to read, too
“wordy”, too few pictures
Phrases:
“ I think that the colour scheme is too dated”,
“It is too hard to read” “There is too much text”,
“It is too hard to navigate”, “I don’t like the
boxes”, “I find myself getting lost”
If you were the website developer, what would
be the first thing you would do to improve the
website?
Phrases:
“Make it more of a feature on the library page”, “Make it
more visible”, “Make it fit the page better, less
landscape”, “Create a more visually interesting home
page”, “Make the boxes easier to see”, “Put referencing as
a separate tab on the library homepage”, “Make the side
bar more functional”
Table 5 continued:
Recurring answers provided during the Post Session
Questionnaire

14. Findings – Qualitative Results
Is there anything that you feel is missing on this
site?
Phrases:
“A title at the top of the page to stop me getting
lost,” “Something that tells me where I am,
maybe at the top of the page so I don’t get lost”,
“The boxes aren’t functional enough, they need
to let you know where you are going”, “Pop
out menus”, “Better links on the sidebar”
If you were to describe this site to a colleague
in a sentence or two, what would you say?
Phrases:
“Invaluable to students, but needs to be promoted
better on the library site”, “Could be a useful tool if I
knew what was being offered in greater detail”, “Useful,
but too difficult to navigate”, “A great place to go for
referencing, but too hard to find your way around”,
“You might need to take a survival kit through if you get
lost”
Would you use SAIL again? Phrases:
All of the respondents interviewed said yes they would
use SAIL again, but would use it more frequently if the
site were changed in the ways they suggested.
Table 5 continued:
Recurring answers provided during the Post Session
Questionnaire

15. Discussion – What can we learn from
this?
This report seeks to bolster the belief that the iterative design
approach is “the only evidenced-based way to improve the quality of
computer-based systems, including Web sites and web applications”
(Usability.gov, 2005, para. 8).
Although this report only contains one usability test, and does not
document the redesign of the SAIL site that will follow this report,
the aim is to provide findings that promote the method of iterative
design usability testing, as the literature reviewed for this project
suggests that “revising a system based solely on one lone usability
study does not lead to better user performance or increased user
satisfaction” (Usability.gov, 2005, para. 8).

16. Discussion - Limitations
Selection bias on behalf of the researcher
Selection bias of Think Aloud Protocol and Post
Session Questionnaire Answers
Research not reaffirming the role of Iterative
Testing in the Usability Process

17. Recommendations
Recommendations for the future redesign of SAIL:
Decrease the amount of clicks a user needs to open the main SAIL page.
Make SAIL easier to navigate by developing side menus, pop out windows, and other
navigational tools that allow the user to see not only the page they are currently
visiting, but other pages that are associated with the module that they are viewing.
Separate the referencing component of SAIL from the current SAIL page to make it a
singular stand alone feature of the library site.
As iterative design promotes design, testing, redesign, and retesting, it is
recommended SAIL be redesigned and retested using this project as a means to frame
the next series of usability tests.