1. FORMAL ASSESSMENT OF CARELOGIC’S REPORTING
FUNCTION: PENTAHO
by
Christina Garner
Professor David Marc – College of St. Scholastica
HIF 6230 Section 700 – Human Factors and Usability – Spring I, 2015

2. Evaluate the usability of CareLogic’s reporting feature,
Pentaho, by identifying issues users may incur and
suggest design alternatives.
Purpose of study

3. Participant M/F Age Occupation EHR Experience
(years)
1 M 35 Product designer in
computer-aided
drafting (CAD)
0
2 F 37 Medical front office
supervisor
10
3 F 30 Medical back office
supervisor
7
Participants
Table 1. Description of study participants
Three participants who use computer applications in their daily work
were recruited for this study. They have no usability evaluation
experience however two are in the medical field.

4. Methods
Participant’s Task
 Create a report of patients’ diagnoses by city using the
following data fields : “City,” “All Diagnoses,” and
“Client ID”
 They were given an outline and logged into the
CareLogic system.
 Participants worked separately in the researcher’s office.
 Studies were conducted February 25-27 with one
participant per day.
 Time on task (TOT):
 36 minutes for Participant 1
 14 minutes for Participant 2
 17 minutes for Participant 3
Usability Methods
 Concurrent probing
 Quasi-concurrent think-aloud
 System Usability Scale (SUS)
 Voice data recorded and notes taken
 An analysis of the data revealed
several common issues.

5. Task
Figure 1. Task given to study participants

6. Desired Result
Figure 2. Partial screenshot of desired report

7. Results and recommendations:
Data Fields
Observation
 Data can be inputted on X or Y
axis but users did not know the
difference.
 High TOT to find data fields
 Two options for inserting data
fields into the table: double-
click or drag
Recommendation
 Rather than have the X and Y
axis lines appear when the data
is dragged, have them there
permanently until data is
inputted. Consider adding an
instruction icon.
 Add search function
 A tip that appears when data is
picked up would be helpful.

8. Results and recommendations: Data
Fields (Con’t)
Observation
 Users said the data headings
should be collapsed.
 Alphabetical order
 Multiple data fields with same
name
Recommendation
 Data field headings should be
collapsed by default.
 Include an “A-Z” clickable link
above data fields in addition to
the aforementioned search
bar.
 Each data field needs a unique
name.

9. Results and recommendations: Tips
and Help Features
Observation
 Error messages
 “Hide Tips” is hidden and
doesn’t return
 How do you move data?
 Several ways to delete or undo
 Count vs. count distinct
Recommendation
 Instruct users how to fix the
problem in error messages
 “Show Tips” should appear after
“Hide Tips”
 Create a better help feature that
guides users through steps of
report creation
 Include instruction on how to
drag data into trash can

10. Results and recommendations: Icons
Observation
 Unclear meaning
 Thinking icon not visible
 Setting system to auto-refresh
in settings
Recommendation
 Include icon definitions in help
feature
 Make thinking icon more
visible, perhaps in middle of
page to suspend user
 Set the default to auto-refresh,
move settings icon

11. Results and recommendations: Other
Observation
 “Show Menu” in CareLogic
increases TOT
 There is a delay before the
name appears when the user
names the report title in
Pentaho
Recommendation
 Omit “Show Menu” if possible.
If not, it should default to
menu shown and give “Hide
Menu” option.
 Decrease delay

12. SUS
The participants were asked to rate
their experience on a scale of one
(strongly disagree) to five (strongly
agree). They were asked 10
questions.
Figure 3. The System Usability Scale. Copyright
1986 by the Digital Equipment Corporation.

13. The SUS produces a score from zero to 100 representing the overall
usability of the application (Hashdoc, 2015). Note the SUS score does not
represent a percentage; however scores can be normalized into
percentages. A score of 68 is above average and a score below 68 is
below average (Usability.gov, 2015).
System Usability Scale (SUS) Results
Participant SUS Score
1 52.5
2 50
3 62.5
Table 2. Participant’s SUS results

14. The following recommendations are considered high priority:
1. Create a free-text search bar for data fields and consider adding “A-Z” search functionality.
2. Data field headings should be collapsed by default. Users should then be able to expand and
collapse headings with ease.
3. Each data field name should be unique to avoid confusion, even if they are under a different
heading.
4. Auto-refresh should always be enabled, with the ability for the user to disable if needed.
5. User “Tips” should be refined with clear how-tos that guide the user through an initial basic
report creation .
6. Omit “Show Menu” in CareLogic or make the button more visible. It should default to menu
shown and give the option to “Hide Menu” if needed.
Conclusions and Priorities

15. Help Feature Example
Figure 4. The first three steps of Salesforce’s TaskRay help
feature. Retrieved from StaffHub, the intranet for the St.
Joseph Health System. Copyright Salesforce 2000-2015.

16. Collapsed vs. Expanded Data
Expanded Collapsed

17. Hashdoc (2015). System Usability Scale. Retrieved from
https://www.hashdoc.com/documents/12150/system-
usability-scale-sus-#!
Usability.gov (2015). System Usability Scale. Retrieved
from http://www.usability.gov/how-to-and-
tools/methods/system-usability-scale.html
References