2. Heuristic Evaluation
• Goal: Identifying usability problems in existing
systems
• Checking compliance with a set of usability
principles
• Performed individually
• Results will be aggregated
2
3. Heuristic Evaluation
• Freely explore the interface or perform
certain tasks
• You can go through the interface several times
• You should specify:
– The problem
– The heuristic(s)
– The severity
3
4. IT security management (ITSM) tools
• IT security management tools are components
in the design, development, and maintenance
of a secure information technology
infrastructure.
– Examples: network firewall, intrusion detection
system, identity and access management system
• Characteristics of the ITSM environment
– Complex, collaborative, people with different
backgrounds
4
6. 1 – Visibility of activity status
Provide users with awareness about the status of the activity
distributed over time and space, including the other users
involved in the activity, their actions, and distribution of work
between them; rules that govern the activity; tools,
information, and material that are used in the activity; and
progress toward the activity objective. Provide
communication channels for transferring the status of the
activity. While providing awareness is crucial, provide
awareness only about what a user needs to know to complete
his actions.
6
7. 2- History of actions and changes on
artifacts
Allow capturing the history of actions and
changes on tools or other artefacts such as
policies, logs, and communication between
users. Provide a means for searching and
analyzing historical information.
7
8. 3- Flexible representation of
information
Allow changing the representation of
information to suit the target audience and
their current task. Support flexible reports.
Allow tools to change the representation of
their input/output for flexible combination
with other tools.
8
9. 4- Rules and constraints
Promote rules and constraints on ITSM activities, but
provide freedom for users to choose different paths
that respect the constraints. Constraints can be
enforced in multiple layers. For example, a tool could
constrain the possible actions based on the task, the
chosen strategy for performing the task (e.g., the order
of performing actions), the social and organizational
structure (e.g., number of subjects involved in the task,
policies, standards), and the competency of the user.
9
10. 5- Planning and dividing work between
users
Facilitate dividing work between the users involved in
an activity. For routine and pre-determined tasks,
allow incorporation of a workflow. For unknown
conditions, allow generation of new work plans and
incorporation of new users.
10
11. 6- Capturing, sharing, and discovery of
knowledge
Allow users to capture and store their knowledge
explicitly by generating documents, web-pages, scripts,
and notes or implicitly by providing access to a history
of their previous actions. Tools could then facilitate
sharing such knowledge with other users. Furthermore,
tools should facilitate discovery of the required
knowledge source including artefacts or a person who
possess the knowledge and provide means of
communicating with the person who possesses the
knowledge.
11
12. 7- Verification of knowledge
For critical ITSM activities, tools should help SPs validate their
knowledge about the actions that are required to perform the
activity. Allow users to validate their knowledge by
performing actions and validating the results on a test system
before applying them to the real system. Allow users to
document the required actions in the form of a note or a
script; this helps the users or their colleagues to review the
required actions before applying them on the system.
12
14. 1 - Visibility of system status
The system should always keep users informed
about what is going on, through appropriate
feedback within reasonable time.
14
15. 2 - Match between system and the real
world
The system should speak the users' language,
with words, phrases and concepts familiar to the
user, rather than system-oriented terms. It
should follow real-world conventions, making
information appear in a natural and logical
order.
15
16. 3 - User control and freedom
Users often choose system functions by mistake
and will need a clearly marked "emergency exit"
to leave the unwanted state without having to
go through an extended dialogue. Support undo
and redo.
16
17. 4 - Consistency and standards
Users should not have to wonder whether
different words, situations, or actions mean the
same thing. Follow platform conventions and
terminology which are familiar to the user.
17
18. 5 - Error prevention
Even better than good error messages is a
careful design which prevents a problem from
occurring in the first place. Either eliminate
error-prone conditions or check for them and
present users with a confirmation option before
they commit to the action.
18
19. 6 - Recognition rather than recall
Minimize the user's memory load by making
objects, actions, and options visible. The user
should not have to remember information from
one part of the dialogue to another. Instructions
for use of the system should be visible or easily
retrievable whenever appropriate.
19
20. 7 - Flexibility and efficiency of use
Accelerators -- unseen by the novice user -- may
often speed up the interaction for the expert
user such that the system can cater to both
inexperienced and experienced users. Allow
users to tailor frequent actions.
20
21. 8 - Aesthetic and minimalist design
Dialogues should not contain information which
is irrelevant or rarely needed. Every extra unit of
information in a dialogue competes with the
relevant units of information and diminishes
their relative visibility.
21
22. 9 - Help users recognize, diagnose, and
recover from errors
Error messages should be expressed in plain
language (no codes), precisely indicate the
problem, and constructively suggest a solution.
22
23. 10 - Help and documentation
Even though it is better if the system can be
used without documentation, it may be
necessary to provide help and documentation.
Any such information should be easy to search,
focused on the user's task, list concrete steps to
be carried out, and not be too large.
23
Notes de l'éditeur
The ITSM version
The goal of heuristic evaluation is to identify usability problems in a system. The heuristic evaluation method employs a set of usability principles, called heuristics, to guide the evaluators in identification of usability problems. Each evaluator inspects the interface and checks the compliance of the interface with the heuristics. Heuristic evaluation is performed individually by each evaluator and then the results from different evaluators are aggregated into a set of usability problems.
During heuristic evaluation, you can freely explore an interface and identify problems or you can use scenarios to focus on a specific part of the system. In this study, we will use scenarios to limit the scope of the evaluation. You can walk through the steps of each scenario and perform them on the interface once or multiple times. Also, you are free to explore other parts of the interface to become familiar with the interface. But we are only interested in identifying problems related to the provided scenarios. For each problem that you identify, you should specify 3 components: First, the problem itself. Second, the scenario in which you identified the problem. Third, the heuristic (or heuristics) with which you found the problem. You might find problems that can’t be associated to a heuristic. For those problems, just record the problem, and scenario.
As we are going to evaluate an IT security management tool, I want to give you a quick background about this class of applications. IT security tools are components in the design, development, and maintenance of a secure information technology infrastructure. Tools like network firewalls, intrusion detection systems, and identity and access management systems are examples of IT security management tools. The environment in which these tools work has certain characteristics. First, it is complex. Everyday, new security issues arise in this environment that need to be addressed in a timely fashion. Many tasks of security administrators are not routine, and require knowledge and expertise. Second, this environment is collaborative. It means that different stakeholders in the organization need to collaborate with each other to perform their tasks. Third, the stakeholders involved in IT security management have different background. There are security administrators, managers, employees, and external contractors who need to collaborate with each other to perform IT security management tasks.
Now I will present the heuristics you will be using today. For each heuristic, I will describe it and give an example of how it can be applied to an IT security management tool, in this case, a network firewall. There are 7 heuristics that I will describe. They are called ITSM heuristics. I will go through them in this tutorial and give examples of how they can be applied to IT security tools.
IT security management activities are distributed over time and space and they include multiple users. These users work in an organizational environment and must comply with the rules in the organization (e.g., the security policy, privacy policy, and other organizational rules) and standards. Therefore, the tool should provide the required awareness about the status of the activity, the other users involved, the progress toward activity objectives, rules, available tools, etc., about the activity, to all of the stakeholders of the tool. It is important to only provide the status that users really need to know, not more.
Historical information could be in the form of use-histories by other people or the subject himself. Use histories can be employed to reflect on work and getting feedback from peers. In IT security, reflecting on work is important as the actions are performed on the system by different stakeholders. Moreover, security regulations require the system to keep a history of actions.
As IT security tools have different stakeholders, tools should be able to present information in the format suitable to the target audience. Furthermore, to address complexity, providing different presentation of data allows the user to view information from different perspectives and help them understand complex scenarios. From a different view point, security admins like to combine their tools together to address different problems. Therefore, tools should be able to present their information in a way that can be used by other tools and also accept inputs from different tools.
As ITSM tool designers can’t predict all the conditions that a tool user might face, they should provide freedom for users to choose the way they want to perform the activities. Therefore, while there should be multiple ways for users to perform activities, constraints should be enforced by the tool so the user can only choose those paths that are not violating any of the constraints.
Because the use of ITSM tools involves multiple stakeholders, the tool should provide facilities for dividing work between different stakeholders. For those tasks that have a routine procedure, incorporation of a workflow in the tool would be a good idea. But for unknown conditions, tool should provide ways for users to generate plans for performing the activity (for example, showing who is available to perform a task or allow a workflow to be created dynamically)
To address problems in the complex and evolving environment of ITSM, a subject needs to use the knowledge and experience of other stakeholders involve in the activity. To facilitate accessing distributed knowledge, ITSM tools should enable their users to express their knowledge in a form of a document, web-page, or script that can be used by other users and also facilitate identification and access to the required knowledge sources for accomplishing the activity. In cases that documenting knowledge is not feasible, a method for finding and starting collaboration with the person who possesses the knowledge should be provided.
Many actions in ITSM are responses to new, unseen and complex situations. These actions should be performed on systems that are critical to the organization. Moreover, the actions are distributed in time and space and the result of an action can't be evaluated in real time. Therefore, the cost of errors in these actions is huge. To find a solution to a new or complex problem, a security admins usually consults different information sources and combines them into a single plan (a plan, a guide document, a check list, etc.) This plan extracted from different sources might not be correct. Therefore, it should be verified before applying it to the system. Therefore, ITSM tools should allow users to rehearse the action on a non-critical, test system, evaluate the outcome of the action, and then apply it on the critical system. If something goes wrong in the rehearsal, the user can re-examine his or her interpretation of the external sources. After successful rehearsal, users can perform rehearsed actions on the critical artifact. To facilitate this process, tools should help creation of a non-critical system from a critical system, and help the process of applying rehearsed plan on the critical system.
Now I will present the heuristics you will be using today. For each heuristic, I will describe it and give an example of how it can be applied to an IT security management tool, in this case, a network firewall. There are 7 heuristics that I will describe. They are called ITSM heuristics. I will go through them in this tutorial and give examples of how they can be applied to IT security tools.
The system should keep users informed about what is going on in the system, through appropriate feedback within reasonable time, so that users can accurately determine the right action to perform on the system and also can determine the result of their actions.
Users can understand the system if it matches what the users know from the real world. [Nielsen heuristic itself with modifications] Therefore, the system should speak the users' language, with words, phrases, and concepts familiar to the user, rather than using system-oriented terms. It should follow real-world conventions, making information appear in a natural and logical order.
Users often make mistakes in choosing their actions. Therefore there should be a way for users to exit an undesirable state. Supporting undo or redo is an example of a shortcut for exiting such an unwanted state.
When using a system, users should not have to wonder whether different words, situations, or actions mean the same thing. It is better if the system can follow platform conventions and terminology which are familiar to the user.
Even better than good error messages is a careful design that prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
Different users might use system in different ways. For example a novice user might prefer to go through more steps that help him do the tasks successfully, but an expert user might prefer a very straightforward way for performing an action that he is familiar with. Therefore, it would be good if the tool provides customization in a way that helps users tailor their frequent actions and choose the way they want to perform their tasks.
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.
