Methods for Improving Organizational Website IA by Barbora Batokova

Methods for Improving
Organizational Website IA
WORLD IA DAY
PITTSBURGH, PA • 2.20.2016
 @WIAD_PGH
BARBORA BATOKOVA
CARNEGIE MELLON SOFTWARE ENGINEERING INSTITUTE
 @bbatokova

DM-0003306 This material has been approved for public release and unlimited distribution.
© 2016 Carnegie Mellon UniversityMethods for Improving Organizational Website IA • Barbora Batokova 2 of 42
Copyright 2016 Carnegie Mellon University
This material is based upon work funded and supported by the Department of Defense under Contract
No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering
Institute, a federally funded research and development center.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL
IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY
KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY
OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF
THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH
RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
[Distribution Statement A] This material has been approved for public release and unlimited distribution.
Please see Copyright notice for non-US Government use and distribution.
This material may be reproduced in its entirety, without modification, and freely distributed in written or
electronic form without requesting formal permission. Permission is required for any other use. Requests
for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.
DM-0003306

Goals
2Introduce you to tree testing and how
to analyze the results.
CASE STUDY
1Provide a brief overview of a 3-stage
design process for improving a site IA.
PROCESS OVERVIEW
OVERVIEW

Could be Improved Good Example Bad Example

THE CHALLENGE

Fill in the Gaps
Survey • Expert Interviews • Competitive Analysis
Assess Current IA
Tree Testing•Think Aloud Protocol•Heatmaps/Click Tracking1
2
Create New IA
Butcher Paper IA•Card Sorting•Click Testing•Tree Testing3
3-Stage Process
OVERVIEW
Developed for SEI’s website redesign project.

Assess
Current IA
1

Home
Work Areas
Cyber-Physical Systems
Research
Our People
Performance & Dependability
Overview
Tools & Methods
Risk Management
Overview
Tools & Methods
Security & Survivability
Overview
Getting Started
Research
People
Software Architecture
Software Product Lines
Engage with Us
Products and Services
Tools & Methods
Research
Consulting
Library
News
Overview
News Stories
Press Releases
Media Coverage
SEI Bulletin
About Us
SEI Overview
SEI Overview
Why Work with an FFRDC
Year in Review
Our Organization
Tree Testing

Methodology
Considerations
Site Tree Size => No. of Tasks => No. of Tests
Participants
• Min 50 for 10 tasks
• Site Visitors / Recruitment Service
1: ASSESS CURRENT IA » TREE TESTING
Software
• Treejack
• User Zoom
• C-Inspector

Site Tree
6-Level Nav
Global > Main > Drop-Down Menus > Standardized Local > Local > Sub-Local
1
2
3
4
5
6
600+
items
}

Organizational Focus
Tasks (15)
# TASK TASK COMPLETION
9. Find out about what our organization does. 81%
4. Get involved with us on a research project. 50%
5. Fund a research and development (R&D) project. 42%
7. Find out about our upcoming conferences. 37%
2. Find out about our latest research. 35%
15. Find out about the history of the organization. 0%
Based on website objectives and basic organizational information.
37% Overall Success Rate Usable75%
Good
75% and above
Very Poor
0–49%
Poor
50–74%

Good
75% and above
Very Poor
0–49%
Poor
50–74%
# TASK TASK COMPLETION
3. Download software packages and tools the SEI has created. 76%
10.
Find out how the SEI is helping to solve the nation’s cybersecurity
challenges. 36%
17.
Learn more about SEI participation at approved targeted academic
conferences. 26%
2. Find out about the outreach and impact the SEI has had. 18%
1.
Find information on what high priority problems the DoD faces and
how we help to solve them. 6%
4.
Find out how the SEI enables new or extended capabilities in
existing military systems. 0%
Tasks (18)
Based on web objectives and technical staff input.
20%Overall Success Rate Usable75%
Technical Focus

TREE TESTING
Analyzing Results
Analyzing Results
For each question, we created a Results Sheet
that summarized the findings.

Results Sheets
Download: http://bit.ly/1RzDIRz
TASK RESULTS
DESTINATIONS
PIE TREE
IMPLICATIONS
FIRST CLICK PATHS
Direct Failure
QUESTION#
%
Indirect Failure
Indirect Success
Direct Success
Overall Success Rate
How many people completed the task
successfully? 75% above is acceptable.
Direct Failure
If a participant goes directly to a wrong
answer, without ever clicking back up the tree.
When someone goes directly to the wrong
information, it suggests they have confidence
they’re on the right track.
Indirect Failure
If a participant has gone back through the
tree at least once before they’ve selected
the wrong answer. When a task receives high
numbers of indirect fails, it gives a clear
message that the labeling and organization of
that part of your tree is confusing people.
Direct Success
If a participant goes directly
to the correct answer.
Indirect Success
If a participant has gone back through the
tree at least once before they’ve selected the
correct answer. This means that organization
and labeling can be improved.
Task Results
Breakdown of success, directness, time taken,
and the overall score calculated for each task.
First Click
Shows which branches were clicked first for
each task, and what % of participants did so.
Destinations
Where did participants end up? / Where did
they think they would find the answer to the
question? The more there is, the more confused
participants were.
Pie Tree
Shows which way participants went at each
junction in the site tree and what they selected
as their final answers. Useful for determining
the breakdowns.
Implications
What does this mean? What are the takeaways?
How can we improve the site tree
based on these results?
Destination Across All Tasks
Number of participants that picked this item as
the correct answer for multiple tasks.
Each column corresponds to questions. If an
item is picked as the correct answer for multiple
questions, it means the item label is vague and
applies to too many things.

Task Results
Breakdown of success, directness, time taken, and the overall score
calculated for each task.




Shows which branches were clicked first for each task, and what
percentage of participants did so.

First Click

Shows which way participants went at each junction in the site tree and
what they selected as their final answers.
Home
Work Areas
Acquisition Support
Our People
Research
Processor Performance
Overview
Engage with Us
Connect
CollaborateCases Examples: Collaboration ResultsSoftware Assurance Curriculum
SEI Training
SEI Partner Network
SEI Certification
Overview
Calendar of EventsTools & Methods
Research
About Us
SEI Overview
SEI Overview
SEI Statistics and History
Year in Review
Our Organization
Our Organization
Director's Office
CERT Division
Our People
Pie Tree
Pies
Lines
Went down the right path
Went down the wrong path
Went back
Nominated as correct answer
Skipped question
Correct path
Incorrect path
Home
Work Areas
Acquisition Support
Overview
Tools and Methods
Methods
Consulting on Acquisition Issues and Needs
Strategic Planning
Case Studies
Cyber-Physical Systems
Overview
Research
Measurement Analysis
Research
Consulting
Case Studies
Military Aircraft Systems
Weapons Systems Management
Process & Performance Improvement
Getting Started
Research
Accelerated Improvement Method (AIM)
Consulting at the SEI
Process Case Studies
IEEE Computer Society/Software Engineer
Watts S. Humphrey Software Process Achi
Risk Management
Research
Tools & Methods
ConsultingSecurity & Survivability
Overview
Research
Systems of Systems
Overview
SEI Training in Systems Interoperability Integration of Software-Intensive Systems
Research
Identity Management in SOA Environments
Tools & Methods
Consulting
Case Studies
Ultra-Large Scale Systems
Overview
Engage with Us
Connect
Consult
ConsultSolutions
Effective Use of Software Architecture Provides a Blueprint for Successful Products
Cases
The Army Strategic Software Improvement Program (ASSIP)
Sponsor
Sponsor
Case Examples: R&D Results
Collaborate
Collaborate
Cases Examples: Collaboration Results
Cyber Intelligence Tradecraft Project
Architecture Evaluation
Develop
SEI Training
SEI Partner Network
Overview
Find an SEI Partner
Partner Organizations
Find a Partner - Search Results
Partner Detail
SEI Certification
Calendar of Events
Tools & Methods
Research
Consulting Library
SEI Digital Library
SEI Podcast Series
Browse by Category
Cybersecurity Engineering
SEI Webinar Series
News
Overview
News Stories
Press Releases
SEI Bulletin
Careers
About Us
SEI Overview
SEI Overview
Director's Message
Year in Review
Our Organization
Our Organization
Software Solutions Division
Architecture Practices


Paths

Direct Failure
Indirect Failure

Indirect Success

Direct Success


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
TASK #
Where did participants end up?


Destinations

Visualizing the Results
WORK AREAS
ACQUISITION SUPPORT
Overview
Getting Started
Research
Tools & Methods
Consulting
Our People
CYBER-PHYSICAL SYSTEMS
Overview
Research
Our People
MEASUREMENT & ANALYSIS
Overview
Getting Started
Research
Tools & Methods
Consulting
Case Studies
ABOUT US
SEI OVERVIEW
DIRECTOR’S MESSAGE
OUR ORGANIZATION
Director’s Office
CERT Divsion
Emerging Technology Center
Software Solutions Division
OUR PEOPLE
Sponsoring and Oversight
Leadership Team
SEI Fellows
YEAR IN REVIEW
PRODUCTS & SERVICES
SEI TRAINING
Overview
Find a Course
Classroom
Onsite Training
E-Learning
Registration
FAQ
Certificates
SEI PARTNER NETWORK
SEI CERTIFICATION
CALENDAR OF EVENTS
TOOLS & METHODS
RESEARCH
CONSULTING
LIBRARY
SEI DIGITAL LIBRARY
SEI PODCAST SERIES
SEI WEBINAR SERIES
NEWS
NEWS STORIES
ANNOUNCEMENTS
PRESS RELEASES
MEDIA COVERAGE
SEI BULLETIN
CAREERS
OVERVIEW
AFFILIATE PROGRAM
ENGAGE WITH US
OVERVIEW
CONNECT
CONSULT
Solutions
Cases
SPONSOR
Case Examples: R&D Results
COLLABORATE
Case: Examples: Collaboration
Results
DEVELOP
CMU SEI CERT DIVISION DIGITAL LIBRARY INSIGHTS
Ambiguous Meaning Problematic Placement
Global > Main > Drop-Down Menus …
Ambiguous Meaning Problematic Placement

Think Aloud
Protocol

Methodology
1: ASSESS CURRENT IA » THINK ALOUD PROTOCOL
Software
usertesting.com, User Zoom, Usability Tools, etc.

What does this mean? What are the takeaways? How do we fix it?
Implications
1: ASSESS CURRENT IA » THINK ALOUD PROTOCOL & TREE TESTING

Heatmaps &
Click Tracking

1: ASSESS CURRENT IA » HEATMAPS / CLICK TRACKING
Software
crazyegg, ClickTale, ClickHeat, etc.
sei.cmu.edu • 60 days 48 mins • 25,000 visits • 15,698 clicks
Methodology

Fill in
the Gaps
2

Survey

Distribution Statement A: Approved for Public Release; Distribution is Unlimited
© 2016 Carnegie Mellon UniversityMethods for Improving Organizational Website IA • Barbora Batokova 27 of 42DM-0003306 This material has been approved for public release and unlimited distribution.
To gather types of information that are currently not
represented on the site.
• What important content is missing?
• What content areas are obsolete and should be archived?
• Are our clients looking for information that we don’t have online?
Benefits
• Quick way to gather a lot of useful data.
• Involve the entire organization and clients in the design process.
Why do a Survey?
2: FILL IN THE GAPS » SURVEY

Expert
Interviews

To identify business needs not being met
within the structure.
• Where is the organization headed and how does the website fit in?
• How can the website IA help express the direction of the organization?
Benefits
• Cheap in terms of effort and budget.
• Quick way to get expert information directly from the source.
• Ensure business needs are being met.
2: FILL IN THE GAPS » EXPERT INTERVIEWS
Why do Expert Interviews?

Competitive
Analysis

Methodology
Competitors
Identify your competitors.
Screenshots
Framework
What are you evaluating?
Spreadsheet
2: FILL IN THE GAPS » COMPETITIVE ANALYSIS

Create
New IA
3

Butcher
Paper IA

3: CREATE NEW IA » BUTCHER PAPER IA
Methodology
Set Specific Goals

Card
Sorting

Methodology
Items Categorization
• Open or closed card sort?
• Hybrid?
Software
• OptimalSort
• User Zoom
• UsabilityTools ...
Results
Similarity Matrix
3: CREATE NEW IA » CARD SORTING
Dendrograms

Click
Testing

Measuring Improvement
Proposed IA
3: CREATE NEW IA » CLICK TESTING

Measuring Improvement
Proposed IA + Heatmap
Compare: Original First Click 72%
118%
Improvement33%News
3: CREATE NEW IA » CLICK TESTING

Home
Work Areas
Acquisition Support
Performanc
Enga
SEI C
Calendar of EventsTools & Methods
Research
About Us
SEI Overview
SEI Overview
SEI Statistics and History
Year in Review
Our Organization
Our Organization
Director's Office
Our People
Tree
Testing

Additional Sources
Web Usability:
http://webusability.com/firstclick-usability-testing/
Boxes and Arrows:
http://boxesandarrows.com/tree-testing/
Universal Methods of Design,
Bella Martin and Bruce Hanington
Optimal Workshop Blog:
http://www.optimalworkshop.com/blog
Communicating Design,
Dan M. Brown
Software Tools Used
• UserTesting
• Optimal Workshop
• Treejack
• Optimal Sort
• crazyegg
• UsabilityHub
Other Software Tools
• ClickHeat
• ClickTale
• Loop11
• Optimizely
• Qualtrics
• SurveyMonkey
• UsabilityTools
• UserZoom
RESOURCES
http://bit.ly/1RzDIRz
Results Sheet Diagram

Thanks!
Do you have any questions?
BARBORA BATOKOVA
bbatokova@sei.cmu.edu
 @bbatokova

Methods for Improving Organizational Website IA by Barbora Batokova

Recommandé

Recommandé

Contenu connexe

Similaire à Methods for Improving Organizational Website IA by Barbora Batokova

Similaire à Methods for Improving Organizational Website IA by Barbora Batokova (20)

Dernier

Dernier (20)

Methods for Improving Organizational Website IA by Barbora Batokova