Beyond the Black Box - An Empirical Approach to Taxonomy Development

1. #AIIM12
Better Than the Black Box:
An Empirical Approach to Taxonomy
Development
Patrick Lambe
Straits Knowledge, Singapore
#AIIM12

2. Three Traps
 Chasing your tail
 Experts know best
K e e p O u t !!
T a x o n o m is t a t W o r k
 Enclosure
#AIIM12

3. Being Empirical Means…
 Warrant - grounding taxonomy in
 Content warrant
 User warrant
 De facto standards in use
 Testing
 Open card sorts
 Closed card sorts
 Scenario-based tests
 Load/balance tests
 Consultation
 Targeted on gaps, technical accuracy, NOT general opinions
 Transparent, tough issue resolution process
For enterprise taxonomies, unmanaged exposure to unfocused opinions
are the Taxonomist’s greatest enemy
#AIIM12

4. 1. Warrant - the role of the
knowledge audit
#AIIM12

5. What is a knowledge audit?
A knowledge audit is a survey-cum-
inventory of the different kinds of knowledge
that are:
currently in use
needed but not currently in use (gaps and
desired improvements)
…to support the work and strategic
objectives of your organisation
Natural
#AIIM12

6. Knowledge Audit
Half day workshop
2-3 operational managers
per dept
QuickTimeª and a
Knowledge maps, culture decompressor
are needed to see this picture.
analysis, pain points
Site visits
Gallery viewing
Ex a mp le o f a gal ler y v ie w ing exe rcis e
#AIIM12

7. Knowledge Maps
Input to activity Div & Activity Output from activity
Subscribing
divisions
Unique code
of knowledge
asset
Type of
knowledge
asset
Description
of knowledge
asset
#AIIM12

8. Expected outcomes
Knowledge maps give us:
 Evidence for how people describe their knowledge assets (content warrant)
 Maps linking knowledge assets to activities (context of use - user warrant)
Site visits give us:
 Observational evidence for organizing principles in use (user warrant)
 Evidence (printouts, photos) of how folders, physical and digital documents,
emails, shared resources are labeled and organized (content and user
warrant)
Analysis of the knowledge maps gives insights into Knowledge gaps, Knowledge
risks, Critical knowledge requiring protection, Sharing and collaboration
opportunities, Knowledge flows and blockages - and helps identify
knowledge priorities, useful in designing the taxonomy
#AIIM12

9. 2. Testing - evidence that people
can use the taxonomy
effectively
#AIIM12

10. Testing Techniques
 Open card sorting - to find “natural
categories” among users and variance
between them - at the start of a
taxonomy design
 Closed card sorting - to test usability of
top level categories - to test a draft
taxonomy
 Scenario based testing - to uncover
navigation, predictability issues - to test
a completed taxonomy - scenarios are
derived from user warrant
The conversations are
 Balance/load testing - to analyze the
as important as the
distribution of content across a
results
populated taxonomy in use
Donna Spencer, Card Sorting Rosenfeld
Media 2009
#AIIM12

11. Case Study
Review of existing taxonomy in use
Public health agency
#AIIM12

12. Balance
53% of taxonomy folders
have no content
76% of populated folders
have fewer than 5
documents
9% of populated folders
have more than 15 items
15% of populated folders
(7% of all taxonomy folders)
have 5-15 items and could
be considered navigable
#AIIM12

13. Balance & Accessibility
35% of content is 5 or
more levels deep
83% of content is 4 or
more levels deep
#AIIM12

14. What we found
From the usability tests:
Avge time Avge Avge Task Confident Consistency
/ task clicks / dead ends abandoned final of decisions
task / task decision
2.5 18 clicks 1.5 25% 46% 35%
minutes folders
opened
#AIIM12

15. Taxonomy design issues
 As Navigation Tool:
 Too narrow and deep
 Unpredictable, different principles of organisation applied in parallel
 As Controlled Vocabulary:
 Same categories repeated across the different parts of the structure -
ambiguity
 No overall design principle to govern additions to taxonomy
 As Classification Scheme:
 Fewer than half the taxonomy is populated, extreme lack of balance in
distribution of content
 Only 35% consistency in usability tests
Solution: break the hierarchy into three facets: Business Activities, Document
Types, Health Topics, using warrant from the existing classification plus
knowledge maps from a knowledge audit
#AIIM12

16. Taxonomy Redesign
 This combination of facets gave a very big reduction in frustration
levels, increased the richness of information captured about
documents, and substantially increased consistency and confidence
Because in a faceted system each document needs between 2 and 3 tags to achieve the same level of
precision as in a single hierarchy, we have divided the actual consistency rate of individual tags by 2.5 to
make a meaningful comparison.
#AIIM12

17. 3. Consultation - transparent
but rigorous issue resolution
process
#AIIM12

18. Consultation Means…
 “We’d like you to examine and Focus the feedback on the areas
comment on the following sections of the taxonomy where testing
of the taxonomy because of your indicates that you need advice
expertise area”
You’ll get these anyway - but
 “We do not want your opinions, we
you’ve made it possible to
want targeted feedback on: Say “no” to opinions
 Gaps without warrant
 Accuracy of language
 Accuracy of structure and
relationships Try to ensure that all suggested
 Consolidating and simplifying changes have some warrant underpinning
categories” them - labels in use, standards in use,
situations requiring this language; and
 “Please give reasons for your that feedback represents the general
suggestions, based on how people user not the specialist user
will actually use these categories”
#AIIM12

19. Issue Resolution
#AIIM12

20. Issue Resolution
#AIIM12

21. Being Empirical Also Means…
 Being better equipped to
manage the politics of
taxonomy development
 Being able to establish
credibility for the taxonomy
development process and
products - including being able
to justify denying requests
 Reducing the risk of project
delays resulting from claims on
the taxonomy from narrowly
informed special interests
#AIIM12

22. #AIIM12
Any Questions?
www.greenchameleon.com
plambe@straitsknowledge.com
#AIIM12