Session at Taxonomy Bootcamp comparing pros and cons of rules-based versus machine-learning approaches to Autoclassification

1. Jeff Fried
CTO
BA Insight
@jefffried
#tbc2016
Rules-Based vs. Document-Based Bake-off

3. Focused on Search and
SharePoint since 2004
Longtime
Search Nerd
• CTO, BA Insight
• Senior PM, Microsoft
• VP, FAST
• SVP, LingoMotors
About Jeff Fried
Passionate About
• Search
• SharePoint
• Search-driven
applications
• Information Strategy
Blog:
BAinsight.com/blog
Technet Column
“A View from the
Crawlspace”
jeff.fried@bainsight.com

4. About BA Insight


– Connectivity
– Applications -
– Classification -
– Analytics


5. Metadata Drives Great User Experiences
Documents from many sources
All client or matter-relevant documents are integrated.
Rich MetaData
Content annotated automatically – concepts,
categories, citations, matters, clients, etc
Navigation Controls
Explore, Discover, Drill-down

6. Manual Tagging is impractical
and remarkably inconsistent

7. Automation
Called: AutoClassification, AutoTagging, Metadata Generation, Text Analytics, ….

8. 8

9. Complicators






–
–
–

–
–

11. 11
Common Techniques across Applications

12. -
-
-
-
-
-
-
-
-
-
-
-

13. Rules-based Approach
Enhanced Content
Enriched with
Metadata and
Content Types
Search Visualization Workflow

14. Name Blood Type Give Birth Can Fly Live in Water Class
human warm yes no no mammals
python cold no no no reptiles
salmon cold no no yes fishes
whale warm yes no yes mammals
frog cold no no sometimes amphibians
komodo cold no no no reptiles
bat warm yes yes no mammals
pigeon warm no yes no birds
cat warm yes no no mammals
leopard shark cold yes no yes fishes
turtle cold no no sometimes reptiles
penguin warm no no sometimes birds
porcupine warm yes no no mammals
eel cold no no yes fishes
salamander cold no no sometimes amphibians
gila monster cold no no no reptiles
platypus warm no no no mammals
owl warm no yes no birds
dolphin warm yes no yes mammals
eagle warm no yes no birds
Rule-based Classifier (Example)
R1: (Give Birth = no)  (Can Fly = yes)  Birds
R2: (Give Birth = no)  (Live in Water = yes)  Fishes
R3: (Give Birth = yes)  (Blood Type = warm)  Mammals
R4: (Give Birth = no)  (Can Fly = no)  Reptiles
R5: (Live in Water = sometimes)  Amphibians

15. Example Rules Engine UI

16. Examples of Rules
Boolean
• “IT” OR “Information Technology” or “MIS”
• (“Expert” OR “Witness”) NOT “police”
• “New York” AND “environmental policy”
• *work
• "legal" -briefs
• "Legal" NEAR(5) "issue“
Property-based
• filetype:docx
• title:"2029 L.P" or title:2030
• footer="BA Insight Confidential" or
footer:proprietary or footer:BA*
Overriding/changing Linguistics
• NOSTEM(“illumination")
• CASE("prerequisites")
• SOUNDLIKE("prerech")
Regular expressions
• title:REGEX([0-4])
• REGEX("b(([w-]+://?|www[.])[^s()<>]+(?:([wd]+)|([^[:punct:]s]|/)))")

17. Controlling scores & thresholds

18. Taxonomy Management is often included with
Auto-Classification Tools

19. 




Where do you get Taxonomies?

20. 20
Semantics! Machine Learning! AI!

22. Key Concepts





23. False positives vs. false negatives
Look at the impact of each in your context

24. Machine Learning Approach

25. Example: identify people as good or
bad from their appearance

26. Decision Tree Classifier

27. Building an accurate classifier

28. 



–
Training and Test Data
28

29. Choosing the algorithm

–

–
–

30. + Easy to get started
+ Transparent and
debuggable
+ Easily controlled (when #
rules not too large)
- Need taxonomies
- Rule maintenance effort
- Harder to cover domain
fully and to switch domains
+ Don’t need taxonomies
+ Improves without manual
maintenance
+ Handles new data
types/domains more easily
- Need a training set
- Opaque, usually can’t debug
- Can’t specify or control
specific examples

31. What would you use for








32. Case Study
Content Identification and Movement

33. Benchmarks





34. Large scale example

35. Combinations of Techniques usually work better

36. Examples of hybrid configurations





37. Example: clustering combined with rules

39. carrot2
Open Source & Platform packages
offer an easy way to play

40. How to get started
 Setup up a metadata framework
– keep it simple
 Develop or acquire managed vocabularies for
critical elements
 Start with rule-driven automation
 Test out ML-based techniques as you grow

41. 41
www.BAinsight.com
Jeff.Fried@BAinsight.com
@jefffried