Slides prepared for a BBL event hosted by the Independent Evaluation Group of the World Bank.

1. Text Analytics:
From Colored Pens and Crumbly Papers to
Custom Machine Classifiers forTwitter
Dr. StuartW. Shulman
Founder & CEO,Texifter
@stuartwshulman
“…a wealth of information creates a poverty of attention.”
- Herbert Simon, 1971

2. Presentation Outline
1. Moving from pen and paper to machine-learning
2. Overview of the spectrum methods
3. Portfolio identification using the five pillars
4. HowTwitter data is relevant to evaluation of theWBG

3. Our Core Philosophy

4. Emergent properties in very well read texts such as
the archetypal “extremist agent of the law”

5. Agenda-Setting in the Progressive Era Print Press

6. Relations between Classes
Rates andTerms for Credit
Farm Profitability
Cost of Living
Soil Fertility
Education
Exploration
Speculation
Coding
Validation

7. Qualitative Methods: Genes,Taste, orTactic?
• Qualitative by birth or choice?
• Some look to words as an alternative to number crunching
• Others rooted in rich and meaningful interpretive traditions
• Another group is fluent in both qual & quant
• Mixed methods open up rather than limits fields of knowledge
• One central goal is valid inferences about phenomena
• Replicable and transparent methods
• Attention to error and corrective measures
• Internal and external validation of results
• Using computers for qualitative data analysis helps, but…
• Rigor still originates with the research design, not the technology
• Software makes better organization and efficiency possible
• Coders enable the researcher to step back while scaling up

8. Purist Pluralist Positivist
A Spectrum of Approaches toWorking with Qualitative Data
Different types of knowledge claims depending where you sit
deep immersion
closeness to data
antipathy to numbers
credible interpretation
in-depth analysis
contextual
subjective
experimental
mixed method
adaptive hybrid
flexible approach
interdisciplinary
quantitative
focus on error
measurement critical
validity and reliability
replication & objectivity
generalization
hypotheses
These choices can be philosophical, ideological, and ethical in nature

9. Stuart W. Shulman. 2003. "An Experiment in Digital Government at the
United States National Organic Program," Agriculture and Human Values
20(3), 253-265.

14. CodingWeb Sites and Focus Groups to Study Agenda-Setting

15. Annotation to Improve Optical Character Recognition

16. Over 13,000 hours of video and audio were recorded of the public spaces in a LTC facility’s dementia unit in
suburban Pittsburgh, PA. A codebook of 80+ codes was developed to categorize the behavior of the consenting
residents and staff (only in relation to patients). 22 coders spent more than 4,400 hours over a period of 22
months coding the video data.The data were coded using the Informedia DigitalVideo Library (IDVL), an
interface designed by computer scientists at Carnegie Mellon University.

18. An Incredibly Important Book

19. Grimmer & Stewart
“Text as Data” Political Analysis (2013)
Volume is a problem for scholars
Coders are expensive
Groups struggle to accurately label text at scale
Validation of both humans and machines is essential
Some models are easier to validate than others
All models are wrong
Automated models enhance/amplify, but don’t replace humans
There is no one right way to do this
“Validate, validate, validate”
“What should be avoided then, is the blind use of
any method without a validation step.”

20. Free, Open-Source,Web-basedText AnalyticsToolkit

21. Original Software Kernel:Tools for Measurement

22. Text Classification
A 2500 year-old problem
Plato argued it would be frustrating; it still is
Software cannot remove the problem

23. Computer Science and National Science Foundation
Influences in a Nutshell: Measure Everything!
Fast?
Reliable?
Accurate?
Valid?

24. Interrater Reliability: A Critical Measurement

25. Adjudication: Creating a Gold Standard

26. CoderRank is our key innovation
Patent issued in 2016
Service Mark issued 2017

27. CoderRank for Enhanced Machine-Learning
CoderRank is to text analytics what PageRank was to search.
Just as Google said not all web pages are created equal,
Texifter argues that not all humans are created equal.When
training machines, it is best to rely most on the humans most
likely to create a valid observation.We proposed a unique way
to rank humans on trust and knowledge vectors.

28. Pronounced “tech-sifter”- the metaphor is of a sifter

31. AvoidTennis Elbow
Items load to the screen and the coder hits the keystroke

32. Keystroke Human Coding
Human coding distributed to individuals, groups & crowds
Data
Code
s

33. The Five Pillars ofText Analytics
Search
Metadata Filtering
De-duplication and Clustering
Human Coding
Machine-Learning

34. Pillar One: Search

35. Pillar One: Defined Multi-term Search

38. PillarTwo: Metadata Filters

41. Slicing big piles of text into smaller, more focused sets is key
AllTextAnalytics are FilteringTechniques

43. Users Drill Into Interactive Displays
Use metadata to examine sub-sets of responses and create reports

44. PillarThree: Duplicate Detection & Clustering

46. Latent Dirichlet Allocation (LDA)Topic Models

47. Topic Composition (SampleTerms and Phrases)
• Topic 1: development; projects; support; financials
• Topic 2: education; training; standards; teaching
• Topic 3: health; coverage; ministry; information
• Topic 4: government; investment; social; policy
• Topic 5: administrative; coordination; market; procedures
• Topic 6: institutional; technical; strengthening; programs
• Topic 7: infrastructure; rehabilitation; maintenance; upgrading
• Topic 8: utility; company; privatization; restructuring; supply

48. Pillar Four: Human Coding (Labeling orTagging)

49. Human Coding Converted into Machine Classifiers
Accumulated human coding becomes
training data via machine-learning

50. Simplified Coding Management

51. Crowdsourcing accelerates the insight generation process
Synchronous & Asynchronous Collaboration

54. Pillar Five: Machine-Learning

55. Our ActiveLearning engine and coding tools combine…
what humans do best… with what computers do best
Humans and machines learning together
Keep humans “in-the-loop” for more accurate results and better insights

56. Boolean Operators Cannot Solve Every Problem
There are language problems well-suited to machine-learning
We are all training classifiers in daily life
Spam filtering gave way to Amazon & Netflix
Humans and machines are constantly learning together

57. Interested in Money Banks?

58. Researching a Politician?

59. Doing Smoking Research?

60. Brands?

61. Studying a SportsTeam?

62. Super Bowl HistoryVersus Political History?

63. Twitter Can Feel Overwhelming

65. Full HistoricalTwitter Access with Free Estimates

66. PowerTrack Operators for Precise Queries

67. Create andTest Rules Self Serve

68. Three Estimates Per Day SentVia Email
# of
Tweets
Cost

69. Twitter Data Should Be Human Coded
Using theTwitter Display
The rush to CSV is a mistake; data is degraded
Data
Data
Live
Data
Live
Data
Data

77. Contents
Network
Time Series
Description
Author Description
Overall Metrics
Top Influencers
Top URLs
Top Domains
Top Hashtags
Top Words
Top Word Pairs
Top Replied-To
Top Mentioned
Top Tweeters
Network
sdonnan
Tweet Follow
WorldBank
Tweet Follow
CraigHammerd
Tweet Follow
bijancbayne
Tweet Follow
YouTube
Tweet Follow
TweetsAnup
Tweet Follow
realDonaldTrum
p
Tweet Follow
Nik_6996
Tweet Follow
jeremyhillman
Tweet Follow
alanBStardmp
Created with NodeXL
(http://nodexl.codeplex.com)
from the Social Media Research Foundation
(http://www.smrfoundation.org)

79. Special Enterprise License Keys
Everyone can have one
Request via an email: info@texifter.com

80. For more information
discovertext.com
@discovertext
Thank-you for listening!
Dr. Stuart Shulman
@stuartwshulman