Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2o4Z4Pq. Casey Stella talks about discovering missing values, values with skewed distributions and discovering likely errors within data, as well as a novel approach to finding data interconnectedness based on usage using unsupervised learning. He describes the impact of these lessons on team construction and how to avoid some of the most painful lessons. Filmed at qconlondon.com. Casey Stella is a committer and PMC member on the Apache Metron project in the engineering team at Hortonworks. Previously, he worked as an architect and senior engineer at a healthcare informatics startup spun out of the Cleveland Clinic, as a developer at Oracle and as a Research Geophysicist in the Oil & Gas industry.

1. Data Cleansing and Data Understanding
Best Practices and Lessons from the Field
Casey Stella
@casey_stella
2017
1

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
practices-data-curate
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. Hi, I’m Casey Stella!
• I’m a Principal Software Engineer at Hortonworks, a Hadoop vendor, writing open
source software
• I work on Apache Metron (Incubating), constructing a platform to do advanced
analytics and data science for cyber security at scale
2

5. Hi, I’m Casey Stella!
• I’m a Principal Software Engineer at Hortonworks, a Hadoop vendor, writing open
source software
• I work on Apache Metron (Incubating), constructing a platform to do advanced
analytics and data science for cyber security at scale
• Prior to this, I was
• Doing data science consulting on the Hadoop ecosystem for Hortonworks
• Doing data mining on medical data at Explorys using the Hadoop ecosystem
• Doing signal processing on seismic data at Ion Geophysical
• A graduate student in the Math department at Texas A&M in algorithmic
complexity theory
2

6. Garbage In =⇒ Garbage Out
“80% of the work in any data project is in cleaning the data.”
— D.J. Patel in Data Jujitsu
3

7. Data Cleansing =⇒ Data Understanding
There are two ways to understand your data
• Syntactic Understanding
• Semantic Understanding
If you hope to get anything out of your data, you have to have a handle on both.
4

8. Syntactic Understanding: True Types
A true type is a label applied to data points xi such that xi are mutually comparable.
• Schemas type != true data type
• A specific column can have many different types
5

9. Syntactic Understanding: True Types
A true type is a label applied to data points xi such that xi are mutually comparable.
• Schemas type != true data type
• A specific column can have many different types
“735” has a true type of integer but could have a schema type of string or double
5

10. Syntactic Understanding: Density
Data density is an indication of how data is clumped together.
6

11. Syntactic Understanding: Density
Data density is an indication of how data is clumped together.
• For numerical data, distributions and statistical characteristics are informative.
6

12. Syntactic Understanding: Density
Data density is an indication of how data is clumped together.
• For numerical data, distributions and statistical characteristics are informative.
• For non-numeric data, counts and distinct counts of a canonical representation are
extremely useful.
6

13. Syntactic Understanding: Density
Data density is an indication of how data is clumped together.
• For numerical data, distributions and statistical characteristics are informative.
• For non-numeric data, counts and distinct counts of a canonical representation are
extremely useful.
• For ALL data, an indication of how “empty” the data is.
6

14. Syntactic Understanding: Density
Data density is an indication of how data is clumped together.
• For numerical data, distributions and statistical characteristics are informative.
• For non-numeric data, counts and distinct counts of a canonical representation are
extremely useful.
• For ALL data, an indication of how “empty” the data is.
Canonical representations are representations which give you an idea at a glance of the
data format
6

15. Syntactic Understanding: Density
Data density is an indication of how data is clumped together.
• For numerical data, distributions and statistical characteristics are informative.
• For non-numeric data, counts and distinct counts of a canonical representation are
extremely useful.
• For ALL data, an indication of how “empty” the data is.
Canonical representations are representations which give you an idea at a glance of the
data format
• Replacing digits with the character ‘d’
• Stripping whitespace
• Normalizing punctuation
6

16. Syntactic Understanding: Density
Data density is an indication of how data is clumped together.
• For numerical data, distributions and statistical characteristics are informative.
• For non-numeric data, counts and distinct counts of a canonical representation are
extremely useful.
• For ALL data, an indication of how “empty” the data is.
Canonical representations are representations which give you an idea at a glance of the
data format
• Replacing digits with the character ‘d’
• Stripping whitespace
• Normalizing punctuation
Data density is an assumption underlying any conclusions drawn from your
data. 6

17. Syntactic Understanding: Density over Time
∆Density
∆t is how data clumps change over time.
7

18. Syntactic Understanding: Density over Time
∆Density
∆t is how data clumps change over time.
This kind of analysis can show
• Problems in the data pipeline
7

19. Syntactic Understanding: Density over Time
∆Density
∆t is how data clumps change over time.
This kind of analysis can show
• Problems in the data pipeline
• Whether the assumptions of your analysis are violated
7

20. Syntactic Understanding: Density over Time
∆Density
∆t is how data clumps change over time.
This kind of analysis can show
• Problems in the data pipeline
• Whether the assumptions of your analysis are violated
∆Density
∆t =⇒
• Automation
7

21. Syntactic Understanding: Density over Time
∆Density
∆t is how data clumps change over time.
This kind of analysis can show
• Problems in the data pipeline
• Whether the assumptions of your analysis are violated
∆Density
∆t =⇒
• Automation
• Outlier Alerting
7

22. Story Time: A Summation over Time Saves Face
The stage: An unnamed startup analyzing clinical effectiveness measurement for
hospital.
8

23. Story Time: A Summation over Time Saves Face
The stage: An unnamed startup analyzing clinical effectiveness measurement for
hospital.
• Think of it as a rules engine that takes medical data and outputs how well doctors
and departments are doing
8

24. Story Time: A Summation over Time Saves Face
The stage: An unnamed startup analyzing clinical effectiveness measurement for
hospital.
• Think of it as a rules engine that takes medical data and outputs how well doctors
and departments are doing
• Insights aren’t trusted if they’re wrong.
8

25. Story Time: A Summation over Time Saves Face
The stage: An unnamed startup analyzing clinical effectiveness measurement for
hospital.
• Think of it as a rules engine that takes medical data and outputs how well doctors
and departments are doing
• Insights aren’t trusted if they’re wrong.
• Correctness depend on good data
8

26. Semantic Understanding: “Do what I mean, not what I say”
Semantic understanding is understanding based on how the data is used rather than
how it is stored.
9

27. Semantic Understanding: “Do what I mean, not what I say”
Semantic understanding is understanding based on how the data is used rather than
how it is stored.
• Finding equivalences based on semantic understanding are often context sensitive.
9

28. Semantic Understanding: “Do what I mean, not what I say”
Semantic understanding is understanding based on how the data is used rather than
how it is stored.
• Finding equivalences based on semantic understanding are often context sensitive.
• May come from humans (e.g. domain experience and ontologies)
9

29. Semantic Understanding: “Do what I mean, not what I say”
Semantic understanding is understanding based on how the data is used rather than
how it is stored.
• Finding equivalences based on semantic understanding are often context sensitive.
• May come from humans (e.g. domain experience and ontologies)
• May come from machine learning (e.g. analyzing usage patterns to find synonyms)
9

30. Semantic Understanding: “Do what I mean, not what I say”
Semantic understanding is understanding based on how the data is used rather than
how it is stored.
• Finding equivalences based on semantic understanding are often context sensitive.
• May come from humans (e.g. domain experience and ontologies)
• May come from machine learning (e.g. analyzing usage patterns to find synonyms)
Semantic understanding may require data science. At the same time, data science will
require semantic understanding.
9

31. Story Time: Very Busy Bags of Mostly Water; Very Confused Boxes of Silicon
The stage: An unnamed insurance company building models to predict disease
10

32. Story Time: Very Busy Bags of Mostly Water; Very Confused Boxes of Silicon
The stage: An unnamed insurance company building models to predict disease
• Doctors and Nurses are busy people
10

33. Story Time: Very Busy Bags of Mostly Water; Very Confused Boxes of Silicon
The stage: An unnamed insurance company building models to predict disease
• Doctors and Nurses are busy people
• Humans suffer from confirmation bias
10

34. Story Time: Very Busy Bags of Mostly Water; Very Confused Boxes of Silicon
The stage: An unnamed insurance company building models to predict disease
• Doctors and Nurses are busy people
• Humans suffer from confirmation bias
• Machines can only interpret what they can see
10

35. Story Time: Very Busy Bags of Mostly Water; Very Confused Boxes of Silicon
The stage: An unnamed insurance company building models to predict disease
• Doctors and Nurses are busy people
• Humans suffer from confirmation bias
• Machines can only interpret what they can see
• Together we can fill in the gaps
10

36. SummarizerCLI

41. Implications for Team Structure
To be successful,
15

42. Implications for Team Structure
To be successful,
• Your data science teams have to be integrally involved in the data transformation
and understanding.
15

43. Implications for Team Structure
To be successful,
• Your data science teams have to be integrally involved in the data transformation
and understanding.
• Your data science teams have to be willing to get their hands dirty
15

44. Implications for Team Structure
To be successful,
• Your data science teams have to be integrally involved in the data transformation
and understanding.
• Your data science teams have to be willing to get their hands dirty
• Your data science teams have to be allowed to get their hands dirty
15

45. Implications for Team Structure
To be successful,
• Your data science teams have to be integrally involved in the data transformation
and understanding.
• Your data science teams have to be willing to get their hands dirty
• Your data science teams have to be allowed to get their hands dirty
• Your data science teams need software engineering chops.
15

46. Questions

47. Questions
Thanks for your attention! Questions?
• Code & scripts for this talk available on my github presentation page.1
• Find me at http://caseystella.com
• Twitter handle: @casey_stella
• Email address: cstella@hortonworks.com
1
http://github.com/cestella/presentations/
16

48. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
practices-data-curate