'Seeing Meaning in the age of Big Data'
Probabilistic modeling and machine learning touches every industry. Extracting meaning from data allows better user interaction, finds patterns that would otherwise be obscured using traditional BI reporting, and leads to defensible decision making.
Statistics notes ,it includes mean to index numbers
Visualizing inference
1. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Visualizing Inference
Seeing Meaning in the Age of Big Data
Alex Morrise
October 9, 2014
2. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
1 Big Data meets Machine Learning
2 Machine Learning Pros and Cons
What is Business Value of ML?
ML vs. BI
ML disclaimer
Personalization
3 Visualizing Inference
ML Infancy
Cases Studies
4 Summary
3. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
“Machine Learning is the antidote to having to
write down billions of business rules”
Machine Learning Uses Include:
User Personalization
4. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
“Machine Learning is the antidote to having to
write down billions of business rules”
Machine Learning Uses Include:
User Personalization
Finding Predictors to a given Objective (Revenue,
Churn, Volatility, Moods, Sentiment, Market Cap,
Stocks, Biomedical, Risk Assessment, etc)
5. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
“Machine Learning is the antidote to having to
write down billions of business rules”
Machine Learning Uses Include:
User Personalization
Finding Predictors to a given Objective (Revenue,
Churn, Volatility, Moods, Sentiment, Market Cap,
Stocks, Biomedical, Risk Assessment, etc)
Fraud Detection (Purchases, Identity Thief, Health
Insurance, Governments)
6. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
“Machine Learning is the antidote to having to
write down billions of business rules”
Machine Learning Uses Include:
User Personalization
Finding Predictors to a given Objective (Revenue,
Churn, Volatility, Moods, Sentiment, Market Cap,
Stocks, Biomedical, Risk Assessment, etc)
Fraud Detection (Purchases, Identity Thief, Health
Insurance, Governments)
Optimizing Decision Flows (Shipping, Markets,
Robotics, Database Migrations, Team resource
allocation and management, etc...)
7. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
“Machine Learning is the antidote to having to
write down billions of business rules”
Machine Learning Uses Include:
User Personalization
Finding Predictors to a given Objective (Revenue,
Churn, Volatility, Moods, Sentiment, Market Cap,
Stocks, Biomedical, Risk Assessment, etc)
Fraud Detection (Purchases, Identity Thief, Health
Insurance, Governments)
Optimizing Decision Flows (Shipping, Markets,
Robotics, Database Migrations, Team resource
allocation and management, etc...)
Key Take Away
Every Industry is Touched by Machine Learning
8. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Companies are being inundated with data:
User Behavioral Data/Click Stream (purchases, views,
engagement, interests)
9. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Companies are being inundated with data:
User Behavioral Data/Click Stream (purchases, views,
engagement, interests)
Sensor Data (Internet of Things, smart meters,
construction, management)
10. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Companies are being inundated with data:
User Behavioral Data/Click Stream (purchases, views,
engagement, interests)
Sensor Data (Internet of Things, smart meters,
construction, management)
B2B (SaaS management tools across all industries,
Salesforce, etc)
11. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Companies are being inundated with data:
User Behavioral Data/Click Stream (purchases, views,
engagement, interests)
Sensor Data (Internet of Things, smart meters,
construction, management)
B2B (SaaS management tools across all industries,
Salesforce, etc)
B2C (Uber, Netflix, google)
12. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Companies are being inundated with data. They all have
one thing in common.
They want a way to capitalize on the real meaning behind
the data.
Bayesian methods allow the discovery of the latent
properties in data, while assessing our confidence in
the models certainty/ignorance.
Hypothesis testing, parameter estimation, confidence
intervals, etc..
13. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Why do businesses need Machine Learning/Data Science?
They have plenty of data to train expert systems
14. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Why do businesses need Machine Learning/Data Science?
They have plenty of data to train expert systems
Traditional BI may not be able to find the correct
patterns
15. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Why do businesses need Machine Learning/Data Science?
They have plenty of data to train expert systems
Traditional BI may not be able to find the correct
patterns
Shifting focus from traditional 20th century business
objectives, companies need to convert their value
propositions into technological currency.
16. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business value of Machine
Learning and Data Science?
Why do businesses need Machine Learning/Data Science?
They have plenty of data to train expert systems
Traditional BI may not be able to find the correct
patterns
Shifting focus from traditional 20th century business
objectives, companies need to convert their value
propositions into technological currency.
Users & Businesses are sophisticated and want
intelligence in their applications
17. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business Value of BI?
Traditional BI tools and methods (Tableaux, Splunk, etc), are
amazing, sophisticated and potentially misleading
Example: Splitting demographic data into seemingly
good piles and running aggregating reporting over
those splits.
18. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business Value of BI?
Traditional BI tools and methods (Tableaux, Splunk, etc), are
amazing, sophisticated and potentially misleading
Example: Splitting demographic data into seemingly
good piles and running aggregating reporting over
those splits.
Wonderful if you want to know what women ages 24-26
are purchasing in Los Angeles this month.
19. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
What is the Business Value of BI?
Traditional BI tools and methods (Tableaux, Splunk, etc), are
amazing, sophisticated and potentially misleading
Example: Splitting demographic data into seemingly
good piles and running aggregating reporting over
those splits.
Wonderful if you want to know what women ages 24-26
are purchasing in Los Angeles this month.
What about Behavior?
20. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Businesses want Automatic Action
Finding Behavior in a Automatic Actionable Way
Finding Behavior requires leveraging the power of
machine learning to tease out the meaning behind the
observations in a crowd sourced way.
21. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Businesses want Automatic Action
Finding Behavior in a Automatic Actionable Way
Finding Behavior requires leveraging the power of
machine learning to tease out the meaning behind the
observations in a crowd sourced way.
Bayesian methods such as Factorizations, Hierarchical
Clustering, and other Topic models, extract meaning
from the data.
22. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Businesses want Automatic Action
Finding Behavior in a Automatic Actionable Way
Finding Behavior requires leveraging the power of
machine learning to tease out the meaning behind the
observations in a crowd sourced way.
Bayesian methods such as Factorizations, Hierarchical
Clustering, and other Topic models, extract meaning
from the data.
Once model is fit, observations of Behavior connect to
Actions in the system (API), yielding an automatic
intelligent way to process information and transactions.
23. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Disclaimer: Value of ML?
Let’s be Honest:
ML can also lead to misleading results when used in the
wrong hands:
Running a Decision Tree on demographic data could
likely split the population by M/F right off the bat.
24. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Disclaimer: Value of ML?
Let’s be Honest:
ML can also lead to misleading results when used in the
wrong hands:
Running a Decision Tree on demographic data could
likely split the population by M/F right off the bat.
This split will lead to misleading results as it tries to
explaining the objective.
25. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Disclaimer: Value of ML?
Let’s be Honest:
ML can also lead to misleading results when used in the
wrong hands:
Running a Decision Tree on demographic data could
likely split the population by M/F right off the bat.
This split will lead to misleading results as it tries to
explaining the objective.
Know your ML tool belt, practice makes perfect, and
treat the job as science (tests, validation, parameter
search, research, etc).
26. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Disclaimer: Value of ML?
Let’s be Honest:
ML can also lead to misleading results when used in the
wrong hands:
Running a Decision Tree on demographic data could
likely split the population by M/F right off the bat.
This split will lead to misleading results as it tries to
explaining the objective.
Know your ML tool belt, practice makes perfect, and
treat the job as science (tests, validation, parameter
search, research, etc).
Answer: Use a Random Forest instead.
27. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
ML is BI
Example: The Retail Vertical
BI reporting can be good for detecting aggregate trends
but fails to personalize.
28. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
ML is BI
Example: The Retail Vertical
BI reporting can be good for detecting aggregate trends
but fails to personalize.
Personalization can find aggregate trends and solve the
question,
“What are the 3 shoes you are highly likely to engage
and ultimately purchase, given your (sparse) purchase
history, time of year, demographic information, etc”
29. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Intermission
We’ve stated use cases for ML
Business will capitalize on including ML in their stack.
Let’s move on to see how we see the meaning behind
the data?
30. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Machine Learning in it’s Infancy
31. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Visualizing Inference
To see the latent properties in your data, construct a Graph
G as follows
Form your data matrix M
32. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Visualizing Inference
To see the latent properties in your data, construct a Graph
G as follows
Form your data matrix M
Factor it using your favorite algorithm, M = WH
33. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Visualizing Inference
To see the latent properties in your data, construct a Graph
G as follows
Form your data matrix M
Factor it using your favorite algorithm, M = WH
Cluster in W and Ht
34. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Visualizing Inference
To see the latent properties in your data, construct a Graph
G as follows
Form your data matrix M
Factor it using your favorite algorithm, M = WH
Cluster in W and Ht
Use the cluster assignments (or some similarity metric
on factors) to make graph G.
35. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
8tracks.com, the Best Music Service on Planet
Earth
36. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
8tracks.com
37. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
8tracks.com
38. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
8tracks.com
39. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Boomtrain.com
Boomtrain.com uses machine learning to inform decisions.
Boomtrain offers an end to end solution using, in part, a real
time novel view into the user base of a given company. By
learning
Users Proclivity to a set of Topics
User Archetypes
Users Derived Meta-Properties
Boomtrain.com exposes this knowledge in an actionable
framework, allowing clients to drive engagement and
retention.
40. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Boomtrain.com
41. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Quid.com (Assessing metrics in Technology and
Innovation)
IdleGames.com (Behavior as Predictor of Demographic
and Monetization)
BeatsMusic.com (Contextualized Music
Recommendation – Understanding the Heart of the
Music)
42. Visualizing
Inference
Alex Morrise
Outline
Big Data
meets
Machine
Learning
Machine
Learning Pros
and Cons
What is Business
Value of ML?
ML vs. BI
ML disclaimer
Personalization
Visualizing
Inference
ML Infancy
Cases Studies
Summary
Machine Learning is the Future
We are just at the onset of a radical transformation in the
way we do, and see, everything
Every business is transforming into a technology
company
They all need intelligence powering their core offerings
Finding better ways to see the meaning behind the data
will drive each of those offerings.