Presented at #H2OWorld 2017 in Mountain View, CA.
Enjoy the video: https://youtu.be/TBJqgvXYhfo.
Learn more about H2O.ai: https://www.h2o.ai/.
Follow @h2oai: https://twitter.com/h2oai.
- - -
Abstract:
Machine learning is at the forefront of many recent advances in science and technology, enabled in part by the sophisticated models and algorithms that have been recently introduced. However, as a consequence of this complexity, machine learning essentially acts as a black-box as far as users are concerned, making it incredibly difficult to understand, predict, or "trust" their behavior. In this talk, I will describe our research on approaches that explain the predictions of ANY classifier in an interpretable and faithful manner.
Sameer's Bio:
Dr. Sameer Singh is an Assistant Professor of Computer Science at the University of California, Irvine. He is working on large-scale and interpretable machine learning applied to natural language processing. Sameer was a Postdoctoral Research Associate at the University of Washington and received his PhD from the University of Massachusetts, Amherst, during which he also worked at Microsoft Research, Google Research, and Yahoo! Labs on massive-scale machine learning. He was awarded the Adobe Research Data Science Faculty Award, was selected as a DARPA Riser, won the grand prize in the Yelp dataset challenge, and received the Yahoo! Key Scientific Challenges fellowship. Sameer has published extensively at top-tier machine learning and natural language processing conferences. (http://sameersingh.org)
6. More Complex: Question Answering
Is there a moustache in the picture?
> Yes
What is the moustache made of?
> Banana
7. Essentially black-boxes!
How can we trust the
predictions are correct?
How do we know they are
not breaking regulations?
How do we avoid
“stupid mistakes”?
Trust
How can we understand and
predict the behavior?
Predict
How do we improve it to
prevent potential mistakes?
Improve
10. Visual Question Answering
What is the moustache made of?
> Banana
What are the eyes made of?
> Bananas
What?
> Banana What is?
> Banana
11. Text Classification
Why did this
happen?
From: Keith Richards
Subject: Christianity is the answer
NTTP-Posting-Host: x.x.com
I think Christianity is the one true religion.
If you’d like to know more, send me a note
12. Applying for a Loan
Machine
Learning
I would like to apply for a loan.
vvvvvvv
vvvvvvv
vvvvvvv
vvvvvvv
vvvvvvv
Here is my information.
vvvvvvv
vvvvvvv
vvvvvvv
vvvvvvv
vvvvvvv
Sorry, your request has been denied
Why? What were the reasons?
Currently
Cannot explain.. [0.25,-4.5,3.5,-10.4,…]
13. How did we get here?
Big Data and Deep Learning
16. Decision trees
X1
X2
X1 > 0.5
X2 > 0.5
You can interpret it…
- X2 is irrelevant if X1<0.5
- Otherwise X2 is enough
17. Looking at the structure
How can we trust the
predictions are correct?
Trust
How can we understand and
predict the behavior?
Predict
How do we improve it to
prevent potential mistakes?
Improve
Test whether the structure
agrees with our intuitions.
Structure tells us exactly what
will happen on any data.
Structure tells you where the
error is, thus how to fix it.
22. Big Data: More Dimensions
Savings
Income
Profession
Loan Amount
Age
Marital
Status
Past defaults
Credit scores
Recent defaults
This easily goes to hundreds
- Images: thousands
- Text: tens of thousands
- Video: millions
- … and so on
26. Looking at the structure
How can we trust the
predictions are correct?
Trust
How can we understand and
predict the behavior?
Predict
How do we improve it to
prevent potential mistakes?
Improve
Test whether the structure
agrees with our intuitions.
Structure tells us exactly what
will happen on any data.
Structure tells you where the
error is, thus how to fix it.
29. LIME: Explain Any Classifier!
Interpretability
Accuracy
Real-world use case Make
everything
interpretable!
30. What an explanation looks like
Why did this
happen?
From: Keith Richards
Subject: Christianity is the answer
NTTP-Posting-Host: x.x.com
I think Christianity is the one true religion.
If you’d like to know more, send me a note
38. Comparing Classifiers
Classifier 1
Classifier 2
Explanations?
Look at Examples?
Deploy and Check?
“I have a gut feeling..”
Accuracy?
Change the model
Different data
Different parameters
Different “features”
…
40. Explanation for a bad classifier
From: Keith Richards
Subject: Christianity is the answer
NTTP-Posting-Host: x.x.com
I think Christianity is the one true religion.
If you’d like to know more, send me a note
After looking at the explanation,
we shouldn’t trust the model!
43. Understanding via Predicting
Users “understand” a model if they can
predict its behavior on unseen instances
Precision is much more
important than Coverage!
Precision
How accurate are the users guesses?
If the users guess wrong, they don’t understand
Coverage
How often do the users make confident guesses?
It’s okay not to be able to guess!
It’s much better not to guess than to guess
confidently, but be completely wrong!
44. Linear Explanations
This movie is not bad. This movie is not very good.
LIMELIME
D
D
…
D
This director is always bad.
This movie is not nice.
This stuff is rather honest.
This star is not bad.
Problem 1: Where is the explanation good?
Explanation is wrong in this region
This explanation is a better approximation
than the other one.
Problem 2: What is the coverage?
Explanation doesn’t apply here
→ Users will make mistakes!
45. Anchors: Precise Counter-factuals
Anchor: ”not bad” →
This movie is not bad.
This audio is not bad.
This novel is not bad.
This footage is not bad.
D(.|A)
Positive
This movie is not very good.
Anchor: ”not good” →
This poster is not ever good.
This picture is not rarely good.
This actor is not incredibly good.
D(.|A)
Negative
anchor
anchor
LIMELIME
D
D
An anchor is a sufficient condition
Clear (and adaptive) coverage Probabilistic guarantee avoids human mistakes
51. What’s a Good Explanation?
We want to understand the models
Compact description
Lines, Decision Trees,
Simple Rules, etc.
When we read them,
we imagine instances
where they apply, and
where they don’t
Directly show useful examples?
What examples describe the behavior?
Closest Counter-example:
How can we change this example
to change the prediction?
52. Adversarial Examples
Goodfellow et al, "Explaining and Harnessing Adversarial Examples", ICLR 2015.
adversary predicted as "2"original MNIST digit "3"
+ .02 x =
adversarial noise
"inputs formed by applying small but
intentionally worst-case perturbations
to examples from the dataset, such
that the perturbed input results in the
model outputting an incorrect answer
with high confidence"
53. Adversarial Examples: Pros
Advantages:
◦ Applicable to any gradient -based classifier
◦ Useful to evaluate the robustness of the model against adversaries
◦ Small perturbations often lead to imperceivable adversarial examples
54. Adversarial Examples: Cons
Disadvantages:
◦ Examples are unnatural
◦ may not look anything you would naturally see in the "wild"
◦ Distance is not always meaningful
◦ E.g. color change or translation/rotation of an image
◦ Cannot be used for structured domains like text, code, etc.:
◦ E.g. replacing/removing words results in sentences that are not grammatical
◦ Do not provide insights into why the sample is an adversary
◦ How is the model working?
◦ How to fix the model?
58. Explanations are important!
How can we trust the
predictions are correct?
Trust
How can we understand and
predict the behavior?
Predict
How do we improve it to
prevent potential mistakes?
Improve
Model Agnostic
Explanations