As organizations increasingly leverage data and machine learning methods, people throughout those organizations need to build a basic "data literacy" in those topics. In this session, data scientist and instructor Brian Lange provides simple, visual, and equation free explanations for a variety of classification algorithms, geared towards helping anyone understand how they work. Now with Python code examples!

1. It’s Not Magic
Brian Lange, Data Scientist + Partner at
Explaining
classification algorithms

2. HEADS UP

4. I work with some really
freakin’ smart people.

5. classification
algorithms

6. popular examples

7. popular examples
-spam filters

8. popular examples
-spam filters

9. popular examples
-spam filters
-the Sorting Hat

10. things to know

11. things to know
- you need data labeled with the correct answers to
“train” these algorithms before they work

12. things to know
- you need data labeled with the correct answers to
“train” these algorithms before they work
- feature = dimension = column = attribute of the data

13. things to know
- you need data labeled with the correct answers to
“train” these algorithms before they work
- feature = dimension = column = attribute of the data
- class = category = label = Harry Potter house

14. BIG CAVEAT
Often times choosing/creating
good features or gathering more
data will help more than
changing algorithms...

15. % of email body that is all-caps
# mentions
of brand
names spam
not spam

16. Linear
discriminants

17. % of email body that is all-caps
# mentions
of brand
names

18. % of email body that is all-caps
# mentions
of brand
names

19. % of email body that is all-caps
# mentions
of brand
names

20. % of email body that is all-caps
# mentions
of brand
names
1 wrong

21. % of email body that is all-caps
# mentions
of brand
names
5 wrong

22. % of email body that is all-caps
# mentions
of brand
names
4 wrong

23. % of email body that is all-caps
# mentions
of brand
names
4 wrong
y = .01x+4

24. terribleness
slope
intercept
a map of terribleness
to find the least terrible line

25. terribleness
slope
intercept
a map of terribleness
to find the least terrible line

26. terribleness
slope
intercept
“gradient descent”

27. terribleness
slope
intercept
“gradient descent”

28. training
data

29. training
data
import numpy as np
X = np.array([[1, 0.1], [3, 0.2], [5, 0.1]…])
y = np.array([1, 2, 1])

30. training
data

31. training
data

32. training
data
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
model = LinearDiscriminantAnalysis()
model.fit(X, y)

33. training
data
trained
model

34. new data
point
trained
model

35. trained
model

36. trained
model
new_point = np.array([1, .3])

37. trained
model
new_point = np.array([1, .3])
print(model.predict(new_point))

38. trained
model
new_point = np.array([1, .3])
print(model.predict(new_point))
1

39. trained
model
new_point = np.array([1, .3])
print(model.predict(new_point))
1
not
spam
prediction

40. trained
model
not
spam
prediction

41. Logistic
regression

42. logistic regression
“divide it with a logistic function”

43. logistic regression
“divide it with a logistic function”

44. logistic regression
“divide it with a logistic function”
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X,y)
predicted = model.predict(z)

45. Support Vector
Machines
(SVM)

46. SVMs (support vector machines)
“*advanced* draw a line through it”

47. SVMs (support vector machines)
“*advanced* draw a line through it”
- better definition of “terrible”

48. SVMs (support vector machines)
“*advanced* draw a line through it”
- better definition of “terrible”
- lines can turn into non-linear
shapes if you transform your data

51. 💩

53. 💩

55. “the kernel trick”

56. “the kernel trick”

57. SVMs (support vector machines)
“*advanced* draw a line through it”
figure credit: scikit-learn documentation

58. % of email body that is all-caps
# mentions
of brand
names

59. % of email body that is all-caps
# mentions
of brand
names

60. % of email body that is all-caps
# mentions
of brand
names

61. % of email body that is all-caps
# mentions
of brand
names

62. % of email body that is all-caps
# mentions
of brand
names

63. % of email body that is all-caps
# mentions
of brand
names

64. SVMs (support vector machines)
“*advanced* draw a line through it”
from sklearn.svm import SVC
model = SVC(kernel='poly', degree=2)
model.fit(X,y)
predicted = model.predict(z)

65. SVMs (support vector machines)
“*advanced* draw a line through it”
from sklearn.svm import SVC
model = SVC(kernel='rbf')
model.fit(X,y)
predicted = model.predict(z)

66. KNN
(k-nearest
neighbors)

67. KNN (k-nearest neighbors)
“what do similar cases look like?”

68. KNN (k-nearest neighbors)
“what do similar cases look like?”
k=1

69. KNN (k-nearest neighbors)
“what do similar cases look like?”
k=2

70. KNN (k-nearest neighbors)
“what do similar cases look like?”
figure credit: scikit-learn documentation

71. KNN (k-nearest neighbors)
“what do similar cases look like?”
k=1

72. KNN (k-nearest neighbors)
“what do similar cases look like?”
k=1

73. KNN (k-nearest neighbors)
“what do similar cases look like?”
k=2

74. KNN (k-nearest neighbors)
“what do similar cases look like?”
k=3

75. KNN (k-nearest neighbors)
“what do similar cases look like?”
figure credit: Burton DeWilde

76. KNN (k-nearest neighbors)
“what do similar cases look like?”
from sklearn.neighbors import NearestNeighbors
model = NearestNeighbors(n_neighbors=5)
model.fit(X,y)
predicted = model.predict(z)

77. Decision tree
learners

78. decision tree learners
make a flow chart of it

79. decision tree learners
make a flow chart of it
x < 3?
yes no
3

80. decision tree learners
make a flow chart of it
x < 3?
yes no
y < 4?
yes no
3
4

81. decision tree learners
make a flow chart of it
x < 3?
yes no
y < 4?
yes no
x < 5?
yes no
3 5
4

82. decision tree learners
make a flow chart of it
x < 3?
yes no
y < 4?
yes no
x < 5?
yes no
3 5
4

83. decision tree learners
make a flow chart of it
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier()
model.fit(X,y)
predicted = model.predict(z)

84. decision tree learners
make a flow chart of it
sklearn.tree.export_graphviz() +
pydot

85. decision tree learners
make a flow chart of it

86. Ensemble
models
(make a bunch of models and combine them)

87. bagging
split training set, train one model each, models “vote”

88. bagging
split training set, train one model each, models “vote”

89. bagging
split training set, train one model each, models “vote”

90. bagging
split training set, train one model each, models “vote”
new data
point

91. bagging
split training set, train one model each, models “vote”
new data
point

92. bagging
split training set, train one model each, models “vote”
new data
point
not
spam
spam
not
spam

93. bagging
split training set, train one model each, models “vote”
new data
point
not
spam
spam
not
spam
not
spam
Final
Answer:

94. bagging
split training set, train one model each, models “vote”

95. bagging
split training set, train one model each, models “vote”

96. other spins on this

97. other spins on this
Random Forest - like bagging, but at each split
randomly constrain features to choose from

98. other spins on this
Random Forest - like bagging, but at each split
randomly constrain features to choose from
Extra Trees - for each split, make it randomly, non-
optimally. Compensate by training a ton of trees

99. other spins on this
Random Forest - like bagging, but at each split
randomly constrain features to choose from
Extra Trees - for each split, make it randomly, non-
optimally. Compensate by training a ton of trees
Voting - combine a bunch of different models of your
design, have them “vote” on the correct answer.

100. other spins on this
Random Forest - like bagging, but at each split
randomly constrain features to choose from
Extra Trees - for each split, make it randomly, non-
optimally. Compensate by training a ton of trees
Voting - combine a bunch of different models of your
design, have them “vote” on the correct answer.
Boosting- train models in order, make the later ones
focus on the points the earlier ones missed

101. voting example
figure credit: scikit-learn documentation

102. other spins on this
Random Forest - like bagging, but at each split
randomly constrain features to choose from
Extra Trees - for each split, make it randomly, non-
optimally. Compensate by training a ton of trees
Voting - combine a bunch of different models of your
design, have them “vote” on the correct answer.
Boosting- train models in order, make the later ones
focus on the points the earlier ones missed

103. from sklearn.ensemble import BaggingClassifier
RandomForestClassifier
ExtraTreesClassifier
VotingClassifier
AdaBoostClassifier
GradientBoostingClassifier

105. which one do I
pick?

106. which one do I
pick?
try a few!

108. Nonlinear
decision
boundary
provide
probability
estimates
tell how important a
feature is to the model

109. Nonlinear
decision
boundary
provide
probability
estimates
tell how important a
feature is to the model
Logistic Regression no yes yes, if you scale

110. Nonlinear
decision
boundary
provide
probability
estimates
tell how important a
feature is to the model
Logistic Regression no yes yes, if you scale
SVMs yes, with kernel no no

111. Nonlinear
decision
boundary
provide
probability
estimates
tell how important a
feature is to the model
Logistic Regression no yes yes, if you scale
SVMs yes, with kernel no no
KNN yes kinda (percent of
nearby points)
no

112. Nonlinear
decision
boundary
provide
probability
estimates
tell how important a
feature is to the model
Logistic Regression no yes yes, if you scale
SVMs yes, with kernel no no
KNN yes kinda (percent of
nearby points)
no
Naïve Bayes yes yes no

113. Nonlinear
decision
boundary
provide
probability
estimates
tell how important a
feature is to the model
Logistic Regression no yes yes, if you scale
SVMs yes, with kernel no no
KNN yes kinda (percent of
nearby points)
no
Naïve Bayes yes yes no
Decision Tree yes no yes (number of times
that feature is used)

114. Nonlinear
decision
boundary
provide
probability
estimates
tell how important a
feature is to the model
Logistic Regression no yes yes, if you scale
SVMs yes, with kernel no no
KNN yes kinda (percent of
nearby points)
no
Naïve Bayes yes yes no
Decision Tree yes no yes (number of times
that feature is used)
Ensemble models yes kinda (% of models
that agree)
yes, depending on
component parts

115. Nonlinear
decision
boundary
provide
probability
estimates
tell how important a
feature is to the model
Logistic Regression no yes yes, if you scale
SVMs yes, with kernel no no
KNN yes kinda (percent of
nearby points)
no
Naïve Bayes yes yes no
Decision Tree yes no yes (number of times
that feature is used)
Ensemble models yes kinda (% of models
that agree)
yes, depending on
component parts
Boosted models yes kinda (% of models
that agree)
yes, depending on
component parts

117. can be updated with new
training data
easy to parallelize?

118. can be updated with new
training data
easy to parallelize?
Logistic Regression kinda kinda

119. can be updated with new
training data
easy to parallelize?
Logistic Regression kinda kinda
SVMs kinda, depending on kernel yes for some kernels,
no for others

120. can be updated with new
training data
easy to parallelize?
Logistic Regression kinda kinda
SVMs kinda, depending on kernel yes for some kernels,
no for others
KNN yes yes

121. can be updated with new
training data
easy to parallelize?
Logistic Regression kinda kinda
SVMs kinda, depending on kernel yes for some kernels,
no for others
KNN yes yes
Naïve Bayes yes yes

122. can be updated with new
training data
easy to parallelize?
Logistic Regression kinda kinda
SVMs kinda, depending on kernel yes for some kernels,
no for others
KNN yes yes
Naïve Bayes yes yes
Decision Tree no no (but it’s very fast)

123. can be updated with new
training data
easy to parallelize?
Logistic Regression kinda kinda
SVMs kinda, depending on kernel yes for some kernels,
no for others
KNN yes yes
Naïve Bayes yes yes
Decision Tree no no (but it’s very fast)
Ensemble models kinda, by adding new
models to the ensemble
yes

124. can be updated with new
training data
easy to parallelize?
Logistic Regression kinda kinda
SVMs kinda, depending on kernel yes for some kernels,
no for others
KNN yes yes
Naïve Bayes yes yes
Decision Tree no no (but it’s very fast)
Ensemble models kinda, by adding new
models to the ensemble
yes
Boosted models kinda, by adding new
models to the ensemble
no

125. Other quirks

126. Other quirks
SVMs have to pick a kernel

127. Other quirks
SVMs have to pick a kernel
KNN you need to define what “similarity” is in a good way.
fast to train, slow to classify (compared to other methods)

128. Other quirks
SVMs have to pick a kernel
KNN you need to define what “similarity” is in a good way.
fast to train, slow to classify (compared to other methods)
Naïve Bayes have to choose the distribution
can deal with missing data

129. Other quirks
SVMs have to pick a kernel
KNN you need to define what “similarity” is in a good way.
fast to train, slow to classify (compared to other methods)
Naïve Bayes have to choose the distribution
can deal with missing data
Decision Tree can provide literal flow charts
very sensitive to outliers

130. Other quirks
SVMs have to pick a kernel
KNN you need to define what “similarity” is in a good way.
fast to train, slow to classify (compared to other methods)
Naïve Bayes have to choose the distribution
can deal with missing data
Decision Tree can provide literal flow charts
very sensitive to outliers
Ensemble models less prone to overfitting than their component parts

131. Other quirks
SVMs have to pick a kernel
KNN you need to define what “similarity” is in a good way.
fast to train, slow to classify (compared to other methods)
Naïve Bayes have to choose the distribution
can deal with missing data
Decision Tree can provide literal flow charts
very sensitive to outliers
Ensemble models less prone to overfitting than their component parts
Boosted models many parameters to tweak
more prone to overfit than normal ensembles
most popular Kaggle winners use these

132. if this sounds cool
datascope.co/careers

133. thanks!
question time…
.cohttp:// @bjlange