1.
工業技術研究院機密資料 禁止複製、轉載、外流 ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE
Machine Learning, Deep
Learning and Data Analysis
簡介
劉得彥
teyen.liu@gmail.com

2.
Outline
2
• Overview of ML, DL and Data Analysis
• What is Machine Learning
▪ Take a Look At Linear Regression
▪ Other ML Algorithms at a Glance
▪ What is Neural Network?
• What is Deep Learning?
• Deep Learning using TensorFlow
• Data Analysis
▪ Case 1, 2 and 3
▪ Multivariate Analysis

3.
My Experience for Machine Learning
3
• 學習過程走了一些冤望路!!
▪ Hope giving you an experience and guideline
• Take courses:
▪ Coursera: Machine Learning ( Got Certificate )
▪ Udemy: Data Science: Deep Learning in Python ( ongoing)
• Study on-line resources:
▪ Youtube、ML/DL tutorials … and so on
▪ https://morvanzhou.github.io/
▪ http://bangqu.com/gpu/blog
▪ http://www.jiqizhixin.com/insights
• Get you hands dirty
▪ Python programming
a. TensorFlow Deep Learning Library
b. Scikit-Learn Library
c. Numby, Pandas, matplotlib, …

4.
From AI to Deep Learning
4
• 推薦觀賞: 人工智能 极客公园 2017 年 大会演讲
▪ Google首席科学家李飛飛 https://www.youtube.com/watch?v=uZ-7DVzRCy8
https://blogs.nvidia.com.tw/2016/07/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/
CPU/
GPU
Big Data
Algorithms
Breakthrough

5.
ML, DL and Data Analysis
5
• Visually Linking
• What we focus today
https://whatsthebigdata.com/2016/10/17/visually-linking-ai-machine-learning-deep-learning-big-data-and-data-science/
迷思??
Data Analysis

6.
WHAT IS MACHINE
LEARNING
6

7.
Machine Learning definition
7
• Arthur Samuel (1959). Machine Learning:
Field of study that gives computers the
ability to learn without being explicitly
programmed.
• Tom Mitchell (1998) Well-posed
Learning Problem: A computer program
is said to learn from experience E with
respect to some task T and some
performance measure P, if its
performance on T, as measured by P,
improves with experience E.

8.
Machine Learning definition
8
• Suppose your email program watches
which emails you do or do not mark as
spam, and based on that learns how to
better filter spam. What is the task T in
this setting?
▪ Classifying emails as spam or not spam. (T)
▪ Watching you label emails as spam or not spam. (E)
▪ The number (or fraction) of emails correctly classified
as spam/not spam. (P)
▪ None of the above—this is not a machine learning
problem

9.
What is Machine Learning ?
• Without writing any
custom code
specific to the
problem
• Feed data to the
generic algorithm
• It builds its own logic

10.
Two styles of Machine Learning
• Supervised Learning 監督式學習
• Unsupervised Learning 非監督式學習
Use the logic to
predict the sales
price
figure out if there is a
pattern or grouping or
something
Features Label

11.
What are machine learning
algorithms?
• Regression Algorithms
▪ Linear Regression
▪ Logistic Regression
▪ LASSO
• Decision Tree Algorithms
▪ Classification and Regression Tree (CART)
▪ Iterative Dichotomiser 3 (ID3)
▪ C4.5 and C5.0 (different versions of a powerful approach)
• Bayesian Algorithms
▪ Naive Bayes
• Clustering Algorithms (unsupervised)
▪ k-Means
• Support Vector Machines
• Principal Component Analysis
• Anomaly Detection
• Recommender Systems
• Artificial Neural Network Algorithms

12.
LET’S TAKE A LOOK AT
LINEAR REGRESSION
12

13.
Linear Regression
13
• The Hypothesis Function
• Cost Function
• Gradient Descent for Multiple Variables
https://www.coursera.org/learn/machine-learning/

14.
Gradient Descent
14
• How to choose learning α
https://www.coursera.org/learn/machine-learning/

15.
Gradient Descent
15
• Convergence of gradient descent with an appropriate
learning rate α
Cost
Function
https://www.coursera.org/learn/machine-learning/

16.
Linear Regression
16
• Training data with linear regression fit
https://www.coursera.org/learn/machine-learning/

17.
OTHER ML ALGORITHMS
AT A GLANCE
17

18.
Logistic Regression
18
• Training data with decision boundary
linear decision boundary
no linear decision boundary
https://www.coursera.org/learn/machine-learning/

19.
Support Vector Machines
19
• The difference between the kernels in SVM
▪ Linear
▪ Polynomial
▪ Gaussian (RBF)
▪ Sigmoid
• SVM (Gaussian Kernel) Decision Boundary
▪ Choose gamma ( auto )
Gaussian (RBF)
Non-linear decision boundary
https://www.coursera.org/learn/machine-learning/

20.
K-Means
20
• The original 128x128 image
with 24-bit color (three 8-bit )
• using K-means (K=16) to use
find the 16 colors that best
group (cluster) the pixels in the
3-dimensional RGB space.
• K=3 and computing centroid
means Iteratively
https://www.coursera.org/learn/machine-learning/
非監督式學習

21.
Principal Component Analysis
• An example to deal with image dimension
reduction and proximate recovery.
Faces Dataset
Recovered faces
Principal components
非監督式學習

22.
WHAT IS NEURAL
NETWORK
(We will review the previous concepts a little bit)
22

23.
ML -- write that program by
ourselves
• To estimate the price of a house
▪ If we could just figure out the perfect weights to use that work
for every house, our function could predict house prices!
▪ How to do that with ML?
def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
price = 0 # a little pinch of this
price += num_of_bedrooms * .841231951398213
price += sqft * 1231.1231231
price += neighborhood * 2.3242341421
price += 201.23432095
return price

24.
ML -- write that program by
ourselves
• Step 1
▪ Initialize weights to 1.0
• Step 2
▪ See the difference and how far off the function is at guessing the
correct price
• Step 3
▪ Repeat Step 2 over and over with every single possible
combination of weights.

25.
ML -- What about that whole “try
every number” bit in Step 3?
θ is what represents your current weights. J(θ) means the ‘cost for your current weights’.
• Clever ways to quickly find good values for those
weights without having to try very many.
• If we graph this cost equation for all possible values of
our weights for number_of_bedrooms and sqft
Gradient Descent

26.
Making Smarter Guesses
26
• We ended up with this simple estimation function
def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
price = 0 # a little pinch of this
price += num_of_bedrooms * 0.123
price += sqft * 0.41
price += neighborhood * 0.57
return price
a linear relationship with the input

27.
If there is more complicated situation?
▪ Different of weights for the different house sizes

28.
What is a Neural Network
• Now we have four different price estimates.
• Let’s combine those four price estimates
into one final estimate.
neurons
This is a neural network

29.
What is a Neural Network
• Human Brains
http://www.slideshare.net/tw_dsconf/ss-62245351

30.
What is a Neural Network
• Different
connections lead
to different
structured
network.
http://www.slideshare.net/tw_dsconf/ss-62245351

31.
Fully Connected Feedforward
Network
http://www.slideshare.net/tw_dsconf/ss-62245351

32.
Fully Connected Feedforward
Network
http://www.slideshare.net/tw_dsconf/ss-62245351

33.
Fully connected feedforward network
• Matrix
Operation
http://www.slideshare.net/tw_dsconf/ss-62245351

34.
Output Layer
• Softmax layer as the output layer

35.
Neural Network Playground
• http://playground.tensorflow.org/

36.
WHAT IS DEEP
LEARNING?
36

37.
What is Deep Learning?
37
• Deep Learning is Large Neural Networks
• Deep Learning attracts lots of attention
http://static.googleusercontent.com/media/research.google.com/en//people/jeff/BayLearn2015.pdf

38.
Why Deep Learning?
• The more data, the more performance.
• Game Changer
▪ DL accuracy/performance is more than 99%

39.
Deep Learning Models
39
• Convolutional Neural Network
▪ Inception-V3
• Recurrent Neural Network
▪ LSTM
• Auto-encoder
• Reinforcement Learning
▪ Q-Learning
▪ Policy Gradient
• Wide and Deep Learning
▪ Recommender system

40.
Deep Learning is not so simple
• Backpropagation
▪ an efficient way to compute Gradient Descent
• Overfitting
• Choosing Loss function
▪ Square Error, Cross Entropy, and so on…
• Mini-Batch
• Too deep ( many hidden layers )
▪ ReLU, MaxOut, …
• Learning Rates
• Momentum
▪ Adam ( optimizer )
• Weight Decay
• Dropout

41.
Backpropagation
41
• A common method of training artificial neural
networks and used in conjunction with an optimization
method such as gradient descent.

42.
Underfitting and Overfitting
• Bias-Variance Tradeoff

43.
Convolutional Neural Network (CNN)
• Why CNN is for image?
▪ The first layer of fully connected network would be very
large

44.
The solution is Convolution

45.
The solution is Convolution

46.
Convolutional Neural Network (CNN)
Adding Even More Steps

47.
Convolutional Neural Network (CNN)

48.
DEEP LEARNING USING
TENSORFLOW
48

49.
Linear Regression in TensorFlow
49
X_data  array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8,
0.9, 1. , 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7,
1.8, 1.9, 2. , 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, …
Y_data  array([ 0. , 0.29999667, 0.59997333, 0.89991 ,
1.19978668, 1.49958339, 1.79928013, 2.09885695,
2.39829388, 2.69757098, 2.99666833, 3.29556602, …

50.
Linear Regression in TensorFlow
50

51.
Linear Regression in TensorFlow
51

52.
MNIST
52
• 手寫數字辨識
• MNIST dataset
▪ 55000 samples (50000 for training, 5000 for testing)
▪ For each sample, it has X, y parts.
▪ X are the image with 28*28 pixels in 8 bit gray scale
▪ Y is the label answer: 0, 1, 2, …, 9

53.
MNIST
53
• X, y can be represented as follows

54.
MNIST
54
• If you want to get the accuracy more than 99%, check it out:
• https://gotocon.com/dl/goto-london-2016/slides/MartinGorner_TensorflowAndDeepLearningWithoutAPhD.pdf
92%

55.
Image Recognition and Retraining
• Inception-v3 model is ready and made by Google
▪ it took Google researchers two weeks to build on a desktop with eight NVidia
Tesla K40s.
▪ It can recognize > 1000 categories
• Retraining
▪ To prepare the new images and categories
▪ Do training and testing

56.
Plate Number Recognition
• There is an example using UK’s Plate Number
and Character to train TensorFlow CNN model
• Take 3 days with GPU Card (GTX 750 TI)
http://matthewearl.github.io/2016/05/06/cnn-anpr/

57.
Technical breakthrough for Deep-ANPR
http://matthewearl.github.io/2016/05/06/cnn-anpr/

58.
Autoencoder
• Enccode the input data (MNIST data) and then decode it back
• It is similar to PCA
autoencoder
original

59.
LSTM (RNN)
• It is a special kind of RNN, capable of learning
long-term dependencies
LSTM
Training with

60.
Recurrent Neural Network
60
RNN Model
training
output
Play this map with Super Mario Maker
https://medium.com/@ageitgey/machine-learning-is-fun-part-2-a26a10b68df3

61.
(BIG) DATA ANALYSIS
61

62.
Data Analysis
62
• The steps to do data analysis
▪ Data Collection
a. From CSV files, database, … and so on.
▪ Data Preprocessing ( very very very important )
a. Regularization, Normalization, …
b. Table Join,
▪ Feature Extraction
a. Reduce the dimensions ….
▪ Feature Selection
a. Select the important features
▪ Machine Learning / Deep Learning
a. To train the model
b. To do the Prediction and Classification by the trained model
c. Apply or implement to system
• But, still needs:
▪ domain experts involved!!
▪ Studying related papers and researches

63.
Analysis Tools and Libraries
63
• Open Sources(Python)
▪ Machine Learning
a. SciKit-Learn
b. NumPy
c. Matplotlib
d. Pandas
▪ Deep Learning
a. TensorFlow
b. Keras
▪ Hadoop & Spark
• Commercial Software ( rare to use…)
▪ PolyAnalyst 6.5
▪ SAS

64.
Data Analysis
64
• In my experience with data analysis, I
belong to a “Rookie”…
• 製造資料科學：從預測性思維到處方性決策
▪ http://www.slideshare.net/tw_dsconf/ss-71780267

65.
Reference
• Machine Learning is Fun! – Medium
• Machine Learning is Fun! Part 2 – Medium
• Machine Learning is Fun! Part 3: Deep Learning and ... - Medium
• Deep Learning Tutorial
• FIRST CONTACT WITH TENSORFLOW
• https://ireneli.eu/2016/02/03/deep-learning-05-talk-about-convolutional-neural-
network%EF%BC%88cnn%EF%BC%89/
• http://www.slideshare.net/tw_dsconf/ss-71780267
• https://morvanzhou.github.io/tutorials/python-basic/
• https://media.readthedocs.org/pdf/python-for-multivariate-analysis/latest/python-for-
multivariate-analysis.pdf
• http://blog.topspeedsnail.com/
• http://www.leiphone.com/news/201702/vJpJqREn7EyoAd09.html
• Python 之機器學習套件 scikit-learn
▪ https://machine-learning-python.kspax.io/
▪ version >= 0.17

66.
Thank You
66
如果你了解了機器學習，看到下列這句話, 你會會心一笑。