Digital Garage & Naviplus 主催
第２回Pythonで実践する深層学習
浅川伸一 asakawa@ieee.org

10/06/2016
2/24
Notice
次回以降で取り上げる話題，データ，プロジェクト，質疑応答のために
slack のチームを作成しました。今回参加されなかった方でも自由に参
加できます。チーム名は deeppython.slack.com です。ご参加ください。
参加ご希望の方は deeplearning.w.python@gmail.com までメールをお
願いします。
＃DLｗPY

10/06/2016
3/24
本日のお品書き
●
Basic knowledge of neurons
● Cross entropy
●
Feedforwards and feedbacks of neural networks
●
Try googleplayground
● Convolutional neural networks
● Try keras

10/06/2016
4/24
前回の補足
最速で理解するには
1. 線形回帰 linear regression
2. ロジスティック回帰 logistic regression
3. 正則化 regularization
4. 多層パーセプトロン multi-layered perceptrons
5. 畳み込みニューラルネットワーク convolutional neural networks
6. リカレントニューラルネットワーク recurrent neural networks
7. 強化学習 reinforcement learning

10/06/2016
5/24
Warning: Stealing Machine Learning Models via
Prediction APIs https://arxiv.org/abs/1609.02943
Machine learning (ML) models may be deemed confidential due to their sensitive training data,
commercial value, or use in security applications. Increasingly often, confidential ML models are
being deployed with publicly accessible query interfaces. ML-as-a-service ("predictive analytics")
systems are an example: Some allow users to train models on potentially sensitive data and charge
others for access on a pay-per-query basis.
The tension between model confidentiality and public access motivates our investigation of model
extraction attacks. In such attacks, an adversary with black-box access, but no prior knowledge of
an ML model's parameters or training data, aims to duplicate the functionality of (i.e., "steal") the
model. Unlike in classical learning theory settings, ML-as-a-service offerings may accept partial
feature vectors as inputs and include confidence values with predictions. Given these practices, we
show simple, efficient attacks that extract target ML models with near-perfect fidelity for popular
model classes including logistic regression, neural networks, and decision trees. We demonstrate
these attacks against the online services of BigML and Amazon Machine Learning. We further show
that the natural countermeasure of omitting confidence values from model outputs still admits
potentially harmful model extraction attacks. Our results highlight the need for careful ML model
deployment and new model extraction countermeasures.

10/06/2016
6/24
Prerequisites
●
Basics about Neural Networks
– How the brain actually works?
– How parallel computation works adapting parameters inspired by
neurons?
– How the brain implements learning algorithms?
● (ペ)What is it a good idea to try to emulate the brain when solving
a recognition task?

10/06/2016
7/24
A schematic neuron
There are many neurotransimtters, but
we deal with those as positive/negative
weights and also positive negative
inputs. (ペ) why?
http://www.mhhe.com/socscience/intro/ibank/set1.htm

10/06/2016
8/24
Why computer vision is so difficult?
http://xkcd.com/1425/

10/06/2016
9/24
Cross entropy

10/06/2016
10/24
playground.tensorflow.org

10/06/2016
11/24
Convnet.js http://cs.stanford.edu/people/karpathy/convnetjs/

10/06/2016
12/24
TensorFlow Tips
●
Computation graph (see http://colah.github.io/posts/2015-08-Backprop/ , その翻訳記事
は http://postd.cc/2015-08-backprop/ )
● Ways of installations (c.f. Tensorflow.org Download and Setup )
– Pip
– Virutalenv
– Anaconda
– Docker
● Let’s try http://playground.tensorflow.org/
● You can also check it out, Karpthy’s convnetjs
● Keras is another choice to consider

10/06/2016
13/24
A computation graph
http://deeplearning.net/software/theano/extending/graphstructures.html

10/06/2016
14/24
Sample code of TensorFlow
import tensorflow as tf
W = tf.get_variable(shape=[], name='W')
b = tf.get_variable(shape=[], name='b')
x = tf.placeholder(shape=[None], dtype=tf.float32, name='x')
y = tf.matmul(W, x) + b
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(sess.run(y, feed_dict={x: x_in}))

10/06/2016
15/24
The difference between placeholder and variable
Since Tensor computations compose graphs then it's better to interpret the two in terms of graphs.
When you launch the graph, variables have to be explicitly initialized before you can run Ops that use
their value. Then during the process of the an operation variables should be constant.
import tensorflow as tf
# Create a variable.
# w = tf.Variable(<initial-value>, name=<optional-name>)
w = tf.Variable(tf.truncated_normal([10, 40]))
v = tf.Variable(tf.truncated_normal([40, 20]))
# Use the variable in the graph like any Tensor.
# The variable should be initialized before this operation!
y = tf.matmul(w, v)
# Assign a new value to the variable with `assign()` or a related method.
w.assign(w + 1.0)
w.assign_add(1.0)
http://stackoverflow.com/questions/36693740/whats-the-difference-between-tf-placeholder-and-tf-variable
tf.Variableはオペレーション実行前に初期化される

10/06/2016
16/24
The difference between placeholder and variable
A placeholder is a handle of a value in the operation and it can be not initialized before the execution of
the graph (launching the graph in the session which does its computation relaying on a highly efficient C+
+ backend).
x = tf.placeholder(tf.float32, shape=(1024, 1024))
# You don't need to initialize it to calculate y, it's different from
# the variable above, the placeholder is a "variable"(not intialized)
# in this operation.
y = tf.matmul(x, x)
with tf.Session() as sess:
# However you should initialize x to execute y for the execution of the graph.
print(sess.run(y)) # ERROR: will fail because x was not fed.
rand_array = np.random.rand(1024, 1024)
print(sess.run(y, feed_dict={x: rand_array})) # Will succeed.
プレースホルダーは初期化されない

10/06/2016
17/24
Convolutional neural networks: LeNet5
LeCun1998 より

10/06/2016
18/24
AlexNet
Krizensky+2012 より

10/06/2016
19/24
GoogLeNet

10/06/2016
20/24
GoogLeNet Inception module

10/06/2016
21/24
畳み込み演算

10/06/2016
22/24
kernels

10/06/2016
23/24
To understand convolution
●
畳み込みの理解には
http://colah.github.io/posts/2014-07-Understanding-Convolutions/

10/06/2016
24/24
What is convolution?
●
Convolution is an operation from signal processing
● Filters, or kernels in machine learning,