Functional CNN in elixir

Functional CNN in Elixir
Masayoshi Tamura
Retrieva, Inc.
© 2020, Retrieva, Inc. All rights reserved.

Agenda
• Self-introduction
• What is Elixir
• What is Convolutional Newral Network(CNN)
• CNN in Elixir
• Functional CNN
• Acceleration
• Tips
• Future work
© 2020, Retrieva, Inc. All rights reserved. 2

Who am I?
Self-introduction
Java
C
C#
C++
JavaScript
Python
Elixir
Haskell
prolog
R
matlab
Lua
php
SAS
SVF
Objective-C
cocos2d-x
AngularJS
jQuery
D3.js
TwitterAPI
Docker
CircleCI
Apache
Tomcat
OpenGL
GLSL
WebGL
OpenCL
Oracle
MySQL
PostgreSQL
SQLServer
memcached
Spring
Seasar2
Servlet
Jenkins
Nginx
WebSphere
RPM
Hadoop
maven
ProtocolBuffer
gRPC
Tensorflow
Chainer
scikit-learn
k-means
Naïve Bayes Classifier
Random Forest
Ray Tracer
Photon Mapper
Dynamic Programing
Greedy Algorithm
Binary Indexed Tree
Segement Tree

My Events in 2019
• Events
|> ElixirConf JP 2019 Lightning Talk
|> Forkwell press interview
|> Elixir Advent Calendar 2019 Day 18
|> fukuoka.ex Advent Calendar 2019 Day 17

First of all,
why Deep Learning in Elixir?

Motivation
• Fast training on the machine without CUDA
• To learn how deep learning is working
• Favorite Language
• No practical library in Elixir so far
• Ecosystem is growing

Resource
• My implementation is based on
“DeepLeening from scratch”
written by Koki Saito[1]
https://github.com/oreilly-japan/deep-learning-from-scratch

What is Elixir

Elixir
• Functional Programming Language
• Pipe |> Operator
• BEAM(Bogdan's Erlang Abstract Machine) = Erlang VM
• Immutable
• Concurrency
• Failure Resistance
• Not assignment but binding
• Availability
• Let it crash

Convolutional Neural Network

Neural Network
Input Layer Hidden Layer Output Layer

Deep Neural Network
Input Layer
Hidden Layers
Output Layer

Convolutional Neural Network
Input Layer Convolution Pooling Output Layer

how to use the network

Means to use the model
• Predict
• Infer the label(s) for the data
• Train
• Fix the parameters to obtain beter predictions
• Partial differentiation
• Backward propagation

Predict(forward)
• Make data pass through layers

Train(forward->backward)
• Make data pass through layers

Train(forward->backward)
• go back and adjust parameters as they should be
t

Update params
• ex). Affine
• 𝑊 =
𝑤11 𝑤12 𝑤13
𝑤21 𝑤22 𝑤23
•
𝜕𝐿
𝜕𝑊
=
𝜕𝐿
𝜕𝑤11
𝜕𝐿
𝜕𝑤12
𝜕𝐿
𝜕𝑤13
𝜕𝐿
𝜕𝑤21
𝜕𝐿
𝜕𝑤22
𝜕𝐿
𝜕𝑤23

Update params
• Backward Propagation
-> chain rule
• ex). Affine
dot +
X(N, 2)
W(2, 3)
X・W(N, 3) Y(N, 3)
B(3,)

Update params
• Backward Propagation
• ex). Affine
dot +
X(N, 2)
W(2, 3)
X・W(N, 3) Y(N, 3)
B(3,)
𝜕𝐿
𝜕𝑌
(N, 3)
𝜕𝐿
𝜕𝑌
(N, 3)
𝜕𝐿
𝜕𝑌
(3,) sum of cols(axis=0)
𝐿11 𝐿12 𝐿13
𝐿21 𝐿22 𝐿23
𝜕𝐿
𝜕𝑋
=
𝜕𝐿
𝜕𝑌
∙ 𝑊 𝑇
𝜕𝐿
𝜕𝑊
= 𝑋 𝑇 ∙
𝜕𝐿
𝜕𝑌

CNN in Elixir

Simple Two Layer Net

• 2 Affine Network
no type config Activation parameters
1 Affine (784, 100) ReLU 78,500
2 Affine (100, 10) Softmax 1,010

1 Affine (784, 100) ReLU 78,500

Affine Layer
• Full-Connected Layer
biasweight
x1
x2
x3

Affine Layer
• Forward
• Y = WX + B

1 Affine (784, 100) ReLU 78,500

ReLU
• Forward
• ℎ 𝑥 =
𝑥, 𝑥 > 0
0, 𝑥 ≤ 0

1 Affine (784, 100) ReLU 78,500

Softmax
• Forward
• 𝑦 𝑘 =
𝑒𝑥𝑝 𝑎 𝑘
𝑖=0
𝑛 𝑒𝑥𝑝 𝑎 𝑖

Softmax
• Forward
• 𝑦 𝑘 =
𝑒𝑥𝑝 𝑎 𝑘
𝑖=0
𝑛 𝑒𝑥𝑝 𝑎 𝑖
=
𝑒𝑥𝑝 𝑎 𝑘+𝐶′
𝑖=0
𝑛 𝑒𝑥𝑝 𝑎 𝑖+𝐶′
Due to overflow,
it is better to subtruct
maximum value in the list

CrossEntropyError
• 𝐸 = − 𝑘 𝑡 𝑘log 𝑦 𝑘

Softmax with CrossEntropyError
• Backward
• 𝑦 𝑘 − 𝑡 𝑘

ReLU
• Backward
• ℎ 𝑑𝑜𝑢𝑡 =
𝑑𝑜𝑢𝑡, 𝑥 > 0 𝑤ℎ𝑒𝑟𝑒 𝑥 𝑖𝑠 𝑡ℎ𝑒 𝑖𝑛𝑝𝑢𝑡 𝑜𝑓 𝑓𝑜𝑟𝑤𝑎𝑟𝑑
0, 𝑥 ≤ 0

Affine Layer
• Backward
• W = W – (XT・dout)
• B = sum(dout, axis=0)
• dx = dout ・WT

Forward
• protocol

Simple ConvNet

Simple ConvNet
• 1 Convolution + 2 Affine Network
1 Convolution2D 30, (5, 5), 1, 0 ReLU 780
2 MaxPooling (2, 2, 0) - 0
3 Affine (4320, 100) ReLU 432,100

im2col
• pick the values in the kernel

im2col

im2col
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
20 21 22 23 24
0 1 5 6

im2col
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
20 21 22 23 24
25 26 27 28 29
30 31 32 33 34
35 36 37 38 39
40 41 42 43 44
45 46 47 48 49
0 1 5 6 25 26 30 31

im2col
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
20 21 22 23 24
25 26 27 28 29
30 31 32 33 34
35 36 37 38 39
40 41 42 43 44
45 46 47 48 49
0 1 5 6 25 26 30 31
1 2 6 7 26 27 31 32

im2col
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
20 21 22 23 24
out_w

im2col
• ex). y=1, x=1
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
20 21 22 23 24

im2col
• ex). y=1, x=1
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
20 21 22 23 24

im2col
• ex). y=1, x=1
5 6 7 8 9
10 11 12 13 14

im2col
• ex). y=1, x=1
6 7 8 9
11 12 13 14

im2col
• ex). y=1, x=1
6 7
11 12

im2col
• ex). y=1, x=1
6 7 11 12

im2col
• ex). y=1, x=1
6 7 11 12 31 32 36 37

Simple ConvNet
2 MaxPooling2D (2, 2, 0) - 0
3 Affine (4320, 100) ReLU 432,100

MaxPooling2D
• Remain maximum value in the kernel
0 3 1 6
10 4 5 2
1 5 7 1
8 4 2 3
3 2 9 4
3 6 2 8

MaxPooling2D
0 3 1 6
10 4 5 2
1 5 7 1
8 4 2 3
3 2 9 4
3 6 2 8

MaxPooling2D
10 4 5 2
1 5 7 1
8 4 2 3
3 2 9 4
3 6 2 8
6

MaxPooling2D
6
10
7
8
9
8

MaxPooling2D
6 10 7 8
9 8 8 6
5 9 7 8
9 7 10 7

MaxPooling2D
• Forward

Simple ConvNet
2 MaxPooling2D (2, 2, 0) - 0
3 Affine (4320, 100) ReLU 432,100

Convolution2D
• Forward
• im2col(2x2 kernel)
0 1 5 6 25 26 30 31
1 2 6 7 26 27 31 32
2 3 7 8 27 28 32 33
3 4 8 9 28 29 33 34

Convolution2D
• Forward
• im2col ・ WT + B
0 1 5 6 25 26 30 31
1 2 6 7 26 27 31 32
2 3 7 8 27 28 32 33
3 4 8 9 28 29 33 34
(N, 8)
(8, out_channel)

Convolution2D
• Forward
0 1 5 6 25 26 30 31
1 2 6 7 26 27 31 32
2 3 7 8 27 28 32 33
3 4 8 9 28 29 33 34
(N, 8)
(8, out_channel)
o1

o1 o2
Convolution2D
• Forward
0 1 5 6 25 26 30 31
1 2 6 7 26 27 31 32
2 3 7 8 27 28 32 33
3 4 8 9 28 29 33 34
(N, 8)
(8, out_channel)

o1 o2 o3
Convolution2D
• Forward
0 1 5 6 25 26 30 31
1 2 6 7 26 27 31 32
2 3 7 8 27 28 32 33
3 4 8 9 28 29 33 34
(N, 8)
(8, out_channel)

• Forward
o1 o2 o3
Convolution2D
0 1 5 6 25 26 30 31
1 2 6 7 26 27 31 32
2 3 7 8 27 28 32 33
3 4 8 9 28 29 33 34
(N, 8)
(8, out_channel)
b1 b2 b3
+

• Forward
o1 o2 o3
Convolution2D
0 1 5 6 25 26 30 31
1 2 6 7 26 27 31 32
2 3 7 8 27 28 32 33
3 4 8 9 28 29 33 34
(N, 8)
(8, out_channel)
b1 b2 b3
+
(N, out_channel)

Convolution2D
• Forward
• reshape+transpose
o11 o12 o13
o21 o22 o23
o31 o32 o33
o41 o42 o43
o11 o12 o13
o21 o22 o23
o31 o32 o33
o41 o42 o43
reshape
(batch, height, width, channel)

Convolution2D
• Forward
• reshape+transpose
o11 o12 o13
o21 o22 o23
o31 o32 o33
o41 o42 o43
o11 o12 o13
o21 o22 o23
o31 o32 o33
o41 o42 o43
reshape
(batch, height, width, channel)
o13 o23
o33 o41
o12 o22
o32 o41
o11 o21
o31 o41
transpose(0, 3, 1, 2)

Convolution2D
• Forward

MaxPooling2D
• Backward
d1 d2 d3 d4
d5 d6 d7 d8
d9 d d d
d d d d
0 3 1 6
10 4 5 2
1 5 7 1
8 4 2 3
3 2 9 4
3 6 2 8
im2col

MaxPooling2D
• Backward
d1 d2 d3 d4
d5 d6 d7 d8
d9 d d d
d d d d
0 3 1 6
10 4 5 2
1 5 7 1
8 4 2 3
3 2 9 4
3 6 2 8
0 3 10 4 1 5 8 4
1 6 5 2 7 1 2 3
3 2 3 6 7 2 12 5
9 4 2 8 8 1 3 8
im2colinput

MaxPooling2D
• Backward

Convolution2D
• Backward
• dw: colT・dout’
d1 d2 d3 d4
d5 d6 d7 d8
d9 d d d
d d d d
d11 d12 d13 d14
d21 d22 d23 d24
d31 d32 d33 d34
d41 d42 d43 d44
d51 d52 d53 d54
d61 d62 d63 d64
reshape+transpose(0, 2, 3, 1)
dout’

Convolution2D
• Backward
• db: sum of columns
d1 d2 d3 d4
d5 d6 d7 d8
d9 d d d
d d d d
d11 d12 d13 d14
d21 d22 d23 d24
d31 d32 d33 d34
d41 d42 d43 d44
d51 d52 d53 d54
d61 d62 d63 d64
dout’

Convolution2D
• Backward
d1 d2 d3 d4
d5 d6 d7 d8
d9 d d d
d d d d
d11 d12 d13 d14
d21 d22 d23 d24
d31 d32 d33 d34
d41 d42 d43 d44
d51 d52 d53 d54
d61 d62 d63 d64

Convolution2D
• Backward
• dx: same as MaxPooing2D
d1 d2 d3 d4
d5 d6 d7 d8
d9 d d d
d d d d
0 3 1 6
10 4 5 2
1 5 7 1
8 4 2 3
3 2 9 4
3 6 2 8
0 3 10 4 1 5 8 4
1 6 5 2 7 1 2 3
3 2 3 6 7 2 12 5
9 4 2 8 8 1 3 8
w
im2colinput

Convolution2D
• Backward

Result
• MNIST(N, 1, 28, 28)

Is this like Functional programming?

Make it functional
• Remove struct!

Make it functional
• Forward

Pros and Cons
• Pros
• Simple(pass only as the function needs)
• No protocol needed
• Less memory usage
• Cons
• Pattern match increases
• Doctest does not work with pattern match?
• Huge result lists are displayed

CIFAR10
2 Convolution2D 32, (3, 3), 1, 1 ReLU 9,248
3 MaxPooling (2, 2, 0) - 0
4 Dropout 0.25 - 0
7 MaxPooling (2, 2, 0) - 0
8 Dropout 0.25 - 0
9 Affine (4096, 512) ReLU 2,097,664
10 Dropout 0.5 - 0

Acceleration

Matrex

Matrex
• Fast matrix manipulation library with CBLAS

Matrex
• Fast matrix manipulation library with CBLAS
• Problem
• Only for matrix
• Huge cost to create Matrex struct

NIF

NIF
• Native Implemented Functions
• Enable to call C/C++ functions

Bench
• Checked by Benchfella

Tips

Use Float for tests
• Numbers are automatically converted to chars

Memory usage
• Explicitly use processes
• Avoid defstruct
• Do NOT rush data
• Enum.reduce consumes bigger amount of memory

Future work

Future work
• Use OpenCL(NIF)
• Implement more layers
• Apply Pelemay
• Apply Cockatrice

Github
• Release in a few days
https://github.com/muramasa8191/broca

Reference
[1] K. Saito, “ゼロから作るDeep Learning”, O’REILLY Japan,
2016

Functional CNN in elixir

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Functional CNN in elixir

Similaire à Functional CNN in elixir (20)

Dernier

Dernier (20)

Functional CNN in elixir