"A fast-paced introduction to Deep Learning (DL) concepts, such as neural networks, back propagation, activation functions, and CNNs. We'll also look at JavaScript-based toolkits (such as TensorFire and deeplearning.js) that leverage the power of WebGL. Basic knowledge of elementary calculus (e.g., derivatives) is recommended in order to derive the maximum benefit from this session.

1. Deep Learning and WebGL
WebGL Meetup 03/27/2018
GTC SJ Convention Center
Oswald Campesato
oswald@perceptrons.io
ocampesato@yahoo.com

2. Overview
intro to AI/ML/DL
linear regression
activation/cost functions
gradient descent
back propagation
hyper-parameters
what are CNNs
JS/WebGL and DL

3. The Data/AI Landscape

4. Gartner 2017: Deep Learning (YES!)

5. The Official Start of AI (1956)

6. Neural Network with 3 Hidden Layers

7. AI/ML/DL: How They Differ
Traditional AI (20th century):
based on collections of rules
Led to expert systems in the 1980s
The era of LISP and Prolog

8. AI/ML/DL: How They Differ
Machine Learning:
Started in the 1950s (approximate)
Alan Turing and “learning machines”
Data-driven (not rule-based)
Many types of algorithms
Involves optimization

9. AI/ML/DL: How They Differ
Deep Learning:
Started in the 1950s (approximate)
The “perceptron” (basis of NNs)
Data-driven (not rule-based)
large (even massive) data sets
Involves neural networks (CNNs: ~1970s)
Lots of heuristics
Heavily based on empirical results

10. AI/ML/DL: Commonality
All of them involve a model
A model represents a system
Goal: a good predictive model
The model is based on:
Many rules (for AI)
data and algorithms (for ML)
large sets of data (for DL)

11. A Basic Model in Machine Learning
Let’s perform the following steps:
1) Start with a simple model (2 variables)
2) Generalize that model (n variables)
3) See how it might apply to a NN

12. Linear Regression
One of the simplest models in ML
Fits a line (y = m*x + b) to data in 2D
Finds best line by minimizing MSE:
m = average of x values (“mean”)
b also has a closed form solution

13. Linear Regression in 2D: example

14. Sample Cost Function #1 (MSE)

15. Linear Regression: example #1
One feature (independent variable):
X = number of square feet
Predicted value (dependent variable):
Y = cost of a house
A very “coarse grained” model
We can devise a much better model

16. Linear Regression: example #2
Multiple features:
X1 = # of square feet
X2 = # of bedrooms
X3 = # of bathrooms (dependency?)
X4 = age of house
X5 = cost of nearby houses
X6 = corner lot (or not): Boolean
a much better model (6 features)

17. Linear Multivariate Analysis
General form of multivariate equation:
Y = w1*x1 + w2*x2 + . . . + wn*xn + b
w1, w2, . . . , wn are numeric values
x1, x2, . . . , xn are variables (features)
Properties of variables:
Can be independent (Naïve Bayes)
weak/strong dependencies can exist

18. Neural Network with 3 Hidden Layers

19. Neural Networks: equations
Node “values” in first hidden layer:
N1 = w11*x1+w21*x2+…+wn1*xn
N2 = w12*x1+w22*x2+…+wn2*xn
N3 = w13*x1+w23*x2+…+wn3*xn
. . .
Nn = w1n*x1+w2n*x2+…+wnn*xn
Similar equations for other pairs of layers

20. Neural Networks: Matrices
From inputs to first hidden layer:
Y1 = W1*X + B1 (X/Y1/B1: vectors; W1: matrix)
From first to second hidden layers:
Y2 = W2*X + B2 (X/Y2/B2: vectors; W2: matrix)
From second to third hidden layers:
Y3 = W3*X + B3 (X/Y3/B3: vectors; W3: matrix)
 Apply an “activation function” to y values

21. Neural Networks (general)
Multiple hidden layers:
Layer composition is your decision
Activation functions: sigmoid, tanh, RELU
https://en.wikipedia.org/wiki/Activation_function
Back propagation (1980s)
https://en.wikipedia.org/wiki/Backpropagation
=> Initial weights: small random numbers

22. Euler’s Function

23. The sigmoid Activation Function

24. The tanh Activation Function

25. The ReLU Activation Function

26. What’s the “Best” Activation Function?
Initially: sigmoid was popular
Then: tanh became popular
Now: RELU is preferred (better results)
Softmax: for FC (fully connected) layers
NB: sigmoid and tanh are used in LSTMs

27. Sample Cost Function #1 (MSE)

28. Sample Cost Function #2

29. Sample Cost Function #3

30. CNNs versus RNNs
CNNs (Convolutional NNs):
Good for image processing
2000: CNNs processed 10-20% of all checks
=> Approximately 60% of all NNs
RNNs (Recurrent NNs):
Good for NLP and audio

31. Convolutional Neural Networks

32. CNNs: Convolution Calculations
https://docs.gimp.org/en/plug-in-
convmatrix.html

33. CNNs: Convolution Matrices (examples)
Sharpen:
Blur:

34. CNNs: Convolution Matrices (examples)
Edge detect:
Emboss:

35. How do Filters Work?
 compute “inner product” with an image
The “padding” parameter
Filter size: 3x3, 5x5, or 1x1 are common
Stride values: 1 or 2 (possibly greater)
“parallelize” multiple filters then merge

36. CNNs: Max Pooling Example

37. CNN in Python/Keras (fragment)
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Flatten, Activation
 from keras.layers.convolutional import Conv2D, MaxPooling2D
 from keras.optimizers import Adadelta
 input_shape = (3, 32, 32)
 nb_classes = 10
 model = Sequential()
 model.add(Conv2D(32, (3, 3), padding='same’,
input_shape=input_shape))
 model.add(Activation('relu'))
 model.add(Conv2D(32, (3, 3)))
 model.add(Activation('relu'))
 model.add(MaxPooling2D(pool_size=(2, 2)))
 model.add(Dropout(0.25))

38. What is TensorFlow?
An open source framework for ML and DL
A “computation” graph
Created by Google (released 11/2015)
Evolved from Google Brain
Linux and Mac OS X support (VM for Windows)
TF home page: https://www.tensorflow.org/

39. What is TensorFlow?
Support for Python, Java, C++
Desktop, server, mobile device (TensorFlow Lite)
CPU/GPU/TPU support
Visualization via TensorBoard
Can be embedded in Python scripts
Installation: pip install tensorflow
TensorFlow cluster:
https://www.tensorflow.org/deploy/distributed

40. TensorFlow Use Cases (Generic)
Image recognition
Computer vision
Voice/sound recognition
Time series analysis
Language detection
Language translation
Text-based processing
Handwriting Recognition

41. What is TensorFlow?
Graph: graph of operations (DAG)
Sessions: contains Graph(s)
lazy execution (default)
operations in parallel (default)
Nodes: operators/variables/constants
Edges: tensors
=> graphs are split into subgraphs and
executed in parallel (or multiple CPUs)

42. TensorFlow Graph Execution
Execute statements in a tf.Session() object
Invoke the “run” method of that object
“eager” execution is now possible
Not part of the mainline yet
Installation: pip install tf-nightly

43. What is a Tensor?
TF tensors are n-dimensional arrays
TF tensors are very similar to numpy ndarrays
scalar number: a zeroth-order tensor
vector: a first-order tensor
matrix: a second-order tensor
3-dimensional array: a 3rd order tensor
https://dzone.com/articles/tensorflow-simplified-
examples

44. TensorFlow “primitive types”
 tf.constant: initialized immediately
 tf.placeholder (a function):
+ initial value is not required
+ assigned value via feed_dict at run time
+ are not modified during training
 tf.Variable (a class):
+ initial value is required
+ updated during training
+ in-memory buffer (saved/restored from disk)
+ can be shared between works (distributed env)

45. TensorFlow: constants (immutable)
 import tensorflow as tf # tf-const.py
 aconst = tf.constant(3.0)
 print(aconst)
# output: Tensor("Const:0", shape=(), dtype=float32)
 sess = tf.Session()
 print(sess.run(aconst))
# output: 3.0
 sess.close()
 # => there's a better way…

46. TensorFlow: constants
import tensorflow as tf # tf-const2.py
aconst = tf.constant(3.0)
print(aconst)
Automatically close “sess”
with tf.Session() as sess:
 print(sess.run(aconst))

47. TensorFlow Arithmetic
import tensorflow as tf # basic1.py
a = tf.add(4, 2)
b = tf.subtract(8, 6)
c = tf.multiply(a, 3)
d = tf.div(a, 6)
with tf.Session() as sess:
print(sess.run(a)) # 6
print(sess.run(b)) # 2
print(sess.run(c)) # 18
print(sess.run(d)) # 1

48. TensorFlow Arithmetic Methods
import tensorflow as tf #tf-math-ops.py
PI = 3.141592
sess = tf.Session()
print(sess.run(tf.div(12,8)))
print(sess.run(tf.floordiv(20.0,8.0)))
print(sess.run(tf.sin(PI)))
print(sess.run(tf.cos(PI)))
print(sess.run(tf.div(tf.sin(PI/4.), tf.cos(PI/4.))))

49. TensorFlow Arithmetic Methods
Output from tf-math-ops.py:
1
2.0
6.27833e-07
-1.0
1.0

50. TF placeholders and feed_dict
import tensorflow as tf # tf-var-multiply.py
a = tf.placeholder("float")
b = tf.placeholder("float")
c = tf.multiply(a,b)
# initialize a and b:
feed_dict = {a:2, b:3}
# multiply a and b:
with tf.Session() as sess:
print(sess.run(c, feed_dict))

51. Deep Learning JavaScript Toolkits
Tensorfire ( https://tenso.rs )
deeplearn.js ( https://deeplearnjs.org )
 keras-js ( https://github.com/transcranial/keras-js )
MXNetJS ( https://github.com/dmlc/mxnet.js )
ML.js ( https://github.com/mljs/ml )
Jeeliz ( https://jeeliz.com )

52. What is TensorFire?
uses WebGL to run NNs in browsers
TensorFire has two parts:
1) a GLSL-based language to write massively parallel
WebGL shaders that operate on 4D tensors
2) a high-level library for importing models trained with
Keras or TensorFlow

53. What is TensorFire?
imitates the structure of TensorFlow & numpy
uses WebGL JavaScript API for 2D/3D graphics
can also perform computations on the GPU
hence it offers significant performance

54. What is TensorFire?
it supports low-precision quantized tensors
supports browsers that don't fully support the
OES_texture_float extension
https://petewarden.com/2015/05/23/why-are-eight-
bits-enough-for-deep-neural-networks

55. What is TensorFire?
works on any GPU (with/without CUDA)
on computers with AMD graphics
the latest 2016 Retina Macbook Pro
https://github.com/tensorfire

56. What is TensorFire?
the Tensorfire demo app:
1) runs the style-transfer neural network
2) renders in your browser
3) as fast as CPU TensorFlow on a desktop
https://tenso.rs/demos/fast-neural-style/
https://tenso.rs/demos/rock-paper-scissors/

57. TensorFire Code Sample
Go to ‘demos’ directory
Go to backprop subdirectory
Open backprop.html
https://github.com/tensorfire/tensorfire/blob/master/demos/bac
kprop/backprop.html

58. What is deeplearn.js?
an open-source library
developed by the Google Brain PAIR team
API closely mirrors TensorFlow eager API
can be used with TypeScript (preferred)
can be used with plain JavaScript

59. What is deeplearn.js?
train NNs in a browser OR
run pre-trained models (inference mode)
A hardware-accelerated machine intelligence
library for the web

60. What is deeplearn.js?
install via yarn/npm:
yarn add deeplearn or npm install deeplearn
available as a global variable named dl
Use a version (e.g. 0.5.0) instead of @latest:
<script src="https://cdn.jsdelivr.net/npm/deeplearn@0.5.0">
</script>
OR
<script src="https://unpkg.com/deeplearn@latest"></script>

61. What is deeplearn.js?
 import * as dl from 'deeplearn';
 const a = dl.tensor1d([1, 2, 3]);
 const b = dl.scalar(2);
 const result = a.add(b);
 //Float(32Array([3,4,5])
 result.data().then(data => console.log(data));
 // Alternatively can use a blocking call:
 //Float(32Array([3,4,5])
 console.log(result.dataSync());

62. deeplearn.js Code Sample
Go to ‘demos’ directory
Go to mnist_eager subdirectory
Open model.ts
https://github.com/PAIR-
code/deeplearnjs/blob/master/demos/mnist_eager/model.ts

63. deeplearn.js Useful Links
port TensorFlow models to deeplearn.js:
https://deeplearnjs.org/demos/mnist/mnist.html
API reference:
https://deeplearnjs.org/docs/api/index.html
Other samples:
https://deeplearnjs.org/index.html#demos

64. Deep Learning, JS, and GPU Support
Matrix comparison of JS toolkits:
 https://towardsdatascience.com/gpu-accelerated-neural-
networks-in-javascript-195d6f8e69ef

65. About Me: Recent Books
1) HTML5 Canvas and CSS3 Graphics (2013)
2) jQuery, CSS3, and HTML5 for Mobile (2013)
3) HTML5 Pocket Primer (2013)
4) jQuery Pocket Primer (2013)
5) HTML5 Mobile Pocket Primer (2014)
6) D3 Pocket Primer (2015)
7) Python Pocket Primer (2015)
8) SVG Pocket Primer (2016)
9) CSS3 Pocket Primer (2016)
10) Android Pocket Primer (2017)
11) Angular Pocket Primer (2017)
12) Data Cleaning Pocket Primer (2018)
13) RegEx Pocket Primer (2018)

66. About Me: Training
=> Deep Learning. Keras, and TensorFlow:
http://codeavision.io/training/deep-learning-workshop
=> Instructor at UCSC (May/2018):
Deep Learning with TensorFlow
=> Mobile and TensorFlow Lite (WIP)
=> R and Deep Learning (WIP)
=> Android for Beginners