1.
Deep Learning with TensorFlow
Rajat Monga
Engineering Director, TensorFlow

2.
How complex are
Deep Learning Systems?

3.
Neural networks

4.
Inception (2015)

5.
That’s not all...
Input
Parameters
Prediction

6.
That’s not all...
Input
Parameters
Labels
Prediction Loss
Update

7.
Parameters
Labels
Prediction Loss
Update
Input
batch
...
That’s not all...

8.
That’s not all...
Input
batch
...
Parameters
Labels
Prediction Loss
Update
Parameter Server

9.
That’s not all...
Input
batch
...
Parameters Update
Model
Worker

10.
UpdateUpdateUpdate
Model
That’s not all...
Input
batch
...
Parameters Update
Model
Model
Model
...

11.
Raspberry
Pi
Android iOS
Portability is a requirement
TPU
GPU
CPU

12.
machine learning gets complex quickly
Modeling complexityHeterogenous
System
Distributed
System

13.
repository
for “machine learning”
category on GitHub
#1

14.
12,000+ commits since Nov, 2015
570+ contributors
1M+ binary downloads
5000+ TensorFlow related repositories on GitHub
#15 most popular repository on GitHub by stars - across all categories
Used in ML classes at many universities:
Toronto, Berkeley, Stanford, ...
Some Stats

15.
TensorFlow powered Cucumber Sorter
From: http://workpiles.com/2016/02/tensorflow-cnn-cucumber/

16.
machine learning gets complex quickly
Modeling complexityHeterogenous
System
Distributed
System

17.
Platforms

18.
Languages

19.
TensorFlow Architecture
CPU GPU Android iOS ...
C++ front end Python front end
...
Kernels
Bindings + Compound Ops
add mul print reshape ...
Core TensorFlow Execution System
Ops

20.
Ecosystem

21.
machine learning gets complex quickly
Modeling complexityHeterogenous
System
Distributed
System

22.
Parallelism in Op implementations
MatMul
...
...

23.
...
...
...
...
...MatMul
MatMul
Task Parallelism in DataFlow graph

24.
Data Parallelism
MatMul
Input
MatMulParam
...
...MatMul
MatMul
...
...

25.
Model Parallelism
MatMul
Matrix
Matrix
MatMul
Split Concat

26.
GPU 0 CPU
Add Mul
biases
learning rate
−=...
...
Distribution across Devices

27.
GPU 0 CPU
Add Mul
biases
learning rate
−=...
...
Distribution
● TensorFlow inserts Send/Recv Ops to transport tensors across devices
● Recv ops pull data from Send ops
Send Recv

28.
GPU 0 CPU
Add Mul
biases
learning rate
−=...
...
Distribution
● TensorFlow inserts Send/Recv Ops to transport tensors across devices
● Recv ops pull data from Send ops
Send Recv
Send Recv
Send
Recv Send
Recv

29.
Just-In-Time Compilation
0x00000000 movq (%rdx), %rax
0x00000003 vmovaps (%rax), %xmm0
0x00000007 vmulps %xmm0, %xmm0, %xmm0
0x0000000b vmovaps %xmm0, (%rdi)
...
TF graphs go in,
Optimized & specialized
assembly comes out.
XLA: “Accelerated Linear Algebra” Compiler

30.
machine learning gets complex quickly
Modeling complexityHeterogenous
System
Distributed
System

31.
MatMul
Add Relu
biases
weights
input
input = ...
biases = tf.get_variable(’biases’, ...)
weights = tf.get_variable(’weights’, ...)
out = tf.matmul(input, weights)
out = tf.add(out, biases)
out = tf.nn.relu(out)

32.
MatMul
Add Relu
biases
weights
input
input = ...
output = tf.contrib.layers.fully_connected(input, ...)

33.
MatMul
Add Relu
biases
weights
input
input = ...
output = tf.contrib.layers.fully_connected(input, ...)

34.
Align cognitive model with programming model
Each box is one line of code!

35.
TensorFlow contains complete algorithms
Linear{Classifier,Regressor}
DNN{Classifier,Regressor}
DNNLinearCombined{Classifier,Regressor}
SVM
KMeansClustering
GMM
...

36.
Simple machine learning
classifier = learn.LinearClassifier(feature_columns=feature_columns,
n_classes=10)
classifier.fit(data, labels, batch_size=100, steps=1000)
classifier.evaluate(eval_data, eval_labels)
Tooling provided for distributed training and evaluation, graphical debugging, and
export to production server (tensorflow/serving).

37.
UpdateUpdateUpdate
Model
Remember this?
Input
batch
...
Parameters Update
Model
Model
Model
...

38.
TensorFlow handles complexity for you...
...so you can focus on your ideas

39.
Rajat Monga
@rajatmonga
Thank You!