SlideShare a Scribd company logo
1 of 39
Download to read offline
@nfmcclure
Introduction to Neural
Networks with Tensorflow
Nick McClure
July 27th, 2016
Seattle, WA
@nfmcclure
Who am I?
• ddd
www.github.com/nfmcclure/tensorflow_cookbook
@nfmcclure
Why Does Any of This Matter?
8
8
19 152
@nfmcclure
Why Neural Networks?
• The human brain can be considered
to be one of the best processors.
(Estimated to contain ~1011
neurons.)
• Studies show that our brain can
process the equivalent of
~20Mb/sec just through the optical
nerves.
• If we can copy this design, maybe
we can solve the “hard for a
computer – easy for humans”
problems.
• Speech recognition
• Facial identification
• Reading emotions
• Recognizing images
• Sentiment analysis
• Driving a vehicle
• Disease diagnosis
@nfmcclure
The Basic Unit: Operational Gate
• F(x,y) = x * y
• How can we change the inputs to decrease the output?
• The intuitive answer is to decrease 3 and/or decrease 4.
• Mathematically, the answer is to use the derivative…
• We know if we increase the (3) input by 1, the output goes up
by a total of 4. (4*4=16)
*
3
4
12
@nfmcclure
Multiple Gates
• F(x, y, z) = (x + y) * z
• If G(a, b) = a + b, then F(x,y,z) = G(x, y) * z
• Or (change notation) F = G*z (we have solved this problem
before)
• Mathematically, we will use the “chain rule” to get the
derivatives:
+
2
3
20
4
*
𝑑𝐹
𝑑𝑥
=
𝑑𝐹
𝑑𝐺
∙
𝑑𝐺
𝑑𝑥
𝑑𝐹
𝑑𝑥
= 𝑧 ∙ 1
@nfmcclure
Loss Functions
• You can’t learn unless you know how bad/good your past
results have been.
• This is designated by a ‘Loss function’.
• The idea is we have a labeled data set and we look at each
data point. We run the data point forward through the
network and then see if we need to increase or decrease
the output.
• We then change the parameters according to the derivatives
in the network.
• We loop through this many times until we reach the lowest
error in our training.
@nfmcclure
Learning Rate
• The learning rate controls how much of a change we make
to our model parameters.
@nfmcclure
Logistic Regression as a Neural
Network
• A neural network is very similar to what we have been
doing, except that we bound the outputs between 0 and 1…
with a sigmoid function. (Called an activation function)
• The sigmoid function is nice, because it turns out that the
derivative is related to itself:
• We can use this fact in our ‘chain rule’ for derivatives.
𝐹(𝑥, 𝑎, 𝑏) = 𝜎(𝑎𝑥 + 𝑏) σ(𝑥) =
1
1 + 𝑒−𝑥
𝑑𝜎
𝑑𝑥
= 𝜎(𝑥) ∙ (1 − 𝜎 𝑥 )
x
a
*
b
+ Output
x
1
Output
a
b
OR
https://github.com/nfmcclure/tensorflow_cookbook/tree/master/03_Linear_Regression/08_Implementing_Logistic_Regression
@nfmcclure
Why Activation Functions?
• Turns out, no matter how many additions and multiplications we pile
on, we will never be able to model non-linear outputs without having
non-linear transformations.
• The sigmoid function is just one example of a gate function.
• There is also the Rectified Linear Unit (ReLU):
• Softplus:
• Leaky ReLU:
• Where 0<a<<1
σ(𝑥) =
1
1 + 𝑒−𝑥
σ(𝑥) = max(𝑥, 0)
σ(𝑥) = ln(1 + 𝑒 𝑥
)
σ(𝑥) = max(𝑥, 𝑎𝑥)
@nfmcclure
Many Layers
• Neural networks can have many ‘hidden layers’.
• We can make them as deep as we want.
• Neural Networks can also have as many inputs or
outputs as we would like as well.
Depth of Neural Network
Fully Connected Layers
@nfmcclure
What is Tensorflow?
• An open source framework for creating
computational graphs.
• Computational graphs are actually extremely
flexible.
• Linear Regression (regular, multiple, logistic, lasso, ridge,
elasticnet,…)
• Support Vector Machines (any kernel)
• Nearest Neighbor methods
• Neural Networks
• ODE/PDE Solvers
@nfmcclure
Why Tensorflow?
• Compare to other open source deep learning
frameworks
Popularity
Speed/Size
Caffe
Others
@nfmcclure
Why Tensorflow?
• Tensorflow is also very portable and flexible!
• Virtually no code change to go from CPU-> GPU or…
@nfmcclure
How is the Tensorflow Community?
• Very active github community
• In approximately 6 months: >5k commits, >250 contributors
• Over 2,000 questions tagged on stackoverflow (>50%
answered within 24 hours)
@nfmcclure
How do Tensorflow Algorithms
Work?
Computational
Graph
Model
Variables
Output
Loss Function
@nfmcclure
A One Hidden Layer NN in Tensorflow
• Iris dataset: Predict Petal width from (Sepal length, Sepal
width, and petal length)
S.L.
S.W.
P.L.
bias
h1
h2
h3
h4
h5
Output
bias
15 weights + 5 biases 5 weights + 1 bias = 26 variables
Computational Graph
Placeholders
https://github.com/nfmcclure/tensorflow_cookbook/tree/master/06_Neural_Networks/04_Single_Hidden_Layer_Network
@nfmcclure
More in Tensorflow: Distributed
• Computational graphs can take very long to train.
• Quicker training across multiple machines.
• Just need a dictionary mapping jobs to network addresses:
job_spec = {“my_workers”: [“worker1.location.com:22”, “worker2.location.com:22”],
“other_workers”: [“gpu_worker.location.com:22”]}
tf.train.ClusterSpec(job_spec)
with tf.device(“/job:my_workers/task:0”):
# do task
with tf.device(“/job:other_workers/task:0”):
# do another task
• Scheduling and communication is done for you (in an
efficient and greedy manner)
@nfmcclure
More in Tensorflow: GPU Capabilities
• Sometimes neural networks can have hundreds of millions
of parameters to train.
• Graphic cards (GPUs) are built to handle matrix calculations
very fast and efficiently.
• It’s quite common to get 10x to 50x speedup when switching the
same code to a GPU
(1) Install GPU version of T.F.
(2) Done.
As a side note, you can also tell T.F. to use
specific devices.
/cpu:0
/gpu:0
/gpu:100
- The CPU of the machine.
- The GPU of the machine.
- The 100th GPU of the machine.
@nfmcclure
More in Tensorflow: Tensorboard
• A built in way to visualize the computational graph,
accuracies, loss functions, gradients…
• (1) Specify a specific way to write and store summaries to a
log file.
• (2) Point Tensorboard at logging directory, and navigate to
0.0.0.0:6006. (Can view during training).
tf.histogram_summary(‘variable_histogram’, variable_value)
tf.scalar_summary(‘loss_value’, loss)
merged = tf.merge_all_summaries()
my_writer = tf.train.SummaryWriter('my_logging_directory', sess.graph)
$tensorboard –logdir=my_logging_directory
@nfmcclure
More in Tensorflow: Tensorboard
Collapsable/Expandable Named Scopes!!
with tf.name_scope(‘hidden1’) as hidden_scope:
#Perform calculations.
@nfmcclure
More in Tensorflow: skflow
• Easier scikit learn type implementation of common
algorithms.
• 3 layer NN:
• RNN:
• Easy to add custom layers by adding in layer functions from
vanilla Tensorflow.
import tensorflow.contrib.learn as skflow
my_model = skflow.TensorFlowDNNClassifier(hidden_units=[N1, N2, N3], n_classes=n_out)
my_model.fit(x_data, y_target)
my_rnn = skflow.TensorFlowRNNClassifier(…)
@nfmcclure
What is a Convolutional Neural
Network?
• A CNN us a large pattern recognition neural
network that uses various parameter reducing
techniques.
• E.g. Alexnet (2012 ImageNet Winner)
@nfmcclure
Reduction of Parameters
• CNNs use tricks to reduce the amount of
parameters to fit.
• Convolution layers: Assume weights are the same on
arrows in the same direction of a window we move
across data.
w1
w2
w3
There are only three
weights in this layer, w1,
w2, and w3
Note: Usually multiple convolutions are performed, creating ‘Layers’.
@nfmcclure
Other Tricks- Pooling
• Layer that is an aggregation across a window from
prior layer.
• Types of pooling:
• Max, min, mean, median, …
Aggregation over
three nodes in
prior layer. E.g.
Max().
@nfmcclure
Other Tricks- Dropout
• Dropout is a way to increase the training strength and regularize
the parameters.
• During training, we randomly select activation functions and
make the output equal to zero.
• Compare this to ‘zoneout’, which randomly replaced activation
functions with prior iteration value.
• This helps the network find multiple pathways for prediction.
Because of this, the network will not rely on just one connection
or edge for prediction.
• This also helps the training of other parameters in the network.
@nfmcclure
Larger Network Shorthand
• Since neural networks can get quite large, we do
not want to write out all the connections nor
nodes.
• Here’s how to convey the meaning in a shorter, less
complicated way:
“Max of a set of features”
E.g., Here, it appears to be
a max of 2 features.
@nfmcclure
What is a Recurrent Neural Network?
• Recurrent Neural Networks (RNN) are networks that can
‘see’ outputs from prior layers. Very helpful for sequences.
• The most common usage is to predict the next letter from
the prior letter. We train this network on large text samples.
U,V,W = Weight Parameters
S = layer
X = input
O = output
@nfmcclure
Regional Convolutional Networks
• We can convolve a whole network over patches of the
image and see what regions score highly for a category.
(Object Detection)
https://www.youtube.com/watch?v=lr1qVxhGUDw
@nfmcclure
Regional CNN + Recurrent NN =
Image Captioning
• We take the object detection output and feed it through a
recurrent neural network to generate text describing the
image.
Microsoft Research 2015
@nfmcclure
Deep Dream (2015)
• We can select specific nodes in an image
recognition pipeline and up weight the output…
https://www.youtube.com/watch?v=DgPaCWJL7XI
http://www.fastcodesign.com/3057368/inside-googles-first-deepdream-art-show
@nfmcclure
Stylenet (2015)
• Train network on a “style image”. Try to reconstruct
another image from that same pretrained network.
https://vimeo.com/139123754
Recoloring of black and white photos
@nfmcclure
CNN for Text (2015)
• Zhang, X., et. al. released a paper (http://arxiv.org/abs/1509.01626) that
showed great results for treating strings as 1-D numerical vectors.
• Ways to get strings into numerical vectors:
• Word level one-hot encoding.
• Word2Vec encoding or similar (GloVe or Doc2Vec or …)
• Character level one-hot encoding.
• Can’t resize vectors like pictures, so instead we pick a max length and
pad many entries with zeros.
• Like many NN results, depend heavily on dataset and architecture.
• Paper shows good results for low # of classes (ratings or sentiment).
@nfmcclure
Parsey McParseface
• Google released a model called ‘Syntaxnet’ in May 2016,
based on Tensorflow.
• Syntaxnet is arguably one of the best part-of-speech parsers
available.
• They trained an English version, called ‘Parsey McParseface’.
• This is a free and open source model to use.
@nfmcclure
The Possibilities are Endless
• Predict age from facial photo - https://how-old.net
• Predict dog breed from photo - https://www.what-dog.net
• Inside a self driving car brain :
https://www.youtube.com/watch?v=ZJMtDRbqH40
• Memory networks: https://arxiv.org/pdf/1410.3916.pdf
• Frodo journeyed to Mount-Doom. Frodo dropped the ring there. Sauron died.
• Frodo went back to the Shire. Bilbo travelled to the Grey-havens.
• Where is the ring? A: Mount-Doom
• Where is Bilbo now? A: Grey-havens
• “Synthia”- Virtual driving school for self driving cars:
• http://www.gizmag.com/synthia-dataset-self-driving-cars/43895/
• Solving hand-written chalk board problems in real time:
• http://www.willforfang.com/computer-vision/2016/4/9/artificial-intelligence-for-
handwritten-mathematical-expression-evaluation
@nfmcclure
<Insert Name Here>-Neural Net
• Last Month (June-July 2016):
• YodaNN: http://arxiv.org/abs/1606.05487
• CMS-RCNN: http://arxiv.org/abs/1606.05413
• Zoneout RNN: https://arxiv.org/abs/1606.01305
• Pixel CNN: http://arxiv.org/abs/1606.05328
• Memory CNN: http://arxiv.org/abs/1606.05262
• DISCO Nets: http://arxiv.org/abs/1606.02556
• This Year (2016):
• Stochastic Depth Net: https://arxiv.org/abs/1603.09382
• Squeeze Net: https://arxiv.org/abs/1602.07360
• Binary Nets: http://arxiv.org/abs/1602.02830
• Last Year:
• PoseNet: http://arxiv.org/abs/1505.07427
• YOLO Nets: http://arxiv.org/abs/1506.02640
• Style Net: http://arxiv.org/abs/1508.06576
• Residual Net: https://arxiv.org/abs/1512.03385
• And so many more…
• What’s Your Architecture?
@nfmcclure
How to Keep ML Current
• Machine learning advances happen more and more
frequently.
• Can’t rely on journals that take multiple years to publish.
• Twitter
• http://arxiv.org/
• Others?
@nfmcclure
More Resources
• More on CNNs:
• https://www.reddit.com/r/MachineLearning/
• https://www.reddit.com/r/deepdream/
• http://karpathy.github.io/
• RNNs:
• http://colah.github.io/posts/2015-08-Understanding-LSTMs/
• Recent Research: (Jan 16-June 16)
• Residual Conv. Nets: https://www.youtube.com/watch?v=1PGLj-
uKT1w
• Determine location of a random image:
https://www.technologyreview.com/s/600889/google-unveils-
neural-network-with-superhuman-ability-to-determine-the-
location-of-almost/
• Stochastic Depth Networks: http://arxiv.org/abs/1603.09382
Pre-trained models: http://www.vlfeat.org/matconvnet/pretrained/
@nfmcclure
Questions?
• Slides:
• http://www.slideshare.net/NicholasMcClure1/introduction-to-
neural-networks-in-tensorflow
• Code and more:
• https://github.com/nfmcclure/tensorflow_cookbook

More Related Content

What's hot

Introduction to TensorFlow Lite
Introduction to TensorFlow Lite Introduction to TensorFlow Lite
Introduction to TensorFlow Lite Koan-Sin Tan
 
Tensorflow presentation
Tensorflow presentationTensorflow presentation
Tensorflow presentationAhmed rebai
 
What is TensorFlow? | Introduction to TensorFlow | TensorFlow Tutorial For Be...
What is TensorFlow? | Introduction to TensorFlow | TensorFlow Tutorial For Be...What is TensorFlow? | Introduction to TensorFlow | TensorFlow Tutorial For Be...
What is TensorFlow? | Introduction to TensorFlow | TensorFlow Tutorial For Be...Simplilearn
 
ChatGPT.pdf
ChatGPT.pdfChatGPT.pdf
ChatGPT.pdfdhatura
 
Introduction to Machine Learning with TensorFlow
Introduction to Machine Learning with TensorFlowIntroduction to Machine Learning with TensorFlow
Introduction to Machine Learning with TensorFlowPaolo Tomeo
 
Introduction to Deep learning
Introduction to Deep learningIntroduction to Deep learning
Introduction to Deep learningleopauly
 
Tensorflow - Intro (2017)
Tensorflow - Intro (2017)Tensorflow - Intro (2017)
Tensorflow - Intro (2017)Alessio Tonioni
 
Deep Learning: Application & Opportunity
Deep Learning: Application & OpportunityDeep Learning: Application & Opportunity
Deep Learning: Application & OpportunityiTrain
 
Deep Learning (DL) from Scratch
Deep Learning (DL) from ScratchDeep Learning (DL) from Scratch
Deep Learning (DL) from ScratchAziz416788
 
Neural networks and deep learning
Neural networks and deep learningNeural networks and deep learning
Neural networks and deep learningJörgen Sandig
 
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...Simplilearn
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural NetworksDatabricks
 
6 games
6 games6 games
6 gamesMhd Sb
 
LangChain Intro by KeyMate.AI
LangChain Intro by KeyMate.AILangChain Intro by KeyMate.AI
LangChain Intro by KeyMate.AIOzgurOscarOzkan
 
A comprehensive guide to prompt engineering.pdf
A comprehensive guide to prompt engineering.pdfA comprehensive guide to prompt engineering.pdf
A comprehensive guide to prompt engineering.pdfAnastasiaSteele10
 
Generative AI and ChatGPT - Scope of AI and advance Generative AI
Generative AI and ChatGPT - Scope of AI and advance Generative AIGenerative AI and ChatGPT - Scope of AI and advance Generative AI
Generative AI and ChatGPT - Scope of AI and advance Generative AIKumaresan K
 

What's hot (20)

Introduction to TensorFlow Lite
Introduction to TensorFlow Lite Introduction to TensorFlow Lite
Introduction to TensorFlow Lite
 
Tensorflow presentation
Tensorflow presentationTensorflow presentation
Tensorflow presentation
 
What is TensorFlow? | Introduction to TensorFlow | TensorFlow Tutorial For Be...
What is TensorFlow? | Introduction to TensorFlow | TensorFlow Tutorial For Be...What is TensorFlow? | Introduction to TensorFlow | TensorFlow Tutorial For Be...
What is TensorFlow? | Introduction to TensorFlow | TensorFlow Tutorial For Be...
 
ChatGPT.pdf
ChatGPT.pdfChatGPT.pdf
ChatGPT.pdf
 
Machine learning workshop
Machine learning workshopMachine learning workshop
Machine learning workshop
 
Introduction to Machine Learning with TensorFlow
Introduction to Machine Learning with TensorFlowIntroduction to Machine Learning with TensorFlow
Introduction to Machine Learning with TensorFlow
 
Introduction to Deep Learning
Introduction to Deep Learning Introduction to Deep Learning
Introduction to Deep Learning
 
Introduction to Deep learning
Introduction to Deep learningIntroduction to Deep learning
Introduction to Deep learning
 
Tensorflow - Intro (2017)
Tensorflow - Intro (2017)Tensorflow - Intro (2017)
Tensorflow - Intro (2017)
 
Deep learning
Deep learningDeep learning
Deep learning
 
Deep Learning: Application & Opportunity
Deep Learning: Application & OpportunityDeep Learning: Application & Opportunity
Deep Learning: Application & Opportunity
 
Deep Learning (DL) from Scratch
Deep Learning (DL) from ScratchDeep Learning (DL) from Scratch
Deep Learning (DL) from Scratch
 
Neural networks and deep learning
Neural networks and deep learningNeural networks and deep learning
Neural networks and deep learning
 
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
 
Training Neural Networks
Training Neural NetworksTraining Neural Networks
Training Neural Networks
 
6 games
6 games6 games
6 games
 
LangChain Intro by KeyMate.AI
LangChain Intro by KeyMate.AILangChain Intro by KeyMate.AI
LangChain Intro by KeyMate.AI
 
A comprehensive guide to prompt engineering.pdf
A comprehensive guide to prompt engineering.pdfA comprehensive guide to prompt engineering.pdf
A comprehensive guide to prompt engineering.pdf
 
Generative AI and ChatGPT - Scope of AI and advance Generative AI
Generative AI and ChatGPT - Scope of AI and advance Generative AIGenerative AI and ChatGPT - Scope of AI and advance Generative AI
Generative AI and ChatGPT - Scope of AI and advance Generative AI
 
Deep learning presentation
Deep learning presentationDeep learning presentation
Deep learning presentation
 

Viewers also liked

Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...
Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...
Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...Altoros
 
Applying Transfer Learning in TensorFlow
Applying Transfer Learning in TensorFlowApplying Transfer Learning in TensorFlow
Applying Transfer Learning in TensorFlowScott Thompson
 
Introduction to Deep Learning with TensorFlow
Introduction to Deep Learning with TensorFlowIntroduction to Deep Learning with TensorFlow
Introduction to Deep Learning with TensorFlowTerry Taewoong Um
 
SocialHRCamp Omaha 2016 Kick Off Presentation - Tech Adoption, Scalability, A...
SocialHRCamp Omaha 2016 Kick Off Presentation - Tech Adoption, Scalability, A...SocialHRCamp Omaha 2016 Kick Off Presentation - Tech Adoption, Scalability, A...
SocialHRCamp Omaha 2016 Kick Off Presentation - Tech Adoption, Scalability, A...SocialHRCamp
 
Multithreading to Construct Neural Networks
Multithreading to Construct Neural NetworksMultithreading to Construct Neural Networks
Multithreading to Construct Neural NetworksAltoros
 
Neuron Mc Culloch Pitts dan Hebb
Neuron Mc Culloch Pitts dan HebbNeuron Mc Culloch Pitts dan Hebb
Neuron Mc Culloch Pitts dan HebbSherly Uda
 
Introduction to Neural networks (under graduate course) Lecture 9 of 9
Introduction to Neural networks (under graduate course) Lecture 9 of 9Introduction to Neural networks (under graduate course) Lecture 9 of 9
Introduction to Neural networks (under graduate course) Lecture 9 of 9Randa Elanwar
 
Introduction to Neural networks (under graduate course) Lecture 2 of 9
Introduction to Neural networks (under graduate course) Lecture 2 of 9Introduction to Neural networks (under graduate course) Lecture 2 of 9
Introduction to Neural networks (under graduate course) Lecture 2 of 9Randa Elanwar
 
A Guided Tour of Machine Learning for Traders by Tucker Balch at QuantCon 2016
A Guided Tour of Machine Learning for Traders by Tucker Balch at QuantCon 2016A Guided Tour of Machine Learning for Traders by Tucker Balch at QuantCon 2016
A Guided Tour of Machine Learning for Traders by Tucker Balch at QuantCon 2016Quantopian
 
Introduction to Neural networks (under graduate course) Lecture 1 of 9
Introduction to Neural networks (under graduate course) Lecture 1 of 9Introduction to Neural networks (under graduate course) Lecture 1 of 9
Introduction to Neural networks (under graduate course) Lecture 1 of 9Randa Elanwar
 
An Introduction to Neural Networks and Machine Learning
An Introduction to Neural Networks and Machine LearningAn Introduction to Neural Networks and Machine Learning
An Introduction to Neural Networks and Machine LearningChris Nicholls
 
Re: 제로부터시작하는텐서플로우
Re: 제로부터시작하는텐서플로우Re: 제로부터시작하는텐서플로우
Re: 제로부터시작하는텐서플로우Mario Cho
 
Teaching Recurrent Neural Networks using Tensorflow (May 2016)
Teaching Recurrent Neural Networks using Tensorflow (May 2016)Teaching Recurrent Neural Networks using Tensorflow (May 2016)
Teaching Recurrent Neural Networks using Tensorflow (May 2016)Rajiv Shah
 
Image Recognition with TensorFlow
Image Recognition with TensorFlowImage Recognition with TensorFlow
Image Recognition with TensorFlowAltoros
 
Proyecto final jonathan lara reyes
Proyecto final jonathan lara reyesProyecto final jonathan lara reyes
Proyecto final jonathan lara reyesJONLR
 
State of open research data open con
State of open research data   open conState of open research data   open con
State of open research data open conAmye Kenall
 

Viewers also liked (20)

Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...
Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...
Deep Learning with TensorFlow: Understanding Tensors, Computations Graphs, Im...
 
Google TensorFlow Tutorial
Google TensorFlow TutorialGoogle TensorFlow Tutorial
Google TensorFlow Tutorial
 
TensorFlow
TensorFlowTensorFlow
TensorFlow
 
Applying Transfer Learning in TensorFlow
Applying Transfer Learning in TensorFlowApplying Transfer Learning in TensorFlow
Applying Transfer Learning in TensorFlow
 
Introduction to Deep Learning with TensorFlow
Introduction to Deep Learning with TensorFlowIntroduction to Deep Learning with TensorFlow
Introduction to Deep Learning with TensorFlow
 
Data Science At Zillow
Data Science At ZillowData Science At Zillow
Data Science At Zillow
 
SocialHRCamp Omaha 2016 Kick Off Presentation - Tech Adoption, Scalability, A...
SocialHRCamp Omaha 2016 Kick Off Presentation - Tech Adoption, Scalability, A...SocialHRCamp Omaha 2016 Kick Off Presentation - Tech Adoption, Scalability, A...
SocialHRCamp Omaha 2016 Kick Off Presentation - Tech Adoption, Scalability, A...
 
Multithreading to Construct Neural Networks
Multithreading to Construct Neural NetworksMultithreading to Construct Neural Networks
Multithreading to Construct Neural Networks
 
Neuron Mc Culloch Pitts dan Hebb
Neuron Mc Culloch Pitts dan HebbNeuron Mc Culloch Pitts dan Hebb
Neuron Mc Culloch Pitts dan Hebb
 
Introduction to Neural networks (under graduate course) Lecture 9 of 9
Introduction to Neural networks (under graduate course) Lecture 9 of 9Introduction to Neural networks (under graduate course) Lecture 9 of 9
Introduction to Neural networks (under graduate course) Lecture 9 of 9
 
Introduction to Neural networks (under graduate course) Lecture 2 of 9
Introduction to Neural networks (under graduate course) Lecture 2 of 9Introduction to Neural networks (under graduate course) Lecture 2 of 9
Introduction to Neural networks (under graduate course) Lecture 2 of 9
 
Neural networks introduction
Neural networks introductionNeural networks introduction
Neural networks introduction
 
A Guided Tour of Machine Learning for Traders by Tucker Balch at QuantCon 2016
A Guided Tour of Machine Learning for Traders by Tucker Balch at QuantCon 2016A Guided Tour of Machine Learning for Traders by Tucker Balch at QuantCon 2016
A Guided Tour of Machine Learning for Traders by Tucker Balch at QuantCon 2016
 
Introduction to Neural networks (under graduate course) Lecture 1 of 9
Introduction to Neural networks (under graduate course) Lecture 1 of 9Introduction to Neural networks (under graduate course) Lecture 1 of 9
Introduction to Neural networks (under graduate course) Lecture 1 of 9
 
An Introduction to Neural Networks and Machine Learning
An Introduction to Neural Networks and Machine LearningAn Introduction to Neural Networks and Machine Learning
An Introduction to Neural Networks and Machine Learning
 
Re: 제로부터시작하는텐서플로우
Re: 제로부터시작하는텐서플로우Re: 제로부터시작하는텐서플로우
Re: 제로부터시작하는텐서플로우
 
Teaching Recurrent Neural Networks using Tensorflow (May 2016)
Teaching Recurrent Neural Networks using Tensorflow (May 2016)Teaching Recurrent Neural Networks using Tensorflow (May 2016)
Teaching Recurrent Neural Networks using Tensorflow (May 2016)
 
Image Recognition with TensorFlow
Image Recognition with TensorFlowImage Recognition with TensorFlow
Image Recognition with TensorFlow
 
Proyecto final jonathan lara reyes
Proyecto final jonathan lara reyesProyecto final jonathan lara reyes
Proyecto final jonathan lara reyes
 
State of open research data open con
State of open research data   open conState of open research data   open con
State of open research data open con
 

Similar to Introduction to Neural Networks in Tensorflow

D3, TypeScript, and Deep Learning
D3, TypeScript, and Deep LearningD3, TypeScript, and Deep Learning
D3, TypeScript, and Deep LearningOswald Campesato
 
Automatic Attendace using convolutional neural network Face Recognition
Automatic Attendace using convolutional neural network Face RecognitionAutomatic Attendace using convolutional neural network Face Recognition
Automatic Attendace using convolutional neural network Face Recognitionvatsal199567
 
D3, TypeScript, and Deep Learning
D3, TypeScript, and Deep LearningD3, TypeScript, and Deep Learning
D3, TypeScript, and Deep LearningOswald Campesato
 
NVIDIA 深度學習教育機構 (DLI): Image segmentation with tensorflow
NVIDIA 深度學習教育機構 (DLI): Image segmentation with tensorflowNVIDIA 深度學習教育機構 (DLI): Image segmentation with tensorflow
NVIDIA 深度學習教育機構 (DLI): Image segmentation with tensorflowNVIDIA Taiwan
 
Machine Learning, Deep Learning and Data Analysis Introduction
Machine Learning, Deep Learning and Data Analysis IntroductionMachine Learning, Deep Learning and Data Analysis Introduction
Machine Learning, Deep Learning and Data Analysis IntroductionTe-Yen Liu
 
Separating Hype from Reality in Deep Learning with Sameer Farooqui
 Separating Hype from Reality in Deep Learning with Sameer Farooqui Separating Hype from Reality in Deep Learning with Sameer Farooqui
Separating Hype from Reality in Deep Learning with Sameer FarooquiDatabricks
 
Java and Deep Learning (Introduction)
Java and Deep Learning (Introduction)Java and Deep Learning (Introduction)
Java and Deep Learning (Introduction)Oswald Campesato
 
TypeScript and Deep Learning
TypeScript and Deep LearningTypeScript and Deep Learning
TypeScript and Deep LearningOswald Campesato
 
Build a simple image recognition system with tensor flow
Build a simple image recognition system with tensor flowBuild a simple image recognition system with tensor flow
Build a simple image recognition system with tensor flowDebasisMohanty37
 
Language translation with Deep Learning (RNN) with TensorFlow
Language translation with Deep Learning (RNN) with TensorFlowLanguage translation with Deep Learning (RNN) with TensorFlow
Language translation with Deep Learning (RNN) with TensorFlowS N
 
V2.0 open power ai virtual university deep learning and ai introduction
V2.0 open power ai virtual university   deep learning and ai introductionV2.0 open power ai virtual university   deep learning and ai introduction
V2.0 open power ai virtual university deep learning and ai introductionGanesan Narayanasamy
 
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017StampedeCon
 
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in RUnderstanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in RManish Saraswat
 
Introduction To Tensorflow
Introduction To TensorflowIntroduction To Tensorflow
Introduction To TensorflowRayyan Khalid
 
deepnet-lourentzou.ppt
deepnet-lourentzou.pptdeepnet-lourentzou.ppt
deepnet-lourentzou.pptyang947066
 
Intro to Neural Networks
Intro to Neural NetworksIntro to Neural Networks
Intro to Neural NetworksDean Wyatte
 

Similar to Introduction to Neural Networks in Tensorflow (20)

D3, TypeScript, and Deep Learning
D3, TypeScript, and Deep LearningD3, TypeScript, and Deep Learning
D3, TypeScript, and Deep Learning
 
Automatic Attendace using convolutional neural network Face Recognition
Automatic Attendace using convolutional neural network Face RecognitionAutomatic Attendace using convolutional neural network Face Recognition
Automatic Attendace using convolutional neural network Face Recognition
 
D3, TypeScript, and Deep Learning
D3, TypeScript, and Deep LearningD3, TypeScript, and Deep Learning
D3, TypeScript, and Deep Learning
 
NVIDIA 深度學習教育機構 (DLI): Image segmentation with tensorflow
NVIDIA 深度學習教育機構 (DLI): Image segmentation with tensorflowNVIDIA 深度學習教育機構 (DLI): Image segmentation with tensorflow
NVIDIA 深度學習教育機構 (DLI): Image segmentation with tensorflow
 
Machine Learning, Deep Learning and Data Analysis Introduction
Machine Learning, Deep Learning and Data Analysis IntroductionMachine Learning, Deep Learning and Data Analysis Introduction
Machine Learning, Deep Learning and Data Analysis Introduction
 
Separating Hype from Reality in Deep Learning with Sameer Farooqui
 Separating Hype from Reality in Deep Learning with Sameer Farooqui Separating Hype from Reality in Deep Learning with Sameer Farooqui
Separating Hype from Reality in Deep Learning with Sameer Farooqui
 
Java and Deep Learning (Introduction)
Java and Deep Learning (Introduction)Java and Deep Learning (Introduction)
Java and Deep Learning (Introduction)
 
TypeScript and Deep Learning
TypeScript and Deep LearningTypeScript and Deep Learning
TypeScript and Deep Learning
 
Java and Deep Learning
Java and Deep LearningJava and Deep Learning
Java and Deep Learning
 
Build a simple image recognition system with tensor flow
Build a simple image recognition system with tensor flowBuild a simple image recognition system with tensor flow
Build a simple image recognition system with tensor flow
 
Language translation with Deep Learning (RNN) with TensorFlow
Language translation with Deep Learning (RNN) with TensorFlowLanguage translation with Deep Learning (RNN) with TensorFlow
Language translation with Deep Learning (RNN) with TensorFlow
 
Development of Deep Learning Architecture
Development of Deep Learning ArchitectureDevelopment of Deep Learning Architecture
Development of Deep Learning Architecture
 
V2.0 open power ai virtual university deep learning and ai introduction
V2.0 open power ai virtual university   deep learning and ai introductionV2.0 open power ai virtual university   deep learning and ai introduction
V2.0 open power ai virtual university deep learning and ai introduction
 
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
 
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in RUnderstanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
 
Deep learning
Deep learningDeep learning
Deep learning
 
Introduction To Tensorflow
Introduction To TensorflowIntroduction To Tensorflow
Introduction To Tensorflow
 
deepnet-lourentzou.ppt
deepnet-lourentzou.pptdeepnet-lourentzou.ppt
deepnet-lourentzou.ppt
 
MXNet Workshop
MXNet WorkshopMXNet Workshop
MXNet Workshop
 
Intro to Neural Networks
Intro to Neural NetworksIntro to Neural Networks
Intro to Neural Networks
 

Recently uploaded

ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimizationarrow10202532yuvraj
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideHironori Washizaki
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Alexander Turgeon
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 

Recently uploaded (20)

ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 

Introduction to Neural Networks in Tensorflow

  • 1. @nfmcclure Introduction to Neural Networks with Tensorflow Nick McClure July 27th, 2016 Seattle, WA
  • 2. @nfmcclure Who am I? • ddd www.github.com/nfmcclure/tensorflow_cookbook
  • 3. @nfmcclure Why Does Any of This Matter? 8 8 19 152
  • 4. @nfmcclure Why Neural Networks? • The human brain can be considered to be one of the best processors. (Estimated to contain ~1011 neurons.) • Studies show that our brain can process the equivalent of ~20Mb/sec just through the optical nerves. • If we can copy this design, maybe we can solve the “hard for a computer – easy for humans” problems. • Speech recognition • Facial identification • Reading emotions • Recognizing images • Sentiment analysis • Driving a vehicle • Disease diagnosis
  • 5. @nfmcclure The Basic Unit: Operational Gate • F(x,y) = x * y • How can we change the inputs to decrease the output? • The intuitive answer is to decrease 3 and/or decrease 4. • Mathematically, the answer is to use the derivative… • We know if we increase the (3) input by 1, the output goes up by a total of 4. (4*4=16) * 3 4 12
  • 6. @nfmcclure Multiple Gates • F(x, y, z) = (x + y) * z • If G(a, b) = a + b, then F(x,y,z) = G(x, y) * z • Or (change notation) F = G*z (we have solved this problem before) • Mathematically, we will use the “chain rule” to get the derivatives: + 2 3 20 4 * 𝑑𝐹 𝑑𝑥 = 𝑑𝐹 𝑑𝐺 ∙ 𝑑𝐺 𝑑𝑥 𝑑𝐹 𝑑𝑥 = 𝑧 ∙ 1
  • 7. @nfmcclure Loss Functions • You can’t learn unless you know how bad/good your past results have been. • This is designated by a ‘Loss function’. • The idea is we have a labeled data set and we look at each data point. We run the data point forward through the network and then see if we need to increase or decrease the output. • We then change the parameters according to the derivatives in the network. • We loop through this many times until we reach the lowest error in our training.
  • 8. @nfmcclure Learning Rate • The learning rate controls how much of a change we make to our model parameters.
  • 9. @nfmcclure Logistic Regression as a Neural Network • A neural network is very similar to what we have been doing, except that we bound the outputs between 0 and 1… with a sigmoid function. (Called an activation function) • The sigmoid function is nice, because it turns out that the derivative is related to itself: • We can use this fact in our ‘chain rule’ for derivatives. 𝐹(𝑥, 𝑎, 𝑏) = 𝜎(𝑎𝑥 + 𝑏) σ(𝑥) = 1 1 + 𝑒−𝑥 𝑑𝜎 𝑑𝑥 = 𝜎(𝑥) ∙ (1 − 𝜎 𝑥 ) x a * b + Output x 1 Output a b OR https://github.com/nfmcclure/tensorflow_cookbook/tree/master/03_Linear_Regression/08_Implementing_Logistic_Regression
  • 10. @nfmcclure Why Activation Functions? • Turns out, no matter how many additions and multiplications we pile on, we will never be able to model non-linear outputs without having non-linear transformations. • The sigmoid function is just one example of a gate function. • There is also the Rectified Linear Unit (ReLU): • Softplus: • Leaky ReLU: • Where 0<a<<1 σ(𝑥) = 1 1 + 𝑒−𝑥 σ(𝑥) = max(𝑥, 0) σ(𝑥) = ln(1 + 𝑒 𝑥 ) σ(𝑥) = max(𝑥, 𝑎𝑥)
  • 11. @nfmcclure Many Layers • Neural networks can have many ‘hidden layers’. • We can make them as deep as we want. • Neural Networks can also have as many inputs or outputs as we would like as well. Depth of Neural Network Fully Connected Layers
  • 12. @nfmcclure What is Tensorflow? • An open source framework for creating computational graphs. • Computational graphs are actually extremely flexible. • Linear Regression (regular, multiple, logistic, lasso, ridge, elasticnet,…) • Support Vector Machines (any kernel) • Nearest Neighbor methods • Neural Networks • ODE/PDE Solvers
  • 13. @nfmcclure Why Tensorflow? • Compare to other open source deep learning frameworks Popularity Speed/Size Caffe Others
  • 14. @nfmcclure Why Tensorflow? • Tensorflow is also very portable and flexible! • Virtually no code change to go from CPU-> GPU or…
  • 15. @nfmcclure How is the Tensorflow Community? • Very active github community • In approximately 6 months: >5k commits, >250 contributors • Over 2,000 questions tagged on stackoverflow (>50% answered within 24 hours)
  • 16. @nfmcclure How do Tensorflow Algorithms Work? Computational Graph Model Variables Output Loss Function
  • 17. @nfmcclure A One Hidden Layer NN in Tensorflow • Iris dataset: Predict Petal width from (Sepal length, Sepal width, and petal length) S.L. S.W. P.L. bias h1 h2 h3 h4 h5 Output bias 15 weights + 5 biases 5 weights + 1 bias = 26 variables Computational Graph Placeholders https://github.com/nfmcclure/tensorflow_cookbook/tree/master/06_Neural_Networks/04_Single_Hidden_Layer_Network
  • 18. @nfmcclure More in Tensorflow: Distributed • Computational graphs can take very long to train. • Quicker training across multiple machines. • Just need a dictionary mapping jobs to network addresses: job_spec = {“my_workers”: [“worker1.location.com:22”, “worker2.location.com:22”], “other_workers”: [“gpu_worker.location.com:22”]} tf.train.ClusterSpec(job_spec) with tf.device(“/job:my_workers/task:0”): # do task with tf.device(“/job:other_workers/task:0”): # do another task • Scheduling and communication is done for you (in an efficient and greedy manner)
  • 19. @nfmcclure More in Tensorflow: GPU Capabilities • Sometimes neural networks can have hundreds of millions of parameters to train. • Graphic cards (GPUs) are built to handle matrix calculations very fast and efficiently. • It’s quite common to get 10x to 50x speedup when switching the same code to a GPU (1) Install GPU version of T.F. (2) Done. As a side note, you can also tell T.F. to use specific devices. /cpu:0 /gpu:0 /gpu:100 - The CPU of the machine. - The GPU of the machine. - The 100th GPU of the machine.
  • 20. @nfmcclure More in Tensorflow: Tensorboard • A built in way to visualize the computational graph, accuracies, loss functions, gradients… • (1) Specify a specific way to write and store summaries to a log file. • (2) Point Tensorboard at logging directory, and navigate to 0.0.0.0:6006. (Can view during training). tf.histogram_summary(‘variable_histogram’, variable_value) tf.scalar_summary(‘loss_value’, loss) merged = tf.merge_all_summaries() my_writer = tf.train.SummaryWriter('my_logging_directory', sess.graph) $tensorboard –logdir=my_logging_directory
  • 21. @nfmcclure More in Tensorflow: Tensorboard Collapsable/Expandable Named Scopes!! with tf.name_scope(‘hidden1’) as hidden_scope: #Perform calculations.
  • 22. @nfmcclure More in Tensorflow: skflow • Easier scikit learn type implementation of common algorithms. • 3 layer NN: • RNN: • Easy to add custom layers by adding in layer functions from vanilla Tensorflow. import tensorflow.contrib.learn as skflow my_model = skflow.TensorFlowDNNClassifier(hidden_units=[N1, N2, N3], n_classes=n_out) my_model.fit(x_data, y_target) my_rnn = skflow.TensorFlowRNNClassifier(…)
  • 23. @nfmcclure What is a Convolutional Neural Network? • A CNN us a large pattern recognition neural network that uses various parameter reducing techniques. • E.g. Alexnet (2012 ImageNet Winner)
  • 24. @nfmcclure Reduction of Parameters • CNNs use tricks to reduce the amount of parameters to fit. • Convolution layers: Assume weights are the same on arrows in the same direction of a window we move across data. w1 w2 w3 There are only three weights in this layer, w1, w2, and w3 Note: Usually multiple convolutions are performed, creating ‘Layers’.
  • 25. @nfmcclure Other Tricks- Pooling • Layer that is an aggregation across a window from prior layer. • Types of pooling: • Max, min, mean, median, … Aggregation over three nodes in prior layer. E.g. Max().
  • 26. @nfmcclure Other Tricks- Dropout • Dropout is a way to increase the training strength and regularize the parameters. • During training, we randomly select activation functions and make the output equal to zero. • Compare this to ‘zoneout’, which randomly replaced activation functions with prior iteration value. • This helps the network find multiple pathways for prediction. Because of this, the network will not rely on just one connection or edge for prediction. • This also helps the training of other parameters in the network.
  • 27. @nfmcclure Larger Network Shorthand • Since neural networks can get quite large, we do not want to write out all the connections nor nodes. • Here’s how to convey the meaning in a shorter, less complicated way: “Max of a set of features” E.g., Here, it appears to be a max of 2 features.
  • 28. @nfmcclure What is a Recurrent Neural Network? • Recurrent Neural Networks (RNN) are networks that can ‘see’ outputs from prior layers. Very helpful for sequences. • The most common usage is to predict the next letter from the prior letter. We train this network on large text samples. U,V,W = Weight Parameters S = layer X = input O = output
  • 29. @nfmcclure Regional Convolutional Networks • We can convolve a whole network over patches of the image and see what regions score highly for a category. (Object Detection) https://www.youtube.com/watch?v=lr1qVxhGUDw
  • 30. @nfmcclure Regional CNN + Recurrent NN = Image Captioning • We take the object detection output and feed it through a recurrent neural network to generate text describing the image. Microsoft Research 2015
  • 31. @nfmcclure Deep Dream (2015) • We can select specific nodes in an image recognition pipeline and up weight the output… https://www.youtube.com/watch?v=DgPaCWJL7XI http://www.fastcodesign.com/3057368/inside-googles-first-deepdream-art-show
  • 32. @nfmcclure Stylenet (2015) • Train network on a “style image”. Try to reconstruct another image from that same pretrained network. https://vimeo.com/139123754 Recoloring of black and white photos
  • 33. @nfmcclure CNN for Text (2015) • Zhang, X., et. al. released a paper (http://arxiv.org/abs/1509.01626) that showed great results for treating strings as 1-D numerical vectors. • Ways to get strings into numerical vectors: • Word level one-hot encoding. • Word2Vec encoding or similar (GloVe or Doc2Vec or …) • Character level one-hot encoding. • Can’t resize vectors like pictures, so instead we pick a max length and pad many entries with zeros. • Like many NN results, depend heavily on dataset and architecture. • Paper shows good results for low # of classes (ratings or sentiment).
  • 34. @nfmcclure Parsey McParseface • Google released a model called ‘Syntaxnet’ in May 2016, based on Tensorflow. • Syntaxnet is arguably one of the best part-of-speech parsers available. • They trained an English version, called ‘Parsey McParseface’. • This is a free and open source model to use.
  • 35. @nfmcclure The Possibilities are Endless • Predict age from facial photo - https://how-old.net • Predict dog breed from photo - https://www.what-dog.net • Inside a self driving car brain : https://www.youtube.com/watch?v=ZJMtDRbqH40 • Memory networks: https://arxiv.org/pdf/1410.3916.pdf • Frodo journeyed to Mount-Doom. Frodo dropped the ring there. Sauron died. • Frodo went back to the Shire. Bilbo travelled to the Grey-havens. • Where is the ring? A: Mount-Doom • Where is Bilbo now? A: Grey-havens • “Synthia”- Virtual driving school for self driving cars: • http://www.gizmag.com/synthia-dataset-self-driving-cars/43895/ • Solving hand-written chalk board problems in real time: • http://www.willforfang.com/computer-vision/2016/4/9/artificial-intelligence-for- handwritten-mathematical-expression-evaluation
  • 36. @nfmcclure <Insert Name Here>-Neural Net • Last Month (June-July 2016): • YodaNN: http://arxiv.org/abs/1606.05487 • CMS-RCNN: http://arxiv.org/abs/1606.05413 • Zoneout RNN: https://arxiv.org/abs/1606.01305 • Pixel CNN: http://arxiv.org/abs/1606.05328 • Memory CNN: http://arxiv.org/abs/1606.05262 • DISCO Nets: http://arxiv.org/abs/1606.02556 • This Year (2016): • Stochastic Depth Net: https://arxiv.org/abs/1603.09382 • Squeeze Net: https://arxiv.org/abs/1602.07360 • Binary Nets: http://arxiv.org/abs/1602.02830 • Last Year: • PoseNet: http://arxiv.org/abs/1505.07427 • YOLO Nets: http://arxiv.org/abs/1506.02640 • Style Net: http://arxiv.org/abs/1508.06576 • Residual Net: https://arxiv.org/abs/1512.03385 • And so many more… • What’s Your Architecture?
  • 37. @nfmcclure How to Keep ML Current • Machine learning advances happen more and more frequently. • Can’t rely on journals that take multiple years to publish. • Twitter • http://arxiv.org/ • Others?
  • 38. @nfmcclure More Resources • More on CNNs: • https://www.reddit.com/r/MachineLearning/ • https://www.reddit.com/r/deepdream/ • http://karpathy.github.io/ • RNNs: • http://colah.github.io/posts/2015-08-Understanding-LSTMs/ • Recent Research: (Jan 16-June 16) • Residual Conv. Nets: https://www.youtube.com/watch?v=1PGLj- uKT1w • Determine location of a random image: https://www.technologyreview.com/s/600889/google-unveils- neural-network-with-superhuman-ability-to-determine-the- location-of-almost/ • Stochastic Depth Networks: http://arxiv.org/abs/1603.09382 Pre-trained models: http://www.vlfeat.org/matconvnet/pretrained/