SlideShare une entreprise Scribd logo

Neural Network Design: Chapter 18 Grossberg Network

•Télécharger en tant que PPTX, PDF•
•
Jason Tsai
Jason Tsai

Lecture for Neural Networks study group held on January 11, 2020. Reference book: http://hagan.okstate.edu/nnd.html Video: https://youtu.be/H4NKgliTFUw Initiated by Taiwan AI Group (https://www.facebook.com/groups/Taiwan.AI.Group/permalink/2017771298545301/)

Technologie
1  sur  44
Chapter 18 Grossberg Network Jason Tsai (蔡志順) January 11, 2020 Mozilla Community Space Taipei
*Copyright Notice: Most materials from this presentation are taken from the book “Neural Network Design 2nd edition” authored by Martin T. Hagan, Howard B. Demuth, Mark Hudson Beale and Orlando De Jesús. The other quoted sources are mentioned in the respective slides. This presentation itself adopts Creative Commons license.
A Quote from this Chapter “Although the original inspiration for the field of artificial neural networks came from biology, at times we forget to look back to biology for new ideas. It will be the blending of biology, mathematics, psychology and other disciplines that will provide the maximum growth in our understanding of neural networks.” 「雖然類神經網路這一領域的最初靈感來自生物學， 但有時我們會忘了回顧生物學以啟迪新的想法。融合 生物學、數學、心理學和其他學科將使我們對神經網 路的理解得以最大的發展。」
Vision: Eyeball & Retina
Visual Pathway
Photograph of the Retina
Blind Spot *Figure adopted from https://bit.ly/39iUtNQ
Test for the Blind Spot Look at the blue circle on the left side of above figure with your right eye while covering your left eye and move your head back and forth perpendicularly to the circle to find the point at which the circle on the right will disappear from your field of vision.
Imperfections in Retinal Uptake
Compensatory Processing
Visual Illusions
Neon Color Spreading *Figure adopted from https://bit.ly/37xDyWg
Oriented Receptive Field *Figure adopted from https://bit.ly/2rPXGDF & https://bit.ly/2tmGg1X
Oriented Receptive Field (cont.)
Vision Normalization
Brightness Contrast
Lateral Inhibition *Figure adopted from https://bit.ly/2yaat37
Leaky Integrate-and-Fire *Figure adopted from Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski “Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition” Cambridge University Press. 2014. Page 11.
Leaky Integrator
*Solution to 1st order ODE *Credit to Schwartz Lyu
Leaky Integrator Response
Shunting Model Here p+, p-, b+, are b- are nonnegative values Refer to slide 28: Analysis of Normalization
Shunting Model Response
Grossberg Competitive Network
Layer 1
Characteristics of Layer 1 • The network is sensitive to relative intensities of the input pattern, rather than absolute intensities. • The output of Layer 1 is a normalized version of the input pattern. • The on-center/off-surround connection pattern and the nonlinear gain control of the shunting model produce the normalization effect. • The operation of Layer 1 explains the brightness constancy and brightness contrast characteristics of the human visual system.
Operation of Layer 1
Analysis of Normalization
Example of Layer 1 Response The response of the network maintains the relative intensities of the inputs, while limiting the total response.
Layer 2
Characteristics of Layer 2 • As in the Hamming and Kohonen networks, the inputs to Layer 2 are the inner products between the prototype patterns (rows of the weight matrix W2) and the output of Layer 1 (normalized input pattern). • The nonlinear feedback enables the network to store the output pattern (pattern remains after input is removed). • The on-center/off-surround connection pattern causes contrast enhancement (large inputs are maintained, while small inputs are attenuated).
Layer 2 Operation
Example of Layer 2 Response
Choice of Transfer Function
Sigmoid Transfer Function • A sigmoid function is faster-than-linear for small signals, approximately linear for intermediate signals and slower-than-linear for large signals. • When a sigmoid transfer function is used in Layer 2, the pattern is contrast enhanced; larger values are amplified, and smaller values are attenuated. • All initial neuron outputs that are less than a certain level decay to 0. • This merges the noise suppression of the faster- than-linear transfer functions with the perfect storage produced by linear transfer functions.
Adaptive Weights W2
Example of W2 Response
Equivalence to Discrete-time Rule
Relation to Kohonen Rule
Relation to Kohonen Rule (Cont.)
Three major differences: • The Grossberg network is a continuous-time network. • Layer 1 of the Grossberg network automatically normalizes the input vectors. • Layer 2 of the Grossberg can perform a “soft” competition, rather than the winner-take-all competition of the Kohonen network. This soft competition allows more than one neuron in Layer 2 to learn. This causes the Grossberg network to operate as a feature map. Grossberg vs. Kohonen Networks
Potential Issues • One key problem of the Grossberg network is the stability of learning. As more inputs are applied to the network, there is no guarantee that the weight matrix will eventually converge (form stable clusters / categories). [Refer to Chapter 19: Adaptive Resonance Theory, ART] • Another problem is the stability of the differential equations that implement Grossberg’s continuous- time competitive recurrent network. The output of a recurrent network could converge, oscillate, or even diverge. [Refer to Chapter 20: Stability]
Basic ART Architecture (Ch.19) Add-ons: Layer 2 to Layer 1 expectations The orienting subsystem (reset) Modified gain control Let’s move on…
References

Recommandé

# Neural network toolbox
# Neural network toolbox # Neural network toolbox
# Neural network toolbox VineetKumar508
 
pattern classification
pattern classificationpattern classification
pattern classificationRanjan Ganguli
 
Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Suraj Aavula
 
Histogram processing
Histogram processingHistogram processing
Histogram processingVisvesvaraya National Institute of Technology, Nagpur, Maharashtra, India
 
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018Universitat Politècnica de Catalunya
 
HML: Historical View and Trends of Deep Learning
HML: Historical View and Trends of Deep LearningHML: Historical View and Trends of Deep Learning
HML: Historical View and Trends of Deep LearningYan Xu
 
Support Vector Machines ( SVM )
Support Vector Machines ( SVM ) Support Vector Machines ( SVM )
Support Vector Machines ( SVM ) Mohammad Junaid Khan
 
Back propagation
Back propagationBack propagation
Back propagationNagarajan
 

Recommandé

# Neural network toolbox
# Neural network toolbox # Neural network toolbox
# Neural network toolbox VineetKumar508
 
pattern classification
pattern classificationpattern classification
pattern classificationRanjan Ganguli
 
Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Suraj Aavula
 
Histogram processing
Histogram processingHistogram processing
Histogram processingVisvesvaraya National Institute of Technology, Nagpur, Maharashtra, India
 
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018Universitat Politècnica de Catalunya
 
HML: Historical View and Trends of Deep Learning
HML: Historical View and Trends of Deep LearningHML: Historical View and Trends of Deep Learning
HML: Historical View and Trends of Deep LearningYan Xu
 
Support Vector Machines ( SVM )
Support Vector Machines ( SVM ) Support Vector Machines ( SVM )
Support Vector Machines ( SVM ) Mohammad Junaid Khan
 
Back propagation
Back propagationBack propagation
Back propagationNagarajan
 
Linear regression
Linear regressionLinear regression
Linear regressionMartinHogg9
 
openCV with python
openCV with pythonopenCV with python
openCV with pythonWei-Wen Hsu
 
Activation function
Activation functionActivation function
Activation functionAstha Jain
 
Pattern recognition and Machine Learning.
Pattern recognition and Machine Learning.Pattern recognition and Machine Learning.
Pattern recognition and Machine Learning.Rohit Kumar
 
Differential Evolution Algorithm (DEA)
Differential Evolution Algorithm (DEA) Differential Evolution Algorithm (DEA)
Differential Evolution Algorithm (DEA) A. Bilal Özcan
 
Understanding Bagging and Boosting
Understanding Bagging and BoostingUnderstanding Bagging and Boosting
Understanding Bagging and BoostingMohit Rajput
 
Dimensionality Reduction
Dimensionality ReductionDimensionality Reduction
Dimensionality Reductionmrizwan969
 
Machine Learning lecture4(logistic regression)
Machine Learning lecture4(logistic regression)Machine Learning lecture4(logistic regression)
Machine Learning lecture4(logistic regression)cairo university
 
Introduction to Machine Learning Classifiers
Introduction to Machine Learning ClassifiersIntroduction to Machine Learning Classifiers
Introduction to Machine Learning ClassifiersFunctional Imperative
 
Artificial Neural Networks for NIU session 2016 17
Artificial Neural Networks for NIU session 2016 17 Artificial Neural Networks for NIU session 2016 17
Artificial Neural Networks for NIU session 2016 17 Prof. Neeta Awasthy
 
Methods of Optimization in Machine Learning
Methods of Optimization in Machine LearningMethods of Optimization in Machine Learning
Methods of Optimization in Machine LearningKnoldus Inc.
 
Regularization in deep learning
Regularization in deep learningRegularization in deep learning
Regularization in deep learningKien Le
 
YOLO
YOLOYOLO
YOLOgeothomas18
 
Texture in image processing
Texture in image processing Texture in image processing
Texture in image processing Anna Aquarian
 
Neuro-fuzzy systems
Neuro-fuzzy systemsNeuro-fuzzy systems
Neuro-fuzzy systemsSagar Ahire
 
Video Segmentation
Video SegmentationVideo Segmentation
Video SegmentationSmriti Jain
 
Perceptron (neural network)
Perceptron (neural network)Perceptron (neural network)
Perceptron (neural network)EdutechLearners
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Gaurav Mittal
 
Coordinate Descent method
Coordinate Descent methodCoordinate Descent method
Coordinate Descent methodSanghyuk Chun
 
Mathematical tools in dip
Mathematical tools in dipMathematical tools in dip
Mathematical tools in dipVisvesvaraya National Institute of Technology, Nagpur, Maharashtra, India
 
Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksAdri Jovin
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithmsaciijournal
 

Contenu connexe

Tendances

Linear regression
Linear regressionLinear regression
Linear regressionMartinHogg9
 
openCV with python
openCV with pythonopenCV with python
openCV with pythonWei-Wen Hsu
 
Activation function
Activation functionActivation function
Activation functionAstha Jain
 
Pattern recognition and Machine Learning.
Pattern recognition and Machine Learning.Pattern recognition and Machine Learning.
Pattern recognition and Machine Learning.Rohit Kumar
 
Differential Evolution Algorithm (DEA)
Differential Evolution Algorithm (DEA) Differential Evolution Algorithm (DEA)
Differential Evolution Algorithm (DEA) A. Bilal Özcan
 
Understanding Bagging and Boosting
Understanding Bagging and BoostingUnderstanding Bagging and Boosting
Understanding Bagging and BoostingMohit Rajput
 
Dimensionality Reduction
Dimensionality ReductionDimensionality Reduction
Dimensionality Reductionmrizwan969
 
Machine Learning lecture4(logistic regression)
Machine Learning lecture4(logistic regression)Machine Learning lecture4(logistic regression)
Machine Learning lecture4(logistic regression)cairo university
 
Introduction to Machine Learning Classifiers
Introduction to Machine Learning ClassifiersIntroduction to Machine Learning Classifiers
Introduction to Machine Learning ClassifiersFunctional Imperative
 
Artificial Neural Networks for NIU session 2016 17
Artificial Neural Networks for NIU session 2016 17 Artificial Neural Networks for NIU session 2016 17
Artificial Neural Networks for NIU session 2016 17 Prof. Neeta Awasthy
 
Methods of Optimization in Machine Learning
Methods of Optimization in Machine LearningMethods of Optimization in Machine Learning
Methods of Optimization in Machine LearningKnoldus Inc.
 
Regularization in deep learning
Regularization in deep learningRegularization in deep learning
Regularization in deep learningKien Le
 
Texture in image processing
Texture in image processing Texture in image processing
Texture in image processing Anna Aquarian
 
Neuro-fuzzy systems
Neuro-fuzzy systemsNeuro-fuzzy systems
Neuro-fuzzy systemsSagar Ahire
 
Video Segmentation
Video SegmentationVideo Segmentation
Video SegmentationSmriti Jain
 
Perceptron (neural network)
Perceptron (neural network)Perceptron (neural network)
Perceptron (neural network)EdutechLearners
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Gaurav Mittal
 
Coordinate Descent method
Coordinate Descent methodCoordinate Descent method
Coordinate Descent methodSanghyuk Chun
 

Tendances (20)

Linear regression
Linear regressionLinear regression
Linear regression
 
openCV with python
openCV with pythonopenCV with python
openCV with python
 
Activation function
Activation functionActivation function
Activation function
 
Pattern recognition and Machine Learning.
Pattern recognition and Machine Learning.Pattern recognition and Machine Learning.
Pattern recognition and Machine Learning.
 
Differential Evolution Algorithm (DEA)
Differential Evolution Algorithm (DEA) Differential Evolution Algorithm (DEA)
Differential Evolution Algorithm (DEA)
 
Understanding Bagging and Boosting
Understanding Bagging and BoostingUnderstanding Bagging and Boosting
Understanding Bagging and Boosting
 
Dimensionality Reduction
Dimensionality ReductionDimensionality Reduction
Dimensionality Reduction
 
Machine Learning lecture4(logistic regression)
Machine Learning lecture4(logistic regression)Machine Learning lecture4(logistic regression)
Machine Learning lecture4(logistic regression)
 
Introduction to Machine Learning Classifiers
Introduction to Machine Learning ClassifiersIntroduction to Machine Learning Classifiers
Introduction to Machine Learning Classifiers
 
Artificial Neural Networks for NIU session 2016 17
Artificial Neural Networks for NIU session 2016 17 Artificial Neural Networks for NIU session 2016 17
Artificial Neural Networks for NIU session 2016 17
 
Methods of Optimization in Machine Learning
Methods of Optimization in Machine LearningMethods of Optimization in Machine Learning
Methods of Optimization in Machine Learning
 
Regularization in deep learning
Regularization in deep learningRegularization in deep learning
Regularization in deep learning
 
YOLO
YOLOYOLO
YOLO
 
Texture in image processing
Texture in image processing Texture in image processing
Texture in image processing
 
Neuro-fuzzy systems
Neuro-fuzzy systemsNeuro-fuzzy systems
Neuro-fuzzy systems
 
Video Segmentation
Video SegmentationVideo Segmentation
Video Segmentation
 
Perceptron (neural network)
Perceptron (neural network)Perceptron (neural network)
Perceptron (neural network)
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)
 
Coordinate Descent method
Coordinate Descent methodCoordinate Descent method
Coordinate Descent method
 
Mathematical tools in dip
Mathematical tools in dipMathematical tools in dip
Mathematical tools in dip
 

Similaire à Neural Network Design: Chapter 18 Grossberg Network

Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksAdri Jovin
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithmsaciijournal
 
Web spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsWeb spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsaciijournal
 
Autoencoders for image_classification
Autoencoders for image_classificationAutoencoders for image_classification
Autoencoders for image_classificationCenk Bircanoğlu
 
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Universitat Politècnica de Catalunya
 
Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Dr.(Mrs).Gethsiyal Augasta
 
Topology ppt
Topology pptTopology ppt
Topology pptboocse11
 
Handwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksHandwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksIRJET Journal
 
DEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONDEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONIRJET Journal
 
Basic Learning Algorithms of ANN
Basic Learning Algorithms of ANNBasic Learning Algorithms of ANN
Basic Learning Algorithms of ANNwaseem khan
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORKESCOM
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural networkDEEPASHRI HK
 
M7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfM7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfArushiKansal3
 
Artificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningArtificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningSujit Pal
 
Neural Networks.pptx
Neural Networks.pptxNeural Networks.pptx
Neural Networks.pptxshahinbme
 

Similaire à Neural Network Design: Chapter 18 Grossberg Network (20)

Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural Networks
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
 
D028018022
D028018022D028018022
D028018022
 
Web spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsWeb spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithms
 
Autoencoders for image_classification
Autoencoders for image_classificationAutoencoders for image_classification
Autoencoders for image_classification
 
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
 
Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
Handwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksHandwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural Networks
 
DEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONDEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTION
 
Basic Learning Algorithms of ANN
Basic Learning Algorithms of ANNBasic Learning Algorithms of ANN
Basic Learning Algorithms of ANN
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORK
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural network
 
M7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfM7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdf
 
Artificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningArtificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep Learning
 
Neural Networks.pptx
Neural Networks.pptxNeural Networks.pptx
Neural Networks.pptx
 

Plus de Jason Tsai

基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介Jason Tsai
 
Neural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksNeural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksJason Tsai
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Jason Tsai
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
Reinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceReinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceJason Tsai
 
Deep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyDeep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyJason Tsai
 
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsDeep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsJason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 

Plus de Jason Tsai (13)

基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介
 
Neural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksNeural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis Networks
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
Reinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceReinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 Neuroscience
 
Deep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyDeep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical Methodology
 
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsDeep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 

Dernier

How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfPrecisely
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 

Dernier (20)

How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 

Neural Network Design: Chapter 18 Grossberg Network

  • 1. Chapter 18 Grossberg Network Jason Tsai (蔡志順) January 11, 2020 Mozilla Community Space Taipei
  • 2. *Copyright Notice: Most materials from this presentation are taken from the book “Neural Network Design 2nd edition” authored by Martin T. Hagan, Howard B. Demuth, Mark Hudson Beale and Orlando De Jesús. The other quoted sources are mentioned in the respective slides. This presentation itself adopts Creative Commons license.
  • 3. A Quote from this Chapter “Although the original inspiration for the field of artificial neural networks came from biology, at times we forget to look back to biology for new ideas. It will be the blending of biology, mathematics, psychology and other disciplines that will provide the maximum growth in our understanding of neural networks.” 「雖然類神經網路這一領域的最初靈感來自生物學， 但有時我們會忘了回顧生物學以啟迪新的想法。融合 生物學、數學、心理學和其他學科將使我們對神經網 路的理解得以最大的發展。」
  • 7. Blind Spot *Figure adopted from https://bit.ly/39iUtNQ
  • 8. Test for the Blind Spot Look at the blue circle on the left side of above figure with your right eye while covering your left eye and move your head back and forth perpendicularly to the circle to find the point at which the circle on the right will disappear from your field of vision.
  • 12. Neon Color Spreading *Figure adopted from https://bit.ly/37xDyWg
  • 13. Oriented Receptive Field *Figure adopted from https://bit.ly/2rPXGDF & https://bit.ly/2tmGg1X
  • 17. Lateral Inhibition *Figure adopted from https://bit.ly/2yaat37
  • 18. Leaky Integrate-and-Fire *Figure adopted from Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski “Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition” Cambridge University Press. 2014. Page 11.
  • 20. *Solution to 1st order ODE *Credit to Schwartz Lyu
  • 22. Shunting Model Here p+, p-, b+, are b- are nonnegative values Refer to slide 28: Analysis of Normalization
  • 26. Characteristics of Layer 1 • The network is sensitive to relative intensities of the input pattern, rather than absolute intensities. • The output of Layer 1 is a normalized version of the input pattern. • The on-center/off-surround connection pattern and the nonlinear gain control of the shunting model produce the normalization effect. • The operation of Layer 1 explains the brightness constancy and brightness contrast characteristics of the human visual system.
  • 29. Example of Layer 1 Response The response of the network maintains the relative intensities of the inputs, while limiting the total response.
  • 31. Characteristics of Layer 2 • As in the Hamming and Kohonen networks, the inputs to Layer 2 are the inner products between the prototype patterns (rows of the weight matrix W2) and the output of Layer 1 (normalized input pattern). • The nonlinear feedback enables the network to store the output pattern (pattern remains after input is removed). • The on-center/off-surround connection pattern causes contrast enhancement (large inputs are maintained, while small inputs are attenuated).
  • 33. Example of Layer 2 Response
  • 34. Choice of Transfer Function
  • 35. Sigmoid Transfer Function • A sigmoid function is faster-than-linear for small signals, approximately linear for intermediate signals and slower-than-linear for large signals. • When a sigmoid transfer function is used in Layer 2, the pattern is contrast enhanced; larger values are amplified, and smaller values are attenuated. • All initial neuron outputs that are less than a certain level decay to 0. • This merges the noise suppression of the faster- than-linear transfer functions with the perfect storage produced by linear transfer functions.
  • 37. Example of W2 Response
  • 40. Relation to Kohonen Rule (Cont.)
  • 41. Three major differences: • The Grossberg network is a continuous-time network. • Layer 1 of the Grossberg network automatically normalizes the input vectors. • Layer 2 of the Grossberg can perform a “soft” competition, rather than the winner-take-all competition of the Kohonen network. This soft competition allows more than one neuron in Layer 2 to learn. This causes the Grossberg network to operate as a feature map. Grossberg vs. Kohonen Networks
  • 42. Potential Issues • One key problem of the Grossberg network is the stability of learning. As more inputs are applied to the network, there is no guarantee that the weight matrix will eventually converge (form stable clusters / categories). [Refer to Chapter 19: Adaptive Resonance Theory, ART] • Another problem is the stability of the differential equations that implement Grossberg’s continuous- time competitive recurrent network. The output of a recurrent network could converge, oscillate, or even diverge. [Refer to Chapter 20: Stability]
  • 43. Basic ART Architecture (Ch.19) Add-ons: Layer 2 to Layer 1 expectations The orienting subsystem (reset) Modified gain control Let’s move on…