Introduction Of Artificial neural network

Madras university,[object Object],Department Of Computer Science,[object Object]

Seminar On,[object Object],Introduction Of ANN, Rules,[object Object],And,[object Object],Adaptive Resonance Theory,[object Object]

GROUP MEMBERS ARE :P.JayaVelJ.Joseph Amal RajM.Kaja Mohinden,[object Object]

ARTIFICIAL NEURAL NETWORK (ANN),[object Object],An artificial neural network (ANN), usually called "neural network" (NN), is a mathematical model or computational model that tries to simulate the structure and/or functional aspects of biological neural networks.,[object Object],It consists of an interconnected group of artificial neurons and processes information using a connectionist approach to computation.,[object Object],In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network during the learning phase. ,[object Object]

ARTIFICIAL NEURAL NETWORK (ANN),[object Object]

ARTIFICIAL NEURAL NETWORK (ANN),[object Object],Why use neural networks?,[object Object],Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques. A trained neural network can be thought of as an "expert" in the category of information it has been given to analyse. This expert can then be used to provide projections given new situations of interest and answer "what if" questions.Other advantages include: ,[object Object],[object Object]

Self-Organisation: An ANN can create its own organisation or representation of the information it receives during learning time.

Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability.

Fault Tolerance via Redundant Information Coding: Partial destruction of a network leads to the corresponding degradation of performance. However, some network capabilities may be retained even with major network damage. ,[object Object]

Reinforcement learning,[object Object]

supervised learning,[object Object]

supervised learning,[object Object],[object Object],Quickprop[fahlman88empirical],[object Object],[object Object]

Equation 2. Error derivative at this epoch The Quickprop algorithm is loosely based on Newton's method. It is quicker than standard backpropagation because it uses an approximation to the error curve, and second order derivative information which allow a quicker evaluation. Training is similar to backprop except for a copy of (eq. 1) the error derivative at a previous epoch. This, and the current error derivative (eq. 2), are used to minimise an approximation to this error curve. ,[object Object]

supervised learning,[object Object],The update rule is given in equation 3:,[object Object],[object Object], This equation uses no learning rate. If the slope of the error curve is less than that of the previous one, then the weight will change in the same direction (positive or negative). However, there needs to be some controls to prevent the weights from growing too large.,[object Object]

Unsupervised learning,[object Object],In unsupervised learning we are given some data x and the cost function to be minimized, that can be any function of the data x and the network's output, f.,[object Object],The cost function is dependent on the task (what we are trying to model) and our a priori assumptions (the implicit properties of our model, its parameters and the observed variables).,[object Object]

Unsupervised learning ,[object Object],As a trivial example, consider the model f(x) = a, where a is a constant and the cost C = E[(x − f(x))2]. Minimizing this cost will give us a value of a that is equal to the mean of the data. ,[object Object],The cost function can be much more complicated. Its form depends on the application: for example, in compression it could be related to the mutual information between x and y, whereas in statistical modelling, it could be related to theposterior probability of the model given the data. (Note that in both of those examples those quantities would be maximized rather than minimized).,[object Object],Tasks that fall within the paradigm of unsupervised learning are in general estimation problems; the applications include clustering, the estimation of statistical distributions, compression and filtering.,[object Object]

Unsupervised learning,[object Object],Unsupervised learning, in contrast to supervised learning, does not provide the network with target output values. This isn't strictly true, as often (and for the cases discussed in the this section) the output is identical to the input. Unsupervised learning usually performs a mapping from input to output space, data compression or clustering.,[object Object]

Reinforcement learning,[object Object],In reinforcement learning, data x are usually not given, but generated by an agent's interactions with the environment. At each point in time t, the agent performs an action yt and the environment generates an observation xt and an instantaneous cost ct, according to some (usually unknown) dynamics.,[object Object],Tasks that fall within the paradigm of reinforcement learning are control problems, games and other sequential decision making tasks.,[object Object]

Reinforcement learning,[object Object],The aim is to discover a policy for selecting actions that minimizes some measure of a long-term cost; i.e., the expected cumulative cost. The environment's dynamics and the long-term cost for each policy are usually unknown, but can be estimated.,[object Object],ANNs are frequently used in reinforcement learning as part of the overall algorithm.,[object Object], ,[object Object]

Neural Network “Learning Rules”:,[object Object],Successful learning in any neural network is dependent on how the connections between the neurons are allowed to change in response to activity. The manner of change is what the majority of researchers call "a learning rule". ,[object Object],However, we will call it a "synaptic modification rule" because although the network learned the sequence, it is not clear that the *connections* between the neurons in the network "learned" anything in particular.,[object Object]

Mathematical synaptic Modification rule,[object Object],There are many categories of mathematical synaptic modification rule which are used to describe how synaptic strengths should be changed in a neural network. Some of these categories include: backpropgration of error, correlative Hebbian, and temporally-asymmetric Hebbian.,[object Object]

Mathematical synaptic modification rule,[object Object],Backpropogation of error states that connection strengths should change throughout the entire network in order to minimize the difference between the actual activity and the "desired" activity at the "output" layer of the network.,[object Object]

Mathematical synaptic Modification rule,[object Object],Correlative Hebbian states that any two interconnected neurons that are active at the same time should strengthen their connections, so that if one of the neurons is activated again in the future the other is more likely to become activated too.,[object Object]

Mathematical synaptic Modification rule,[object Object],Temporally-asymmetric Hebbian is described in more detail in the example below, but essentially emphasizes the importants of causality: if a neuron realiably fires before another, its connection to the other neuron should be strengthened. Otherwise, it should be weakened. ,[object Object]

Neural Network “Learning Rules”:,[object Object],The Delta Rule,[object Object],The Pattern Associator,[object Object],The Hebb Rule,[object Object]

The Delta Rule,[object Object],A generalized form of the delta rule, developed by D.E. Rumelhart, G.E. Hinton, and R.J. Williams, is needed for networks with hidden layers. They showed that this method works for the class of semilinear activation functions (non-decreasing and differentiable).,[object Object],Generalizing the ideas of the delta rule, consider a hierarchical network with an input layer, an output layer and a number of hidden layers.,[object Object]

The Delta Rule,[object Object],. We will consider only the case where there is one hidden layer. The network is presented with input signals which produce output signals that act as input to the middle layer. Output signals from the middle layer in turn act as input to the output layer to produce the final output vector. This vector is compared to the desired output vector. Since both the output and the desired output vectors are known, the delta rule can be used to adjust the weights in the output layer. ,[object Object]

The Delta Rule,[object Object],Can the delta rule be applied to the middle layer? Both the input signal to each unit of the middle layer and the output signal are known. What is not known is the error generated from the output of the middle layer since we do not know the desired output. To get this error, backpropagate through the middle layer to the units that are responsible for generating that output. The error genrated from the middle layer could be used with the delta rule to adjust the weights.,[object Object]

The Pattern Associator,[object Object],A pattern associator learns associations between input patterns and output patterns. One of the most appealing characteristics of such a network is the fact that it can generate what it learns about one pattern to other similar input patterns. Pattern associators have been widely used in distributed memory modeling.,[object Object]

The Pattern Associator,[object Object],The pattern associator is one of the more basic two-layer networks. Its architecture consists of two sets of units, the input units and the output units.,[object Object],Each input unit connects to each output unit via weighted connections.,[object Object],Connections are only allowed from input units to output units. ,[object Object]

The Pattern Associator,[object Object],The effect of a unit ui in the input layer on a unit uj in the output layer is determined by the product of the activation ai of ui and the weight of the connection from ui to uj. The activation of a unit uj in the output layer is given by: SUM(wij * ai).,[object Object]

Adaptive Resonance Theory (ART) ,[object Object],Discrete Bidirectional Associative Memory ,[object Object],Kochen Self Organization Map,[object Object],Counter Propagation Network (CPN) ,[object Object],Perceptron,[object Object],Vector Representation,[object Object],ADALINE (Adaptive Linear Neuron or later Adaptive Linear Element) ,[object Object],Madaline (Multiple Adaline) ,[object Object],Backpropagation, or propagation of error,[object Object]

Adaptive Resonance Theory (ART) ,[object Object],Adaptive Resonance Theory (ART) is a theory developed by Stephen Grossberg and Gail Carpenter on aspects of how the brain processes information. It describes a number of neural network models which use supervised and unsupervised learning methods, and address problems such as pattern recognition and prediction.,[object Object]

Discrete Bidirectional Associative Memory ,[object Object]

Kochen Self Organization Map,[object Object],The self-organizing map (SOM) invented by TeuvoKohonen performs a form of unsupervised learning.,[object Object], A set of artificial neurons learn to map points in an input space to coordinates in an output space. The input space can have different dimensions and topology from the output space, and the SOM will attempt to preserve these.,[object Object], ,[object Object]

Kochen Self Organization Map,[object Object],If an input space is to be processed by a neural network, the first issue of importance is the structure of this space. A neural network with real inputs computes a function f defined from an input space A to an output space B. The region where f is defined can be covered by a Kohonen network in such a way that when, for example,an input vector is selected from the region a1, only one unit in the network fires. Such a tiling in which input space is classified in subregions is also called a chart or map of input space. Kohonen networks learn to create maps of the input space in a self-organizing way.,[object Object]

Kochen Self Organization Map-Advantages,[object Object],Probably the best thing about SOMs that they are very easy to understand. It’s very simple, if they are close together and there is grey connecting them, then they are similar. If there is a black ravine between them, then they are different. Unlike Multidimensional Scaling or N-land, people can quickly pick up on how to use them in an effective manner.,[object Object],Another great thing is that they work very well. As I have shown you they classify data well and then are easily evaluate for their own quality so you can actually calculated how good a map is and how strong the similarities between objects are. ,[object Object]

Kochen Self Organization Map,[object Object]

Perceptron,[object Object],The perceptron is a type of artificial neural network invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt. It can be seen as the simplest kind of feedforward neural network: a linear classifier.,[object Object],The Perceptron is a binary classifier that maps its input x (a real-valued vector) to an output value f(x) (a single binary value) across the matrix.,[object Object],where w is a vector of real-valued weights and is the dot product (which computes a weighted sum). b is the 'bias', a constant term that does not depend on any input value.,[object Object]

ADALINE,[object Object],Definition,[object Object],Adaline is a single layer neural network with multiple nodes where each node accepts multiple inputs and generates one output. Given the following variables:,[object Object],x is the input vector,[object Object],w is the weight vector,[object Object],n is the number of inputs,[object Object],θ some constant,[object Object],y is the output,[object Object],then we find that the output is . If we further assume that,[object Object],xn + 1 = 1,[object Object],wn + 1 = θ then the output reduces to the dot product of x and w ,[object Object], ,[object Object]

Madaline ,[object Object],Madaline (Multiple Adaline) is a two layer neural network with a set of ADALINEs in parallel as its input layer and a single PE (processing element) in its output layer. For problems with multiple input variables and one output, each input is applied to one Adaline. For similar problems with multiple outputs, madalines in parallel can be used. The madaline network is useful for problems which involve prediction based on multiple inputs, such as weather forecasting (Input variables: barometric pressure, difference in pressure. Output variables: rain, cloudy, sunny).,[object Object]

Backpropagation,[object Object],Backpropagation, or propagation of error, is a common method of teaching artificial neural networks how to perform a given task. It was first described by Arthur E. Bryson and Yu-Chi Ho in 1969,[1][2] but it wasn't until 1986, through the work of David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, that it gained recognition, and it led to a “renaissance” in the field of artificial neural network research.,[object Object],It is a supervised learning method, and is an implementation of the Delta rule. It requires a teacher that knows, or can calculate, the desired output for any given input. It is most useful for feed-forward networks (networks that have no feedback, or simply, that have no connections that loop). The term is an abbreviation for "backwards propagation of errors". Backpropagation requires that the activation function used by theartificial neurons (or "nodes") is differentiable.,[object Object]

Backpropagation,[object Object],Backpropagation,[object Object],Calculation of error ,[object Object],dk = f(Dk) -f(Ok),[object Object]

Network Structure –Back-propagation Network,[object Object],Oi Output Unit,[object Object],Wj,i,[object Object],ajHidden Units,[object Object],Wk,j,[object Object],IkInput Units,[object Object]

Counter propagation network (CPN) (§ 5.3),[object Object],Basic idea of CPN,[object Object],Purpose: fast and coarse approximation of vector mapping,[object Object],not to map any given x to its with given precision,,[object Object],input vectors x are divided into clusters/classes.,[object Object],each cluster of x has one output y, which is (hopefully) the average of for all x in that class.,[object Object],Architecture: Simple case: FORWARD ONLY CPN, ,[object Object],y,[object Object],z,[object Object],x,[object Object],1,[object Object],1,[object Object],1,[object Object],y,[object Object],v,[object Object],z,[object Object],w,[object Object],x,[object Object],j,[object Object],j,k,[object Object],k,[object Object],k,i,[object Object],i,[object Object],y,[object Object],z,[object Object],x,[object Object],m,[object Object],p,[object Object],n,[object Object],from hidden (class) to output,[object Object],from input to hidden (class),[object Object]

[object Object],training sample (x, d ) where is the desired precise mapping,[object Object],Phase1: weights coming into hidden nodes are trained by competitive learning to become the representative vector of a cluster of input vectors x: (use only x, the input part of(x, d )),[object Object], 1. For a chosen x, feedforward to determined the winning,[object Object], 2.,[object Object], 3. Reduce , then repeat steps 1 and 2 until stop condition is met,[object Object],Phase 2: weights going out of hidden nodes are trained by delta rule to be an average output of where x is an input vector that causes to win (use both x andd). ,[object Object], 1. For a chosen x, feedforward to determined the winning,[object Object], 2. (optional) ,[object Object], 3.,[object Object], 4. Repeat steps 1 – 3 until stop condition is met ,[object Object]

Adaptive Resonance Theory,[object Object]

Adaptive Resonance Theory,[object Object],Adaptive Resonance Theory (ART) was developed by Grossberg (1976),[object Object],Input vectors which are close to each other according to a specific similarity measure should be mapped to the same cluster,[object Object],ART adapts itself by storing input patterns, and tries to match best the input pattern ,[object Object],45,[object Object]

Adaptive Resonance Theory 1 (ART 1),[object Object],ART 1 is a binary classification model. ,[object Object],Various other versions of the model have evolved from ART 1,[object Object],Pointers to these can be found in the bibliographic remarks,[object Object],The main network comprises the layers F1, F2 and the attentional gain control as the attentional subsystem,[object Object],The attentional vigilance node forms the orienting subsystem,[object Object]

ART 1: Architecture,[object Object],…,[object Object],…,[object Object],Attentional Subsystem,[object Object],Orienting Subsystem,[object Object],F2,[object Object],-,[object Object],-,[object Object],+,[object Object],+,[object Object],F1,[object Object],-,[object Object],+,[object Object],-,[object Object],G,[object Object],A,[object Object],+,[object Object],+,[object Object],+,[object Object],I,[object Object]

ART 1: 2/3 Rule,[object Object],J,[object Object],…,[object Object],F2,[object Object],Si(yj),[object Object],vji,[object Object],si,[object Object],-,[object Object],sG,[object Object],G,[object Object],F1,[object Object],+,[object Object],l,[object Object],li,[object Object],Three kinds of inputs to each F1 neuron decide when the neuron fires,[object Object],[object Object]

Top-down feedback through outstar weights vji

Gain control signal sG,[object Object]

Adaptive Resonance Theory (ART) ,[object Object],[object Object]

Motivations: Previous methods have the following problems:Number of class nodes is pre-determined and fixed. ,[object Object],[object Object]

Some nodes may have empty classes.

no control of the degree of similarity of inputs grouped in one class. Training is non-incremental: ,[object Object],[object Object]

adding new samples often requires re-train the network with the enlarged training set until a new stable state is reached.,[object Object]

To achieve these, we need:,[object Object],a mechanism for testing and determining (dis)similarity between x and .,[object Object],a control for finding/creating new class nodes.,[object Object],need to have all operations implemented by units of local computation.,[object Object],Only the basic ideas are presented,[object Object],Simplified from the original ART model,[object Object],Some of the control mechanisms realized by various specialized neurons are done by logic statements of the algorithm,[object Object]

ART1 Architecture,[object Object]

Working of ART1,[object Object],3 phases after each input vector x is applied,[object Object],Recognition phase: determine the winner cluster for x,[object Object],Using bottom-up weights b,[object Object],Winner j* with max yj* = bj*ּx,[object Object],x is tentatively classified to cluster j*,[object Object],the winner may be far away from x (e.g., |tj* - x| is unacceptably large),[object Object]

Working of ART1 (3 phases),[object Object],Comparison phase: ,[object Object],Compute similarity using top-down weights t: ,[object Object], vector:,[object Object],If (# of 1’s ins)|/(# of 1’s inx) > ρ, accept the classification, update bj* and tj*,[object Object],else: remove j* from further consideration, look for other potential winner or create a new node with x as its first patter. ,[object Object]

Weight update/adaptive phase,[object Object],Initial weight: (no bias),[object Object], bottom up: top down:,[object Object],When a resonance occurs with,[object Object],If k sample patterns are clustered to node jthen,[object Object], = pattern whose 1’s are common to all these k samples ,[object Object]

Introduction Of Artificial neural network

Example ,[object Object],for input x(1),[object Object],Node 1 wins,[object Object]

Notes,[object Object],Classification as a search process,[object Object],No two classes have the same b and t,[object Object],Outliers that do not belong to any cluster will be assigned separate nodes,[object Object],Different ordering of sample input presentations may result in different classification.,[object Object],Increase of r increases # of classes learned, and decreases the average class size.,[object Object],Classification may shift during search, will reach stability eventually.,[object Object],There are different versions of ART1 with minor variations,[object Object],ART2 is the same in spirit but different in details.,[object Object]

R,[object Object],G1,[object Object],G2,[object Object],ART1 Architecture,[object Object],+,[object Object],+,[object Object],-,[object Object],-,[object Object],+,[object Object],+,[object Object],+,[object Object],+,[object Object]

cluster units: competitive, receive input vector x through weights b: to determine winner j.,[object Object], input units: placeholder or external inputs,[object Object], interface units: ,[object Object],pass s to x as input vector for classification by ,[object Object],compare x and ,[object Object],controlled by gain control unit G1,[object Object],Needs to sequence the three phases (by control units G1, G2, and R),[object Object]

R = 0: resonance occurs, update and,[object Object],R = 1: fails similarity test, inhibits J from further computation,[object Object]

ART clustering algorithms,[object Object],[object Object]

Fuzzy ART,[object Object],Layer1 consists of neurons that are connected to the neurons in Layer 2 through weight vectors.,[object Object],Thenumber of neurons in Layer 1 depends on the characteristics of the input data.,[object Object],The Layer 2 represent clusters.,[object Object]

Fuzzy ART Architecture ,[object Object]

Fuzzy ART FMEA,[object Object],FMEA values are evaluated separately with severity, detection and occurrence values,[object Object],The aim is to apply Fuzzy ART algorithm to FMEA method and by performing FMEA on test problems, most favorable parameter combinations (α , β and ρ) are investigated.,[object Object]

Hand-worked Example,[object Object],Cluster the vectors 11100, 11000, 00001, 00011,[object Object],Low vigilance: 0.3,[object Object],High vigilance: 0.7,[object Object]

Hand-worked Example:  = 0.3,[object Object]

ART 1: Clustering Application,[object Object], = 0.3,[object Object]

Hand-worked Example:  = 0.7,[object Object]

ART 1: Clustering Application,[object Object], = 0.7,[object Object]

Neurophysiological Evidence for ARTMechanisms,[object Object],The attentional subsystem of an ART network has been used to model aspects of the inferotemporal cortex,[object Object],Orienting subsystem has been used to model a part of the hippocampal system, which is known to contribute to memory functions,[object Object],The feedback prevalent in an ART network can help focus attention in models of visual object recognition,[object Object]

Other Applications,[object Object],Aircraft Part Design Classification System.,[object Object],See text for details.,[object Object]

Ehrenstein Pattern Explained by ART !,[object Object],The bright disc disappears ,[object Object],when the alignment of the dark lines is disturbed!,[object Object],Generates a circular illusory contour – a circular disc of enhanced brightness,[object Object]

78,[object Object],Other Neurophysiological Evidence,[object Object],Adam Sillito [University College, London],[object Object],Cortical feedback in a cat tunes cells in its LGN to respond best to lines of a specific length. ,[object Object],Chris Redie [MPI Entwicklungsbiologie, Germany],[object Object],Found that some visual cells in a cat’s LGN and cortex respond best at line ends— more strongly to line ends than line sides. ,[object Object],Sillito et al. [University College, London],[object Object], Provide neurophysiological data suggesting that the cortico-geniculate feedback closely resembles the matching and resonance of an ART network. ,[object Object],Cortical feedback has been found to change the output of specific LGN cells, increasing the gain of the input for feature linked events that are detected by the cortex. ,[object Object]

Computational Experiment,[object Object],Anon-binary,[object Object],dataset of FMEA is,[object Object],used to evaluate the,[object Object],performance of the,[object Object],Fuzzy ART neural,[object Object],network on different,[object Object],test problems,[object Object],79,[object Object]

80,[object Object],Computational Experiment,[object Object],For acomprehensive,[object Object],analysis of the effects,[object Object],of parameters on the,[object Object],performance of Fuzzy,[object Object],ART in FMEA case, a,[object Object],number of levels of,[object Object],parameters are,[object Object],considered.,[object Object]

81,[object Object],Computational Experiment,[object Object],The Fuzzy ART neural network method is applied to determine the most favorable parameter (α, β and ρ) combinations during application of FMEA on test problems,[object Object]

82,[object Object],Results,[object Object],For any test problem 900 solutions are obtained. ,[object Object],The β-ρ interactions for parameter combinations are considered where solutions are obtained. For each test problem, all the combinations are evaluated and frequency distribution of clusters are constituted,[object Object]

Introduction Of Artificial neural network

Introduction Of Artificial neural network

Recommandé

Contenu connexe

Tendances

Tendances(20)

En vedette

En vedette(20)

Similaire à Introduction Of Artificial neural network

Similaire à Introduction Of Artificial neural network(20)

Plus de Nagarajan

Plus de Nagarajan(17)

Dernier

Dernier(20)

Introduction Of Artificial neural network