23 Apr 2015•0 j'aime•4,391 vues

Télécharger pour lire hors ligne

Signaler

Formation

Multi Layer Neural Network

Artificial neural network for machine learninggrinu

Artifical Neural Network and its applicationsSangeeta Tiwari

Artificial Neural NetworkAtul Krishna

Deep Learning - CNN and RNNAshray Bhandare

Convolutional Neural Network Models - Deep LearningBenha University

Classification using back propagation algorithmKIRAN R

- 1. Multilayer Neural Network Presented By: Sunawar Khan Reg No: 813/FBAS/MSCS/F14
- 2. 2
- 3. Learning of ANN’s Type of Neural Network Neural Network Architecture Basic of Neural Network Introduction Logistic Structure Model Building Development of ANN Model Example 3
- 4. 4
- 5. ANN (Artificial Neural Networks) and SVM (Support Vector Machines) are two popular strategies for classification. They are well-suited for continuous-valued inputs and outputs, unlike most decision tree algorithms. Neural network algorithms are inherently parallel; parallelization techniques can be used to speed up the computation process. These factors contribute toward the usefulness of neural networks for classification and prediction in data mining. 5
- 6. An Artificial Neural Network (ANN) is an information processing paradigm that is inspired by biological nervous systems. It is composed of a large number of highly interconnected processing elements called neurons. An ANN is configured for a specific application, such as pattern recognition or data classification 6
- 7. Ability to derive meaning from complicated or imprecise data Extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques Adaptive learning Real Time Operation 7
- 8. Conventional computers use an algorithmic approach. But neural networks works similar to human brain and learns by example. 8
- 9. 9 Some numbers… The human brain contains about 10 billion nerve cells (neurons) Each neuron is connected to the others through 10000 synapses Properties of the brain It can learn, reorganize itself from experience It adapts to the environment It is robust and fault tolerant
- 10. 10 Soma Axon Axon Synapse Synapse Dendrites Dendrites Soma Two Interconnected Brain Cells (Neurons)
- 11. 11 Biology Neural Network Soma Node Dendrites Input Axon Output Synapses Weight Slow Fast Many Neurons(109) Few Neurons(~100)
- 12. 12 w1 w2 wn x1 x2 xn . . . Y Y1 Yn Y2 Inputs Weights Outputs . . . Neuron (or PE) n i iiWXS 1 )( Sf Summation Transfer Function bias Artificial Neuron )sign(y n 0i kii xw Ɵk Composed of Basic Units and Links
- 13. 13 Processing element (PE) Network architecture Hidden layers Parallel processing Network information processing Inputs Outputs Connection weights Summation function
- 14. 14 The following are the basic characteristics of neural network: • Exhibit mapping capabilities, they can map input patterns to their associated output patterns • Learn by examples • Robust • system and fault tolerant • Process the capability to generalize. Thus, they can predict new outcomes from past trends
- 15. 15 An n-dimensional input vector x is mapped into variable y by means of the scalar product and a nonlinear function mapping The inputs to unit are outputs from the previous layer. They are multiplied by their corresponding weights to form a weighted sum, which is added to the bias associated with unit. Then a nonlinear activation function is applied to it. Ɵk f weighted sum Input vector x output y Activation function weight vector w w0 w1 wn x0 x1 xn )sign(y ExampleFor n 0i kii xw bias
- 16. Several network Architecture Exists Feed forward backpropogation Recurrent Associative memory Probabilistic Self-organizing feature maps Hopfield networks 16
- 17. In a feed forward network, information flows in one direction along connecting pathways, from the input layer via hidden layers to the final output layer. There is not feedback, the output of any layers of any layer does not affect that same or preceding layer. 17
- 18. Output x１ x2 … xn bias biaswij Input layer Hidden layers Output layer … … … wjk wkl Transfer function is a sigmoid or any squashing function that is differentiable 1 1 Figure : Multi-layered Feed-forward neural network layout 18
- 19. How can I design the topology of the neural network? Before training can begin, the user must decide on the network topology by specifying the number of units in the input layer, the number of hidden layers (if more than one), the number of units in each hidden layer, and the number of units in the output layer. 19
- 20. First decide the network topology : # of units in the input layer, # of hidden layer(if>1), # of units in each hidden layer, and # of units in the output layer. Normalizing the input values for each attribute measured in the training tuples to [0.0-1.0] One input unit per domain value, each initialized to 0 Output, if for classification and more than two classes, one output unit per class is used. Once a network has been trained and its accuracy is unacceptable, repeat the training process with a different network topology or a different set of initial weights 20
- 21. Number of Hidden Layers Number of Hidden Nodes Number of output Nodes 21
- 22. 22 Output x１ x2 … wij Input Layer Hidden Layers … … … wjk wkl Output Layers
- 23. Back Propagation described by Arthur E. Bryson and Yu-Chi Ho in 1969, but it wasn't until 1986, through the work of David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams , that it gained recognition, and it led to a “renaissance” in the field of artificial neural network research. The term is an abbreviation for & quot; backwards propagation of errors & quot;. 23
- 24. Iteratively process a set of training tuples & compare the network's prediction with the actual known target value For each training tuple, the weights are modified to minimize the mean squared error between the network's prediction and the actual target value From a statistical point of view, networks perform nonlinear regression Modifications are made in the “backwards” direction: from the output layer, through each hidden layer down to the first hidden layer, hence “backpropagation” Steps Initialize weights (to small random #s) and biases in the network Propagate the inputs forward (by applying activation function) Backpropagate the error (by updating weights and biases) Terminating condition (when error is very small, etc.) 24
- 25. FEED FORWARD NETWORK activation flows in one direction only: from the input layer to the output layer, passing through the hidden layer. Each unit in a layer is connected in the forward direction to every unit in the next layer. 25
- 26. Several Nerual Network Types Is Multi Layer Perception Radial Base Function Kohenen Self Organizing Feature 26
- 27. Linear Separable: Linear inseparable: Solution? yx yx yx 27
- 28. Back propagation is a multilayer feed forward network with one layer of z-hidden units. The y output units has b(i) bias and Z-hidden unit has b(h) as bias. It is found that both the output and hidden units have bias. The bias acts like weights on connection from units whose output is always 1. The input layer is connected to the hidden layer and output layer is connected to the output layer by means of interconnection weights. 28
- 29. The architecture of back propagation resembles a multi-layered feed forward network. The increasing the number of hidden layers results in the computational complexity of the network. As a result, the time taken for convergence and to minimize the error may be very high. The bias is provided for both the hidden and the output layer, to act upon the net input to be calculated. 29
- 30. The training algorithm of back propagation involves four stages. Initialization of weights- some small random values are assigned. Feed forward- each input unit (X) receives an input signal and transmits this signal to each of the hidden units Z 1 ,Z 2 ,……Zn Each hidden unit then calculates the activation function and sends its signal Z i to each output unit. The output unit calculates the activation function to form the response of the given input pattern. Back propagation of errors- each output unit compares activation Y k with its target value T K to determine the associated error for that unit. Based on the error, the factor δ O (O=1,……,m) is computed and is used to distribute the error at output unit Y k back to all units in the previous layer. Similarly, the factor δ H (H=1,….,p) is compared for each hidden unit H j. Updating of the weights and biases 30
- 31. 31
- 32. 32
- 33. Mr.Khan 33 33
- 34. Output nodes Input nodes Hidden nodes Output vector Input vector: xi wij jIj e O 1 1 34 XIO ii ji i iji OwO Phase 1: Propagate the Input Forward
- 35. Output nodes Input nodes Hidden nodes Output vector Input vector: xi wij ))(1( jjjjj OTOOErr jk k kjjj wErrOOErr )1( 35 Phase 2: Back propagate the Error
- 36. Output nodes Input nodes Hidden nodes Output vector Input vector: xi Oj 36 ijijij OErrlww )( jjj Errl)( Phase 3: Weights/Bias Update
- 37. Output nodes Input nodes Hidden nodes Output vector Input vector: xi wij jIj e O 1 1 ))(1( jjjjj OTOOErr jk k kjjj wErrOOErr )1( ijijij OErrlww )( jjj Errl)( 37 XIO ii ji i iji OwO Oj
- 38. 38 An n-dimensional input vector x is mapped into variable y by means of the scalar product and a nonlinear function mapping The inputs to unit are outputs from the previous layer. They are multiplied by their corresponding weights to form a weighted sum, which is added to the bias associated with unit. Then a nonlinear activation function is applied to it. Ɵk f weighted sum Input vector x output y Activation function weight vector w w0 w1 wn x0 x1 xn )sign(y ExampleFor n 0i kii xw bias
- 39. The inputs to the network correspond to the attributes measured for each training tuple. Inputs are fed simultaneously into the units making up the input layer. They are then weighted and fed simultaneously to a hidden layer The number of hidden layers is arbitrary, although usually only one The weighted outputs of the last hidden layer are input to units making up the output layer, which emits the network's prediction The network is feed-forward in that none of the weights cycles back to an input unit or to an output unit of a previous layer 39
- 40. 40
- 41. 41
- 42. 42
- 43. 43
- 44. 44
- 45. 45
- 46. 46
- 47. 47
- 48. 48
- 49. 49
- 50. 50
- 51. 51
- 52. 52
- 53. 53
- 54. A process by which a neural network learns the underlying relationship between input and outputs, or just among the inputs Supervised learning For prediction type problems E.g., backpropagation Unsupervised learning For clustering type problems Self-organizing E.g., adaptive resonance theory 54
- 55. 55
- 56. Three-step process: 1. Compute temporary outputs 2. Compare outputs with desired targets 3. Adjust the weights and repeat the process 56
- 57. 57
- 58. Running Cost of Forward Back-Propagation Single Complete Hidden Layer The total running time is O(M + E) (where E is the number of edges) Normally the inputs and output unit is fixed, while we allowed to vary the number n of hidden units. Cost is o(n) Note: that this is only the cost of running forward and backward propagation on a single example. It is not the cost of training an entire network, which takes multiple epochs. It is in fact possible for the number of epochs needed to be exponential in n. 58
- 59. Efficiency of backpropagation: each epoch (one iteration through the training set) takes O(|D| * w), with |D| tuples and w weights, but # of epochs can be exponential to n, the number of inputs, in the worst case Network pruning - Simplify the network structure by removing weighted links that have the least effect on the trained network - Then perform link, unit, or activation value clustering Sensitivity analysis: assess the impact that a given input variable has on a network output. 59
- 60. 60 Weakness Long training time Require a number of parameters typically best determined empirically, e.g., the network topology or “structure.” Poor interpretability: Difficult to interpret the symbolic meaning behind the learned weights and of “hidden units” in the network Strength High tolerance to noisy data Ability to classify untrained patterns Well-suited for continuous-valued inputs and outputs Successful on an array of real-world data, e.g., hand-written letters Algorithms are inherently parallel Techniques have recently been developed for the extraction of rules from trained neural networks
- 61. In this question, I'd like to know specifically what aspects of an ANN (specifically, a Multilayer Perceptron) might make it desirable to use over an SVM? The reason I ask is because it's easy to answer the opposite question: Support Vector Machines are often superior to ANNs because they avoid two major weaknesses of ANNs: (1) ANNs often converge on local minima rather than global minima, meaning that they are essentially "missing the big picture" sometimes (or missing the forest for the trees) (2) ANNs often overfit if training goes on too long, meaning that for any given pattern, an ANN might start to consider the noise as part of the pattern. SVMs don't suffer from either of these two problems. However, it's not readily apparent that SVMs are meant to be a total replacement for ANNs. So what specific advantage(s) does an ANN have over an SVM that might make it applicable for certain situations? I've listed specific advantages of an SVM over an ANN, now I'd like to see a list of ANN advantages 61
- 62. Activation function - A mathematical function applied to a node’s activation that computes the signal strength it outputs to subsequent nodes. Activation - The combined sum of a node’s incoming, weighted signals. Backpropagation - A supervised learning algorithm which uses data with associated target output to train an ANN. Connections - The paths signals follow between ANN nodes. Connection weights - Signals passing through a connection are multiplied by that connection’s weight. Dataset - See Input pattern Epoch - One iteration through the backpropagation algorithm (presentation of the entire training set once to the network). Feedforward ANN - An ANN architecture where signal flow is in one direction only. 62
- 63. Let wxy be the connection weight between node x, in one layer and node y, in the following layer. Let ax be the net activation in node x. Let F(ax) be an activation function that accepts node x’s net activation as input – the function is applied to the net activation before node x propagates its signals onwards to the proceeding layer. Activation functions are varied; I used one of form . Let be an input vector (‘network stimuli data’) that exists in space where n equals the number of input layer nodes. The input layer takes its activation values from the raw input vector element values without having the activation function applied to them so the nth input node’s activation will be nth input vector’s value. The activation of any non-input layer node, y, in the network then is: where s is the number of nodes in the previous layer. The signal y passes on along forward-bound connections is simply, . Net activations in the output layer are run through the activation function, however these nodes do not generate further forward signals. 63
- 64. Generalization - A well trained ANN’s ability to correctly classify unseen input patterns by finding their similarities with training set patterns Input vector - See Input pattern Input pattern - An ANN’s input – there may be no pattern, per se. The pattern has as many elements as the network has input layer nodes. Maximum training error - Criteria for deciding when to stop training an ANN. Network architecture - The design of an ANN: the number of units and their pattern of connection. Output - The signal a node passes on to subsequent layers (before being multiplied by a connection weight). In the output layer, the set of each node’s output is the ANN’s output for a particular input vector. Training - Process allowing an ANN to learn. Training set - The set of input patterns used during network training. Each input pattern has an associated target output. Weights - See Connection weights. 64
- 65. 65