Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Machine learning with py torch

135 vues

Publié le

Introducing PyTorch and build Feed Forward Neural Network at Facebook Developer Circles Jakarta meetup

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Machine learning with py torch

  1. 1. Machine Learning With
  2. 2. Hi, I’m Riza
  3. 3. Agenda • Introduction PyTorch • Getting Started with PyTorch • Handling Datasets • Build Simple Neural network with PyTorch
  4. 4. Introduction
  5. 5. Scientific computing package to replace NumPy to use the power of GPU. A deep learning research platform that provide maximum flexibility and speed
  6. 6. A complete Python rewrite of Machine Learning library called Torch, written in Lua Chainer — Deep learning library, huge in NLP community, big inspiration to PyTorch Team HIPS Autograd - Automatic differentiation library, become on of big feature of PyTorch In need of dynamic execution
  7. 7. January 2017 PyTorch was born 🍼 July 2017 Kaggle Data Science Bowl won using PyTorch 🎉 August 2017 PyTorch 0.2 🚢 September 2017 fast.ai switch to PyTorch 🚀 October 2017 SalesForce releases QRNN 🖖 November 2017 Uber releases Pyro 🚗 December 2017 PyTorch 0.3 release! 🛳 2017 in review
  8. 8. Killer Features Just Python On Steroid Dynamic computation allows flexibility of input Best suited for research and prototyping
  9. 9. Summary • PyTorch is Python machine learning library focus on research purposes • Released on January 2017, used by tech companies and universities • Dynamic and pythonic way to do machine learning
  10. 10. Getting Started
  11. 11. Installa7on $ pip install pytorch torchvision -c pytorch # macos $ pip install pytorch-cpu torchvision-cpu -c pytorch # linux More: https://pytorch.org/get-started/locally/
  12. 12. Tensors Scalar Rank: 0 Dimension: () 3.14Scalar
  13. 13. Tensors Scalar import torch pi = torch.tensor(3.14) print(x) # tensor(3.1400)
  14. 14. Tensors Vector Rank: 1 Dimension: (3,) [3, 4, 8]Vector
  15. 15. Tensors Vector import torch vector = torch.Tensor(3,) print(vector) # tensor([ 0.0000e+00, 3.6893e+19, -7.6570e-25])
  16. 16. Tensors Matrix Rank: 2 Dimension: (2, 3) [[1, 2, 3], [4, 5, 6]]Matrix
  17. 17. Tensors Matrix import torch matrix = torch.Tensor(2, 3) print(matrix) # tensor([[0.0000e+00, 1.5846e+29, 2.8179e+26], # [1.0845e-19, 4.2981e+21, 6.3828e+28]])
  18. 18. Tensors Tensor Rank: 3 Dimension: (2, 2, 3) [[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]Tensor
  19. 19. Tensors Tensor import torch tensor = torch.Tensor(2, 2, 3) print(tensor) # tensor([[[ 0.0000e+00, 3.6893e+19, 0.0000e+00], # [ 3.6893e+19, 4.2039e-45, 3.6893e+19]], # [[ 1.6986e+06, -2.8643e-42, 4.2981e+21], # [ 6.3828e+28, 3.8016e-39, 2.7551e-40]]])
  20. 20. Operators import torch x = torch.Tensor(5, 3) # Randomize Tensor y = torch.rand(5, 3) # Add print(x + y) # or print(torch.add(x, y)) # Matrix Multiplication a = torch.randn(2, 3) b = torch.randn(3, 3) print(torch.mm(a, b)) https://pytorch.org/docs/stable/tensors.html
  21. 21. Working With import torch a = torch.ones(5) print(a) # tensor([1., 1., 1., 1., 1.]) b = a.numpy() print(b) # [1. 1. 1. 1. 1.] import numpy as np a = np.ones(5) b = torch.from_numpy(a) np.add(a, 1, out=a) print(a) "#[2. 2. 2. 2. 2.] print(b) #tensor([2., 2., 2., 2., 2.], dtype=torch.float64)
  22. 22. Working With GPU import torch x = torch.Tensor(5, 3) y = torch.rand(5, 3) if torch.cuda.is_available(): x = x.cuda() y = y.cuda() x + y
  23. 23. Summary • Tensor is like rubiks or multidimentional array • Scalar, Vector, Matrix and Tensor is the same with different dimension • We can use torch.Tensor() to create a tensor.
  24. 24. Autograd
  25. 25. Differen7a7on Refresher y = f(x) = 2xIF THEN dy dx = 2 IF THENy = f(x1, x2,…,xn) [ dy dx1 , dy dx2 , . . . , dy dxn ] Is the gradient of y w.r.t [x1, x2, …, xn]
  26. 26. Autograd • Calculus chain rule on steroid • Derivative of function within a function • Complex functions can be written as many compositions of simple functions • Provides auto differentiation on all tensor operations • In torch.autograd module
  27. 27. Variable • Crucial data structure, needed for automatic differentiation • Wrapper around Tensor • Records reference to the creator function
  28. 28. Variable import torch from torch.autograd import Variable x = Variable(torch.FloatTensor([11.2]), requires_grad=True) y = 2 * x print(x) # tensor([11.2000], requires_grad=True) print(y) # tensor([22.4000], grad_fn=<MulBackward>) print(x.data) # tensor([11.2000]) print(y.data) # tensor([22.4000]) print(x.grad_fn) # None print(y.grad_fn) # <MulBackward object at 0x10ae58e48> y.backward() # Calculates the gradients print(x.grad) # tensor([2.])
  29. 29. Summary • Autograd provides auto differentiation on all tensor operations, inside torch.autograd module • Variable is wrapper around Tensor that will records reference to the creator function
  30. 30. Handling DataSets
  31. 31. Dataset Collection of training examples
  32. 32. Epochs One pass of the entire dataset through your model

  33. 33. Batch A subset of training examples passed through your model at a time
  34. 34. Itera7on A single pass of a batch
  35. 35. Example 1,000 images of dataset 1 epochs Batch size of 50 Leads to 20 iterations
  36. 36. Access Dataset CIFAR10 dataset via torchvision import torch import torchvision import torchvision.transforms as transforms import matplotlib.pyplot as plt
  37. 37. Access Dataset Transform the data transform = transforms.Compose([ transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
  38. 38. Access Dataset Prepare train data trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform) print(len(trainset.train_data)) # 5000 print(trainset.train_labels[1]) # 9 = Truck image
  39. 39. Access Dataset Prepare train loader trainloader = torch.utils.data.DataLoader( trainset, batch_size=10, shuffle=True, num_workers=2)
  40. 40. Access Dataset Iterate data and train the model for i, data in enumerate(trainloader): data, labels = data print(type(data)) # <class 'torch.Tensor'> print(data.size()) # torch.Size([10, 3, 32, 32]) print(type(labels)) # <class 'torch.Tensor'> print(labels.size()) # torch.Size([10]) # Model training happens here""...
  41. 41. Datasets • COCO — large-scale object detection, segmentation, and captioning dataset.  • MNIST — handwritten digit database • Fashion-MNIST — fashion product database • LSUN — Large-scale Image Dataset  • Much more… Others
  42. 42. Summary • We can use existing dataset provided by torch and torchvision such as CIFAR10 • Dataset is training example, epochs is on pass througout the model, batch is subset of training 
 model and iteration is a single pass of one batch
  43. 43. Neural Network
  44. 44. Neural Network Input Hidden 1 Hidden 2 Output Layers
  45. 45. Neural Network A Neuron Output ip1 ip2 ip3 w1 w2 w3 ip1*w1 ip2*w2 ip3*w3 bias fn() Output = fn(w1 * ip1 + w2 * ip2 + w3* ip3 + bias) Input Vector Weights Vector Activation Function
  46. 46. Ac7va7on Func7ons Sigmoid f(x) = 1 1 + e−x
  47. 47. Ac7va7on Func7ons Tanh f(x) = ex − e−x ex + e−x
  48. 48. Ac7va7on Func7ons Rectified Linear Unit f(x) = max(x,0)
  49. 49. Summary • Neural net is a collection of neurons that related to 
 each other consist of input, weight, bias and output. • To generate an output we need to activate it using activation function such as Sigmoid, tanh or ReLU.
  50. 50. Our First Neural Network
  51. 51. Feed Forward NN The Iris Classify flower based on it’s structure
  52. 52. Feed Forward NN The Model
  53. 53. Feed Forward NN The Dataset Table 1 sepal_length_cm sepal_width_cm petal_length_cm petal_width_cm class 5.1 3.5 1.4 0.2 Iris-setosa 4.9 3.0 1.4 0.2 Iris-setosa 7.0 3.2 4.7 1.4 Iris-versicolor 6.4 3.2 4.5 1.5 Iris-versicolor 6.9 3.1 4.9 1.5 Iris-versicolor 6.4 2.8 5.6 2.2 Iris-virginica 6.3 2.8 5.1 1.5 Iris-virginica 6.1 2.6 5.6 1.4 Iris-virginica
  54. 54. The Iris import torch import torch.nn as nn import matplotlib.pyplot as plt from torch.autograd import Variable from data import iris Import things
  55. 55. The Iris class IrisNet(nn.Module): def "__init"__(self, input_size, hidden1_size, hidden2_size, num_classes): super(IrisNet, self)."__init"__() self.layer1 = nn.Linear(input_size, hidden1_size) self.act1 = nn.ReLU() self.layer2 = nn.Linear(hidden1_size, hidden2_size) self.act2 = nn.ReLU() self.layer3 = nn.Linear(hidden2_size, num_classes) def forward(self, x): out = self.layer1(x) out = self.act1(out) out = self.layer2(out) out = self.act2(out) out = self.layer3(out) return out model = IrisNet(4, 100, 50, 3) print(model) Create Module and Instance
  56. 56. The Iris batch_size = 60 iris_data_file = 'data/iris.data.txt' train_ds, test_ds = iris.get_datasets(iris_data_file) train_loader = torch.utils.data.DataLoader(dataset=train_ds, batch_size=batch_size, shuffle=True) test_loader = torch.utils.data.DataLoader(dataset=test_ds, batch_size=batch_size, shuffle=True) DataLoader
  57. 57. Loss Func7on Output/Prediction Actual Result Loss Function Loss Score Input
  58. 58. Loss Func7on In PyTorch • L1Loss • MSELoss • CrossEntropyLoss • BCELoss • SoftMarginLoss • More: https://pytorch.org/docs/stable/nn.html?#loss-functions
  59. 59. Loss Func7on CrossEntropyLoss Measures the performance of a classification model whose output is a probability value between 0 and 1. 🍎 🍌 🍍 Prediction 0.02 0.88 0.1 Actual 🍌 Loss Score 0.98 0.12 0.9
  60. 60. Op7mizer Func7on Output/Prediction Actual Result Loss Function Loss Score Input Calculate Gradients Optimizer Iteration
  61. 61. Op7mizer Func7on In PyTorch • Socastic Gradient Descend (SGD) • Adam • Adadelta • RMSprop
  62. 62. Back to Iris Let’s Train The Neural Network net = IrisNet(4, 100, 50, 3) # Loss Function criterion = nn.CrossEntropyLoss() # Optimizer learning_rate = 0.001 optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate, nesterov=True, momentum=0.9, dampening=0)
  63. 63. Back to Iris Let’s Train The Neural Network num_epochs = 500 for epoch in range(num_epochs): train_correct = 0 train_total = 0 for i, (items, classes) in enumerate(train_loader): # Convert torch tensor to Variable items = Variable(items) classes = Variable(classes)
  64. 64. Back to Iris Let’s Train The Neural Network net.train() # Training mode optimizer.zero_grad() # Reset gradients from past operation outputs = net(items) # Forward pass loss = criterion(outputs, classes) # Calculate the loss loss.backward() # Calculate the gradient optimizer.step() # Adjust weight based on gradients train_total += classes.size(0) _, predicted = torch.max(outputs.data, 1) train_correct += (predicted "== classes.data).sum() print('Epoch %d/%d, Iteration %d/%d, Loss: %.4f' %(epoch+1, num_epochs, i+1, len(train_ds)"//batch_size, loss.data[0]))
  65. 65. Back to Iris Let’s Train The Neural Network net.eval() # Put the network into evaluation mode train_loss.append(loss.data[0]) train_accuracy.append((100 * train_correct / train_total)) # Record the testing loss test_items = torch.FloatTensor(test_ds.data.values[:, 0:4]) test_classes = torch.LongTensor(test_ds.data.values[:, 4]) outputs = net(Variable(test_items)) loss = criterion(outputs, Variable(test_classes)) test_loss.append(loss.data[0]) # Record the testing accuracy _, predicted = torch.max(outputs.data, 1) total = test_classes.size(0) correct = (predicted "== test_classes).sum() test_accuracy.append((100 * correct / total))
  66. 66. Back to Iris Let’s Train The Neural Network import torch import torch.nn as nn import matplotlib.pyplot as plt from torch.autograd import Variable from data import iris # Create the module class IrisNet(nn.Module): def "__init"__(self, input_size, hidden1_size, hidden2_size, num_classes): super(IrisNet, self)."__init"__() self.layer1 = nn.Linear(input_size, hidden1_size) self.act1 = nn.ReLU() self.layer2 = nn.Linear(hidden1_size, hidden2_size) self.act2 = nn.ReLU() self.layer3 = nn.Linear(hidden2_size, num_classes) def forward(self, x): out = self.layer1(x) out = self.act1(out) out = self.layer2(out) out = self.act2(out) out = self.layer3(out) return out # Create a model instance model = IrisNet(4, 100, 50, 3) print(model) # Create the DataLoader batch_size = 60 iris_data_file = 'data/iris.data.txt' train_ds, test_ds = iris.get_datasets(iris_data_file) print('# instances in training set: ', len(train_ds)) print('# instances in testing/validation set: ', len(test_ds)) train_loader = torch.utils.data.DataLoader(dataset=train_ds, batch_size=batch_size, shuffle=True) test_loader = torch.utils.data.DataLoader(dataset=test_ds, batch_size=batch_size, shuffle=True) # Model net = IrisNet(4, 100, 50, 3) # Loss Function criterion = nn.CrossEntropyLoss() # Optimizer learning_rate = 0.001 optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate, nesterov=True, momentum=0.9, dampening=0) # Training iteration num_epochs = 500 train_loss = [] test_loss = [] train_accuracy = [] test_accuracy = [] for epoch in range(num_epochs): train_correct = 0 train_total = 0 for i, (items, classes) in enumerate(train_loader): # Convert torch tensor to Variable items = Variable(items) classes = Variable(classes) net.train() # Training mode optimizer.zero_grad() # Reset gradients from past operation outputs = net(items) # Forward pass loss = criterion(outputs, classes) # Calculate the loss loss.backward() # Calculate the gradient optimizer.step() # Adjust weight/parameter based on gradients train_total += classes.size(0) _, predicted = torch.max(outputs.data, 1) train_correct += (predicted "== classes.data).sum() print('Epoch %d/%d, Iteration %d/%d, Loss: %.4f' %(epoch+1, num_epochs, i+1, len(train_ds)"//batch_size, loss.data[0])) net.eval() # Put the network into evaluation mode train_loss.append(loss.data[0]) train_accuracy.append((100 * train_correct / train_total)) # Record the testing loss test_items = torch.FloatTensor(test_ds.data.values[:, 0:4]) test_classes = torch.LongTensor(test_ds.data.values[:, 4]) outputs = net(Variable(test_items)) loss = criterion(outputs, Variable(test_classes)) test_loss.append(loss.data[0]) # Record the testing accuracy _, predicted = torch.max(outputs.data, 1) total = test_classes.size(0) correct = (predicted "== test_classes).sum() test_accuracy.append((100 * correct / total))
  67. 67. Summary • Created feed forward neural network to predict a type of flower • Start from read the dataset and dataloader • Choose a loss function and optimizer • Train and evaluation
  68. 68. What’s Next?! • pytorch.org/tutorials • course.fast.ai • coursera.org/learn/machine-learning • kaggle.com/datasets
  69. 69. That’s All From me github.com/rizafahmi slideshare.net/rizafahmi rizafahmi@gmail.com twi8er.com/rizafahmi22 facebook.com/rizafahmi

×