Get behind the keyboard for an immersive experience with autonomous cars at Robocar Rally 2018. In this workshop you will get hands-on-experience with machine learning and robo-racing cars. Developers with no prior machine learning or robotic experience will learn new skills and apply their knowledge in a fun and exciting way. You will join a pit crew where you will build, train, and race a robo-racing car! Start your engines, the race is on!

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer – The MGM Speedway
re:Invent 2018
Speaker Name
Job Title
Company/Org Name
A I M 2 0 6

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Introducing AWS DeepRacer ~ 5 min.
• Reinforcement Learning – an introduction ~ 15 min.
• Lab 1 – Pre-Season: Building a Reward Function for AWS DeepRacer ~ 30 min.
• AWS DeepRacer Service and Advanced RL ~ 15 min.
• Lab 2 – Qualifiers: Create an AWS DeepRacer RL Model ~ 22 min.
• AWS DeepRacer, under the hood ~ 10 min.
• AWS DeepRacer League and MGM Speedway ~ 10 min.

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer Origin
How can we put
Reinforcement Learning
in the hands of all
developers? literally

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Robotic autonomous
race car
Announcing AWS DeepRacer: An exciting way for developers to get hands-on
experience with reinforcement learning
Racing LeagueVirtual simulator, to
train and experiment

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reinforcement Learning in the broader AI context
Reinforcement
Learning
Supervised
Learning
Unsupervised
Learning

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Machine Learning Overview
SUPERVISED UNSUPERVISED REINFORCEMENT

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reinforcement Learning use cases
AUTONOMOUS CARS FINANCIAL TRADING DATACENTER COOLINGFLEET LOGISTICS

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is Reinforcement Learning?

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How does training happen? VALUE FUNCTION
POLICY FUNCTION

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RL Algorithms – Value Approximation
CHALLENGE Explore all (state, action, outcome) combinations in a real
world race to create the value function

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RL Algorithms – Value Approximation
CHALLENGE Explore all (state, action, outcome) combinations in a real
world race to create the value function.
VERDICT Not possible
SOLUTION Approximate the value function using supervised learning.
Use data from exploration to create a model capable of estimating the
likely cumulative reward for all possible actions in a state.
The action with the highest cumulative reward,
is the best action.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RL Algorithms – Policy Approximation
ALTERNATIVE Try to directly map actions to
states without using a value function.
HOW Use supervised learning to train a policy
function model that returns the best action
for any state.
How do we know this is a good action?
Because it gave the highest reward. Keep
improving our model in the direction of the
highest reward.
FAMOUS EXAMPLE
Dota2 Challenge - OpenAI Five

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer Console

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lab 1 – Reward Function Constructors Championship
OBJECTIVE Console overview and take part in the reward function constructors championship
TIME 30 min.
1. Find the lab content here:
https://github.com/aws-samples/aws-deepracer-workshops
2. Select: Lab 1

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. AWS Cloud
AWS
DeepRacer
NAT gateway
VPC
AWS DeepRacer
Models
Simulation
video
Metrics
AWS DeepRacer Simulation Architecture

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reinforcement Learning Algorithms Compared
Value Approximation Policy Approximation
Advantages
More stable performance when it works, and tends to
converge on global optimum
Effective in continuous action spaces, can learn stochastic policies,
and faster convergence
Disadvantages
Difficult to converge if too many (state, action)
combinations, slower convergence in general, and can’t
learn stochastic properties
Typically converges to a local rather than global optimum, high
variance in estimating the gradient adversely affects stability, and
evaluating a policy is generally inefficient
Examples Q-Learning, Deep Q Network, Deep Double Q Network Policy Gradient, Proximal Policy Optimization (PPO)

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer Neural Network Architecture
An overview of the network architecture that AWS DeepRacer uses:
Output

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lab 2 – Create an AWS DeepRacer RL Model
• Objective: Start training your AWS DeepRacer RL Model for the MGM
Speedway
• Time: 22 min.
1. Find the lab content here:
https://github.com/aws-samples/aws-deepracer-workshops
2. Select: Lab 2

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer Car Specifications
CAR 18th scale 4WD with monster truck chassis
CPU Intel Atom™ Processor
MEMORY 4GB RAM
STORAGE 32GB (expandable)
WI-FI 802.11ac
CAMERA 4 MP camera with MJPEG
DRIVE BATTERY 7.4V/1100mAh lithium polymer
COMPUTE BATTERY 13600mAh USB-C PD
SENSORS Integrated accelerometer and gyroscope
PORTS 4x USB-A, 1x USB-C, 1x Micro-USB, 1x HDMI
SOFTWARE Ubuntu OS 16.04.3 LTS, Intel® OpenVINO™
toolkit, ROS Kinetic

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ROS Msg Node
Stored File
ROS Nodes
Model
Optimizer
Video
M-JPEG
Web Server
Video
Inference
Results
Web
Server
Publisher
Autonomous
Drive
Control
Node
Optimized
Model
Media engine
Camera
Model
Inference
engine
Manual
Drive
Navigation
Node
Servo & Motor
AWS DeepRacer Software Architecture

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simulation-to-Real Domain Transfer
SIM-to-REAL CHALLENGE
Train model using simulated images, but race
car using real world images
STRATEGIES
Environment Control
Domain Randomization
Modularity and Abstraction

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer League
• The world’s first global, autonomous racing league for developers

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Participating at the MGM Speedway
Location MGM Grand Garden Arena
Date and Time
Wednesday Nov 28 | 11:30 am – Midnight
Thursday Nov 29 | 11:30 am – 5:30 pm
Race, Learn, Win Prizes and More at the
2018 re:Invent Speedway!

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer Next Steps
AT RE:INVENT
Finish training your model and go to MGM Speedway.
Collect your AWS DeepRacer at the swag counter in the Venetian. Included is a voucher
for shipping, if you choose, that can ONLY be used at the Venetian FedEx.
AFTER RE:INVENT
The account you received today will expire on 12/1/2018.
Sign up for whitelisting access at https://aws.amazon.com/deepracer
Use Amazon SageMaker RL Notebook to continue training your AWS DeepRacer RL
models.
KEEP TRAINING AND KEEP RACING!

34. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Speaker Name
Contact information

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.