Slide shown at the ML meetup Victoria 2020 about the AWS opensource project "Amazon SageMaker for Battlesnake AI" https://github.com/awslabs/sagemaker-battlesnake-ai

1. A battlesnake reinforcement
learning starter pack
Jonathan Chung
Xavier Raffin
©2020 Amazon Web Services, Inc. or its affiliates, All rights reserved

2. Battlesnake

3. Battlesnake

4. Battlesnake reinforcement learning starter pack
Agent Environment
Training your own battlesnake
reinforcement learning model
2. Reinforcement learning module
Build custom rules on top of an
existing model
3. Heuristics module
One-click deployment of an
existing model
1. One-click deploy module

5. Reinforcement learning module
Agent Environment

6. Reinforcement learning module
Agent Environment
Actions
Rewards
State

7. Reinforcement learning module
Agent Environment
Actions
Rewards
State
Accelerate, move left, move right
Road image
#Kms driven

8. Reinforcement learning module
Agent Environment
Actions
Rewards
State
Accelerate, move left, move right
Road image
#Kms driven

9. Reinforcement learning module
Agent Environment
Actions
Rewards
State
Position of pieces
Position of all pieces
Winning/losing

10. Reinforcement learning module
Agent Environment
Actions
Rewards
State
Direction of movement
Pos. snakes&food
Reward?

11. Reinforcement learning module
Direction of movement
Pos. snakes+food
Reward per snake
Multiagent reinforcement learning
environment
snakes

12. Reinforcement learning module
Training routine for the module (deep Q learning)
for epi in episodes:
state = env.reset()
while agent.is_alive():
if prob < eps:
action = agent.get_random_action()
else:
action = agent.get_next_best_action(state)
next_state, reward = env.step(action)
memory.append(next_state, state, action, reward)
agent.learn(memory)
Agent Environment
Actions
Rewards
State

13. Reinforcement learning module
Training routine for the module (deep Q learning)
for epi in episodes:
state = env.reset()
while agents.agents_alive() > 1:
actions = []
for agent in agents:
if prob < eps:
action = agent.get_random_action()
else:
action = agent.get_next_best_action(state)
actions.append(action)
next_state, reward = env.step(actions)
memory.append(next_state, state, actions, reward)
for agent in agents:
agent.learn(memory)
Agent
Environment
Actions
Rewards
State
Agent
Agent

14. Reinforcement learning module
Environment:
Agent
Environment
Actions
Rewards
State
Agent
Agent

15. Reinforcement learning module
State representation:
the positions of the snakes and food
Agent
Environment
Actions
Rewards
State
Agent
Agent

16. Reinforcement learning module
State representation:
the positions of the snakes and food
-1 -1 -1 -1 -1 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 0 1 0 0 -1
-1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1
-1 0 5 1 1 -1
-1 0 0 0 1 -1
-1 0 0 0 1 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1
-1 1 0 0 0 -1
-1 5 0 1 0 -1
-1 0 0 5 0 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
Food position Target snake position Other snake position
Agent
Environment
Actions
Rewards
State
Agent
Agent

17. Reinforcement learning module
State representation:
the positions of the snakes and food
-1 -1 -1 -1 -1 -1
-1 0 0 0 0 -1
-1 0 0 1 0 -1
-1 0 0 5 0 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1
-1 1 5 1 1 -1
-1 5 0 0 1 -1
-1 0 0 0 1 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
Food position Snake position Other snake positions
Agent
Environment
Actions
Rewards
State
Agent
Agent
-1 -1 -1 -1 -1 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 0 1 0 0 -1
-1 -1 -1 -1 -1 -1

18. Reinforcement learning module
State representation:
the positions of the snakes and food
-1 -1 -1 -1 -1 -1
-1 1 0 0 0 -1
-1 5 0 0 0 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1
-1 0 5 1 1 -1
-1 0 0 1 1 -1
-1 0 0 5 1 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
Food position Snake position Other snake positions
Agent
Environment
Actions
Rewards
State
Agent
Agent
-1 -1 -1 -1 -1 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 0 1 0 0 -1
-1 -1 -1 -1 -1 -1

19. Reinforcement learning module
Environment: rewards
+1 every turn the snake lives
Agent
Environment
Actions
Rewards
State
Agent
Agent

20. Reinforcement learning module
Agent
Agent??
Agent
Environment
Actions
Rewards
State
Agent
Agent

21. Reinforcement learning module
Agent
Neural
network
Agent
Environment
Actions
Rewards
State
Agent
Agent

22. Reinforcement learning module
Agent
Neural
network
15
10
Predicted total expected
reward
13
0

23. Reinforcement learning module
Learning
Neural
network
t = 0
t = 1
Neural
network
New reward = 1
48
84
New predicted
total expected reward
1510
130
Predicted total expected reward

24. Reinforcement learning module
Learning
New predicted total expected reward ≃ neural network (new reward, predicted total expected reward)
t = 0
Predicted total expected rewards
t = 1 New predicted total expected rewards
New reward = 1

25. Reinforcement learning module
Learning
- Total Expected reward (Q)
Qupdated(st) ← Qold(st) + ⍺ (reward + ɣ ·[max Q(st+1) – Q(st)])
Qt + Reward = 1 → Qt+1

26. Reinforcement learning module
Customization opportunities
• Environment representations
• Neural network design
• Rewards design

27. -1 -1 -1 -1 -1 -1
-1 1 0 0 0 -1
-1 5 0 1 0 -1
-1 0 0 5 0 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
Reinforcement learning module
Environment representation:
-1 -1 -1 -1 -1 -1
-1 0 5 1 1 -1
-1 0 0 0 1 -1
-1 0 0 0 1 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1
-1 0 0 0 0 -1
-1 0 0 1 0 -1
-1 0 0 0 0 -1
-1 0 1 0 0 -1
-1 -1 -1 -1 -1 -1

28. -1 -1 -1 -1 -1 -1
-1 1 0 0 0 -1
-1 5 0 1 0 -1
-1 0 0 5 0 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
Reinforcement learning module
Neural network design:
-1 -1 -1 -1 -1 -1
-1 0 5 1 1 -1
-1 0 0 0 1 -1
-1 0 0 0 1 -1
-1 0 0 0 0 -1
-1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1
-1 0 0 0 0 -1
-1 0 0 1 0 -1
-1 0 0 0 0 -1
-1 0 1 0 0 -1
-1 -1 -1 -1 -1 -1
Snake
health Snake ID
Turn
count
Neural
network

29. Reinforcement learning module
Rewards design
• Surviving another turn
• Eating food
• Starving
• Winning the game
• Losing the game
• Hitting a
wall/snake/yourself
• Performing a forbidden
move
• Eating another snake
• Forcing another snake to
hit your body

30. Reinforcement learning module
Amazon Sagemaker

31. Reinforcement learning module
Amazon Sagemaker

32. Reinforcement learning module
Amazon Sagemaker
Agent
Environment
Actions
Rewards
State
Agent
Agent Snake model
Training

33. Reinforcement learning module
Amazon Sagemaker
• Neural network
parameters
• Learning parameters
• Environment
configurations
Agent
Environment
Actions
Rewards
State
Agent
Agent Best snake
Hyper parameter
optimisation

34. Reinforcement learning module
Amazon Sagemaker
-1 -1 -1 -1 -1 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 0 0 0 0 -1
-1 0 1 0 0 -1
-1 -1 -1 -1 -1 -1
Agent
Environment
Actions
Rewards
State
Agent
Agent
0 0 0 0
0 0 0 0
0 0 0 0
0 1 0 0
Snake with
Border
Snake without
border

35. Reinforcement learning module
Amazon Sagemaker
Agent
Environment
Actions
Rewards
State
Agent
Agent

36. Battlesnake reinforcement learning starter pack
Agent Environment
Training your own battlesnake
reinforcement learning model
2. Reinforcement learning module
Build custom rules on top of an
existing model
3. Heuristics module
One-click deployment of an
existing model
1. One-click deploy module

37. Custom rules with the heuristic module
Provides a starting point for you to build upon
Direction of movement
Pos. snakes+food
Reward?
Trained AI snake model

38. Custom rules with the heuristic module
Provides a starting point for you to build upon
Direction of movement
Pos. snakes+food
Reward?
Trained AI snake model

39. Custom rules with the heuristic module
Code

40. Custom rules with the heuristic module
Code

41. Custom rules with the heuristic module
Situation simulation

42. Battlesnake reinforcement learning starter pack
Agent Environment
Training your own battlesnake
reinforcement learning model
2. Reinforcement learning module
Build custom rules on top of an
existing model
3. Heuristics module
One-click deployment of an
existing model
1. One-click deploy module

43. One-click deployment with sagemaker
• After you train your own model
• After writing your custom rules
• Using the existing pretrained snake

44. One-click deployment with sagemaker