This is a deep-dive straight into the guts of running a low-latency multiplayer game, such as a first-person shooter, on a global scale. We dive into architectures that enable you to split apart your back-end APIs from your game servers, and Auto Scale them independently. See how to run game servers in multiple AWS regions such as China and Frankfurt, and integrate them with your central game stack. We'll even demo this in action, using AWS CloudFormation and Chef to deploy Unreal Engine game servers. In the second half, hear from PlayFab, who built the backend for the Top-10 free-to-play PC shooter Loadout. PlayFab reveals details about their architecture, including AWS Elastic Beanstalk setup, Amazon DynamoDB and Amazon RDS patterns, data sharding, and use of multiple Availability Zones. Finally, PlayFab highlights challenges they faced when deploying to AWS China, and how they solved them.

1. November 14, 2014 | Las Vegas, NV
Nate Wiger, Principal Gaming Solutions Architect, AWS
James Gwertzman, CEO, PlayFab

2. [Game] Online Problems Continue; Features Disabled as [Game Studio] Works on Fixes
“We are working hard to get everyone online as soon as possible,” [Game Studio] says.
Days later, [Game Studio] leaving disconnected players stranded
[Game] has a great online mode –if you can get online, that is
“This is total bull**** I paid for a preorder and I can’t even connect WTF [Game Studio]??”

4. http://bit.ly/verizon-latencyhttp://bit.ly/superdata-latency

7. Tokyo
Oregon
Frankfurt
Virginia

8. VPC Subnet
VPC Subnet
Availability Zone
A
Availability Zone B
VPC Subnet
VPC Subnet
Auto Scaling group
WEB
VPC Subnet
WEB
JOBS

9. Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
EC2
EC2
EC2

10. Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over EC2
EC2

11. Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over
⑦ Write via HTTP API
EC2
EC2

12. VPC Private Subnet
VPC Public Subnet
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME

13. http://bit.ly/ec2-enhanced-nethttp://bit.ly/linux- tuning

14. VPC Private Subnet
VPC Private Subnet
Availability Zone A
Availability Zone B
10.1.0.13
10.2.0.16
TCP 5672
rabbitmq-node1
rabbitmq-node2
TCP 4369
& 25672

15. https://www.rabbitmq.com/clustering.htmlhttps://www.rabbitmq.com/ha.html

16. 5672rabbitmq-node1rabbitmq-node2

17. VPC Private Subnet
VPC Public Subnet
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
Auto Scaling group

18. VPC Private Subnet
VPC Public Subnet
Availability Zone A
GAME GAME GAME
Auto Scaling group

19. UnrealServerunreal_linux.template.jsontruemy-bucket-name Unreal-123.zipsshkeyname1.2.3.4/24 MyPass123

23. Tokyo
Oregon
Frankfurt
Virginia
?
?

24. VPC Subnet
Availability Zone A Availability Zone B
VPC Subnet
Auto Scaling group
WEB WEB
Oregon
Tokyo
VPC Subnet
JOBS

26. VPC Private Subnet
VPC Public Subnet
Availability Zone A
S3 Bucket
Amazon CloudWatch Logs
GAME GAME GAME

27. VPC Private Subnet
VPC Public Subnet
Availability Zone A
Amazon
Kinesis
S3 Bucket
Amazon Redshift
GAME GAME GAME

29. China
Oregon
Frankfurt
Auth
China
Leaderboard

31. +
Tools for operating & monetizing live games
Complete set of cloud-basedbackend services
Eco-system of pre-integrated third-party services
.. and more
+

35. AWS cloud: PlayFab Web Services
US
EU
Hardware Game Servers
Hardware Game Servers
Game Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2 (API handling)
Matchmaker
Instance
Instance
Game Server Monitor
DynamoDB
Amazon RDS
Amazon S3
Elastic Load Balancing
Amazon Redshift
Reports service
Instance
Logs
Matchmaker (Secondary)
Instance
US
EU
SA
AUS
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers

36. Client
Server
Authentication
Title-Wide Data Management
Shared Group Data
Authentication
AddUsernamePassword
GetCatalogItems
AddSharedGroupMembers
AuthenticateSessionTicket
LoginWithAndroidDeviceID
GetStoreItems
CreateSharedGroup
Account Management
LoginWithFacebook
GetTitleData
GetSharedGroupData
GetUserAccountInfo
LoginWithGameCenter
GetTitleNews
RemoveSharedGroupMembers
SendPushNotification
LoginWithGoogleAccount
AddUserVirtualCurrency
UpdateSharedGroupData
Player Data Management
LoginWithIOSDeviceID
Player Item Management
Server-Side Game Logic
GetLeaderboard
LoginWithPlayFab
ConsumeItem
GetLogicServerUrl
GetLeaderboardAroundUser
LoginWithSteam
GetUserInventory
GetUserData
RegisterPlayFabUser
RedeemCoupon
GetUserInternalData
SendAccountRecoveryEmail
SubtractUserVirtualCurrency
GetUserReadOnlyData
Account Management
UnlockContainerItem
GetUserStatistics
GetAccountInfo
StartPurchase
UpdateUserData
GetPlayFabIDsFromFacebookIDs
PayForPurchase
UpdateUserInternalData
GetUserCombinedInfo
ConfirmPurchase
UpdateUserReadOnlyData
LinkFacebookAccount
PurchaseItem
UpdateUserStatistics
LinkGameCenterAccount
Friend List Management
Title-Wide Data Management
LinkSteamAccount
AddFriend
GetCatalogItems
UnlinkFacebookAccount
GetFriendsList
GetTitleData
UnlinkGameCenterAccount
RemoveFriend
SetTitleData
UnlinkSteamAccount
SetFriendTags
Player Item Management
UpdateEmailAddress
IOS-Specific APIs
AddUserVirtualCurrency
UpdatePassword
RegisterForIOSPushNotification
GetUserInventory
UpdateUserTitleDisplayName
ValidateIOSReceipt
GrantItemsToUsers
Player Data Management
Matchmaking APIs
SubtractUserVirtualCurrency
GetFriendLeaderboard
GetCurrentGames
Matchmaking APIs
GetLeaderboard
GetGameServerRegions
NotifyMatchmakerPlayerLeft
GetLeaderboardAroundCurrentUser
Matchmake
RedeemMatchmakerTicket
GetUserData
StartGame
Steam-Specific APIs
GetUserReadOnlyData
Android-Specific APIs
AwardSteamAchievement
GetUserStatistics
AndroidDevicePushNotificationRegistration
UpdateUserData
ValidateGooglePlayPurchase
UpdateUserStatistics
Analytics
LogEvent

37. API Request:
Response from Web Services:

38. Game Client
Amazon Route 53
(3.playfabapi.com)
API Request:
Elastic Load
Balancing
Logs
Amazon Redshift
Amazon S3 bucket
Reports service
Logs
Instance
Logs include title-specific endpoint for analytics
AWS wildcard DNS (*.playfabapi.com)
Log event:

39. API NAME
TOTAL CALLS
AVERAGE BACKEND TIME (S)
AVERAGE RESPONSE TIME (MS)
AVERAGE REQUEST TIME (MS)
TOTAL REQUEST TIME (M)
AVERAGE RECIEVED (BYTES)
AVERAGE SENT (BYTES)
TOTAL ERRORS
adminapi_userinfo
93,081,685
0.03
0.04
0.04
59.83
0
452
1979
gameserver_usercustomdatareadonly
1,582,348
0.06
0.04
0.04
1.06
42
902
290
matchmaker_userinfo
1,138,842
0.20
0.04
0.04
0.81
0
10046
319
matchmaker_playerjoined
1,138,247
0.01
0.04
0.04
0.79
64
2
93
matchmaker_playerleft
1,136,797
0.01
0.04
0.04
0.80
64
2
400
matchmaker_authuser
736,607
0.01
0.03
0.04
0.52
0
48
107
gameserver_usercustomdata
297,735
0.19
0.04
0.04
0.20
0
4080
0
gameserver_usercustomdatainternal
297,469
0.08
0.04
0.04
0.20
0
237
2

40. Instance
Health Check
Auto Scaling group
Auto Scaling service
Instance
Matchmaker (Primary)
Instance
Amazon Route 53
(matchmaker.playfabapi.com)
Elastic Load
Balancing
Matchmaker (Secondary)
1.Auto Scaling is set to keep two instances up.
2.Elastic Beanstalk Health Check terminates bad instance.
3.Amazon Route 53 fails over all traffic to secondary server.
4.Auto Scaling detects failure and starts new instance.

41. Amazon EC2 (API handling)
Elastic Load Balancing
CloudWatch
Health Check
Email Alerts
On-call Engineer Mobile Alerts

42. Amazon EC2 API Server
Server SSD
Amazon Redshift
Local Reports service
Local event logs
Instance
Events: logins and purchases
Reports service
Logs
Customer’s game management dashboard
Analytics
Server Logs
RabbitMQmessage queue
Logs
Instance
Logstashservice
Logstash
Instance
ElasticSearchInstance
Instance

43. DynamoDB
Amazon RDS
Amazon S3
Player data:
•Accounts
•Profile
•Inventory
•Save data
•Player stats
•Player currency values
•Auditing
•Log files
•Game replays
Amazon Redshift
•Analytics data

44. Amazon DynamoDB
•Player data: accounts, profile, inventory, save data
User:
UserID: Hash
TitleID: range key
-DisplayName
-Linked accounts
-…
Custom user data:
UserID: Hash
TitleID: range key
-Key/value 1
-Key/value 2
-…
Inventory:
UserID: Hash
ItemID: range key
-titleID
-Catalog Item
-…
…

45. AZ 1
AZ 2
PlayFab Web Services
PlayFab Web Services
N. California
AZ 1
AZ 2
PlayFab Web Services
PlayFab Web Services
Oregon
AZ 1
PlayFab Web Services
Beijing
•Original deployment: us-west-1
•Expanded to us-west-2 in July
•Coming soon: cn-north-1
Amazon EC2 Game Server
Singapore
Game Client
Instance

46. Player accounts are shared across all regions:
•Cross-regional wrapper for user.getsets order of databases and regions to check
•user.getthen checks for player account data
•High availability, transparent to customers and players
// Returns the first matching player record, searching DynamoDB and SimpleDB
// in both regions in the specified order.
// Updates the player record in the store and region from which it was loaded.
// Creates a player record in the default store and region.

48. AWS cloud: PlayFab Web Services
US
EU
Hardware Game Servers
Hardware Game Servers
Game Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2 (API handling)
Matchmaker
Instance
Instance
Game Server Monitor
DynamoDB
Amazon RDS
Amazon S3
Elastic Load Balancing
Amazon Redshift
Reports service
Instance
Logs
Matchmaker (Secondary)
Instance
US
EU
SA
AUS
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers
Amazon EC2 Game Servers

49. http://bit.ly/awsevalshttp://playfab.com