Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Julien Simon
Principal Technical Evangelist, AWS...
Docker on Amazon Web Services
Amazon EC2 Container Service (ECS)
• https://aws.amazon.com/ecs/
• Launched in 04/2015
• No ...
ECS adoption
https://www.datadoghq.com/blog/3-clear-trends-in-ecs-adoption/
November 2016
Companies
Selected ECS customers
Container Partners
https://aws.amazon.com/containers/partners/
ECS Scheduling
The problem
Given a certain amount of
computing power and memory,
how can we best manage
an arbitrary number of apps
runni...
Amazon ECS: Under the Hood
ALB ALB
AZ 1 AZ 2
user / scheduler
https://github.com/aws/amazon-ecs-agenthttp://www.allthingsd...
Scheduling on ECS: two options so far
1. Let ECS handle scheduling through Services
• Task Definition
• ECS transcription ...
ECS Placement Engine
Placement Engine: giving developers more
control
ALB ALB
AZ 1 AZ 2
user / scheduler
Placement Engine
Constraints
& Strateg...
Placement Constraints
Name Example
AMI ID attribute:ecs.ami-id == ami-eca289fb
Availability Zone attribute:ecs.availabilit...
Example: Constraint on Instance Family/Type
Example: Constraint on Availability Zone
Example: Combining Multiple Constraints
Placement Strategies
Binpacking Spread Affinity Distinct Instance
Placement Strategy Chaining
Spread tasks across Zones
and Binpack within each Zone
Placing Tasks
Anatomy of Task Placement
Cluster Constraints
Custom Constraints
Placement Strategies
Apply Filter
Satisfy CPU, memory, an...
g2.2xlarge t2.small g2.2xlarge g2.2xlarge
Placement: Targeting Instance Type
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge
t2.small
t2.small t2.medium
us-east-1aus-east-1d
Plac...
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2...
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2...
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge t2.small
us-east-1aus-east-1d
g2.2xlarge t2.medium
t2...
Running a Service
t2.medium t2.small t2.small
us-east-1aus-east-1d
t2.medium t2.micro t2.small
us-east-1c
Placement: Multiple Services on a ...
t2.medium g2.2xlarge t2.micro t2.small
t2.small t2.small g2.2xlarge t2.small
t2.small t2.small
g2.2xlarge t2.small
Placeme...
Event Stream & Blox
Amazon ECS: Under the Hood
ALB ALB
AZ 1 AZ 2
user / scheduler
Placement Engine
Event Stream
Consuming Real-time Events
Handling ECS events with Blox
scheduler cluster state service
https://blox.github.io/
Amazon ECS: Under the Hood
ALB ALB
AZ 1 AZ 2
user / scheduler
Scheduler
Cluster State Service
Placement Engine
Event Stream
t2.small t2.small t2.small
Blox: Daemon Scheduler
t2.small t2.small t2.small
scheduler cluster state service
t2.small t2.s...
Demo: Deploying Blox on AWS
Creating Clusters
Create an ECS cluster for Blox
CF template: https://github.com/blox/blox/blob/dev/deploy/aws/conf/cloudf...
Listing Task Definitions
Grab the ARN for an nginx Task Definition, which the
Daemon Scheduler will manage on ‘WebCluster’...
Creating an Environment
$ ./blox-create-environment.py --environment WebEnvironment --cluster
WebCluster --task-definition...
Listing Environments
$ ./blox-list-environments.py --stack Blox --apigateway --region eu-central-1
== Blox Demo CLI - List...
Creating a Deployment
$ ./blox-create-deployment.py --environment WebEnvironment
--deployment-token "17248257-08ec-4438-88...
Listing Deployments
$ ./blox-list-deployments.py --environment WebEnvironment --stack Blox
--apigateway --region eu-centra...
Scaling a Deployment
$ ecs-cli ps
Name State Ports TaskDefinition
26313cbe-d929-49de-9cc3-873bf5f32a91/nginx RUNNING nginx...
Additional resources
ECS customers
https://www.mapbox.com/blog/switch-to-ecs/
https://segment.com/blog/rebuilding-our-infr...
Thank you!
Julien Simon
Principal Technical Evangelist, AWS
@julsimon
Prochain SlideShare
Chargement dans…5
×

sur

Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 1 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 2 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 3 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 4 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 5 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 6 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 7 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 8 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 9 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 10 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 11 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 12 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 13 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 14 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 15 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 16 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 17 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 18 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 19 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 20 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 21 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 22 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 23 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 24 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 25 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 26 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 27 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 28 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 29 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 30 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 31 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 32 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 33 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 34 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 35 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 36 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 37 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 38 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 39 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 40 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 41 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 42 Advanced Task Scheduling with Amazon ECS - Julien Simon Slide 43
Prochain SlideShare
Introduction to Unikernels at first Paris Unikernels meetup
Suivant
Télécharger pour lire hors ligne et voir en mode plein écran

0 j’aime

Partager

Télécharger pour lire hors ligne

Advanced Task Scheduling with Amazon ECS - Julien Simon

Télécharger pour lire hors ligne

Retrouvez la présentation de Julien Simon de AWS lors de la conférence du Paris Container Day

  • Soyez le premier à aimer ceci

Advanced Task Scheduling with Amazon ECS - Julien Simon

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Julien Simon Principal Technical Evangelist, AWS @julsimon Advanced Task Scheduling with Amazon ECS
  2. 2. Docker on Amazon Web Services Amazon EC2 Container Service (ECS) • https://aws.amazon.com/ecs/ • Launched in 04/2015 • No additional charge Amazon EC2 Container Registry (ECR) • https://aws.amazon.com/ecr/ • Launched in 12/2015 • Free tier: 500MB / month for a year • $0.10 / GB / month + outgoing traffic Both services are available in US, EU and APAC
  3. 3. ECS adoption https://www.datadoghq.com/blog/3-clear-trends-in-ecs-adoption/ November 2016
  4. 4. Companies Selected ECS customers
  5. 5. Container Partners https://aws.amazon.com/containers/partners/
  6. 6. ECS Scheduling
  7. 7. The problem Given a certain amount of computing power and memory, how can we best manage an arbitrary number of apps running in Docker containers? http://tidalseven.com
  8. 8. Amazon ECS: Under the Hood ALB ALB AZ 1 AZ 2 user / scheduler https://github.com/aws/amazon-ecs-agenthttp://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  9. 9. Scheduling on ECS: two options so far 1. Let ECS handle scheduling through Services • Task Definition • ECS transcription of the Docker Compose file • Versioning • cpu_shares, mem_limit • Number of desired containers 2. Implement a custom scheduler with the ECS API • Describe cluster state • Select a specific ECS instance according to custom logic • Run task on this instance • Coursera use case: www.youtube.com/watch?v=a45J6xAGUvA
  10. 10. ECS Placement Engine
  11. 11. Placement Engine: giving developers more control ALB ALB AZ 1 AZ 2 user / scheduler Placement Engine Constraints & Strategies
  12. 12. Placement Constraints Name Example AMI ID attribute:ecs.ami-id == ami-eca289fb Availability Zone attribute:ecs.availability-zone == us-east-1a Instance Type attribute:ecs.instance-type == t2.small Distinct Instances type=“distinctInstance” Custom attribute:stack == prod
  13. 13. Example: Constraint on Instance Family/Type
  14. 14. Example: Constraint on Availability Zone
  15. 15. Example: Combining Multiple Constraints
  16. 16. Placement Strategies Binpacking Spread Affinity Distinct Instance
  17. 17. Placement Strategy Chaining Spread tasks across Zones and Binpack within each Zone
  18. 18. Placing Tasks
  19. 19. Anatomy of Task Placement Cluster Constraints Custom Constraints Placement Strategies Apply Filter Satisfy CPU, memory, and port requirements Filter for location, instance-type, AMI, or custom attribute constraints Identify instances that meet spread or binpack placement strategy Select final container instances for placement
  20. 20. g2.2xlarge t2.small g2.2xlarge g2.2xlarge Placement: Targeting Instance Type
  21. 21. g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small t2.small t2.medium us-east-1aus-east-1d Placement: Targeting Instance Type & Zone
  22. 22. g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small us-east-1aus-east-1d g2.2xlarge t2.medium t2.micro t2.small us-east-1c Placement: Availability Zone Spread
  23. 23. g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small us-east-1aus-east-1d g2.2xlarge t2.medium t2.micro t2.small us-east-1c Placement: Spread across Zone and Binpack
  24. 24. g2.2xlarge t2.small t2.micro t2.medium t2.medium t2.small g2.2xlarge t2.small us-east-1aus-east-1d g2.2xlarge t2.medium t2.micro t2.small us-east-1c Placement: Affinity and Anti-Affinity
  25. 25. Running a Service
  26. 26. t2.medium t2.small t2.small us-east-1aus-east-1d t2.medium t2.micro t2.small us-east-1c Placement: Multiple Services on a Cluster
  27. 27. t2.medium g2.2xlarge t2.micro t2.small t2.small t2.small g2.2xlarge t2.small t2.small t2.small g2.2xlarge t2.small Placement: Services – Distinct Instances
  28. 28. Event Stream & Blox
  29. 29. Amazon ECS: Under the Hood ALB ALB AZ 1 AZ 2 user / scheduler Placement Engine Event Stream
  30. 30. Consuming Real-time Events
  31. 31. Handling ECS events with Blox scheduler cluster state service https://blox.github.io/
  32. 32. Amazon ECS: Under the Hood ALB ALB AZ 1 AZ 2 user / scheduler Scheduler Cluster State Service Placement Engine Event Stream
  33. 33. t2.small t2.small t2.small Blox: Daemon Scheduler t2.small t2.small t2.small scheduler cluster state service t2.small t2.small t2.small
  34. 34. Demo: Deploying Blox on AWS
  35. 35. Creating Clusters Create an ECS cluster for Blox CF template: https://github.com/blox/blox/blob/dev/deploy/aws/conf/cloudformation_template.json à CloudWatch Event Rule + SQS queue à 3 containers: Daemon Scheduler + Cluster State Service + etcd à REST API exposing the Daemon Scheduler API Create another ECS cluster managed by Blox $ ecs-cli configure --cluster WebCluster $ ecs-cli up --keypair admin --capability-iam --size 3 --instance-type t2.micro Invoke the scheduler API ‘demo-cli’ tool: https://github.com/blox/blox/tree/dev/deploy/demo-cli
  36. 36. Listing Task Definitions Grab the ARN for an nginx Task Definition, which the Daemon Scheduler will manage on ‘WebCluster’. $ ./list-task-definitions.py --region eu-central-1 == Blox Demo CLI - List Task Definitions == { "taskDefinitionArns": [ "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/BloxFramework:2", "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:1", "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2" ] }
  37. 37. Creating an Environment $ ./blox-create-environment.py --environment WebEnvironment --cluster WebCluster --task-definition "arn:aws:ecs:eu-central-1:ACCOUNT:task- definition/nginx:2" --stack Blox --apigateway --region eu-central-1 == Blox Demo CLI - Create Blox Environment == HTTP Response Code: 200 { "taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2", "deploymentToken": "17248257-08ec-4438-888f-e0ac28397653", "health": "healthy", "name": "WebEnvironment", "instanceGroup": { "cluster": "arn:aws:ecs:eu-central-1:ACCOUNT:cluster/WebCluster" } }
  38. 38. Listing Environments $ ./blox-list-environments.py --stack Blox --apigateway --region eu-central-1 == Blox Demo CLI - List Blox Environments == HTTP Response Code: 200 { "items": [ { "taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2", "deploymentToken": "17248257-08ec-4438-888f-e0ac28397653", "health": "healthy", "name": "WebEnvironment", "instanceGroup": { "cluster": "arn:aws:ecs:eu-central-1:ACCOUNT:cluster/WebCluster" } } ] }
  39. 39. Creating a Deployment $ ./blox-create-deployment.py --environment WebEnvironment --deployment-token "17248257-08ec-4438-888f-e0ac28397653" --stack Blox --apigateway --region eu-central-1 == Blox Demo CLI - Create Blox Deployment == HTTP Response Code: 200 { "status": "pending", "environmentName": "WebEnvironment", "id": "7a05ea99-27a9-4339-a7a6-f4120065aea3", "failedInstances": [], "taskDefinition": "arn:aws:ecs:eu-central-1:613904931467:task-definition/nginx:2” }
  40. 40. Listing Deployments $ ./blox-list-deployments.py --environment WebEnvironment --stack Blox --apigateway --region eu-central-1 == Blox Demo CLI - List Blox Deployments == HTTP Response Code: 200 { "items": [ { "status": "completed", "environmentName": "WebEnvironment", "id": "7a05ea99-27a9-4339-a7a6-f4120065aea3", "failedInstances": [], "taskDefinition": "arn:aws:ecs:eu-central-1:ACCOUNT:task-definition/nginx:2" } ] }
  41. 41. Scaling a Deployment $ ecs-cli ps Name State Ports TaskDefinition 26313cbe-d929-49de-9cc3-873bf5f32a91/nginx RUNNING nginx:2 98442432-fd5c-434d-b93c-0737bd06aaab/nginx RUNNING nginx:2 ce9bf217-4b34-4f31-9c7b-a8c3402f1ffd/nginx RUNNING nginx:2 $ ecs-cli scale --size 4 --capability-iam $ ecs-cli ps Name State Ports TaskDefinition 26313cbe-d929-49de-9cc3-873bf5f32a91/nginx RUNNING nginx:2 98442432-fd5c-434d-b93c-0737bd06aaab/nginx RUNNING nginx:2 c404ac9a-0948-4cc8-b5b0-2238ccdf4035/nginx RUNNING nginx:2 ce9bf217-4b34-4f31-9c7b-a8c3402f1ffd/nginx RUNNING nginx:2
  42. 42. Additional resources ECS customers https://www.mapbox.com/blog/switch-to-ecs/ https://segment.com/blog/rebuilding-our-infrastructure/ Tech articles by Werner Vogels, CTO, Amazon.com http://www.allthingsdistributed.com/2014/11/amazon-ec2-container-service.html http://www.allthingsdistributed.com/2015/04/state-management-and-scheduling-with-ecs.html http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html Blog articles & videos https://aws.amazon.com/blogs/compute/amazon-ec2-container-service-at-aws-reinvent-2016-wrap-up/ https://aws.amazon.com/blogs/compute/continuous-deployment-to-amazon-ecs-using-aws-codepipeline- aws-codebuild-amazon-ecr-and-aws-cloudformation/
  43. 43. Thank you! Julien Simon Principal Technical Evangelist, AWS @julsimon

Retrouvez la présentation de Julien Simon de AWS lors de la conférence du Paris Container Day

Vues

Nombre de vues

764

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

4

Actions

Téléchargements

13

Partages

0

Commentaires

0

Mentions J'aime

0

×