Amazon ECS Deep Dive
•Télécharger en tant que PPTX, PDF•
8 j'aime•4,900 vues
Running and managing large scale applications with microservices architectures is difficult and often requires operating complex container management infrastructure. Amazon EC2 Container Service (ECS) is a highly scalable, high performance service for running and managing Docker applications. In this webinar, we will walk through a number of patterns and tools used by our customers to run their applications on Amazon ECS. We will show you how to set up, manage and scale your Amazon ECS resources, keep them secure and deploy your applications to an Amazon ECS cluster. We will also provide best practices for monitoring, logging and service discovery. Learning Objectives: • Learn how to set up and manage Amazon ECS for production applications • Learn how to schedule containers on production clusters using Amazon ECS Who Should Attend: •Developers, DevOps, Sys Admin
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Amazon ECS Deep Dive
Similaire à Amazon ECS Deep Dive (20)
Plus de Amazon Web Services
Plus de Amazon Web Services (20)
Dernier
Dernier (20)
Amazon ECS Deep Dive
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. October 25th, 2016 Amazon EC2 Container Service Deep Dive Peter Dalbhanjan, Solutions Architect
- 2. Agenda • ECS Infrastructure Setup • ECS Infrastructure Management • PaaS on ECS • Q & A
- 3. Amazon ECS Benefits • Easily Manage Clusters for any scale • Flexible Container Placement • Designed for use with other AWS Services • Extensible
- 5. Amazon ECS Infrastructure Setup • Amazon ECS Cluster • AWS CloudFormation • Amazon ECS CLI • AWS OpsWorks • Amazon ECR
- 6. ECS Cluster Setup with AWS CloudFormation "Resources" : { "ECSCluster": { "Type": "AWS::ECS::Cluster" }, "ECSAutoScalingGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "VPCZoneIdentifier" : { "Ref" : "SubnetID" }, "LaunchConfigurationName" : { "Ref" : "ContainerInstances" }, "MinSize" : "1", "MaxSize" : { "Ref" : "MaxSize" }, "DesiredCapacity" : { "Ref" : "DesiredCapacity" } }, […] }, Autoscaling Group ECS Cluster
- 7. "ContainerInstances": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "commands" : { "01_add_instance_to_cluster" : { "command" : { "Fn::Join": [ "", [ "#!/bin/bashn", "echo ECS_CLUSTER=", { "Ref": "ECSCluster" }, " >> /etc/ecs/ecs.config" ] ] } } }, […] } } } ECS Cluster Setup with AWS CloudFormation Launch Configuration
- 8. "taskdefinition": { "Type": "AWS::ECS::TaskDefinition", "Properties" : { "ContainerDefinitions" : [ { "Name": "simple-app", "Cpu": "10", "Essential": "true", "Image":"httpd:2.4", "Memory":"300", "MountPoints": [{ "ContainerPath": "/usr/local/apache2/htdocs", "SourceVolume": "my-vol” }], "PortMappings": [ { "HostPort": 80, "ContainerPort": 80 } ] }, ECS Cluster Setup with AWS CloudFormation { "Name": "busybox", "Cpu": 10, "Command": [ "/bin/sh -c " while true; do echo '<html> <head> <title>Amazon ECS Sample App</title> <style>..... > /usr/local/apache2/htdocs/index.html ; sleep 1; done"” ], "EntryPoint": [ "sh", "-c"], "Essential": false, "Image": "busybox", "Memory": 200, "VolumesFrom": [ { "SourceContainer": "simple-app" } ] } ],
- 9. ECS Cluster Setup with Amazon ECS CLI • Simplifies creating, updating, and monitoring clusters and tasks • Supports Docker Compose • Available on github https://github.com/aws/a mazon-ecs-cli
- 10. ECS Cluster Setup with Amazon ECS CLI # Build cluster and container instances $ ecs-cli scale --size 2 --capability-iam --keypair demo-user # Create task definition and start tasks $ ecs-cli compose up # See running tasks $ ecs-cli compose ps # Start tasks as ECS service $ ecs-cli compose --project-name wordpress-test service start # See the progress of task state $ ecs-cli compose --project-name wordpress-test service ps
- 11. ECS Cluster Setup with AWS OpsWorks • Update OpsWorks IAM role to allow ecs:* actions • Add instances to layer (24/7, time-based, load-based) • Manage security updates, user permission and access Note: • One ECS Cluster layer per stack • An ECS Cluster can only be associated with one stack
- 12. Amazon ECR Setup
- 13. Amazon ECR Setup • You have read and write access to the repositories you create in your default registry, i.e. <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com • Repository names can support namespaces, e.g. team- a/web-app. • Repositories can be controlled with both IAM user access policies and repository policies.
- 14. Amazon ECR Setup # Authenticate Docker to your Amazon ECR registry > aws ecr get-login > docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us- east-1.amazonaws.com > docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us- east-1.amazonaws.com # Create a repository called ecr-demo > aws ecr create-repository --repository-name ecr-demo # Push an image to your repository > docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecr-demo:v1
- 15. Amazon ECR Docker Credential Helper • Available today - https://github.com/awslabs/amazon- ecr-credential-helper • Place the docker-credential-ecr-login binary on your PATH • Set the contents of ~/.docker/config.json file to be: { "credsStore": "ecr-login" } • Push and pull images from ECR without docker login
- 17. Amazon ECS Infrastructure Management • Monitoring and Logging • Automatic Scaling • Service Discovery • Security
- 19. Monitoring with Amazon CloudWatch • Metric data sent to CloudWatch in 1-minute periods and recorded for a period of two weeks • Available metrics: CPUReservation, MemoryReservation, CPUUtilization, MemoryUtilization • Available dimensions: ClusterName, ServiceName
- 20. Monitoring with Amazon CloudWatch
- 21. Monitoring with Amazon CloudWatch
- 22. Monitoring with Amazon CloudWatch Use the Amazon CloudWatch Monitoring Scripts to monitor additional metrics, e.g. disk space: # Edit crontab > crontab -e # Add command to report disk space utilization to CloudWatch every five minutes */5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used -- disk-space-avail --disk-path=/ --from-cron
- 23. CloudWatch Logs with awslogs driver Amazon CloudWatch Logs Amazon CloudWatch Logs Amazon CloudWatch Logs Amazon CloudWatch Logs Amazon S3 Amazon Kinesis AWS Lambda Amazon Elasticsearch Service Amazon ECS Store Stream Process Search
- 25. Configuring Logging in Task Definition "containerDefinitions": [ { "memory": 300, "portMappings": [ { "hostPort": 80, "containerPort": 80 } ], "entryPoint": [ "sh", "-c" ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "awslogs-test", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "nginx" } }, "name": "simple-app", "image": "httpd:2.4", "command": [ "/bin/sh -c "echo 'Congratulations! Your application is now running on a container in Amazon ECS.' > /usr/local/apache2/htdocs/index.html && httpd-foreground"" ], "cpu": 10 } ], "family": "cw-logs-example" }
- 26. Monitoring Amazon ECS with Datadog
- 27. Monitoring Amazon ECS with Sysdig Cloud
- 29. Setup ECS Cluster with AutoScaling Create LaunchConfiguration • Pick instance type depending on resource requirements, e.g. memory or CPU • Use latest Amazon Linux ECS- optimized AMI, other distros available Create AutoScaling group and set to cluster initial size
- 30. AutoScaling your Amazon ECS Cluster • Create CloudWatch alarm on a metric, e.g. MemoryReservation • Configure scaling policies to increase and decrease the size of your cluster
- 31. AutoScaling your Amazon ECS services
- 32. AutoScaling your Amazon ECS services
- 34. Service Discovery using ELB • Automation built using CloudWatch Events, Lambda and Route53 private hosted zones • Route53 is used as service registry • Lambda is used to add/remove records based on Service API’s from ECS • Available on github https://github.com/awslabs /ecs-refarch-service- discovery
- 35. Service Discovery using ELB
- 36. Service Discovery using DNS • Install an agent (ecssd_agent.go) on container instances • The agent registers service name, IP and port into Route53 private hosted zone • lambda_health_check.py used for cleanup • Available on github https://github.com/awslabs/ service-discovery-ecs-dns
- 37. Service Discovery using DNS
- 38. Service Discovery with Weaveworks • DNS interface for cross-host container communication • Gossip protocol to share grouped updates • Overlay network between hosts
- 39. Service Discovery and Configuration Management with Consul ECSCluster consul-server ECS Instance consul-agent registrator ECS Instance Back end 1 Back end 2 consul-agent registrator ECS Instance Front end ECSCluster
- 40. Security
- 41. IAM Roles for ECS Tasks { "family": “signup-app", "taskRoleArn": "arn:aws:iam::123456789012:role/DynamoDB RoleForTask", "volumes": [], "containerDefinitions": [{ "environment": [ ... ], "name": “signup-web", "mountPoints": [], "image": “amazon/signup-web", "cpu": 25, "portMappings": [ ... ], "entryPoint": [ ... ], "memory": 100, "essential": true, "volumesFrom": [] } ]}
- 42. Logging Amazon ECS API with AWS CloudTrail { "eventVersion": "1.03", "userIdentity": {…}, "eventTime": "2015-10-12T13:57:33Z", "eventSource": "ecs.amazonaws.com", "eventName": "CreateCluster", "awsRegion": "eu-west-1", "sourceIPAddress": "54.240.197.227", "userAgent": "console.amazonaws.com", "requestParameters": { "clusterName": "ecs-cli" }, Create Cluster event
- 43. Logging Amazon ECS API with AWS CloudTrail "responseElements": { "cluster": { "clusterArn": "arn:aws:ecs:eu-west- 1:560846014933:cluster/ecs-cli", "pendingTasksCount": 0, "registeredContainerInstancesCount": 0, "status": "ACTIVE", "runningTasksCount": 0, "clusterName": "ecs-cli", "activeServicesCount": 0 } }, […] Create Cluster event
- 44. Image Vulnerability Scanning with Twistlock
- 45. Secrets Management • Option 1: Task Definition Environment Variables • Easy to get Started • Configuration stored Directly into Task Definition • Version in Immutable Definition; Easy Rollback • Not Great for Secrets • Option 2: Encrypted DynamoDB or S3 • Use Environment Variables to Provide Pointer • Use AWS Encryption Clients to Securely Store • Use VPC-Endpoints, IAM Policies, and IAM Roles to Restrict Access
- 46. Secrets Management Task ECS Cluster Container instance
- 47. PaaS on ECS
- 48. AWS Elastic Beanstalk • Elastic Beanstalk uses Amazon ECS to coordinate deployments to multi-container Docker environments • Dockerrun.aws.json file that describes how to deploy containers. • Takes care of tasks including cluster creation, task definition and execution
- 49. Convox
- 50. Convox # Initialize your app and create default manifest > convox init # Locally build and run your app as declared in the manifest > convox start # Create app > convox apps create my_app # Deploy app, output ELB DNS name > convox deploy [...] web: http://my_app-1234567890.us-east-1.elb.amazonaws.com
- 51. Remind Empire • Offers a control layer on top of Amazon ECS that provides a Heroku like workflow • Any tagged Docker image can be deployed to Empire as an app • When you deploy a Docker image to Empire, it will extract a Procfile from the WORKDIR • Each process type in the Procfile maps directly to an ECS Service
- 52. Remind Empire • Get started by launching CloudFormation stack • Use the emp client to start developing your app # tell empire client where it can find the API $ export EMPIRE_API_URL=http://empire-60-LoadBala-…elb.amazonaws.com/ # login to empire using your github credentials $ emp login # run your first app $ emp deploy remind101/acme-inc:master # check what’s running $ emp apps acme-inc Jun 15 20:42[...]
- 53. Additional Resources • ECS CLI – http://bit.ly/2eKy3I6 • ECR Docker Credential Helper – http://bit.ly/2dD02xo • AutoScaling – http://amzn.to/2eohA2a • ECS integration with ALB to support Dynamic ports and Path-based routing: http://amzn.to/2exhh07 • Service Discovery • Service Discovery using ELB – http://bit.ly/2dAN6Dw • Service Discovery using DNS – http://bit.ly/2eI831D