When it comes to managing the security of your AWS environment, traditional, on-premise, perimeter-only tactics must evolve to be environment-aware, data-centric, and automated wherever possible. Speed of detection and agility in recovery are your new challenges and AWS Config, Cloudwatch, and Lambda are your new allies that help address them. Learn about high-speed security incident response and recovery at the push of a button perhaps. This talk provides an overview with detailed examples of configuration management, event notification, and automatic execution to rapidly detect and react to potential security concerns within your AWS environment. Speaker: Don Bailey, Principal Security Engineer, Amazon Web Services & Joshua Du Lac, Senior Security Consultant, Amazon Web Services

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Don “Beetle” Bailey, Security Consultant & Josh Du Lac, Professional Services
Amazon Web Services
Enforcing Your Security Policy
at Scale
Technical 301

2. What to Expect from This Session?
• Iteration of previous re:Invent talks
• Concrete Examples of Potential Events and How you
can Handle them Manually
• Ideas for Increasing Security Agility through Automation
• Specific AWS Mechanisms to Leverage, Code
• New Services and Features for Security Geeks

3. Previous Talks
YouTube search for…
• “Intrusion Detection in the Cloud” … 2014
• “Incident Response (IR) in the Cloud” … 2014
• “Wrangling Security Events in The Cloud” … 2015
Quick! Take a picture!
FYI – “Enforcing Your Security Policy at Scale” Session

4. “Intrusion Detection in the Cloud” …
• AWS-­Specific Areas to Monitor for Security-­Concerning
events
• Prerequisites
• Key Concepts, such as Security Role, Write-­once Storage
• Key services to Leverage, Events and Behaviors to look for
• Example detection of Key Configuration changes,
Resource usage Anomalies
• YouTube search “Intrusion Detection in the Cloud”

5. “Incident Response (IR) in the Cloud” …
• Ensuring your existing IR Process considers AWS
• More Prerequisites
• Mechanisms for Mitigation and Investigation
• Tactics specific to AWS IR, such as Constraining Exposed
AWS Credentials
• Tactics analogous to traditional IR, modified for AWS, such as
Amazon EC2 instance memory dumping, analysis
• YouTube search “Incident Response in the Cloud”

6. “Wrangling Security Events in The Cloud” …
• Types of Security Events to be wary of and Prepared for
• Absent Protection, knowing how to Detect -­> Recover
• Where to Gather supporting data to Investigate -­> Protect
• Step by step manual Security Event Recovery
• Services, features, code for AUTOMATED recovery
• YouTube search “Wrangling Security Events in The Cloud”
• Here is an Encore with some Updates …

7. Protect, Detect, React, Recover, etc.
Protect
Detect
Recover
Investigate

8. AWS = Agility for Security Geeks
• Ability to Programmatically Inventory Environment —
knowing what you need to protect is key
• Awareness of what’s Happening, what’s Changing, from
AWS API activity to Application Behavior
• Detection and Alerting Mechanisms, freedom to Create
and Flexibility to Configure and tune what’s appropriate
for YOU
• Analysis and Response, via the same platform, natively
or with AWS Partner Solutions

9. Example Events of Concern, Signatures
• Configuration changes that Impact Ability to Detect or
Understand Events
• Activities that are Inconsistent with Expectations
• Activities that Violate Policy
• Resources no longer Available
• Resources more Available than Desired
• Event Detection Signatures! = Commercial Product, and
may require careful thought vs. Operations to Develop

10. AWS CloudTrail
• Records AWS API calls for your account and Delivers
log files to you.
• Turn it ON!
http://docs.aws.amazon.com/awscloudtrail/latest/usergui
de/cloudtrail-­user-­guide.html

15. CloudTrail Events
• A record in JSON format that contains information about
requests for resources in your account.
• Describes which service was accessed, what action was
performed, and any parameters for the action.
• Helps you determine who made the request.
• The event data is enclosed in a Records array.
http://docs.aws.amazon.com/awscloudtrail/latest/usergui
de/send-­cloudtrail-­events-­to-­cloudwatch-­logs.html

16. Example CloudTrail event
"Records": [{
"eventVersion": "1.0",
"userIdentity": {
"type": "IAMUser",
"principalId": "EX_PRINCIPAL_ID",
"arn": "arn:aws:iam::123456789012:user/Alice",
"accountId": "123456789012",
"accessKeyId": "EXAMPLE_KEY_ID",
"userName": "Alice"
},
"eventTime": "2015-03-24T21:11:59Z",
"eventSource": "iam.amazonaws.com",
"eventName": "CreateUser",
"awsRegion": "us-east-1",
"sourceIPAddress": ”55.55.55.55",
"userAgent": "aws-cli/1.3.2 Python/2.7.5 Windows/7",
"requestParameters": {
"userName": "Bob"
},
"responseElements": {
"user": {
"createDate": "Mar 24, 2015 9:11:59 PM",
"userName": "Bob",
"arn": "arn:aws:iam::123456789012:user/Bob",
"path": "/",
"userId": "EXAMPLEUSERID"
}
....

20. CloudTrail OFF
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAI5WIMUDR2UZUI62VO",
"arn": "arn:aws:iam::000123456789:user/reinvent-sec308",
"accountId": "000123456789",
"accessKeyId": "AKIAIRAHHRD3PHLUFJLQ",
"userName": "reinvent-sec308"
},
"eventTime": "2015-09-23T00:41:45Z",
"eventSource": "cloudtrail.amazonaws.com",
"eventName": "StopLogging",
"awsRegion": "us-west-2",
"sourceIPAddress": “55.55.55.55",
"userAgent": "aws-cli/1.7.25 Python/2.7.5 Darwin/13.4.0",
"requestParameters": {
"name": "CloudTrail-Default"
},
"responseElements": null,
....

21. Amazon CloudWatch Logs
• Monitor, store, and access your log files from Amazon
EC2 instances, AWS CloudTrail, or other sources.
• Enable in the AWS Management Console, CLI, or via
AWS CloudFormation.
• Monitor and alarm for specific phrases, values, or
patterns.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/
DeveloperGuide/WhatIsCloudWatchLogs.html

26. CloudTrail -­> CloudWatch Alarms
• Downloadable and editable example CloudFormation template from
AWS
• Contains predefined CloudWatch metric filters and alarms that
enable you to receive email notifications when certain security-­
related API calls are made in your AWS account
• Amazon S3 bucket events, network events, Amazon EC2 events,
AWS CloudTrail, and AWS Identity and Access Management (IAM)
events
http://docs.aws.amazon.com/awscloudtrail/latest/userguide/use-­
cloudformation-­template-­to-­create-­cloudwatch-­alarms.html

27. CloudTrail OFF Event – Detect
"CloudTrailStopMetricFilter": {
"Type": "AWS::Logs::MetricFilter",
"Properties": {
"LogGroupName": { "Ref" : "LogGroupName" },
"FilterPattern": ”{ ($.eventName = StopLogging) }",
"MetricTransformations": [
{
"MetricNamespace": "CloudTrailMetrics",
"MetricName": "CloudTrailEventCount",
"MetricValue": "1"
}
]
}
},

28. CloudTrail OFF Event – Detect
"CloudTrailStoppedAlarm": {
"Type": "AWS::CloudWatch::Alarm",
"Properties": {
"AlarmName" : ”CloudTrailStoppedAlarm",
"AlarmDescription" : "Alarms when StopLogging API call is made",
"AlarmActions" : [{ "Ref" : "AlarmNotificationTopic" }],
"MetricName" : "CloudTrailEventCount",
"Namespace" : "CloudTrailMetrics",
"ComparisonOperator" : "GreaterThanOrEqualToThreshold",
"EvaluationPeriods" : "1",
"Period" : "300",
"Statistic" : "Sum",
"Threshold" : "1"
}
},

29. CloudTrail OFF Event – Recover

30. CloudTrail OFF Event – Investigate
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAI5WIMUDR2UZUI62VO",
"arn": "arn:aws:iam::000123456789:user/reinvent-sec308",
"accountId": "000123456789",
"accessKeyId": "AKIAIRAHHRD3PHLUFJLQ",
"userName": "reinvent-sec308"
},
"eventTime": "2015-09-23T00:41:45Z",
"eventSource": "cloudtrail.amazonaws.com",
"eventName": "StopLogging",
"awsRegion": "us-west-2",
"sourceIPAddress": "55.55.55.55",
"userAgent": "aws-cli/1.7.25 Python/2.7.5 Darwin/13.4.0",
"requestParameters": {
"name": "CloudTrail-Default"
},
"responseElements": null,
....

31. CloudTrail OFF Event – Protect
Deny Permissions for CloudTrail in IAM Groups or Roles
{
"Sid": "Stmt0001",
"Effect": "Deny",
"Action": [
"cloudtrail:DeleteTrail",
"cloudtrail:StopLogging"
],
"Resource": [
"*"
]
}

32. CloudTrail OFF – Automated Recovery
• We know how to detect CloudTrail OFF.
• We know we don’t want it OFF. Ever.
• We know that the immediate response to learning that
CloudTrail is OFF is to turn it back ON. Always.
• Shouldn’t we be able to automate doing that?
• If only there was “Do Something aaS” ...

33. Let’s use AWS Lambda!
• Runs your code in response to events
• Python, Node.js, Java
• Automatically manages compute resources for you
• Create new back-­end services where compute
resources are automatically triggered based on custom
requests.
• You can read CloudTrail events with AWS Lambda
http://docs.aws.amazon.com/lambda/latest/dg/welcome.html

34. Automate Incident Response?
• Most, if not all, of the pieces to automate IR exist in AWS
• Automated IR = Even greater security agility
• Detect -­> Protect programmatically
• Lambda-­fy your IR!

35. CloudWatch Events – NEW TO SYDNEY!!
• Amazon CloudWatch Events delivers a near real-­time
stream of system events that describe AWS resource
changes to a target (such as AWS Lambda)
• Using simple rules that you can quickly set up, you can
match events and route them to one or more target
functions or streams

36. CloudWatch Events – Components
• Events
• EC2 state change (such as AutoScaling launch or terminate)
• CloudTrail read/write API calls & Management Console logins
• Your own code can publish application-­level events
• Scheduled basis (periodic or cron-­style scheduling)
• Rules
• Match incoming events and route them to one or more targets
for processing
• Targets
• Are specified in rules and receive matching events

37. Detecting with AWS CloudWatch Events

38. Detecting with AWS CloudWatch Events

39. Detecting with AWS CloudWatch Events

40. Detecting with AWS CloudWatch Events

41. Detecting with AWS CloudWatch Events

42. Detecting with AWS CloudWatch Events

43. Logging with AWS Lambda
from __future__ import print_function
import json
def lambda_handler(event, context):
print(json.dumps(event, indent=2))

44. AWS CloudWatch Logs

45. Notifying with AWS Lambda
sns_topic = "arn:aws:sns:us-east-1:350419227465:reporter-topic"
subject = 'EVENT: ' + event["detail"]["eventName"]
message = "What happened? " + event["detail"]["eventName"] + "n"
"What service? " + event["detail"]["eventSource"] + "n"
"Where? " + event["detail"]["awsRegion"] + "n"
"When? " + event["detail"]["eventTime"] + "n"
"Who? " + str(json.dumps(event["detail"]["userIdentity"], indent=2))
sns = boto3.client('sns')
sns_response = sns.publish(
TopicArn = sns_topic,
Message = message,
Subject = subject,
MessageStructure = 'string'
)

46. Example Notification from Amazon SNS

47. Responding to Events in Lambda
cloudtrail = boto3.client('cloudtrail')
trail_arn = event["detail"]["requestParameters"]["name"]
ct_response = cloudtrail.start_logging(
Name = trail_arn
)

48. Responding to Events in Lambda

49. Choosing CloudWatch Event Rules
• What could you automatically respond to?

50. Automated Incident Response Diagram
AWS
CloudTrail
Amazon
CloudWatch
Events
AWS
Lambda
Amazon
Simple
Notification
Service
AWS API
Endpoints
Your Staff Amazon S3
Bucket
Your Security
Team
AWS IAM
Role
AWS API
Your SaaS
Tools

51. AWS Config
• AWS resource inventory, configuration history, and
configuration change notifications
• Discover existing AWS resources
• Export inventory of your AWS resources with all configuration
details
• Determine how a resource was configured at any point in
time
• Security geeks should LOVE it!
http://aws.amazon.com/documentation/config/

60. Open Security Group Event – Detect
• Subscribe to AWS Config notification topic.
• Filter notifications for creation of security groups that
might be concerning. You could look for the following,
individually or combined:
• “SecurityGroup” and “Created” within subject
• changeType : “CREATE” within body
• resourceType: "AWS::EC2::SecurityGroup” within body

62. Open Security Group Event – Detect
"groupId": "sg-7dc0d21a",
...
"ipPermissions": [
{
"ipProtocol": "-1",
"fromPort": null,
"toPort": null,
"userIdGroupPairs": [],
"ipRanges": [
"0.0.0.0/0"
],
"prefixListIds": []
}
],
...

63. Open Security Group Event – Recover
• If responding soon enough to the creation of a new
security group and no instances, simply delete the
security group.
• Otherwise, assign running instances to another security
group, and then delete the offending security group.
• You can’t delete a default security group, but you can
change its rules back to something sane, including no
rules.

67. Delete Open Security Group – AWS CLI
aws ec2 delete-security-group --no-dry-run --group-id sg-d3bda2b4

68. Open Security Group Event – Investigate
• Revisit the AWS Config change notification.
• Note time, action, and security group ID to correlate to
principal and source IP of EC2 API call via AWS
CloudTrail.
• If possible, engage principal to understand intent or
determine if unexplained, such as by external actor and
potentially malicious.

69. Open Security Group Event – Protect
• Appropriately constrain or deactivate associated
credentials as warranted.
• Security group changes, particularly within production,
should not be a frequent event, so maintain high
vigilance.

70. Lambda – Automated Open Security Group Delete
var snsMsgString = JSON.stringify(event.Records[0].Sns.Message);
var snsMsgObject = getSNSMessageObject(snsMsgString);
if (snsMsgObject.configurationItemDiff.changeType == 'CREATE' &&
snsMsgObject.configurationItem.resourceType == 'AWS::EC2::SecurityGroup' &&
snsMsgObject.configurationItem.configuration.ipPermissions[0].ipProtocol == '-1' &&
snsMsgObject.configurationItem.configuration.ipPermissions[0].ipRanges == '0.0.0.0/0')
{
var params = {
DryRun: false,
GroupId: snsMsgObject.configurationItem.resourceId,
};
ec2.deleteSecurityGroup(params, function(err, data) {
context.succeed(snsMsgObject);
});
}

72. AWS Config Rules

73. AWS Config Rules

74. AWS Config Rules

75. AWS Config Rules
function hasExpectedSecurityGroup(expectedSecurityGroupId, securityGroups) {
for (var i = 0; i < securityGroups.length; i++) {
var securityGroup = securityGroups[i];
if (securityGroup.groupId === expectedSecurityGroupId) {
return true;
}
}
return false;
}

76. AWS Config Rules
config.putEvaluations(putEvaluationsRequest, function (err, data) {
if (err) {
context.fail(err);
} else {
context.succeed(data);
}
});

77. AWS Config Rules

78. AWS Config Rules Community Repository
Visit https://github.com/awslabs/aws-­config-­rules

80. VPC Flow Logs
• Choose to collect for VPC, VPC subnet, or Elastic
Network Interface (ENI)
• SRC and DST IP addresses, ports, IANA protocol
number, packet and byte counts, time of flow, action
(ACCEPT or REJECT).
• Create metrics to ID trends and patterns
• Create alarms that will fire if certain types of traffic are
detected!

81. Leverage VPC Flow Logs for Event Detection!
• Reviewing your application’s NORMAL flows may enable
you to constrain security groups further
• Once constrained, pay particular attention to REJECT
based on egress traffic
• Home in on certain hosts, eg. infrequently used jump
hosts, pay attention to ACCEPT even
• Key AWS partners speak VPC Flow Logs!

82. Security Event Response … Practice makes perfect!
• IR Game Day…YAY!
• Humans practicing exercising good judgment under pressure
• Tabletop First…yay?
• Humans talking about exercising good judgment
• YouTube search “Harden Your Architecture with Security
Incident Response Simulations”
• Push-­button testing / recovery?

86. Buy an IoT Starter Kit
Intel® Edison and Grove IoT Starter Kit Powered by AWS

87. Example Button #1

88. Example Button #2

90. AWS Security Best Practices Whitepaper
• Help for designing security infrastructure and
configuration of your AWS environment
• High-­level guidance for:
• Managing accounts, users, groups, and roles
• Managing OS-­level access to instances
• Securing your data, OS, apps, and infrastructure
• Managing security monitoring, auditing, alerting, and incident
response
https://media.amazonwebservices.com/AWS_Security_Best_Practices.pdf

92. External Resources – Reading, Training
• SANS Reading Room, Incident Response
http://www.sans.org/reading-­room/whitepapers/incident
• FIRST
http://www.first.org/resources/guides
• CERT, Incident Management
http://www.cert.org/incident-­management/publications/

93. External Resources – IR Tools, Frameworks
• Mozilla Investigator (MIG)
http://mig.mozilla.org/
• Netflix Fully Integrated Defense Operations (FIDO)
http://techblog.netflix.com/2015/05/introducing-­fido-­
automated-­security.html

95. AWS Support for Security Concerns
• AWS Support is the one-­stop shop for AWS customers,
for any concerns, including security related.
• If AWS Support cannot immediately address your
concerns, they will escalate internally to the appropriate
technical team, AWS Security included.
https://aws.amazon.com/support

96. AWS Security Resources
• AWS Security Blog
http://blogs.aws.amazon.com/security/
• AWS Security Center
https://aws.amazon.com/security
• Contact the AWS security team
aws-­security@amazon.com

97. Summary
• Security Agility with AWS
• Threat vs. policy-­driven concerns, enumerate, create
signatures, detection mechanisms
• Automate IR where you can … with buttons, even?
• TWO ways to get more practice, but you only get to
choose ONE
• Enforce your security policy AT SCALE.
• We (AWS and our technology partners) are here to help!

98. AWS Training & Certification
Intro Videos & Labs
Free videos and labs to
help you learn to work
with 30+ AWS services
– in minutes!
Training Classes
In-­person and online
courses to build
technical skills –
taught by accredited
AWS instructors
Online Labs
Practice working with
AWS services in live
environment –
Learn how related
services work
together
AWS Certification
Validate technical
skills and expertise –
identify qualified IT
talent or show you
are AWS cloud ready
Learn more: aws.amazon.com/training

99. Your Training Next Steps:
ü Visit the AWS Training & Certification pod to discuss your
training plan & AWS Summit training offer
ü Register & attend AWS instructor led training
ü Get Certified
AWS Certified? Visit the AWS Summit Certification Lounge to pick up your swag
Learn more: aws.amazon.com/training

100. Thank You!