Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
04.03.19
O S L O

Analyzing your web and application logs
S3, Cloudfront, Lambda and the Elasticsearch Service
Javier Ramirez
AWS Tech Evangelist
@supercoco9
B A R 2
O S L O
04.03.19

Agenda
Hosting up a very simple web
Hosting static sites on AWS
Scaling out any website with a Content Delivery Network
The Elasticsearch Service
Kibana Dashboards
Using Lambda for serverless log ingestion
Demo time

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why logs?

What produces data?
• Metering
Records
• Mobile Apps
• IoT Sensors
Web
Clickstream
• Enterprise
Documents
• Application
Logs
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client denied
by server configuration:
/export/home/live/ap/htd
ocs/test

Logs, logs and more logs
Logs are important:
• Debugging
• Working out user flow
• Monitoring
Centralising those logs is even more important

Operational analytics

The explosion of machine-generated data
Transition from IT
to DevOps
Increase in IoT and
Mobile Devices
Cloud-based
architectures
Machine-generated data is growing 10x faster than business data
Source: insideBigData - The Exponential Growth of Data, February 16, 2017

Operational analytics requirements/challenges

Let’s host a static website on AWS
so we can generate some traffic logs

Not just for Geocities web pages

How to enable hosting in S3

Why hosting in S3 is cool!
• Simple static website hosting with a simple
workflow
• 2 clicks to enable
• Scales
• Fast
• Supports your own FQDN
• You can extend it with lambda
functions for dynamic content
Developer
Users
Resolve DNS
Fetch site
direct from
S3
Push Code

Scaling out with CloudFront
so we can handle high volume traffic more efficiently

Amazon Global Network
• Redundant 100GbE network
• Redundant private capacity
between all Regions except China
166 Global CloudFront PoPs
a e
o
q
i
h
Paris
Sweden
AWS GovCloud East
First 5 years: 4 regions
2016–2020: 13 regions
Next 5 years: 7 regions
A W S
REGIONS
2 0 R e g i o n s 6 1 A Z s
d
m
c
g
b
n
s
k
v
i
i
i
i i
i
i
i
Milan
i
Cape Town

Going global
• Simple to enable
• Speeds up your site for international users
• Choice of regions
• Allows you to enter multiple CNAME’s
• Integrated with Route 53 alias records
• Supports SSL certs from certificate manager

The Amazon Elasticsearch Service

Amazon Elasticsearch Service is a cost-effective
managed service that makes it easy to deploy,
manage, and scale open-source Elasticsearch and
Kibana for log analytics, full-text search, and more.

Easy to Use
Deploy a production-ready Elasticsearch
cluster in minutes
Simplifies time-consuming management
tasks such as software patching, failure
recovery, backups, and monitoring
Open
Get direct access to the Elasticsearch open-
source API
Fully compatible with the open-source
Elasticsearch API, for all code and
applications
Secure
Secure Elasticsearch clusters with AWS
Identity and Access Management (IAM)
policies with fine-grained access control
access for users and endpoints
Automatically applies security patches
without disruption, keeping Elasticsearch
environments secure
Available
Provides high availability using Zone
Awareness, which replicates data between
two Availability Zones
Monitors the health of clusters and
automatically replaces failed nodes,
without service disruption
AWS Integrated
Integrates with Amazon Kinesis Firehose,
AWS IoT, and Amazon CloudWatch Logs for
seamless data ingestion
AWS CloudTrail for auditing, AWS Identity
and Access Management (IAM) for security,
and AWS CloudFormation for cloud
orchestration
Scalable
Scale clusters from a single node up to 20
nodes
Configure clusters to meet performance
requirements by selecting from a range of
instance types and storage options,
including SSD-powered EBS volumes
Amazon Elasticsearch Service Benefits

Application Monitoring & Root-cause Analysis
CASE STUDY: EXPEDIA
Logs, lots and lots of logs. How to cost effectively monitor logs?
Require centralized logging infrastructure
Did not have the man power to manage infrastructure
P R O B L E M
Quick insights: Able to identify and troubleshoot issues in real-time
Secure: Integrated w/ AWS IAM
Scalable: Cluster sizes are able to grow to accommodate additional log sources
B E N E F I T S
Streaming AWS CloudTrail logs, application logs, and Docker startup
logs to Elasticsearch
Created centralized logging service for all team members
Using Kibana for visualizations and for Elasticsearch queries
S O L U T I O N

Leading enterprises trust Amazon Elasticsearch Service for
their search and analytics applications
Media &
Entertainment
Online
Services
Technology Other

What does it do?
Application DataServer, application,
network, AWS, and
other logs
Amazon Elasticsearch Service Domain
with index(es)
1. Send data as JSON via REST APIs
2. Data is indexed - all fields
searchable, including nested JSON
3. Queries, via REST APIs, allow fielded
matching, Boolean expressions,
include sorting and analysis
1
2 3
Application users, analysts, DevOps,
security

{ "title": "Star Wars",
"plot": "Luke Skywalker joins forces with a
Jedi Knight, a cocky pilot, a wookiee and two
droids to save the universe from the Empire's
world-destroying battle-station, while also
attempting to rescue Princess Leia from the evil
Darth Vader.",
"year": 1977,
"actors": [
"Mark Hamill",
"Harrison Ford",
"Carrie Fisher”
],
"directors": [
"George Lucas”
],
"rating": 8.7,
"genres" : [
"Action",
"Adventure",
"Fantasy",
"Sci-Fi”
]
}
Elasticsearch works with
structured JSON containing
fields and values

Elasticsearch creates an index for each field
Doc
Name Value
Name Value
Name Value
Name Value
Name Value
Name Value
Fields Analysis
Field indices
Term 1
Term 2
Term 3
Term 4
Term 5
Term 6
Term 7
1,
4,
8,
12,
30,
42,
58,
100.
..
Posting lists

Query your data
{
"query": {
"match": {
"title": "iron man"
}
}
}
Title Score
Iron Man 10.56436
Iron Man 2 8.631084
Iron Man 3 8.631084
Iron Sky 6.387543
The Man with the Iron
Fists
6.185517
3
The Man in the Iron Mask 6.185517
3
The Iron Giant 5.218624
The Iron Lady 5.218624
77 hits

You can search for values by field, with Boolean
expressions to get scored, sorted results
1,
4,
8,
12,
30,
42,
58,
100.
...
1,
4,
8,
12,
30,
42,
58,
100.
...
Term 1
Term 2
Term 3
Term 4
Term 5
Term 6
Term 7
Analysis
1,
4,
8,
12,
30,
42,
58,
100
...
Posting lists
Field1:value
Field2:value
Field3:value
Term 1
Term 2
Term 3
Term 4
Term 5
Term 6
Term 7
Term 1
Term 2
Term 3
Term 4
Term 5
Term 6
Term 7
1,
12,
58,
100
58,
12,
1,
100
Result
Key Idea

You can analyze field values to get statistics
and build visualizations
58,
12,
1,
100,
115
123,
214,
947
Result
GET
GET
POST
GET
PUT
GET
GET
POST
Field
Data
Buckets
GET
POST
PUT
5
2
1
Counts
Key Idea

ElasticSearch Service with Kibana

Get the right set up

Easy to use and scalable
AWS SDK
AWS CLI Elasticsearch
data nodes
Elasticsearch
master nodes
Amazon Elasticsearch Service domain
Developer
Amazon Cognito
Amazon
CloudWatch
AWS
CloudTrail

Security
• Public endpoints – IAM
• Private endpoints – IAM
and security groups
• Encryption

Use IAM for public endpoints
{ "Version": "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Principal": {
"AWS":
"arn:aws:iam::XXXX:root”
},
"Action": "es:*",
"Resource":
"arn:aws:es:
us-west-2:XXXX:domain/YYYY/*”
}
]
}
• To grant access for Kibana, use
a CIDR Condition
• To grant read-only access to
an account or role, specify an
es:HttpGet Action
• To limit a user to a specific
index, or API add it to the
Resource
• For Admin access, specify
administrative es:* Actions

Add security groups for private endpoints
Specify a subnet and security
group to apply CIDR restrictions
on inbound/outbound traffic
security group
security group
Amazon Elasticsearch Service
Data Master
Data
Master
IAM
IAM

Architecture: Web Serving
Public
Subnet
Internet
Gateway
ALB
Private
Subnet
Httpd
Amazon ES
ENI
Amazon ES
Domain

Encrypt your data
• Encrypted data at rest on
Amazon ES instances
• Both EBS and ephemeral
store
• Encrypted automatic
snapshots

Data is stored in indexes, distributed across shards
Amazon Elasticsearch Service domain
ID
Field: value
Field: value
Field: value
Field: value
Index
Shards
Instances
• Shards are primary or
replica
• Primary shard count
can’t be changed
• Elasticsearch distributes
shards to instances
elastically
• Primary and replica are
distributed to different
instances

How many instances?
• Index size is approximately source size
• Double this if you are deploying an index replica
• Instance count based on storage
requirements
• Either local storage or up to 1.5 TB of Amazon
Elastic Block Store (EBS) per instance
Example: a 2 TB corpus will need 4 instances
Assuming a replica and using EBS
Given 1.5 TB of storage per instance, this gives 6TB of storage

Instance type recommendations
Instance Workload
T2 Entry point. Dev and test. OK for dedicated masters.
M3, M4 Equal read and write volumes.
R3, R4 Read-heavy or workloads with high memory
demands (e.g., aggregations).
C4 High concurrency/indexing workloads
I2,I3 Up to 1.6 TB of SSD instance storage.

It is elastic
• Run out of storage space?
• add data nodes or
• increase EBS volume size.
• Need more compute or RAM?
• increase the instance type, or
• add more data nodes.
Amazon Elasticsearch Service is designed to apply those configuration
changes without incurring downtime.

Full text search
CASE STUDY: MirrorWeb
Make the UK Government and UK Parliament’s web archives searchable
Large scale ingestion scenario: 120 TB of data (1.2 MM 100MB files),
duplicates and bad data, Warc format
P R O B L E M
Scalability: Started on a 9-node, R4.4Xlarge cluster for fast ingest, reduced to 6 R4.Xlarge instances for search.
Able to reconfigure the cluster with no down time
Cost effective: Indexed 1.4 billion documents for $337
Fast: 146 MM docs per hour indexed. 14x faster than the previous best for this data set (using Hadoop)
B E N E F I T S
S O L U T I O N
Amazon
S3
(Storage)
Amazon
EC2
(Filtering)
Amazon
EC2
(Extraction)
Amazon ES
(Search)
For more on this case, see http://tinyurl.com/ybqwbolq

Deployment of indices to a cluster
• Index 1
– Shard 1
– Shard 2
– Shard 3
• Index 2
– Shard 1
– Shard 2
– Shard 3
Amazon ES cluster
1
2
3
1
2
3
1
2
3
1
2
3
Primary Replica
1
3
3
1
Instance 1,
Master
2
1
1
2
Instance 2
3
2
2
3
Instance 3

Cluster with dedicated masters
Amazon ES cluster
1
3
3
1
Instance 1
2
1
1
2
Instance 2
3
2
2
3
Instance 3Dedicated master nodes
Data nodes: queries and updates

Master node recommendations
Number of data nodes Master node instance type
< 10 m3.medium+
< 20 m4.large+
<= 50 c4.xlarge+
50-100 c4.2xlarge+
Always use an odd number of masters, >= 3

Cluster with zone awareness
Amazon ES cluster
1
3
Instance 1
2
1 2
Instance 2
3
2
1
Instance 3
Availability Zone 1 Availability Zone 2
2
1
Instance 4
3
3

Scaling Amazon Elasticsearch Service
• Scaling Amazon Elasticsearch Service means scaling Elasticsearch
• Amazon Elasticsearch Service delivers seamless deployment
changes
Storage
Type, amount per-
instance, and count
Parallelism
Add and remove
instances
Throughput and Capacity
Change instance types

To scale correctly, iterate!
• Set config
• Instance type, storage
• Monitor via CloudWatch and
ES APIs
• CloudWatch for CPU, JVM,
and Disk
• ES for indexing, merges,
latencies
• Slow Logs for bottlenecks
• Adjust for workload
Configure
Cluster
Run your
workload
Gather
metrics

Use ES to analyze ES performance
https://aws.amazon.com/blogs/database/analyzing-amazon-elasticsearch-service-slow-logs-using-amazon-
cloudwatch-logs-streaming-and-kibana/

Data pattern for time series data (or logs)
Amazon ES cluster
logs_01.21.2017
logs_01.22.2017
logs_01.23.2017
logs_01.24.2017
logs_01.25.2017
logs_01.26.2017
logs_01.27.2017
Shard 1
Shard 2
Shard 3
host
ident
auth
timestamp
etc.
Each index has
multiple shards
Each shard contains
a set of documents
Each document contains
a set of fields and values
One index per day

Kinesis Firehose
CloudWatch Logs
Logstash
Amazon Lambda
Transform
Kinesis Firehose/
Streams
CloudWatch Logs
Amazon Elasticache/Redis
Kafka Rabbit MQLogstash
Amazon S3
Buffer
Kinesis Firehose/
Streams
Logstash
Worker nodes
Deliver
Kinesis Agent
CloudWatch Logs Agent
Beats
Fluentd
Application
Logstash
Collect
CloudWatch Logs
Amazon Lambda

Small-ish hosted use cases
Public Subnet
Internet Gateway
ALB
Private
Subnet
Filebeat
Redis ENI
Kibana
Amazon ES ENI Amazon ES Domain
Redis Cluster

XL hosted use cases
• Ingest supported through high-
volume technologies like Spark or
Kinesis
• Up to 60 TB of data today
• R3.8xlarge + 640GB data nodes
• 3x m3.xlarge master nodes

Large or serverless use cases
• Data flows from instances and
applications via Lambda
• SigV4 signing via Lambda/roles
• Up to 5 TB of data
• r3.2xlarge + 512 GB EBS data
nodes
• 3x m3.medium master nodes

Lambda for serverless log ingestion

What serverless is not about
Credit: Blue Waters main data center room in June 2010. Still empty except for power distribution cabinets, by Daniel Schwen
https://commons.wikimedia.org/wiki/File:Blue_Waters_main_data_center_room.jpg

The function
https://gitlab.com/ric_harvey/cf-es-log-ingester

Permissions
Cluster access
Account ID

Permissions
IAM Roles
Permissions to allow:
Access to ES – (get granular and lock it down to a single cluster)
Access S3 - (read only permissions to one bucket)
CloudWatch Logs – (push logs from lambda app)

Demo

Thank you
Javier Ramirez
@supercoco9

Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf

Similaire à Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

Analyzing your web and application logs with Cloudfront and ElasticSearch Service.pdf