Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.
30. Customize log data
%D = The time taken to serve the request, in microseconds
%T = The time taken to serve the request, in seconds
%v = The canonical ServerName of the server serving the request
%{Foobar}C = The contents of cookie Foobarin the request sent to the server
%{Foobar}n = The contents of note Foobarfrom another module
Source: https://httpd.apache.org/docs/2.2/mod/mod_log_config.html
Apache LogFormat:
37. Instance failure. Filled disks. Auto Scaling actions.
https://secure.flickr.com/photos/eurleif/186807023
38.
39. syslog-ng,rsyslog, nxlog
Pros:
•Open source
–Linux, Windows,and almost everything else!
•Both variants of syslogd
–Add filtering, flexible configuration, TCP as a transport
•Runs as an OS process
•Typically take the centralized data and feed into another analytics tool
•Can often accept logs from third-party sources like network devices
Central logging instance
virtual private cloud
App instances
Etc.
instances
Web instances
40. syslog-ng, rsyslog, nxlog
Cons:
•No built-in analytics/dashboard abilities
•Typically centralized host can become a single point of failure
•Potentially more difficult to scale
–Federate logs to different centralized hosts?
Central logging instance
virtual private cloud
App instances
Etc.
instances
Web instances
41. Splunk
Pros:
•Enterprise grade
•Extremely scalable
•Fault tolerance and load balancing built in
•Security of data built in
•Can technically accept data from other third-party sources as well
•Full log forwarding, analyzing, dashboardingstack + third-party apps
Splunkindexer
virtual private cloud
Splunkindexer
App instances
Etc.
instances
Web instances
42. Splunk
Cons:
•Enterprise-grade pricing
•Enterprise-grade licensing
•Indexer resources become an important part of capacity planning
A great option for Enterprises and large shops!
Splunkindexer
virtual private cloud
Splunkindexer
App instances
Etc.
instances
Web instances
43. Logstash
Pros:
• Open source
• Extremely scalable
• Fault tolerance built in
• Support offerings from Elasticsearch!
• Active code base and ecosystem
• Pluggable
• Ties in with other tools for
dashboarding/analytics
virtual private cloud
App
instances
Etc.
instances
Web
instances
Redis
Elasticsearch Elasticsearch Elasticsearch
Logstash
indexer
Logstash
indexer
44. Logstash
Cons:
• “ELK Stack” has many moving pieces
• Lot of DIY to getting it set up
• Very quickly changing/improving
technology stack
Most popular open source option today!
virtual private cloud
App
instances
Etc.
instances
Web
instances
Redis
Elasticsearch Elasticsearch Elasticsearch
Logstash
indexer
Logstash
indexer
45. SaaSoptions
Pros:
•Hosted
•Very easy to get started with
•No concerns about scaling yourself
•Flexible pricing methods
•Support
•Either their agents or syslog to them
•Built-in dashboards/analytics tools
•Constantly adding features/capabilities
virtual private cloud
NAT instance
App instances
Etc.
instances
Web instances
46. SaaSoptions
Cons:
•Data leaving your control/infrastructure
•Some restrictiveness in flexibility of the dashboards, collection agents, archive limits
SaaSmakes a lot of sense if you are small and trying to move fast and should be focusing on product first!
virtual private cloud
NAT instance
App instances
Etc.
instances
Web instances
55. •Can process log files on the fly outputting metric data to numerous services:
–CloudWatch
–Ganglia
–Graphite via statsd
–Boundary
–DataDog
–many others!
•Runs as a constantly running daemon
•Little bit easier than Logster
•Can do metric output and full log centralization at the same time!
input {
file {
path => "/var/log/apache/access.log"
type => "apache-access” }
}
filter {
grok{
type => "apache-access"
pattern => "%{COMBINEDAPACHELOG}” }
}
output {
statsd{
# Count one hit every event by response
increment => "apache.response.%{response}” }
} (from: http://logstash.net/docs/1.4.2/tutorials/metrics-from-logs)
Logstash
79. Don’t do this by hand! Make use of tools:
Build basic log centralization into every AMI!
directory "/opt/aws/cloudwatch" do
recursive true
end
remote_file"/opt/aws/cloudwatch/awslogs-agent- setup.py" do
source "https://s3.amazonaws.com/aws-cloudwatch/ downloads/latest/awslogs-agent-setup.py"
mode "0755"
end
execute "Install CloudWatchLogs agent" do
command "/opt/aws/cloudwatch/awslogs-agent- setup.py-n -r us-west-2 -c /etc/cwlogs.cfg"
not_if{ system "pgrep-f aws-logs-agent-setup" }
end