Elastic Stack @ Swisscom Application Cloud

Elastic Stack @ Swisscom Application Cloud
Swisscom (Schweiz) AG
Bremgartner Lucas
13.06.2017
C1 - Public

> Introduction
> What is Swisscom Application Cloud / What is the Elastic Stack
> Use of Elastic Stack @ Swisscom Application Cloud
> Process Logs with Logstash @ Swisscom Application Cloud
> Testing growing Logstash Configurations
2Agenda
BremgartnerLucas,INI-DEV-DIG-TCL-PFD-ELR
ElasticStack@SwisscomAppCloud.pptxC1-
Public
08.06.17

Home of Cloud Native Applications

08.06.17
4
Public
What is Swisscom Application Cloud / What is the Elastic Stack
Kibana
User Interface
ElasticsearchStore, Index,
& Analyze
Ingest
Logstash Beats
Elastic Stack
Swisscom
Developer Portal

5
developer.swisscom.com
www.mycloud.ch
Internal AppCloud
iAPC
Use Cases

> Lucas Bremgartner, Cloud Developer @ Swisscom Application Cloud
Quick notes:
> Elasticsearch user since version 0.9.x.
> My current «goto» programming language is Go
Open Source:
> Logstash Community Maintainer
> Contributor to logstash-filter-verifier (LFV)
> Maintainer of pigeon (PEG grammar parser generator for Go)
> Author of logstash-config (parser for Logstash configuration, written in Go)
08.06.17
6
Public
Introduction

> ELK as a Service
– Available in marketplace, containing Elasticsearch, Logstash and Kibana
– Intended use-case: collect logs from apps running in Application Cloud and visualize them
with Kibana
> Elasticsearch Enterprise
– Currently under development
– Intended use-case: scalable Elasticsearch clusters as a service
– Open for all Elasticsearch use cases (classical full-text search, log management, geo
location search, etc.)
> Elastic Stack for Log Management of the Infrastructure
– Classical pipeline with Filebeat, Logstash, Elasticsearch and Kibana
08.06.17
7
Public
Use of Elastic Stack @ Swisscom Application Cloud

14.06.2017
8
Public
Process Logs with Logstash
@ Swisscom Application Cloud

> Application instances in Cloud Foundry are ephemeral, storing logs on local disk is not a good
idea
> With multiple instances of the app running in parallel, an aggregated log stream is needed
> The 12 factor apps methodology defines for log data:
– «A twelve-factor app never concerns itself with routing or storage of its output stream. It
should not attempt to write to or manage logfiles. Instead, each running process writes its
event stream, unbuffered, to stdout»
> Cloud Foundry collects and ships the log events of the application and makes the log events
available through the API: cf logs <app>
> Cloud Foundry also allow to stream the logs to a customer provided service (syslog or https)
08.06.17
9
Public
Application Logs in Swisscom Application Cloud

App deployed by
Customer
App deployed by
Customer
Service by
Swisscom
AppCloud
05.09.16
10
BremgartnerLucas,ENT-NTC-PHC-PFD-ELR
ELKEnterprise.pptxC2-Internal
Stream Application Logs in Cloud Foundry
Logstash KibanaElasticsearch
ES Dashboards
(e.g. Cerebro,
Kopf)
Logstash
Logstash
House-
Keeping
(e.g. curator)
App
App logs to stdout,
CF log facility forwards
via customer provided
service to Logstash
App
App

> Buildpacks provide framework and runtime support for your applications.
> Buildpacks typically examine user-provided artifacts to determine what dependencies to
download and how to configure applications to communicate with bound services.
> This is done by three entrypoints:
– bin/detect: determines whether or not to apply the buildpack to an app.
– bin/compile: builds a droplet by packaging the app dependencies, assuring that the app
has all the necessary components needed to run.
– bin/release: provides feedback metadata to Cloud Foundry indicating how the app
should be executed.
08.06.17
11
Public
What is a Cloud Foundry Buildpack

08.06.17
12
Public
Demo

14.06.2017
13
Public
Testing growing Logstash Configurations

Elasticsearch KibanaRabbitMQLogstash
(Shipper to
RabbitMQ)
Filebeat on
Edge Nodes
08.06.17
14
Public
Log Management @ Swisscom Application Cloud
Logstash
(Filter)

> Every application/service/daemon has its own log format, which needs to be tackled with a
specific set of Logstash filters.
> While adding more and more log formats, the complexity increases and changes to the
configuration become more and more delicate.
> With new software versions (lifecycle), also changed log patterns may occur, which need to be
processed in parallel to the old one.
> Integrate the testing of the Logstash configuration into the CI pipeline.
> Additionally to the Logstash configuration, also the Elasticsearch mapping needs to be
maintained.
> The Elasticsearch mapping could become a quite large (JSON file), which is a pain to update
(unhandy, error prone, etc.).
> Undocumented Elasticsearch mappings are harder to understand and to maintain (especially if
this is not done on a regular bases)
08.06.17
15
Public
Challenges

> Logstash is an open source, server-side data processing pipeline
that ingests data from a multitude of sources simultaneously,
transforms it, and then sends it to your favorite “stash.”
> Logstash follows the classical input–process–output (IPO) pattern,
the process stage is called «filter».
> A long list of different input, filter and output plugins is available,
which allow to adopt Logstash to a wide variety of use cases.
> A Logstash configuration is like a program which is applied to
every log event.
08.06.17
16
Public
Logstash

> LFV provides unit test kind of functionality for Logstash filter configurations
> Run test input against a given Logstash configuration and compare the result with the
expected value
08.06.17
17
Public
Logstash Filter Verifier
Logstash
filter
config
LFV Logstash
Test
cases
Kudos to @magnusbaeck for developing and maintaining Logstash Filter Verifier (LFV)
«If you get something wrong (… in the Logstash config …) you might have millions of
incorrectly parsed events before you realize your mistake. » – Magnus Bäck

Logstash Filter Verifier testsuite file:
{
"fields": {},
"codec": "line"
"ignore": [ "@version", "host" ],
"testcases": [ {
"input": [
"2017/06/12 08:12:58 WARN message e361827a-990e-
4237-8ea3-047f292f1d14 (1534 bytes) from <mind-blowing-
musa@dagger.com> to <epic_williams@centaur.com> could not
be sent, will retry"
],
"expected": [ {
"@timestamp": "2017-06-12T08:12:58.000Z",
"severity": "WARN",
"from": "mind-blowing-musa@dagger.com",
"to": "epic_williams@centaur.com",
"message": "could not be sent, will retry",
"size": 1534
} ]
} ] }
08.06.17
18
Public
Example
Additional fields, provided by the
source or added by the input plugin
Codec to decode input data (usually
one of line or json_lines)
Fields to be ignored, when the result
is compared
Testcases:
• provided input
• expected log event provided by Logstash

08.06.17
19
Public
Demo

14.06.2017
20
Public
Thanks & Questions?

Software & Tools:
> Logstash Buildpack for Swisscom Application Cloud
https://github.com/swisscom/cf-buildpack-logstash
> Kibana Buildpack for Swisscom Application Cloud
https://github.com/swisscom/cf-buildpack-kibana
> Logstash Filter Verifier (LFV)
https://github.com/magnusbaeck/logstash-filter-verifier
> Logstash Config Check
https://github.com/breml/logstash-config
Additional Links:
> 12 Factor Apps: https://12factor.net/
> Grok Debugger: https://grokdebug.herokuapp.com/
> ./jq: https://stedolan.github.io/jq/
> jsondiff: https://github.com/yudai/gojsondiff/
> dockerize: https://github.com/jwilder/dockerize
08.06.17
21
Public
Links

Elastic Stack @ Swisscom Application Cloud

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Elastic Stack @ Swisscom Application Cloud

Similaire à Elastic Stack @ Swisscom Application Cloud (20)

Dernier

Dernier (20)

Elastic Stack @ Swisscom Application Cloud