Soumettre la recherche
Mettre en ligne
K8s monitoring with elk
•
6 j'aime
•
1,091 vues
윤
윤종원 윤종원
Suivre
k8s monitoring with elk
Lire moins
Lire la suite
Ingénierie
Signaler
Partager
Signaler
Partager
1 sur 34
Télécharger maintenant
Télécharger pour lire hors ligne
Recommandé
Cloud Native 오픈소스 서비스 소개 및 Serverless로 실제 게임 개발하기
Cloud Native 오픈소스 서비스 소개 및 Serverless로 실제 게임 개발하기
Jinwoong Kim
Data(?)Ops with CircleCI
Data(?)Ops with CircleCI
Jinwoong Kim
Serverless technologies with Kubernetes
Serverless technologies with Kubernetes
Provectus
Spark day 2017 - Spark on Kubernetes
Spark day 2017 - Spark on Kubernetes
Yousun Jeong
Kubernetes 101 Workshop
Kubernetes 101 Workshop
Vishal Biyani
Fact-Based Monitoring - PuppetConf 2014
Fact-Based Monitoring - PuppetConf 2014
Puppet
All the troubles you get into when setting up a production ready Kubernetes c...
All the troubles you get into when setting up a production ready Kubernetes c...
Jimmy Lu
Orchestrating Docker Containers with Google Kubernetes on OpenStack
Orchestrating Docker Containers with Google Kubernetes on OpenStack
Trevor Roberts Jr.
Recommandé
Cloud Native 오픈소스 서비스 소개 및 Serverless로 실제 게임 개발하기
Cloud Native 오픈소스 서비스 소개 및 Serverless로 실제 게임 개발하기
Jinwoong Kim
Data(?)Ops with CircleCI
Data(?)Ops with CircleCI
Jinwoong Kim
Serverless technologies with Kubernetes
Serverless technologies with Kubernetes
Provectus
Spark day 2017 - Spark on Kubernetes
Spark day 2017 - Spark on Kubernetes
Yousun Jeong
Kubernetes 101 Workshop
Kubernetes 101 Workshop
Vishal Biyani
Fact-Based Monitoring - PuppetConf 2014
Fact-Based Monitoring - PuppetConf 2014
Puppet
All the troubles you get into when setting up a production ready Kubernetes c...
All the troubles you get into when setting up a production ready Kubernetes c...
Jimmy Lu
Orchestrating Docker Containers with Google Kubernetes on OpenStack
Orchestrating Docker Containers with Google Kubernetes on OpenStack
Trevor Roberts Jr.
Intro to kubernetes
Intro to kubernetes
Faculty of Technical Sciences, University of Novi Sad
OpenStack Storage Overview
OpenStack Storage Overview
Bharat Kumar Kobagana
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
OpenStack Korea Community
Kubernetes 101
Kubernetes 101
Crevise Technologies
Micro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and Ansible
Bamdad Dashtban
Storage based on_openstack_mariocho
Storage based on_openstack_mariocho
Mario Cho
Kubernetes Introduction
Kubernetes Introduction
Peng Xiao
Scaling Microservices with Kubernetes
Scaling Microservices with Kubernetes
Deivid Hahn Fração
Cisco UCS loves Kubernetes, Docker and OpenStack Kolla
Cisco UCS loves Kubernetes, Docker and OpenStack Kolla
Vikram G Hosakote
What's new in Kubernetes
What's new in Kubernetes
Daniel Smith
CoreOS: The Inside and Outside of Linux Containers
CoreOS: The Inside and Outside of Linux Containers
Ramit Surana
Cloud data center and openstack
Cloud data center and openstack
Andrew Yongjoon Kong
Implement Advanced Scheduling Techniques in Kubernetes
Implement Advanced Scheduling Techniques in Kubernetes
Kublr
Kubernetes: My BFF
Kubernetes: My BFF
Jonathan Yu, P.Eng.
Scaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container Service
Ben Hall
AKS
AKS
girish goudar
Kubernetes 101 for Developers
Kubernetes 101 for Developers
Ross Kukulinski
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Jakob Karalus
Laying OpenStack Cinder Block Services
Laying OpenStack Cinder Block Services
Kenneth Hui
[OpenInfra Days Korea 2018] Day 2 - E4 - 딥다이브: immutable Kubernetes architecture
[OpenInfra Days Korea 2018] Day 2 - E4 - 딥다이브: immutable Kubernetes architecture
OpenStack Korea Community
Elk for applications on k8s
Elk for applications on k8s
Che-Chia Chang
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
DataWorks Summit/Hadoop Summit
Contenu connexe
Tendances
Intro to kubernetes
Intro to kubernetes
Faculty of Technical Sciences, University of Novi Sad
OpenStack Storage Overview
OpenStack Storage Overview
Bharat Kumar Kobagana
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
OpenStack Korea Community
Kubernetes 101
Kubernetes 101
Crevise Technologies
Micro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and Ansible
Bamdad Dashtban
Storage based on_openstack_mariocho
Storage based on_openstack_mariocho
Mario Cho
Kubernetes Introduction
Kubernetes Introduction
Peng Xiao
Scaling Microservices with Kubernetes
Scaling Microservices with Kubernetes
Deivid Hahn Fração
Cisco UCS loves Kubernetes, Docker and OpenStack Kolla
Cisco UCS loves Kubernetes, Docker and OpenStack Kolla
Vikram G Hosakote
What's new in Kubernetes
What's new in Kubernetes
Daniel Smith
CoreOS: The Inside and Outside of Linux Containers
CoreOS: The Inside and Outside of Linux Containers
Ramit Surana
Cloud data center and openstack
Cloud data center and openstack
Andrew Yongjoon Kong
Implement Advanced Scheduling Techniques in Kubernetes
Implement Advanced Scheduling Techniques in Kubernetes
Kublr
Kubernetes: My BFF
Kubernetes: My BFF
Jonathan Yu, P.Eng.
Scaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container Service
Ben Hall
AKS
AKS
girish goudar
Kubernetes 101 for Developers
Kubernetes 101 for Developers
Ross Kukulinski
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Jakob Karalus
Laying OpenStack Cinder Block Services
Laying OpenStack Cinder Block Services
Kenneth Hui
[OpenInfra Days Korea 2018] Day 2 - E4 - 딥다이브: immutable Kubernetes architecture
[OpenInfra Days Korea 2018] Day 2 - E4 - 딥다이브: immutable Kubernetes architecture
OpenStack Korea Community
Tendances
(20)
Intro to kubernetes
Intro to kubernetes
OpenStack Storage Overview
OpenStack Storage Overview
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
Kubernetes 101
Kubernetes 101
Micro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and Ansible
Storage based on_openstack_mariocho
Storage based on_openstack_mariocho
Kubernetes Introduction
Kubernetes Introduction
Scaling Microservices with Kubernetes
Scaling Microservices with Kubernetes
Cisco UCS loves Kubernetes, Docker and OpenStack Kolla
Cisco UCS loves Kubernetes, Docker and OpenStack Kolla
What's new in Kubernetes
What's new in Kubernetes
CoreOS: The Inside and Outside of Linux Containers
CoreOS: The Inside and Outside of Linux Containers
Cloud data center and openstack
Cloud data center and openstack
Implement Advanced Scheduling Techniques in Kubernetes
Implement Advanced Scheduling Techniques in Kubernetes
Kubernetes: My BFF
Kubernetes: My BFF
Scaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container Service
AKS
AKS
Kubernetes 101 for Developers
Kubernetes 101 for Developers
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Laying OpenStack Cinder Block Services
Laying OpenStack Cinder Block Services
[OpenInfra Days Korea 2018] Day 2 - E4 - 딥다이브: immutable Kubernetes architecture
[OpenInfra Days Korea 2018] Day 2 - E4 - 딥다이브: immutable Kubernetes architecture
Similaire à K8s monitoring with elk
Elk for applications on k8s
Elk for applications on k8s
Che-Chia Chang
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
DataWorks Summit/Hadoop Summit
KOCOON – KAKAO Automatic K8S Monitoring
KOCOON – KAKAO Automatic K8S Monitoring
issac lim
ELK stack at weibo.com
ELK stack at weibo.com
琛琳 饶
Druid + Superset (資料的快速通道)
Druid + Superset (資料的快速通道)
二文 郭
DOD 2016 - Stefan Thies - Monitoring and Log Management for Docker Swarm and...
DOD 2016 - Stefan Thies - Monitoring and Log Management for Docker Swarm and...
PROIDEA
Stream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NET
confluent
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
SolarWinds Loggly
"How about no grep and zabbix?". ELK based alerts and metrics.
"How about no grep and zabbix?". ELK based alerts and metrics.
Vladimir Pavkin
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
Amazon Web Services
How to improve ELK log pipeline performance
How to improve ELK log pipeline performance
Steven Shim
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltStack
Logstash
Logstash
琛琳 饶
ログ収集プラットフォーム開発におけるElasticsearchの運用
ログ収集プラットフォーム開発におけるElasticsearchの運用
LINE Corporation
Elks for analysing performance test results - Helsinki QA meetup
Elks for analysing performance test results - Helsinki QA meetup
Anoop Vijayan
Search and analyze data in real time
Search and analyze data in real time
Rohit Kalsarpe
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.
Renzo Tomà
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek
PROIDEA
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic Stack
Jakub Hajek
Vinetalk: The missing piece for cluster managers to enable accelerator sharing
Vinetalk: The missing piece for cluster managers to enable accelerator sharing
VINEYARD - Versatile Integrated Accelerator-based Heterogeneous Data Centres
Similaire à K8s monitoring with elk
(20)
Elk for applications on k8s
Elk for applications on k8s
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
KOCOON – KAKAO Automatic K8S Monitoring
KOCOON – KAKAO Automatic K8S Monitoring
ELK stack at weibo.com
ELK stack at weibo.com
Druid + Superset (資料的快速通道)
Druid + Superset (資料的快速通道)
DOD 2016 - Stefan Thies - Monitoring and Log Management for Docker Swarm and...
DOD 2016 - Stefan Thies - Monitoring and Log Management for Docker Swarm and...
Stream Processing with Apache Kafka and .NET
Stream Processing with Apache Kafka and .NET
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
"How about no grep and zabbix?". ELK based alerts and metrics.
"How about no grep and zabbix?". ELK based alerts and metrics.
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
How to improve ELK log pipeline performance
How to improve ELK log pipeline performance
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
Logstash
Logstash
ログ収集プラットフォーム開発におけるElasticsearchの運用
ログ収集プラットフォーム開発におけるElasticsearchの運用
Elks for analysing performance test results - Helsinki QA meetup
Elks for analysing performance test results - Helsinki QA meetup
Search and analyze data in real time
Search and analyze data in real time
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic Stack
Vinetalk: The missing piece for cluster managers to enable accelerator sharing
Vinetalk: The missing piece for cluster managers to enable accelerator sharing
Dernier
S1S2 B.Arch MGU - HOA1&2 Module 3 -Temple Architecture of Kerala.pptx
S1S2 B.Arch MGU - HOA1&2 Module 3 -Temple Architecture of Kerala.pptx
SCMS School of Architecture
A Study of Urban Area Plan for Pabna Municipality
A Study of Urban Area Plan for Pabna Municipality
Morshed Ahmed Rahath
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Cara Menggugurkan Kandungan 087776558899
Double Revolving field theory-how the rotor develops torque
Double Revolving field theory-how the rotor develops torque
BhangaleSonal
Orlando’s Arnold Palmer Hospital Layout Strategy-1.pptx
Orlando’s Arnold Palmer Hospital Layout Strategy-1.pptx
MuhammadAsimMuhammad6
AIRCANVAS[1].pdf mini project for btech students
AIRCANVAS[1].pdf mini project for btech students
vanyagupta248
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
Arindam Chakraborty, Ph.D., P.E. (CA, TX)
Online electricity billing project report..pdf
Online electricity billing project report..pdf
Kamal Acharya
Block diagram reduction techniques in control systems.ppt
Block diagram reduction techniques in control systems.ppt
NANDHAKUMARA10
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ssuser89054b
Employee leave management system project.
Employee leave management system project.
Kamal Acharya
Online food ordering system project report.pdf
Online food ordering system project report.pdf
Kamal Acharya
A CASE STUDY ON CERAMIC INDUSTRY OF BANGLADESH.pptx
A CASE STUDY ON CERAMIC INDUSTRY OF BANGLADESH.pptx
maisarahman1
Standard vs Custom Battery Packs - Decoding the Power Play
Standard vs Custom Battery Packs - Decoding the Power Play
Epec Engineered Technologies
Design For Accessibility: Getting it right from the start
Design For Accessibility: Getting it right from the start
Quintin Balsdon
Unleashing the Power of the SORA AI lastest leap
Unleashing the Power of the SORA AI lastest leap
RishantSharmaFr
FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced Loads
FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced Loads
Arindam Chakraborty, Ph.D., P.E. (CA, TX)
Computer Networks Basics of Network Devices
Computer Networks Basics of Network Devices
ChandrakantDivate1
Computer Lecture 01.pptxIntroduction to Computers
Computer Lecture 01.pptxIntroduction to Computers
MairaAshraf6
"Lesotho Leaps Forward: A Chronicle of Transformative Developments"
"Lesotho Leaps Forward: A Chronicle of Transformative Developments"
mphochane1998
Dernier
(20)
S1S2 B.Arch MGU - HOA1&2 Module 3 -Temple Architecture of Kerala.pptx
S1S2 B.Arch MGU - HOA1&2 Module 3 -Temple Architecture of Kerala.pptx
A Study of Urban Area Plan for Pabna Municipality
A Study of Urban Area Plan for Pabna Municipality
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Double Revolving field theory-how the rotor develops torque
Double Revolving field theory-how the rotor develops torque
Orlando’s Arnold Palmer Hospital Layout Strategy-1.pptx
Orlando’s Arnold Palmer Hospital Layout Strategy-1.pptx
AIRCANVAS[1].pdf mini project for btech students
AIRCANVAS[1].pdf mini project for btech students
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
Online electricity billing project report..pdf
Online electricity billing project report..pdf
Block diagram reduction techniques in control systems.ppt
Block diagram reduction techniques in control systems.ppt
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Employee leave management system project.
Employee leave management system project.
Online food ordering system project report.pdf
Online food ordering system project report.pdf
A CASE STUDY ON CERAMIC INDUSTRY OF BANGLADESH.pptx
A CASE STUDY ON CERAMIC INDUSTRY OF BANGLADESH.pptx
Standard vs Custom Battery Packs - Decoding the Power Play
Standard vs Custom Battery Packs - Decoding the Power Play
Design For Accessibility: Getting it right from the start
Design For Accessibility: Getting it right from the start
Unleashing the Power of the SORA AI lastest leap
Unleashing the Power of the SORA AI lastest leap
FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced Loads
FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced Loads
Computer Networks Basics of Network Devices
Computer Networks Basics of Network Devices
Computer Lecture 01.pptxIntroduction to Computers
Computer Lecture 01.pptxIntroduction to Computers
"Lesotho Leaps Forward: A Chronicle of Transformative Developments"
"Lesotho Leaps Forward: A Chronicle of Transformative Developments"
K8s monitoring with elk
1.
k8s monitoring &
alert with elasticsearch Kubernetes Korea Group Meetup 2018.11.23 윤종원 (sabper@gmail.com)
2.
... • SI •
• (?) ( ...) • ( ??!!) • infra ... ,, • k8s ...
3.
4.
... • k8s ELK
stack • k8s • k8s cpu, memory, disk resource • alarm … • ELK k8s log / monitoring
5.
prometheus !!! • ... •
node-exporter, kube-state-metrics, … • !! - ? • .. • ... ( ... ) • [OpenInfra Days Korea 2018] OpenInfra monitoring with Prometheus
6.
ELK • ELK stack
.. • ES ELK stack • - ... • ...!!! • ( )
7.
for k8s pod
8.
k8s application
9.
? • k8s , •
kubectl logs -f pod-name • - • , • ... • ,,, (reponse time per sec, request per sec …) -> -> ( ) -> ` ` !!
10.
k8s filebeat to
kafka pod container log system log ingress-nginx log with add_kubernetes_metadata container_name, conatiner_label, node_name …
11.
filebeat k8s deploy
!! https://github.com/elastic/examples/tree/master/MonitoringKubernetes filebeat - k8s
12.
,,, ... ...
13.
filebeat multiline java exception
1 row multi row -> filebeat multi row 2018-08-30 09:44:22.847 [pool-2-thread-1] INFO com.barogo.dispatch.util.Util:46 APIPoint getCall0Riders 0 141 Aug 31, 2018 5:52:48 PM com.amazonaws.http.AmazonHttpClient executeHelper INFO: Unable to execute HTTP request: Connection reset java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:209) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ConfigMap - filebeat input - input type docker - docker log - multiline pattern - java multiline example - Test multiline pattern
14.
filebeat drop_event k8s pod
healthcheck access log 2018-11-21T00:08:41.294Z - info: GET 200 /healthz HTTP/1.1 kube-probe/1.10 [time 0.318 ms] "Request-Body" :{} "Response-Body" : "" 2018-11-21T00:08:41.485Z - info: GET 200 /healthz HTTP/1.1 kube-probe/1.10 [time 0.317 ms] "Request-Body" :{} "Response-Body" : "" 2018-11-21T00:08:44.485Z - info: GET 200 /healthz HTTP/1.1 kube-probe/1.10 [time 0.338 ms] "Request-Body" :{} "Response-Body" : "" 2018-11-21T00:08:47.485Z - info: GET 200 /healthz HTTP/1.1 kube-probe/1.10 [time 0.371 ms] "Request-Body" :{} "Response-Body" : "" ConfigMap - processors - drop_event - condition - - Test multiline pattern
15.
Why? Logstash &
Kafka • metadata • , id • logstash order_no, uid filed • • filebeat , • filebeat output logstash , filebeat -> kafka -> logstash -> elasticsearch cloud kibana
16.
logstash - json
parsing flask_response {"timestamp": 1533106987681, "gps": [{"lat": 37.5208423071, "lon": 127.0370946609}, {"lat": 37.5168702957094, "lon": 127.038314337406}], "route": [{"lat": 37.5208423071, "lon": 127.0370946609}, {"lat": 37.52101092506974, "lon": 127.037041062907}, {"lat": 37.5211672, "lon": 127.0375327}, {"lat": 37.5204709, "lon": 127.0376081}, {"lat": 37.5197674, "lon": 127.0377019}, {"lat": 37.5194147, "lon": 127.0377445}, {"lat": 37.5192907, "lon": 127.0377578}, {"lat": 37.5194287, "lon": 127.0383417}, {"lat": 37.5188587, "lon": 127.0385572}, {"lat": 37.5183706, "lon": 127.0387465}, {"lat": 37.5178821, "lon": 127.0389347}, {"lat": 37.5173377, "lon": 127.0391453}, {"lat": 37.5172205, "lon": 127.0387591}, {"lat": 37.51708223867784, "lon": 127.03825608200496}, {"lat": 37.5168702957094, "lon": 127.038314337406}], "second": 135, "distance": 718, "call_status": "rest", "order_no": 1, "uid": "uid", "platform": "PostmanRuntime", "endpoint": "route", "duplicates": "remove", "nearest_node_within": 150, "smoothing_node_within": 2.5, "st_ed": 0.1368551254272461, "st_sp1": -2.5033950805664062e-05, "sp2_sp3": -0.026725292205810547, "sp3_sp4": -1.430511474609375e-06, "sp4_sp5": -0.0015358924865722656, "sp5_sp6": -0.019712209701538086, "sp6_sp7": -0.08764910697937012} { "timestamp":1533106987681, "gps":[ { "lat":37.5208423071, "lon":127.0370946609 }, { "lat":37.5168702957094, "lon":127.038314337406 } ], "route":[ { "lat":37.5208423071, "lon":127.0370946609 }, { "lat":37.52101092506974, "lon":127.037041062907 }, { "lat":37.5168702957094, "lon":127.038314337406 } ], "second":135, "distance":718, "call_status":"rest", "order_no":1, "uid":"uid", "platform":"PostmanRuntime", "endpoint":"route", "duplicates":"remove", "nearest_node_within":150, "sp5_sp6":-0.019712209701538086, "sp6_sp7":-0.08764910697937012 } json prefix flask_response json parsing dissect message `{` multiline pattern
17.
logstash - custom
field , uid custom filed 2018-11-20T09:14:30.207Z - info: [1147] [WjZtOadDkhVsl0FpFnFaFJEMLAI3] [newcall-new-single] Dispatch Result From dispatch-Response SQS { order_no: '1147', users: [ { uid: 'WjZtOadDkhVsl0FpFnFaFJEMLAI3', order_routes: [ { order_no: '1147', order_type: 'pickup', order_status: 'pickup', receiption_dt: '2018-11-20T09:14:28.448Z', grok filter (grok test) grok filter - order_no, uid, command message2
18.
Log Data -
kibana • kubernetes metadata • cloud metadata • custom data
19.
filebeat nginx-ingress • pod
nginx-ingress • response time per sec, request per sec • nginx-ingress acceess log !?!?! ConfigMap filebeat autodiscover namespace ingress-nginx pod nginx module
20.
logstash - nginx •
filebeat nginx module field filed • service container_name, response time • metadata ingress • pod, namespace, • nginx • service_name, namespace, response / request time
21.
logstash nginx 121.135.235.252 -
- [22/Nov/2018:08:15:28 +0000] 1542874528.663 "POST /api/v1/location/group HTTP/1.1" 200 3823 "https://dev-admin.mvmt.delivery/rider/control" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36" 930 0.348 [staging-dev-node-admin-api-svc-80] 100.118.136.176:3500 44425 0.348 200 1. grok pattern nginx 2. [staging-dev-node-admin-api-svc-80] namespace, service_name 3. service_name namespace filed filed
22.
23.
Resource Monitoring for k8s
24.
resource • k8s •
• • k8s - ,, • k8s • , !
25.
metricbeat k8s to
ES cloud module - kubernetes metricset : fetch from kubelet, kube-stat-metrics
26.
metricbeat - k8s metricbeat
k8s deploy !! - !! https://github.com/elastic/examples/tree/master/MonitoringKubernetes
27.
Monitoring k8s state_pod metricset
: state_deployment
28.
Monitoring k8s node metricset
: node
29.
Monitoring k8s container metricset
: container
30.
Alert for k8s
31.
watcher - xpack •
alarm • ES cloud xpack - watcher • xpack -> elastalert • watcher • : trigger • es query: input • : condition • slack, email noti : action
32.
? error message log
- metricbeat metricset : event type: Warning -
33.
• app /
k8s resource , alarm , • - • aggregation ? • ? • logstash, filebeat, metribeat ? • k8s ? - aws dns ? reigon ?? • • ( ) ... • infra
Télécharger maintenant