메타트론 디스커버리의 확장팩인 Anomaly detection을 소개합니다. Introduce an anomaly detection application, an ex-pack for the Metatron Discovery

1. Metatron Anomaly
powered by

2. T A B L E O F C O N T E N T S
Metatron Anomaly
01. Intro
02. Metatron Anomaly
03. Feature
04. Conclusion

3. 01. Intro

4. What is Anomaly
For System Monitoring

5. Why we made it
Machine Learning Scientist
Tagged Data Learning
System
But,,,,
still hard !!!!
Complex Data
Solution

6. So,we made Metatron Anomaly
Easy
Machine
Learning
Anomaly
Management
Big Data
Analyze

7. 02. Metatron Anomaly

8. Visualize
Alarm
Report
Analytics
Data Flow
Machine Learning
Anomaly Management
Automated Anomaly Detection
Anomaly Analytics, Alarm

9. Key Concept

10. Our Menu
알람 룰 만들기 Create Alarm Rule
알람 리스트
Alarm list
알람 룰 리스트
Alarm Rule list
Alarm Rule
Detail
Alarm Detail
통계
Statistics
모든 알람의 요약 및
통계를 보고 빠르게
전체 상황을 파악
알람을 확인&검색
하거나 각 알람 별
디테일을 확인
만들어진 알람 룰의
디테일을 확인하거나
룰을 수정

11. 03. Feature

12. Feature1.
Machine Learning

13. TO-BE
Why do weneed machine learning?
많은 기기의 다양한 상태를 이런식으로 관
리하기가 어려워 machine learning을 통해
서 이를 분석하고 그에 따른 적절한 모델을
적용함으써보다 쉽고 빠르게 더 상세한 감
시를 할 수 있음
AS-IS
기준 범위를 지정하고 그 범위를 벗
어나면 이상 상태로 판단하지만, 좀
더 상세한 감시가 필요해짐에 따라
시간을 분할하여 이를 각각 범위를
설정하여 판단함
설비의 상태나 환경이 일부 변할때
마다 이러한 일이 반복적으로 일어나
게 됨

14. TO-BEAS-IS
Why do we need machine learning?
큰 문제가 발생한 이후에 그에 대한
대처를 수행함으로써 실제로 문제가
발생하게 됨.
이러한 문제의 발생 자체를 막고자
하는 니즈가 있음
상세한 분석을 통해 데이타의 오류 형태
등으로 전조현상을 발견하고, 미리 오류를
발견함으로 선재적 대응이 가능함

15. How machine learning worksin
1. 정답 없이 사용가능한 Unsupervised Learning 사용
3. All-in-one solution 제공
2.Druid를 사용하여 분석환경을 별개로 구성할 필요없음
4.사이언티스트가 없이도 시스템에서 자동 모델 제안
Scientist

16. Select
model
Predict
evaluati
on
Anomaly
score
Anomaly
detect
Bias
MAD
MAPE
MSE
SAE
MAE
SMAPE
MAPE
MASE
MAPEE
NaiveForecastingModel
MovingAverageModel
WeightedMovingAverageModel
RegressionModel
MultipleLinearRegressionModel
PolynomialRegressionModel
SimpleExponentialSmoothingModel
DoubleExponentialSmoothingModel
TripleExponentialSmoothingModel
OlympicModel
SpectralSmoother
Abnormal Predicted Model Create Process (UnsupervisedLearning)
Predict
train
Predict
train
Predict
train
Predict
train
과거 데이터로 11종 예측 모델 적용
(지속적으로 추가 예정)
가장 오차가 적은
모델 선정
오차를
수치화 함
선정된 모델로
현재 데이터로 예측
스레시홀드를 넘을 때
알람 발생

17. 재학습(Re-Learning)
데이터가 달라져서
모델 정확도가 떨어지는 현상이 발생
재학습이 필요한 이유 Data Science Lifecycle
데이터 확인 → 학습 (machine learning) → 이상 탐지 → 원인 분석 → Feedback
재학습 (machine learning) → 보다 정확한 이상 탐지

18. Extension
Druid
Extension
Metatron
Flow
Zeppelin
3가지 알고리즘 적용방식

19. Feature2.
Anomaly Management

20. Anomaly Management

21. UI
Alarm detail & Report
Alarm Statistics
Alarm Search Alarm rule detail
Alarm Create

22. 04. Conclusion

23. Use case
case 1
각 measure 별로 독립적으로 발생함.
각 value 별로 training 하고 예측을 수행함으로써
이상 징후를 발견한다.
case 2 :
각 measure가 같은 패턴으로 발생함.
같은 measure끼리 비교하여 차이점을 찾음으로써
이상 징후를 발견한다.

24. Metatron Architecture
Anomaly Integrator MDM
Application
Tool
Solution
Target MarketingMachine VisionAPM

25. DashBoard/Chart
Workbench – SQL Explorer
Embedded Analytics
Data Prep. – Data Transformer
Workspace
Data Lineage
MDM
[별첨] Metatron Discovery

26. Metatron Discovery 연계로 인한 장점

27. Metatron Anomaly의 장점

28. Roadmap

29. The End

30. 시스템 구조
Client Server Druid
Alarm
rule page
Query
generate
Data
process
Alarm
notice
Predict
train
Predict
evaluatio
n
Anomaly
score
Anomaly
detect
Alarm
schedule
Data
process
Alarm
page
Alarm
History
Data
process
1 2
34
1
2
3
1
4
2
3
Predict
evaluatio
nModel
manage
Select
model

31. 외부 분석 연계(notebook)
Client Server Druid
Alarm
rule page
Query
generate
Data
process
Alarm
schedule
1 2
36
zeppelin
Predict
training
Anomaly
detect
Model
manage
4
5
Predict
evaluatio
n

32. 외부 분석 연계(metatron-stream)
Client Server
Metatron-
stream
Alarm
rule page
Query
generate
Data
process
Alarm
notice
Predict
training
Predict
evaluatio
n
Anomaly
score
Anomaly
detect
Alarm
schedule
Data
process
Alarm
page
Alarm
History
Data
process
1 2
34
1
2
3
1
4
2
3
Model
manage
Druid

33. Predict
evaluati
on
Alarm Rule Create Process
Select Data Source Select Measure Auto Model training & Recommend
Adjust Alarm Threshold Complete Alarm Rule

34. Metatron Anomaly -DRUID
powered by

35. 시스템 구조
Client Server Druid
Alarm
rule page
Data
process
Alarm
notice
Predict
train
Predict
evaluatio
n
Anomaly
score
Anomaly
detect
Alarm
schedule
Data
process
Alarm
page
Alarm
History
Data
process
1 2
34
1
2
3
1
4
2
3
Predict
evaluatio
nModel
manage
Recommend
model

36. Machine learning algorithm call
Metatron anomaly
Druid
extension/druid-egads
egads openforecast

37. Add anomaly extension
1. Extension 폴터에 파일을 넣고
2. druid.extensions.loadList에 druid-egads를 추가

38. Processchange for alarm
Druid Egads
Predict
train
Predict
evaluatio
n
Target
Data
1
2
3Predict
Data
Druid Egads
Predict
train
Training
Data
1
2
3*
4*
Predict
evaluatio
nPredict
Data
Evaluate
Data
Trained
Model

39. Druid anomaly query
{
"queryType" : "groupBy",
"dataSource" : "passengers_01",
"virtualColumns": [
{
"type": "expr",
"outputName": "user_defined.basetime",
"expression": "TIMESTAMP( time_format( __t
ime, 'yyyy-MM'), 'yyyy-MM' ) "
}
],
"dimensions" : [
{
"type": "default",
"dimension": "user_defined.basetime",
"outputName": "basetime"
}
],
"limitSpec" : {
"type" : "default",
"limit" : 1000,
"columns": [
{
"dimension": "basetime",
"direction": "ascending",
"dimensionOrder": "alphanumeric"
}
]
},
"granularity" : "MONTH",
"aggregations" : [
{
"type": "sum",
"name": "value",
"fieldName": "Value",
"inputType": "double"
}
],
"context" : {
"postProcessing" :
{
"type": "list",
"processors": [
{
"type": "rowToMap",
"timestampColumn": "__time"
},
{
"type": "anomaly",
"timestampColumn": "basetime",
"metricColumn": "value",
"predictColumn": "predict",
"anomalyColumn": "isAnomaly",
"tsModel": "TripleExponentialSmoothingMo
del",
"tsModelColumn": "appliedModel",
"adModel": "ExtremeLowDensityModel",
"parameters":
{
"THRESHOLD": "mape#20"
},
"timeGranularity": "MONTH"
}
]
}
},
"intervals" : [ "1949-01-01/1970-01-01" ]
}
"queryType" : "groupBy",
"type": "anomaly",
"tsModel": "TripleExponen
tialSmoothingModel"

40. Druid anomaly query return
[ {
"basetime" : 2493072000000,
"value" : 112.0,
"__time" : "1949-01-01T00:00:00.000Z",
"predict" : 112.0,
}, {
"basetime" : 2495750400000,
"value" : 118.0,
"__time" : "1949-02-01T00:00:00.000Z",
"predict" : -1034.9606,
"isAnomaly" : true
}, {
"basetime" : 2498169600000,
"value" : 132.0,
"__time" : "1949-03-01T00:00:00.000Z",
"predict" : -1145.7456,
"isAnomaly" : true
}, {
"basetime" : 2500848000000,
"value" : 129.0,
"__time" : "1949-04-01T00:00:00.000Z",
"predict" : -1096.7128,
"isAnomaly" : true
"basetime" : 2495750400000,
"value" : 118.0,
"__time" : "1949-02-01T00:00:00.000Z",
"predict" : -1034.9606,
"isAnomaly" : true

41. Druid anomaly training query
{
"queryType" : "groupBy",
"dataSource" : "passengers_01",
"virtualColumns": [
{
"type": "expr",
"outputName": "user_defined.basetime",
"expression": "TIMESTAMP( time_format( __t
ime, 'yyyy-MM'), 'yyyy-MM' ) "
}
],
"dimensions" : [
{
"type": "default",
"dimension": "user_defined.basetime",
"outputName": "basetime"
}
],
"limitSpec" : {
"type" : "default",
"limit" : 1000,
"columns": [
{
"dimension": "basetime",
"direction": "ascending",
"dimensionOrder": "alphanumeric"
}
]
},
"granularity" : "MONTH",
"aggregations" : [
{
"type": "sum",
"name": "value",
"fieldName": "Value",
"inputType": "double"
}
],
"context" : {
"postProcessing" :
{
"type": "list",
"processors": [
{
"type": "rowToMap",
"timestampColumn": "__time"
},
{
"type": "anomaly",
"runType": "training",
"timestampColumn": "basetime",
"metricColumn": "value",
"predictColumn": "predict",
"anomalyColumn": "isAnomaly",
"tsModel": "TripleExponentialSmoothingMo
del",
"tsModelColumn": "appliedModel",
"adModel": "ExtremeLowDensityModel",
"parameters":
{
"THRESHOLD": "mape#20"
},
"timeGranularity": "MONTH"
}
]
}
},
"intervals" : [ "1949-01-01/1970-01-01" ]
}
"queryType" : "groupBy",
"type": "anomaly",
"tsModel": "TripleExponen
tialSmoothingModel"
"runType": "training"

42. Druid anomaly training query return
[ {
"basetime" : 2493072000000,
"value" : 112.0,
"__time" : "1949-01-01T00:00:00.000Z",
"predict" : 112.0,
"appliedModel" : {
"period" : 12,
"startseasonal" :
[ 0.8598057626467811, 0.8546643735355421, 0.9890938674177445, 0.9581315969670006, 0.963956303602938, 1.100
7168862639305, 1.2300053054336821, 1.2338619760370966, 1.0820471920359958, 0.9491716457381539, 0.833161634
6697758, 0.9453834556513592 ],
"alpha" : 1.0,
"name" : "TripleExponentialSmoothingModel",
"startbase" : 126.66666666666667,
"range" : 455.0,
"starttrend" : 1.0833333333333333,
"type" : "Forecast",
"beta" : 0.9999999999999999,
"gamma" : 0.0
}
}, {
"basetime" : 2495750400000,
"value" : 118.0,
"__time" : "1949-02-01T00:00:00.000Z",
"predict" : -1034.9606,
"isAnomaly" : true
}, {
"basetime" : 2498169600000,
"value" : 132.0,
"__time" : "1949-03-01T00:00:00.000Z",
"predict" : -1145.7456,
"isAnomaly" : true
}, {
"basetime" : 2500848000000,
"value" : 129.0,
"__time" : "1949-04-01T00:00:00.000Z",
"predict" : -1096.7128,
"isAnomaly" : true
},
"name" : "TripleExponentialSmoothingModel",
"period" : 12,
"startbase" : 126.6666,
"starttrend" : 1.08333,
"startseasonal" :[……],
"alpha" : 1.0,
"beta" : 0.999999,
"gamma" : 0.0,

43. Druid anomaly evaluate query
{
"queryType" : "groupBy",
"dataSource" : "passengers_01",
"virtualColumns": [
{
"type": "expr",
"outputName": "user_defined.basetime",
"expression": "TIMESTAMP( time_format( __t
ime, 'yyyy-MM'), 'yyyy-MM' ) "
}
],
"dimensions" : [
{
"type": "default",
"dimension": "user_defined.basetime",
"outputName": "basetime"
}
],
"limitSpec" : {
"type" : "default",
"limit" : 1000,
"columns": [
{
"dimension": "basetime",
"direction": "ascending",
"dimensionOrder": "alphanumeric"
}
]
},
"granularity" : "MONTH",
"aggregations" : [
{
"type": "sum",
"name": "value",
"fieldName": "Value",
"inputType": "double"
}
],
"context" : {
"postProcessing" :
{
"type": "list",
"processors": [
{
"type": "rowToMap",
"timestampColumn": "__time"
},
{
"type": "anomaly",
"runType": "evaluate",
"timestampColumn": "basetime",
"metricColumn": "value",
"predictColumn": "predict",
"anomalyColumn": "isAnomaly",
"tsModel": "TripleExponentialSmoothingModel",
"tsModelColumn": "appliedModel",
"adModel": "ExtremeLowDensityModel",
"parameters":
{
"period" : 12,
"startseasonal" :
[ 0.8598057626467811, 0.8546643735355421, 0.989093867
4177445, 0.9581315969670006, 0.963956303602938, 1.100
7168862639305, 1.2300053054336821, 1.23386197603709
66, 1.0820471920359958, 0.9491716457381539, 0.8331616
346697758, 0.9453834556513592 ],
"alpha" : 1.0,
"name" : "TripleExponentialSmoothingModel",
"startbase" : 126.66666666666667,
"starttrend" : 1.0833333333333333,
"type" : "Forecast",
"beta" : 0.9999999999999999,
"gamma" : 0.0,
"THRESHOLD": "mape#20"
},
"timeGranularity": "MONTH"
}
]
}
},
"intervals" : [ "1949-01-01/1970-01-01" ]
}
"queryType" : "groupBy",
"type": "anomaly",
"tsModel": "TripleExponen
tialSmoothingModel"
"runType": " evaluate "
"name" : "TripleExponentialSmoothingModel",
"period" : 12,
"startbase" : 126.6666,
"starttrend" : 1.08333,
"startseasonal" :[……],
"alpha" : 1.0,
"beta" : 0.999999,
"gamma" : 0.0,

44. Druid anomaly evaluate query return
[ {
"basetime" : 2493072000000,
"value" : 112.0,
"__time" : "1949-01-01T00:00:00.000Z",
"predict" : 112.0,
}, {
"basetime" : 2495750400000,
"value" : 118.0,
"__time" : "1949-02-01T00:00:00.000Z",
"predict" : -1034.9606,
"isAnomaly" : true
}, {
"basetime" : 2498169600000,
"value" : 132.0,
"__time" : "1949-03-01T00:00:00.000Z",
"predict" : -1145.7456,
"isAnomaly" : true
}, {
"basetime" : 2500848000000,
"value" : 129.0,
"__time" : "1949-04-01T00:00:00.000Z",
"predict" : -1096.7128,
"isAnomaly" : true
"basetime" : 2495750400000,
"value" : 118.0,
"__time" : "1949-02-01T00:00:00.000Z",
"predict" : -1034.9606,
"isAnomaly" : true

45. deploy
broker
Druid
Historica
l
Historica
l
Historica
l
Historica
l
discovery
metatron

46. deploy
broker
Druid
Historica
l
Historica
l
Historica
l
Historica
l
discovery
metatron
broker
extension
egads
anomaly

47. Open source