Monitoring of SmartNews

1. Monitoring
of
SmartNews
2016/03/24 GREE Tech Talk #10

2. Self Introduction
• Nobutoshi Ogata
• Manager, Site Reliability Engineering
• @nobu666
• ❤ Whiskey, Cat, Heavy Metal
• Entrusted dev.(10y) ➡ GREE
infrastructure devision(3y) ➡ Some
startup(1y) ➡ SmartNews(2015/05-)

4. SmartNews

5. 16,000,000+
downloads
worldwide

9. Before Datadog
• We used:
• munin
• growthforecast
• cloudwatch
• Wanted to centralized management !

10. After Datadog - Phase1
• OK, we can manage centrally
• But...?
• We're respecting the free development of engineers !
• Problem that the monitoring setting is leaked out "

11. Phase2
• Introduce Interferon
• Datadog DSL
• Well, we can monitor all resources automatically
• But...?
• Unmaintained in active !
• Can't feel free to mute from Web UI "
• Lack of flexibility #

12. Phase3
• Integrated itamae
• Our engineers were used to write chef
• Easy to override default settings
• It's asynchronous. Feel free to mute from Web UI
• Integrated dogaws @takus
• Yet another Datadog CloudWatch Integragion
• We are used in combination with itamae

13. Datadog tips
• Event collect and easy overlay
• Provisioning
• Deploy
• etc

14. Datadog tips
• Easiness anomary detection
• Can't compared over 24hours until quite recently
• We request to be able to compare more longer period.
Thank Datadog for implementing !
• This is a closed function. If you want to use it, ask
Datadog support "

15. For example
• Comapare Kinesis records count EWMA
pct_change(median(last_1h),
1w_ago):ewma_20(avg:aws.kinesis.incoming_records{env
:production,cost:smartnews} by {name}) > 50
• Compare application warn log
change(median(last_1h),1w_ago):
sum:app.log.warn{env:production} by
{autoscaling_group} > 25

16. Talk more?
• Join our free lunch in Tokyo office !
• Ask me later "

17. We're hiring!
Only two people on Site Reliability
Engineering Team !
• スマニューのSite Reliability Engineer
募集！
• http://about.smartnews.com/en/
careers/