The lines between Development and Operations people have gotten blurry and lots of skills needs to be held by both sides. In the talk we'll talk about all of the considerations that are needed to be taken when creating a development and production environment, mentioning Continuous Integration, Continuous Deployment and the Buzzword "DevOps", also talking about some real implementations in the industry. Of course how can we leave out the real enabler of the whole deal, "The Cloud", Giving us a tool set that makes life much easier when implementing all of these practices.

1. Thinking DevOps in the era of the
Cloud
Demi Ben-Ari

2. About Me
Demi Ben-Ari, Co-Founder & VP R&D @ Panorays
● Google Developer Expert
● Co-Founder of Communities:
○ “Big Things” - Big Data, Data Science, DevOps
○ Google Developer Group Cloud
○ Ofek Alumni Association
In the Past:
● Sr. Data Engineer - Windward
● Team Leader & Sr. Java Software Engineer,
Missile defense and Alert System - “Ofek” – IAF

3. Agenda
● A lot of (NOT) funny Jokes
● DevOps? WTF?
● Process and Environment
● Monitoring pipeline solutions
○ Metrics, Datastores, Dashboards, Alerting
● Conclusions

4. So What is “DevOps” actually?

5. What is “DevOps”? - A Glance to the past
Development
Software Engineering
Front End
Backend
QA
UX
Operations
System Admin
Database Admin
Support

6. What is “DevOps”? - (Hopefully) Today...
DevOps
Cross-functional teams
Operators automating
systems
Developers operating systems

7. What is “DevOps”?
Deliver High Quality Working Software Faster
and More Safely

8. Running environments
● Development – Testing – Production
○ Don’t you need more?
Be as flexible as you can
○ Ease of creation and Deployment
○ Identical or Not?
● Virtual Machines vs Bare Metal
● Configuration

9. Did someone
say Containers?

10. Docker Environments
● Docker?
● Orchestration?
VS
● Wait, What about local mode?
○ Minikube vs Docker Engine

11. Process

12. DevOps - Keep It Simple Stupid!
● Linux
○ Bash scripts
○ Crontab
● Automation via Jenkins
● Continuous Deployment – with every GIT push

13. Build Automation - What should I check?
● Maven
○ Sonatype Nexus artifact management
●
○ Deploy and Script generation scripts
○ Per Environment Testing
○ Data Validation
○ Scheduled Tasks

14. Workflow Management
● Luigi
● Airflow
● Crontab + Jenkins (KISS again)

15. Testing
● Unit
○ JUnit tests.
● End to End
○ Simulate the full execution of an application on a single JVM (local mode) – Real input,
Real output
● Functional
○ Stand alone application
○ Running on the cluster
○ Minimal coverage – Shows working data flow

16. Mandatory Components
● Source Control
● Artifact Repository
● Testing
● Reporting
● Loging
● Monitoring
● Deployment management

17. Testing - Let’s talk some CI/CD

18. Environment Summary

19. Monitoring

20. Monitor What?
Product
Data
Infrastructure
Biz

21. Monolith Structure
OS CPU Memory Disk
Processes Java
Application
Server
Database
Web Server
Load
Balancer
Users - Other Applications
Monitoring
System
UI
Many times...all of this was on a single physical server!

22. Distributed Microservices Architecture
Service A
Queue
DB
Service B
DBCache
Cache DBService C
Web
Server
DB
Analytics Cluster
Master
Slave Slave Slave
Monitoring System???

23. Some basic concepts

24. Basic Concepts
● Monitoring
● White-box
○ internals
● Black-box
○ behavior
● Dashboard
● Alert
● Root cause
● Node and machine
● Deploy
○ Any change to a
service’s running
software or its
configuration.
● KPI - Key Performance
Indicator

25. Monitoring Stack - Let’s fill in the blanks
Alerting
Metrics Collection
Datastore
Dashboard
Data Monitoring
Log Monitoring

26. Data flow and Environment
(Our Use Case)

27. Structure of the Data
● Maritime Analytics Platform
● Geo Locations + Metadata
● Arriving over time
● Different types of messages being reported by satellites
● Encoded (For compression reasons)
● Might arrive later than actually transmitted

28. Environment Description
Cluster
Dev Testing
Live
Staging
ProductionEnv
OB1K
RESTful Java Services

29. Situations & Problems
https://imgflip.com/i/1ap5kr
http://kingofwallpapers.com/otter/otter-004.jpg

30. MongoDB + Spark
Worker 1
Worker 2
….
….
…
…
Worker N
Spark
Cluster
Master
Write
Read
MasterSharded
MongoDB
Replica Set

31. Cassandra + Spark
Worker 1
Worker 2
….
….
…
…
Worker N
Cassandra
Cluster
Spark
Cluster
Write
Read

32. Cassandra + Serving
Cassandra
Cluster
Write
Read
UI Client
UI Client
UI Client
UI Client
Web
ServiceWeb
ServiceWeb
ServiceWeb
Service

33. So...Solutions
Let’s Start!

34. Monitoring
Operation System

35. Monitoring Operation System Metrics
● What to measure:
○ CPU
○ Memory
○ Disk Space
● How to measure:
○ CollectD or StatsD reporting to Graphite
○ New Relic
■ Nice and easy UI
■ Even the free account gives great tool
■ Alerting of thresholds

36. Some help
from “the Cloud”

37. AWS’s CloudWatch / GCP StackDriver

38. Report to Where?
● We chose:
● Graphite (InfluxDB) + Grafana
● Can correlate System and
Application metrics in one
place :)

39. Connections
Connections…
http://www.mememaker.net/meme/connections-connections-everywhere2/

40. Drivers to Datastores
● Actions they usually do:
○ Open connection
○ Apply actions
■ Select, Insert, Update, Delete
○ Close connection
● Do you monitor each?
○ Hint: Yes!!!! Hell Yes!!!
● Creating a wrapper in any programming language and reporting the
metrics
○ Count, execution times, errors…
○ Infrastructure code that will give great visibility

41. Monitoring
Cassandra

42. Monitoring Cassandra
● OpsCenter - by DataStax

43. Monitoring Cassandra
● Is the enough?...
We can connect it to Graphite also (Blog: “Monitoring the
hell out of Cassandra”)
● Plug & Play the metrics to Graphite - Internal Cassandra
mechanism
● Back to the basics: dstat, iostat, iotop, jstack

44. Monitoring Cassandra

45. Monitoring
Your Data
https://memegenerator.net/instance/53617544

46. Data Questions? What should be measure
● Did all of the computation occur?
○ Are there any data layers missing?
● How much data do we have? (Volume)
● Is all of the data in the Database?
● Data Quality Assurance

47. Data Answers!
● KISS (Keep it simple stupid)
● Jenkins + Maven (JUnit) for the rescue
● Creating a maven “monitoring” project.
○ Running scheduled tasks, each for the relevant data source
■ Database data existence
■ S3 files existence
■ Data flow that keeps on coming from sensors
■ (Any other data source that you can imagine…)
○ Scheduled task that write amount metrics to Graphite -> Dashboards

48. Data Answers!
● The method doesn’t really matter, as long as you:
○ Can follow the results over time
○ Know what your data flow, know what might fail
○ It’s easy for anyone to add more monitoring
(For the ones that add the new data each time…)
○ It don’t trust others to add monitoring
(It will always end up the DevOps’s “fault” -> No
monitoring will be applied)

49. Logging?
Monitoring?
https://lh4.googleusercontent.com/DFVcH-E5XKj8cbhEtI0qabmf_wwVqWWvk0pK5H5rnC_kVxY2tXClKfzV-LvAH61YRLJUEvtO9amjWfjcY4Z57VBYCuQ9
5_hdAVEHgLAuepJiArH0wJERWuzzmgnPysCiIA

50. ELK - Elasticsearch + Logstash + Kibana
http://www.digitalgov.gov/2014/05/07/analyzing-search-data-in-real-time-to-drive-decisions/

51. Did someone say
“Dashboard”?
http://www.funpic.hu/_files/pictures/original/86/71/27186.jpg

52. Redash
● http://redash.io/
● Open Source: https://github.com/getredash/redash
● Came out as one of many Open source tool by
Everything.me
● Created and Maintained by Arik Fraimovich (You rock!)
● Written in Python
● Has an on-premise and hosted solution
●‫רןאאקמ‬

53. Redash - Screenshots

54. Alerting

55. Alerting
● Syren - Open source
● Reporting to:
○ Email, Flowdock, HipChat, HTTP,
Hubot, IRCcat, PagerDuty,
Pushover, SLF4J, Slack,
SNMP, Twilio

56. Summary - Monitoring Stack
Alerting
Metrics Collection
Datastore
Dashboard
Data Monitoring
Log Monitoring

57. So...Who does the monitoring in our company?
https://imgflip.com/i/18kvv1

58. Not my problem...or is it?
https://cdn.meme.am/instances/500x/22605665.jpg

59. Conclusions
● Correlating Application and System metrics!!!!
● Ask the right monitoring questions -> answer with
Dashboards
● KISS - simple is key, what’s hard, we tend not to do at all
● Alert about what you can actually react to
○ (And to the relevant person)
● Measure whatever you can
○ only way to know if you’re improving
● Monitor your business KPIs too

60. Conclusions
● DevOps requires Culture also
● Take responsibility
○ Production is everybody's responsibility
● Leverage the Cloud and the possibilities of being Flexible

61. Questions?

62. ● LinkedIn
● Twitter: @demibenari
● Blog:
http://progexc.blogspot.com/
● demi.benari@gmail.com
● “Big Things” Community
Meetup, YouTube, Facebook,
Twitter
● GDG Cloud