Talking about the ease of use and handling Big Data technologies in the Cloud. Using Google Cloud Platform and Amazon Web Services and all of the tools around it. Showing the problems and how we can solve them with simple tools.

1. Big Data made easy in the era of Cloud
Demi Ben-Ari - VP R&D @ Panorays

2. About Me
Demi Ben-Ari, Co-Founder & VP R&D @ Panorays
! Google Developer Expert
! Co-Founder of Communities:
○ “Big Things” - Big Data, Data Science, DevOps
○ Google Developer Group Cloud
○ Ofek Alumni Association
In the Past:
! Sr. Data Engineer - Windward
! Team Leader & Sr. Java Software Engineer,
Missile defence and Alert System - “Ofek” – IAF

3. Automate the Security Management of Third Parties
Capture the
Hacker’s View
Get Realtime
Ratings
Comply with
Regulations

4. Say “Distributed”, Say “Big Data”,
Say….

5. What is Big Data (IMHO)? And What to Monitor?
! Systems involving the “3 Vs”:
What are the right questions we want to ask?
○ Volume - How much?
○ Velocity - How fast?
○ Variety - What kind? (Difference)

6. What had happened in the last years?
! Storage got cheaper
! The capacity of Data grew exponentially
! Cloud service providers grew rapidly
! Connectivity got much easier
! Cloud made “by demand” computation possible
! “Compute” started moving to the “Data” and not the other way.

7. Situations & Problems
https://imgflip.com/i/1ap5kr
http://kingofwallpapers.com/otter/otter-004.jpg

8. MongoDB + Spark
Worker 1
Worker 2
….
….
…
…
Worker N
Spark
Cluster
Master
Write
Read
MasterSharded
MongoDB
Replica Set

9. Cassandra + Spark
Worker 1
Worker 2
….
….
…
…
Worker N
Cassandra
Cluster
Spark
Cluster
Write
Read

10. Cassandra + Serving
Cassandra
Cluster
Write
Read
UI Client
UI Client
UI Client
UI Client
Web
ServiceWeb
ServiceWeb
ServiceWeb
Service

11. Distributed Microservices Architecture
Service A
Queue
DB
Service B
DBCache
Cache DBService C
Web
Server
DB
Analytics Cluster
Master
Slave Slave Slave
Monitoring System???

12. Did someone
say Containers?

13. Docker Environments
! Docker?
! Orchestration?
VS
! Wait, What about local mode?
○ Minikube vs Docker Engine

14. Problems
! Multiple physical servers
! Multiple logical services
! Want Scaling => More Servers

15. Data flow and Environment
(Use Case)

16. Structure of the Data
! Maritime Analytics Platform
! Geo Locations + Metadata
! Arriving over time
! Different types of messages being reported by satellites
! Encoded (For compression reasons)
! Might arrive later than actually transmitted

17. Data Flow Diagram
External
Data
Source
Analytics
Layers
Data Pipeline
Parsed
Raw
Entity Resolution
Process
Building insights
on top of the entities
Data Output
Layer
Anomaly Detection
Trends
UI for End Users

18. Environment Description
Cluster
Dev Testing
Live
Staging
ProductionEnv
OB1K
RESTful Java Services

19. Monitoring
Your Data
https://memegenerator.net/instance/53617544

20. Data Questions? What should be measure
! Did all of the computation occur?
○ Are there any data layers missing?
! How much data do we have? (Volume)
! Is all of the data in the Database?
! Data Quality Assurance

21. Conclusions
! Keep all of the Data that you can
! In its most raw form
! Duplicating Data is not a bad thing
! By demand compute with save you much time and money
! Find the relevant tool to solve each problem
! Not one tool that will solve all of them (No such thing)
! Use the cloud as an auxiliary tool
! Will boost your productivity by much

22. Questions?

23. ! LinkedIn
! Twitter: @demibenari
! Blog: http://
progexc.blogspot.com/
! demi.benari@gmail.com
! “Big Things” Community
Meetup, YouTube, Facebook, Twitter
! GDG Cloud