Lessons learned building a big data analytics engine, from proprietary to open source by Álvaro Santamaria & Joel Brunger After spending four years building a proprietary all-in-one streaming analytics engine for financial services, it became clear that open-source was starting to pull ahead. Alvaro will talk about the challenges of creating an IT operations solution for financial services; what to build, what not to build, and how to use open source tools to get past the infrastructure and focus on the business problems that matter.

1. Álvaro Santamaría
Data Scientist – ITRS
@dofideas
Joel Brunger
System Engineer - MapR
@joelbrunger
Lessons Learned with Visualisation
and Machine Learning for Big Data

3. Make real-time decisions
based on historical wisdom

5. Jay Krepps – 2013
“The Log: What every software engineer should know about real-time data's unifying abstraction”
The uber-system Lego-like, OS-based

7. 1. Big-data viz is not a matter of “front-end”

8. 2. 𝜅arao 𝜅e visualisations

9. 3. Information extraction

10. 3. Information extraction

11. 3. Information extraction

12. 3. Information extraction

13. 3. Information extraction
GROUP BY country,
timestamp window of 10 minutes
SELECT count(),
average(temperature),
median(temperature),
max(temperature),
...
tdigest(temperature)

14. 3. Information extraction

15. 4. Hold state…

16. 4. Hold state…

17. 5. … and deliver (state).

18. 6. Build your pipeline

20. J on the Beach
All Data, One Platform, Every Cloud
Limitless Possibilities

21. What is MapR?
MapR is the industry’s leading data platform for AI and Analytics.

22. Rendezvous Architecture
MapR is the industry’s leading data platform for AI and Analytics.

23. Rendezvous Architecture
MapR is the industry’s leading data platform for AI and Analytics.

24. Rendezvous Architecture
MapR is the industry’s leading data platform for AI and Analytics.

25. Rendezvous Architecture
MapR is the industry’s leading data platform for AI and Analytics.

26. Rendezvous Architecture
The Decoy is design to just collect data inputs

27. Rendezvous Architecture
MapR is the industry’s leading data platform for AI and Analytics.

28. Rendezvous Architecture
Introducing the Canary

29. Rendezvous Architecture
MapR is the industry’s leading data platform for AI and Analytics.

30. Why did ITRS choose MapR for ‘Gateway Hub’
MapR is the industry’s leading data platform for AI and Analytics.
Ø Simplicity (integrated platform)
Ø Real-time
Ø Processing must be performed in the cluster
Ø Enterprise features

31. MapR enables ITRS ‘Gateway Hub’ to provide the following benefits
MapR is the industry’s leading data platform for AI and Analytics.
Ø Smarter monitoring
Ø Additional features, application and Services
Ø Global Data Fabric
Ø Support ML in real-time

32. Thank you
JOnTheBeach 2018

33. Álvaro Santamaría
Data Scientist – ITRS
@dofideas
Joel Brunger
System Engineer - MapR
@joelbrunger
Lessons Learned with Visualisation
and Machine Learning for Big Data