London Scala Meetup - Omnia
•
0 j'aime•355 vues
Omnia: distributed & reactive platform for data management
Recommandé
Recommandé
Contenu connexe
Similaire à London Scala Meetup - Omnia
Similaire à London Scala Meetup - Omnia (20)
Dernier
Dernier (20)
London Scala Meetup - Omnia
- 1. WELCOME TO THE LONDON SCALA UG MEET UP @ WILLIAM HILL
- 2. Luis Ángel Vicente Sánchez Software Engineer @ Innovation Team OMNIA DISTRIBUTED & REACTIVE PLATFORM FOR DATA MANAGEMENT Site https://developer.williamhill.com Twi$er https://twitter.com/lvicentesanchez
- 3. WHY BUILDING A DATA MANAGEMENT PLATFORM?
- 4. WHY BUILDING A DATA MANAGEMENT PLATFORM?
- 5. • Lots of information flowing in the system. • What if we could make sense of all of it...? • ... and on real-time? • We would be in a great position to innovate. WHY BUILDING A DATA MANAGEMENT PLATFORM?
- 6. BUT…
- 9. BUT…
- 10. BUT… SCALABILITY
- 11. BUT… SCALABILITY
- 12. BUT…
- 15. REQUIREMENTS
- 23. OMNIA
- 24. • It’s a lambda architecture. OMNIA
- 25. • It’s a lambda architecture. OMNIA
- 26. • It’s a lambda architecture. OMNIA Input Chronos DataSource NeoCortex Speed Layer Fates Batch Layer Hermes ServingLayer Output
- 27. CHRONOS
- 28. • Collects data from multiple data sources. • Initially a PUSH system… • ... but that introduces dependencies on 3rd parties. • Evolved to a PUSH-PULL system. CHRONOS
- 32. CHRONOS STREAM
- 34. CHRONOS Adapter STREAM PUSH: • HTTP • TCP • WEB SOCKETS PULL: • HTTP POLL • JMS • SSE
- 35. CHRONOS Adapter STREAM PUSH: • HTTP • TCP • WEB SOCKETS PULL: • HTTP POLL • JMS • SSE
- 36. CHRONOS Adapter Converter STREAM PUSH: • HTTP • TCP • WEB SOCKETS PULL: • HTTP POLL • JMS • SSE
- 37. CHRONOS Adapter Converter STREAM PUSH: • HTTP • TCP • WEB SOCKETS PULL: • HTTP POLL • JMS • SSE INCIDENT: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON …} }
- 38. CHRONOS Adapter Converter STREAM PUSH: • HTTP • TCP • WEB SOCKETS PULL: • HTTP POLL • JMS • SSE INCIDENT: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON …} } Observable[Incident]
- 39. CHRONOS Adapter Converter STREAM PUSH: • HTTP • TCP • WEB SOCKETS PULL: • HTTP POLL • JMS • SSE INCIDENT: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON …} } Observable[Incident]
- 40. CHRONOS Adapter Converter Persistor STREAM PUSH: • HTTP • TCP • WEB SOCKETS PULL: • HTTP POLL • JMS • SSE INCIDENT: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON …} } Observable[Incident]
- 41. CHRONOS Adapter Converter Persistor STREAM PUSH: • HTTP • TCP • WEB SOCKETS PULL: • HTTP POLL • JMS • SSE INCIDENT: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON …} } Observable[Incident]
- 42. CHRONOS
- 43. CHRONOS Chronos 1 Chronos 2 Chronos 3 Chronos N ….…
- 44. CHRONOS Chronos 1 (SSE, Bets placed) (SSE, Twitter) Chronos 2 (JMS, Deposits) Chronos 3 (HTTP, Events) Chronos N ….…
- 45. CHRONOS Chronos 1 (SSE, Bets placed) (SSE, Twitter) Chronos 2 (JMS, Deposits) Chronos 3 (HTTP, Events) Chronos N ….…
- 46. CHRONOS Chronos 1 (SSE, Bets placed) (SSE, Twitter) Chronos 2 (JMS, Deposits) Chronos 3 (HTTP, Events) Chronos N ….…
- 47. CHRONOS Chronos 1 (SSE, Bets placed) (SSE, Twitter) Chronos 3 (HTTP, Events) Chronos N (SSE, Twitter) ….… Chronos 2 (JMS, Deposits) (SSE, Bets placed)
- 48. FATES
- 50. • Long term memory. • Records all incidents as timelines, i.e. customer bets, customer payments, everything that happens in a match… • … or any combination of those. FATES
- 53. FATES Customer: 123 Login Deposit Bet placed … LogoutTimelines Bets Deposits Events Session Fates Batch Layer
- 54. FATES Customer: 123 Login Deposit Bet placed … Logout Event: 78 Started Bet placed Penalty … GoalTimelines Bets Deposits Events Session Fates Batch Layer
- 55. • Creates views using the timelines and/or other views. • These views are created using: – Timeline analysis – Logical reasoning… – ... or any other ML technique FATES
- 57. • Timelines and views persisted to Cassandra. • Column family name: <category>_tl or <category>_vw • Column family key: ((entityId, date), timestamp) FATES
- 58. NEOCORTEX
- 60. • Short term memory • Built on top of Spark: performant, distributed, real-time computations. • Built on top of abstractions that hide low level details: Timeline, View, Stream, Sink, Monoid, Group, Ring… NEOCORTEX
- 61. NEOCORTEX
- 62. HERMES
- 63. • Full duplex serving layer. • Can be queried from any backend… • ... and we provide a JS library to provide efficient data access from the browser. Inspired by Falcor and GraphML HERMES
- 66. INFRASTRUCTURE
- 68. USE CASE: BET SUGGESTIONS
- 69. • To provide customers with bet suggestions to make their experience more relevant/personal • Offered to new and existing customers. • Hot bets and personalized suggestions. USE CASE: BET SUGGESTIONS
- 70. USE CASE: BET SUGGESTIONS
- 72. • To provide customers with a more engaging experience by gamifying parts of the customer journey. • Appeal to attributes like status, mastery, belonging. • Achievements and rewards, leveling up, leaderboard, … USE CASE: GAMIFICATION