SlideShare une entreprise Scribd logo
1  sur  23
NRT Event Processing
Outline
•  Introduction
•  Our Snowplow Setup
•  Example NRT Use Cases
•  Radio Campaign
•  Telephony System
Simply Business
•  Largest UK business insurance provider
•  More than 400.000 policy holders
•  Using BML, tech and data to disrupt the
business insurance market
Data ’n’ Analytics
•  5 Data Engineers
•  3 Business Intelligence Developers
•  3 Data Analysts
•  1 Data Scientist
•  1 Director of Data Science
•  And hiring! :-)
Our Snowplow Setup
Snowplow Setup
Trackers	
   Collector	
   Enrichment	
   Modeling	
   Storage	
  
•  Trackers, collectors and storage are 100% upstream Snowplow
•  Enrichment:
•  Spark apps that use scala-common-enrich as a library
•  We add our own enrichments after the default ones
•  We perform NRT identity stitching and sessionization	
  
•  Modeling: mix of Spark and SQL jobs
•  Storage: Spark apps that use scala-hadoop-shred as a library
Why ?
•  We wanted a near real-time pipeline, but KCL was too rigid:
•  Provision, set up and monitor the machines
•  Configuration is difficult for complex DAGs
•  In contrast, Spark:
•  Once set up, the cluster is a PaaS
•  Allows streaming, batch, ML and graph workloads
•  Allows analysts and data scientists to use Python
Radio Campaign
The Radio Campaign
•  We’re running a radio campaign in Birmingham, Manchester and
London
•  People that get a quote starting from our radio landing pages get
£25 discount
	
  	
  	
  	
  
The Banner
•  The questionnaire to get quotes can be quite long to complete
•  We wanted to reassure our customers that they would get the
discount
•  We wanted to display a banner at the top through all the pages of
the questionnaire
	
  	
  	
  	
  
The Banner
Our Infrastructure
Spark	
  Stream	
  
NRT	
  Enrichment	
  
Scala	
  Stream	
  
Collector	
   Kinesis	
  
MongoDB	
  
Visitor	
  API	
  QuoBng	
  App	
  
HTTP	
  
On average, it takes 2.5s for an event to be available in the Visitor API
Benefits of NRT Snowplow
•  Our quoting app does not need to know about marketing, user
landing pages, etc.
•  Our Mongo table with active sessions’ events becomes a view of our
event log
•  Can be reused for many other use cases: analytics on read!
	
  	
  	
  	
  
Telephony System
Telephony System
•  We have a call center in Northampton with around 200 consultants
•  We used an off-the-shelf telephony system
•  It worked well for a long time, but:
•  Was not very well integrated with our systems
•  Quite rigid, we couldn’t adapt it to all our needs
•  We had daily reports and they contained aggregated data
	
  	
  	
  	
  
Telephony System
•  We decided to replace it with a home grown, Twilio-based solution
•  Components:
•  Contact Strategy Manager
•  Voice Channel Manager
•  Communication is event-based
•  We transform those events into Snowplow’s unstructured
•  Spark Streaming app to insert the events into Redshift every 2min
	
  	
  	
  	
  
The Infrastructure
Spark	
  Stream	
  
NRT	
  Enrichment	
  
Scala	
  Stream	
  
Collector	
   Kinesis	
   Kinesis	
  
RedshiD	
  
Spark	
  Stream	
  
Shredder	
  
Looker	
  
Contact	
  Strategy	
  
Manager	
  
Voice	
  Channel	
  
Manager	
  
Event	
  
Translator	
  
Events
Example call when viewed as sequence of events:
	
  	
  	
  	
  
Benefits of NRT Snowplow
•  Event Sourcing is great for reporting and analytics: ensures that
data quality remains high
•  Team managers now have a NRT view of what teams are doing
•  You can aggregate and drill down on the data as appropriate
•  Leveraging our data platform: Snowplow pipeline, Redshift & Looker
•  Leveraging our existing skills: everyone knows how to use Looker
	
  	
  	
  	
  
Sum Up
The Infrastructure
Spark	
  Stream	
  
NRT	
  Enrichment	
  
Scala	
  Stream	
  
Collector	
   Kinesis	
  
MongoDB	
  
Kinesis	
  
RedshiD	
  
Spark	
  Stream	
  
Shredder	
  
Visitor	
  API	
   Looker	
  ApplicaBons	
  
NRT Benefits
•  We can dynamically alter the website while the user is still using it
•  We can provide insights on live processes
•  Multiple uses to improve conversion:
•  Instant inclusion/exclusion from remarketing lists
•  Abandoned cart emails/calls
•  Social proofing (3 more people are also watching…)
•  …
	
  	
  	
  	
  
Questions?
@dani_sola
dani.sola@simplybusiness.co.uk

Contenu connexe

Tendances

Snowplow Analytics: from NoSQL to SQL and back again
Snowplow Analytics: from NoSQL to SQL and back againSnowplow Analytics: from NoSQL to SQL and back again
Snowplow Analytics: from NoSQL to SQL and back againAlexander Dean
 
2016 09 measurecamp - event data modeling
2016 09 measurecamp - event data modeling2016 09 measurecamp - event data modeling
2016 09 measurecamp - event data modelingyalisassoon
 
Big data meetup budapest adding data schemas to snowplow
Big data meetup budapest   adding data schemas to snowplowBig data meetup budapest   adding data schemas to snowplow
Big data meetup budapest adding data schemas to snowplowyalisassoon
 
Flows in the Service Console, Gotta Go with the Flow! by Duncan Stewart
Flows in the Service Console, Gotta Go with the Flow! by Duncan StewartFlows in the Service Console, Gotta Go with the Flow! by Duncan Stewart
Flows in the Service Console, Gotta Go with the Flow! by Duncan StewartSalesforce Admins
 
Snowplow presentation for Amsterdam Meetup #3
Snowplow presentation for Amsterdam Meetup #3Snowplow presentation for Amsterdam Meetup #3
Snowplow presentation for Amsterdam Meetup #3Snowplow Analytics
 
Understanding event data
Understanding event dataUnderstanding event data
Understanding event datayalisassoon
 
Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016
Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016
Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016yalisassoon
 
Snowplow the evolving data pipeline
Snowplow   the evolving data pipelineSnowplow   the evolving data pipeline
Snowplow the evolving data pipelineyalisassoon
 
Modelling event data in look ml
Modelling event data in look mlModelling event data in look ml
Modelling event data in look mlyalisassoon
 
Snowplow at DA Hub emerging technology showcase
Snowplow at DA Hub emerging technology showcaseSnowplow at DA Hub emerging technology showcase
Snowplow at DA Hub emerging technology showcaseyalisassoon
 
How to evolve your analytics stack with your business using Snowplow
How to evolve your analytics stack with your business using SnowplowHow to evolve your analytics stack with your business using Snowplow
How to evolve your analytics stack with your business using SnowplowGiuseppe Gaviani
 
How we use Hive at SnowPlow, and how the role of HIve is changing
How we use Hive at SnowPlow, and how the role of HIve is changingHow we use Hive at SnowPlow, and how the role of HIve is changing
How we use Hive at SnowPlow, and how the role of HIve is changingyalisassoon
 
Memrise presentation @ London Snowplow meetup
Memrise presentation @ London Snowplow meetup Memrise presentation @ London Snowplow meetup
Memrise presentation @ London Snowplow meetup idan_by
 
How Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with SnowplowHow Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with SnowplowGiuseppe Gaviani
 
Using Snowplow for A/B testing and user journey analysis at CustomMade
Using Snowplow for A/B testing and user journey analysis at CustomMadeUsing Snowplow for A/B testing and user journey analysis at CustomMade
Using Snowplow for A/B testing and user journey analysis at CustomMadeyalisassoon
 
Viewbix tracking journey
Viewbix tracking journeyViewbix tracking journey
Viewbix tracking journeyidan_by
 
Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising
Tapjoy: Building a Real-Time Data Science Service for Mobile AdvertisingTapjoy: Building a Real-Time Data Science Service for Mobile Advertising
Tapjoy: Building a Real-Time Data Science Service for Mobile AdvertisingSingleStore
 
Snowplow: evolve your analytics stack with your business
Snowplow: evolve your analytics stack with your businessSnowplow: evolve your analytics stack with your business
Snowplow: evolve your analytics stack with your businessyalisassoon
 
Cap intro oct2014 pdf
Cap intro oct2014 pdfCap intro oct2014 pdf
Cap intro oct2014 pdfMarkku Ranta
 

Tendances (20)

Snowplow Analytics: from NoSQL to SQL and back again
Snowplow Analytics: from NoSQL to SQL and back againSnowplow Analytics: from NoSQL to SQL and back again
Snowplow Analytics: from NoSQL to SQL and back again
 
2016 09 measurecamp - event data modeling
2016 09 measurecamp - event data modeling2016 09 measurecamp - event data modeling
2016 09 measurecamp - event data modeling
 
Big data meetup budapest adding data schemas to snowplow
Big data meetup budapest   adding data schemas to snowplowBig data meetup budapest   adding data schemas to snowplow
Big data meetup budapest adding data schemas to snowplow
 
Flows in the Service Console, Gotta Go with the Flow! by Duncan Stewart
Flows in the Service Console, Gotta Go with the Flow! by Duncan StewartFlows in the Service Console, Gotta Go with the Flow! by Duncan Stewart
Flows in the Service Console, Gotta Go with the Flow! by Duncan Stewart
 
Snowplow presentation for Amsterdam Meetup #3
Snowplow presentation for Amsterdam Meetup #3Snowplow presentation for Amsterdam Meetup #3
Snowplow presentation for Amsterdam Meetup #3
 
Understanding event data
Understanding event dataUnderstanding event data
Understanding event data
 
Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016
Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016
Analytics at Carbonite: presentation to Snowplow Meetup Boston April 2016
 
Snowplow the evolving data pipeline
Snowplow   the evolving data pipelineSnowplow   the evolving data pipeline
Snowplow the evolving data pipeline
 
Modelling event data in look ml
Modelling event data in look mlModelling event data in look ml
Modelling event data in look ml
 
Snowplow at DA Hub emerging technology showcase
Snowplow at DA Hub emerging technology showcaseSnowplow at DA Hub emerging technology showcase
Snowplow at DA Hub emerging technology showcase
 
How to evolve your analytics stack with your business using Snowplow
How to evolve your analytics stack with your business using SnowplowHow to evolve your analytics stack with your business using Snowplow
How to evolve your analytics stack with your business using Snowplow
 
How we use Hive at SnowPlow, and how the role of HIve is changing
How we use Hive at SnowPlow, and how the role of HIve is changingHow we use Hive at SnowPlow, and how the role of HIve is changing
How we use Hive at SnowPlow, and how the role of HIve is changing
 
Memrise presentation @ London Snowplow meetup
Memrise presentation @ London Snowplow meetup Memrise presentation @ London Snowplow meetup
Memrise presentation @ London Snowplow meetup
 
How Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with SnowplowHow Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with Snowplow
 
Using Snowplow for A/B testing and user journey analysis at CustomMade
Using Snowplow for A/B testing and user journey analysis at CustomMadeUsing Snowplow for A/B testing and user journey analysis at CustomMade
Using Snowplow for A/B testing and user journey analysis at CustomMade
 
Viewbix tracking journey
Viewbix tracking journeyViewbix tracking journey
Viewbix tracking journey
 
Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising
Tapjoy: Building a Real-Time Data Science Service for Mobile AdvertisingTapjoy: Building a Real-Time Data Science Service for Mobile Advertising
Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising
 
Cap server log file analytics
Cap server log file analyticsCap server log file analytics
Cap server log file analytics
 
Snowplow: evolve your analytics stack with your business
Snowplow: evolve your analytics stack with your businessSnowplow: evolve your analytics stack with your business
Snowplow: evolve your analytics stack with your business
 
Cap intro oct2014 pdf
Cap intro oct2014 pdfCap intro oct2014 pdf
Cap intro oct2014 pdf
 

En vedette

Snowplow at Sigfig
Snowplow at SigfigSnowplow at Sigfig
Snowplow at Sigfigyalisassoon
 
Introducing Tupilak, Snowplow's unified log fabric
Introducing Tupilak, Snowplow's unified log fabricIntroducing Tupilak, Snowplow's unified log fabric
Introducing Tupilak, Snowplow's unified log fabricAlexander Dean
 
How Incuda builds user journey models with Snowplow
How Incuda builds user journey models with SnowplowHow Incuda builds user journey models with Snowplow
How Incuda builds user journey models with SnowplowGiuseppe Gaviani
 
Data science as a service
Data science as a serviceData science as a service
Data science as a serviceidan_by
 
The culture trip snowplow implementation
The culture trip snowplow implementationThe culture trip snowplow implementation
The culture trip snowplow implementationidan_by
 
Streetlife's real time analytics stack
Streetlife's real time analytics stackStreetlife's real time analytics stack
Streetlife's real time analytics stackidan_by
 

En vedette (6)

Snowplow at Sigfig
Snowplow at SigfigSnowplow at Sigfig
Snowplow at Sigfig
 
Introducing Tupilak, Snowplow's unified log fabric
Introducing Tupilak, Snowplow's unified log fabricIntroducing Tupilak, Snowplow's unified log fabric
Introducing Tupilak, Snowplow's unified log fabric
 
How Incuda builds user journey models with Snowplow
How Incuda builds user journey models with SnowplowHow Incuda builds user journey models with Snowplow
How Incuda builds user journey models with Snowplow
 
Data science as a service
Data science as a serviceData science as a service
Data science as a service
 
The culture trip snowplow implementation
The culture trip snowplow implementationThe culture trip snowplow implementation
The culture trip snowplow implementation
 
Streetlife's real time analytics stack
Streetlife's real time analytics stackStreetlife's real time analytics stack
Streetlife's real time analytics stack
 

Similaire à Simply Business - Near Real Time Event Processing

Building a [micro]services platform on AWS
Building a [micro]services platform on AWSBuilding a [micro]services platform on AWS
Building a [micro]services platform on AWSShaun Pearce
 
From no services to Microservices
From no services to MicroservicesFrom no services to Microservices
From no services to MicroservicesJoão Cavalheiro
 
HOP! Airlines Jets to Real Time
HOP! Airlines Jets to Real TimeHOP! Airlines Jets to Real Time
HOP! Airlines Jets to Real Timeconfluent
 
SplunkLive! Utrecht 2016 - NXP
SplunkLive! Utrecht 2016 - NXPSplunkLive! Utrecht 2016 - NXP
SplunkLive! Utrecht 2016 - NXPSplunk
 
Rakuten’s Journey with Splunk - Evolution of Splunk as a Service
Rakuten’s Journey with Splunk - Evolution of Splunk as a ServiceRakuten’s Journey with Splunk - Evolution of Splunk as a Service
Rakuten’s Journey with Splunk - Evolution of Splunk as a ServiceRakuten Group, Inc.
 
Moving to microservices – a technology and organisation transformational journey
Moving to microservices – a technology and organisation transformational journeyMoving to microservices – a technology and organisation transformational journey
Moving to microservices – a technology and organisation transformational journeyBoyan Dimitrov
 
Correlate Log Data with Business Metrics Like a Jedi
Correlate Log Data with Business Metrics Like a JediCorrelate Log Data with Business Metrics Like a Jedi
Correlate Log Data with Business Metrics Like a JediTrevor Parsons
 
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...Spark Summit
 
Simply Business' Data Platform
Simply Business' Data PlatformSimply Business' Data Platform
Simply Business' Data PlatformDani Solà Lagares
 
Architecture for Scale [AppFirst]
Architecture for Scale [AppFirst]Architecture for Scale [AppFirst]
Architecture for Scale [AppFirst]AppFirst
 
Cloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark AnalyticsCloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark Analyticsamesar0
 
Best Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterBest Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterInfluxData
 
MongoDB .local London 2019: Nationwide Building Society: Building Mobile Appl...
MongoDB .local London 2019: Nationwide Building Society: Building Mobile Appl...MongoDB .local London 2019: Nationwide Building Society: Building Mobile Appl...
MongoDB .local London 2019: Nationwide Building Society: Building Mobile Appl...MongoDB
 
Suning OpenStack Cloud and Heat
Suning OpenStack Cloud and HeatSuning OpenStack Cloud and Heat
Suning OpenStack Cloud and HeatQiming Teng
 
freebeersessions #26 Scaling Up and Out Using Open Source at Netstar
freebeersessions #26 Scaling Up and Out Using Open Source at Netstarfreebeersessions #26 Scaling Up and Out Using Open Source at Netstar
freebeersessions #26 Scaling Up and Out Using Open Source at NetstarQuintin de Kok
 
Developing multi-functional “sensor” web service platform for citizen sensing
Developing multi-functional “sensor” web service platform for citizen sensingDeveloping multi-functional “sensor” web service platform for citizen sensing
Developing multi-functional “sensor” web service platform for citizen sensingSnowflake Software
 
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsScaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsRandy Shoup
 
Kinesis @ lyft
Kinesis @ lyftKinesis @ lyft
Kinesis @ lyftMian Hamid
 
Qwasi Splunk and NCR Integration: Business Analytics
Qwasi Splunk and NCR Integration: Business AnalyticsQwasi Splunk and NCR Integration: Business Analytics
Qwasi Splunk and NCR Integration: Business AnalyticsTimur Bagirov
 
PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...
PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...
PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...Lucidworks
 

Similaire à Simply Business - Near Real Time Event Processing (20)

Building a [micro]services platform on AWS
Building a [micro]services platform on AWSBuilding a [micro]services platform on AWS
Building a [micro]services platform on AWS
 
From no services to Microservices
From no services to MicroservicesFrom no services to Microservices
From no services to Microservices
 
HOP! Airlines Jets to Real Time
HOP! Airlines Jets to Real TimeHOP! Airlines Jets to Real Time
HOP! Airlines Jets to Real Time
 
SplunkLive! Utrecht 2016 - NXP
SplunkLive! Utrecht 2016 - NXPSplunkLive! Utrecht 2016 - NXP
SplunkLive! Utrecht 2016 - NXP
 
Rakuten’s Journey with Splunk - Evolution of Splunk as a Service
Rakuten’s Journey with Splunk - Evolution of Splunk as a ServiceRakuten’s Journey with Splunk - Evolution of Splunk as a Service
Rakuten’s Journey with Splunk - Evolution of Splunk as a Service
 
Moving to microservices – a technology and organisation transformational journey
Moving to microservices – a technology and organisation transformational journeyMoving to microservices – a technology and organisation transformational journey
Moving to microservices – a technology and organisation transformational journey
 
Correlate Log Data with Business Metrics Like a Jedi
Correlate Log Data with Business Metrics Like a JediCorrelate Log Data with Business Metrics Like a Jedi
Correlate Log Data with Business Metrics Like a Jedi
 
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
 
Simply Business' Data Platform
Simply Business' Data PlatformSimply Business' Data Platform
Simply Business' Data Platform
 
Architecture for Scale [AppFirst]
Architecture for Scale [AppFirst]Architecture for Scale [AppFirst]
Architecture for Scale [AppFirst]
 
Cloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark AnalyticsCloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark Analytics
 
Best Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterBest Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise Cluster
 
MongoDB .local London 2019: Nationwide Building Society: Building Mobile Appl...
MongoDB .local London 2019: Nationwide Building Society: Building Mobile Appl...MongoDB .local London 2019: Nationwide Building Society: Building Mobile Appl...
MongoDB .local London 2019: Nationwide Building Society: Building Mobile Appl...
 
Suning OpenStack Cloud and Heat
Suning OpenStack Cloud and HeatSuning OpenStack Cloud and Heat
Suning OpenStack Cloud and Heat
 
freebeersessions #26 Scaling Up and Out Using Open Source at Netstar
freebeersessions #26 Scaling Up and Out Using Open Source at Netstarfreebeersessions #26 Scaling Up and Out Using Open Source at Netstar
freebeersessions #26 Scaling Up and Out Using Open Source at Netstar
 
Developing multi-functional “sensor” web service platform for citizen sensing
Developing multi-functional “sensor” web service platform for citizen sensingDeveloping multi-functional “sensor” web service platform for citizen sensing
Developing multi-functional “sensor” web service platform for citizen sensing
 
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsScaling Your Architecture with Services and Events
Scaling Your Architecture with Services and Events
 
Kinesis @ lyft
Kinesis @ lyftKinesis @ lyft
Kinesis @ lyft
 
Qwasi Splunk and NCR Integration: Business Analytics
Qwasi Splunk and NCR Integration: Business AnalyticsQwasi Splunk and NCR Integration: Business Analytics
Qwasi Splunk and NCR Integration: Business Analytics
 
PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...
PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...
PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...
 

Dernier

Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
A PowerPoint Presentation on Vikram Lander pptx
A PowerPoint Presentation on Vikram Lander pptxA PowerPoint Presentation on Vikram Lander pptx
A PowerPoint Presentation on Vikram Lander pptxatharvdev2010
 
Transport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MITransport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MIRomil Mishra
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
Arti Languages Pre Seed Pitchdeck 2024.pdf
Arti Languages Pre Seed Pitchdeck 2024.pdfArti Languages Pre Seed Pitchdeck 2024.pdf
Arti Languages Pre Seed Pitchdeck 2024.pdfwill854175
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
Dublin_mulesoft_meetup_API_specifications.pptx
Dublin_mulesoft_meetup_API_specifications.pptxDublin_mulesoft_meetup_API_specifications.pptx
Dublin_mulesoft_meetup_API_specifications.pptxKunal Gupta
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfHCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfROWELL MARQUINA
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024BookNet Canada
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 

Dernier (20)

Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
A PowerPoint Presentation on Vikram Lander pptx
A PowerPoint Presentation on Vikram Lander pptxA PowerPoint Presentation on Vikram Lander pptx
A PowerPoint Presentation on Vikram Lander pptx
 
Transport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MITransport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MI
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
Arti Languages Pre Seed Pitchdeck 2024.pdf
Arti Languages Pre Seed Pitchdeck 2024.pdfArti Languages Pre Seed Pitchdeck 2024.pdf
Arti Languages Pre Seed Pitchdeck 2024.pdf
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
Dublin_mulesoft_meetup_API_specifications.pptx
Dublin_mulesoft_meetup_API_specifications.pptxDublin_mulesoft_meetup_API_specifications.pptx
Dublin_mulesoft_meetup_API_specifications.pptx
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfHCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 

Simply Business - Near Real Time Event Processing

  • 2. Outline •  Introduction •  Our Snowplow Setup •  Example NRT Use Cases •  Radio Campaign •  Telephony System
  • 3. Simply Business •  Largest UK business insurance provider •  More than 400.000 policy holders •  Using BML, tech and data to disrupt the business insurance market
  • 4. Data ’n’ Analytics •  5 Data Engineers •  3 Business Intelligence Developers •  3 Data Analysts •  1 Data Scientist •  1 Director of Data Science •  And hiring! :-)
  • 6. Snowplow Setup Trackers   Collector   Enrichment   Modeling   Storage   •  Trackers, collectors and storage are 100% upstream Snowplow •  Enrichment: •  Spark apps that use scala-common-enrich as a library •  We add our own enrichments after the default ones •  We perform NRT identity stitching and sessionization   •  Modeling: mix of Spark and SQL jobs •  Storage: Spark apps that use scala-hadoop-shred as a library
  • 7. Why ? •  We wanted a near real-time pipeline, but KCL was too rigid: •  Provision, set up and monitor the machines •  Configuration is difficult for complex DAGs •  In contrast, Spark: •  Once set up, the cluster is a PaaS •  Allows streaming, batch, ML and graph workloads •  Allows analysts and data scientists to use Python
  • 9. The Radio Campaign •  We’re running a radio campaign in Birmingham, Manchester and London •  People that get a quote starting from our radio landing pages get £25 discount        
  • 10. The Banner •  The questionnaire to get quotes can be quite long to complete •  We wanted to reassure our customers that they would get the discount •  We wanted to display a banner at the top through all the pages of the questionnaire        
  • 12. Our Infrastructure Spark  Stream   NRT  Enrichment   Scala  Stream   Collector   Kinesis   MongoDB   Visitor  API  QuoBng  App   HTTP   On average, it takes 2.5s for an event to be available in the Visitor API
  • 13. Benefits of NRT Snowplow •  Our quoting app does not need to know about marketing, user landing pages, etc. •  Our Mongo table with active sessions’ events becomes a view of our event log •  Can be reused for many other use cases: analytics on read!        
  • 15. Telephony System •  We have a call center in Northampton with around 200 consultants •  We used an off-the-shelf telephony system •  It worked well for a long time, but: •  Was not very well integrated with our systems •  Quite rigid, we couldn’t adapt it to all our needs •  We had daily reports and they contained aggregated data        
  • 16. Telephony System •  We decided to replace it with a home grown, Twilio-based solution •  Components: •  Contact Strategy Manager •  Voice Channel Manager •  Communication is event-based •  We transform those events into Snowplow’s unstructured •  Spark Streaming app to insert the events into Redshift every 2min        
  • 17. The Infrastructure Spark  Stream   NRT  Enrichment   Scala  Stream   Collector   Kinesis   Kinesis   RedshiD   Spark  Stream   Shredder   Looker   Contact  Strategy   Manager   Voice  Channel   Manager   Event   Translator  
  • 18. Events Example call when viewed as sequence of events:        
  • 19. Benefits of NRT Snowplow •  Event Sourcing is great for reporting and analytics: ensures that data quality remains high •  Team managers now have a NRT view of what teams are doing •  You can aggregate and drill down on the data as appropriate •  Leveraging our data platform: Snowplow pipeline, Redshift & Looker •  Leveraging our existing skills: everyone knows how to use Looker        
  • 21. The Infrastructure Spark  Stream   NRT  Enrichment   Scala  Stream   Collector   Kinesis   MongoDB   Kinesis   RedshiD   Spark  Stream   Shredder   Visitor  API   Looker  ApplicaBons  
  • 22. NRT Benefits •  We can dynamically alter the website while the user is still using it •  We can provide insights on live processes •  Multiple uses to improve conversion: •  Instant inclusion/exclusion from remarketing lists •  Abandoned cart emails/calls •  Social proofing (3 more people are also watching…) •  …