Adding a Data Reservoir to your Oracle Data Warehouse for Customer 360-Degree Analysis

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : info@rittmanmead.com
W : www.rittmanmead.com
Adding a Data Reservoir to Your  
Oracle Data Warehouse for  
Customer 360-Degree Analysis 
Mark Rittman, CTO, Rittman Mead
UKOUG Tech’15, Birmingham, December 2015

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
What Is This Presentation About…?
•What is a Hadoop-based data reservoir, and why might you add one to a data warehouse?
•How do you load, process and integrate one with your data warehouse using Oracle tools?
•How can you use it for what’s termed “Customer 360-degree insight?”
schema-on-read vs schema on write
real-time data ingestion
agile data provisioning vs. curated data
combining Hadoop, NoSQL and Oracle
omni-channel marketing
machine learning & decision engines
attitudinal vs behavioural data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
About Me
•Mark Rittman, Oracle ACE Director, Oracle BI, DW & Big Data
•14 Years Experience with Oracle Technology
•Regular columnist for Oracle Magazine
•Author of two Oracle Press Oracle BI books
•Oracle Business Intelligence Developers Guide
•Oracle Exalytics Revealed
•Writer for Rittman Mead Blog : 
http://www.rittmanmead.com/blog
•Past Editor of Oracle Scene Magazine, 
BIRT SIG Chair, ODTUG Board Member
•Co-founder and CTO for Rittman Mead

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
… Or as I say at Parties…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
15+ Years in Oracle BI and Data Warehousing
•Started back in 1997 on a bank Oracle DW project
•Our tools were Oracle 7.3.4, SQL*Plus, PL/SQL  
and shell scripts
•Went on to use Oracle Developer/2000 and Designer/2000
•Our initial users queried the DW using SQL*Plus
•And later on, we rolled-out Discoverer/2000 to everyone else
•And life was fun…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
The Oracle-Centric DW Architecture
•Over time, this data warehouse architecture developed
•Added Oracle Warehouse Builder to  
automate and model the DW build
•Oracle 9i Application Server (yay!)  
to deliver reports and web portals
•Data Mining and OLAP in the database
•Oracle 9i for in-database ETL (and RAC)
•Data was typically loaded from  
Oracle RBDMS and EBS
•It was turtles Oracle all the way down…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Traditional Three-Layer Relational Data Warehouses
Staging Foundation / 
ODS
Performance / 
Dimensional
ETL ETL
BI Tool (OBIEE) 
with metadata 
layer
OLAP / In-Memory 
Tool with data load 
into own database
Direct 
Read
Data 
Load
Traditional structured 
data sources
Data 
Load
Data 
Load
Data 
Load
Traditional Relational Data Warehouse
•Three-layer architecture - staging, foundation and access/performance
•All three layers stored in a relational database (Oracle)
•ETL used to move data from layer-to-layer

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
ETL Largely Batch-Based and with Single Route through DW
•All data lands in Staging layer, processed and then thrown-away
‣Too expensive to store all incoming granular data online - selected data stored as summary
•Processed through Foundation layer and then Access and Performance
•ETL development an expensive, manual task
•But this approach provided accurate numbers 
that every could trust, and navigate around

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
And Now … Everyone’s Talking About Big Data
•Explosion in volume and variety of data that’s now available
•New, cheap and open-source technology 
makes it economic to store + process it
•Users want more data stored in the DW,  
but budgets for IT are getting smaller
•Analytics and analysis has gone beyond 
tabular reports and dashboards, and requires 
new platforms to enable new approaches
•Which is actually rather scary…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Meanwhile, in the real world…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Who is my customer?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
More Complete Data About
Your Customers
Advanced analytics and
machine learning
More Attributes and
Activities Stored at Scale
True 360°Customer Profile
Connect disparate data
Targeted, personalized
customer treatment
Customer 360-Degree Insight

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Combines transactions + master data with granular behavioural & attitudinal data
Adding “Who” and “Why” to Customer Datasets
Single Customer View
Enriched  
Customer Profile
Correlatin
g
Modeling
Machine 
Learning
Scoring
“How” 
Interaction Data
Voice + Chat
Transcripts In-person 
dialogs
Webserver 
logs
Blogs
Surveys
Social Media
“Why” 
Attitudinal Data
“What” 
Behavioural Data
Transaction 
History
Retail 
Activity
Payment 
History
Basket Analysis
Attributes
Segments
Relationships
“Who” 
Descriptive Data
Demographics

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
But Wait … Isn’t This Just Data Warehousing & Data Mining?
•Data warehouses were conceived as a single source of reporting truth
•Formally accept, model and integrate data to provide analytical reporting platform
•Well-established design patterns for long-term data storage
•Stored in structured, indexed, optimised “schema on write” storage
•Data moved through layers via formal ETL
•Extreme Performance, Highly Secure
•Analytic SQL, In-Database Analytics
‣So why not use for this Customer 360 data?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Back to the real world again…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Consider How Consumers Used to Be Marketed To…
• Marketing used to be generic,  
one-way “broadcasting” to public
• Then Web 2.0 gave customers 
a voice, they could talk back…
• But they expected an immediate answer
• More work, but more intimate relationship
• Big data, smart technology + complex algorithms 
makes a “360-degree view of customers  
now possible
• Customers volunteer much data themselves
• But equilibrium of relationship now moved 
irrevocably to the customer

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Customer Touch-points Now Cover Many More Channels
•The days of a single, high-street retail channel are long gone
•Prospects often now find you via web searches, social media connections
•Shopping and browsing “on the go”  
using mobile devices, wearables
•Web increasingly the main sales channel
•“Order and go” collection at stores
•Call centre helplines,
•Customer service desks
•Forums, blogs, product reviews 
and other user-generated content
1980s 1990s 2000s 2010s
Empowered
Employees
Digital is
Humanized
Knowledge
Everywhere
Internet of
Things
Mobile as
Primary Channel
Cross-Channel
Service
WHAT’S NEXT

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Consumers Now Drive Their Own Purchase Decisions

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Datasets for Marketing Need to Reflect Today’s Consumer

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Customers Share Data with You .. With Expectations
•Customers now share huge amounts of data willingly, and perhaps unknowingly
‣Through your channels and applications - with potential privacy issue
‣Through tweeting, posting on Facebook and other social networks
‣But they also want to be in control
-Ability to delete their data
-Understand what data you hold
-For what purposes
-And how it was collected

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
What is Customer 360-Degree Analysis?
•Gather together all meaningful information about the customer (“360-degree view”)
•Organizing, matching, profiling & storing every interaction in real time
•Matched and combined; factual, interpreted, learned
‣Across all channels, and on public forums and social media
•Captures interactions across all-touch points and all channels
‣Including activity on social networks, forums, blogs
•Typically stored and processed in a Hadoop “data reservoir”
•Dynamic customer profiles with segmentation,  
behavioural analysis “at scale”
•Downstream feeds into DW, CRM and other systems

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Connect the Silos, Understand Customers, Drive Decisions
execute smarterlisten better
consumption logs,
clickstream & devices
demographic, user and
credit data
customer contacts and
service cases
transactions and
subscriptions
content metadata,
ratings, comments
marketing campaign
response
social media 
activity
programmatic 
advertising
audience
acquisition, retention
multi-channel 
marketing
targeted  
promotions
next best 
offer
personalized
content
product & service 
strategy
content acquisition
learn faster
Enriched  
Customer Profile
Correlating
Modeling
Scoring
Micro-Segments
History
Preferences

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
But … Isn’t This Just CRM?
•Typically built for call centres, sales automation
•Core data is customer service activity
•Supplemented by purchase history
•CRM system typically system of record for 
service activity, with links to transactions
‣LoB application focused on particular tasks

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Customer 360-Degree view typically used as central data store for digital marketing
•Provides key data for real-time decision engines, next-best offer, personalisation
Customer 360-Degree View as Driver of Digital Marketing
?
?
?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Customer 360-Degree View Powering Marketing + Offers
Data Transfer Data Access
Real-Time Context 
 Environmental
User Journey
Offer Feedback
Enriched  
Customer Profile
Correlatin
g
Modeling
Machine 
Learning
Scoring
Real Time 
Offers &
Suggestions
 
Up-Sell / Cross-
Sell
Decisioning
Service
Self-Learning 
Predictive
Models

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Typically Stored on Flexible, Scalable Hadoop + NoSQL
Voice + Chat
Transcripts
Call Center LogsChat Logs iBeacon Logs Website LogsCRM Data Transactions Social FeedsDemographics
Real-time Feeds, 
batch and API
$50k
Hadoop
Node
$50k
Hadoop
Node
$50k
Hadoop
Node
Hadoop
Node
Hadoop
Node
$50k$50k
Hadoop
Node
Hadoop
Node
$50k
Enriched  
Customer
Profile Modeling
Scoring
Hadoop Data
Reservoir 
Raw customer data stored at detail 
Enriched and processed for insights
$50k

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Architected using “Data Reservoir” Design Pattern
•Data for customer 360 system typically landed into a Hadoop & NoSQL-based
•Applies aggregation, joining and machine-learning processes to extract insights
Data Factory
Data Reservoir
Business
Intelligence Tools
Hadoop Platform
File Based
Integration
Stream
Based
Integration
Data streams
Discovery & Development Labs
Safe & secure Discovery and Development
environment
Data sets and
samples
Models and
programs
Marketing /
Sales Applications
Models
Machine
Learning
Segments
Operational Data
Transactions
Customer
Master ata
Unstructured Data
Voice + Chat
Transcripts
ETL Based
Integration
Raw
Customer Data
Data stored in
the original
format (usually
files) such as
SS7, ASN.1,
JSON etc.
Mapped
Customer Data
Data sets
produced by
mapping and
transforming
raw data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Data Factory
Data Reservoir
Business
Intelligence Tools
Hadoop Platform
File Based
Integration
Stream
Based
Integration
Data streams
environment
Data sets and
samples
Models and
programs
Marketing /
Sales Applications
Models
Machine
Learning
Segments
Operational Data
Transactions
Customer
Master ata
Unstructured Data
Voice + Chat
Transcripts
ETL Based
Integration
Raw
Customer Data
Data stored in
the original
format (usually
files) such as
SS7, ASN.1,
JSON etc.
Mapped
Customer Data
Data sets
produced by
mapping and
transforming
raw data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
So What is a Data Reservoir?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
What Does it Do?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
And Does it Replace My Data Warehouse?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
A technical digression…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Data from Real-Time, Social & Internet Sources is Strange
Enriched  
Customer Profile
Correlatin
g
Modeling
Machine 
Learning
Scoring
•Typically comes in non-tabular form
•JSON, log files, key/value pairs
•Users often want it speculatively
‣Haven’t though through final
purpose
•Schema can change over time
‣Or maybe there isn’t even one
•But the end-users want it now
‣Not when your ETL team are next
free

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Data Warehouse Loading Requires Formal ETL and Modeling
$1m
Analytic
DBMS Node
ETL
Data Model
ETL 
Developer
Data Modeller
Curated Data
ETL Development takes time, is fragile, but results in well-curated data
But what about data whose schema is now known?
Or final use has not yet been determined?
Dimensional data modelling gives structure to the data for business users
But also restricts how that data can be analysed
What if the end-user is better placed to apply that schema?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
… And Are Limited in What They Can Store (Economically)
$1m
Analytic
DBMS Node
DB
Instance
Compute
ETL
Data Model
ETL 
Developer
Data Modeller
$1m
Analytic
DBMS Node
Compute
$1m
Analytic
DBMS Node
Compute
$1m
Analytic
DBMS Node
Single DB Instance
Compute

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Analytic
DBMS Node
Analytic
DBMS Node
Sharded Databases Can Scale Further - At Even More Cost
$1m
Analytic
DBMS Node
Compute
Data Model
ComputeCompute
DB Shard DB Shard DB Shard
Complex Shard-Aware ETL
A-F O-R S-T
$1m $1m
Analytic
DBMS Node
Compute
DB Shard
Analytic
DBMS Node
Compute
DB Shard
Analytic
DBMS Node
Compute
DB Shard
Analytic
DBMS Node
Compute
DB Shard
$1m$1m $1m $1m
G-J K-N U-W X-Z
.. and adding more nodes means re-sharding the dataset
Also rules out mixed-workload DBs with OLTP

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Exadata Combines Best of Both … Again At Cost
Data Model
Compute
DBMS Node
Compute
Storage Cell
Storage
Compute Offload
Query offloading Filtered, projected columns only returned
Storage Cell
Storage
Compute Offload
Storage Cell
Storage
Compute Offload
Storage Cell
Storage
Compute Offload
Storage Cell
Storage
Compute Offload
Storage Cell
Storage
Compute Offload
Storage Cell
Storage
Compute Offload
Storage Cell
Storage
Compute Offload
ETL
Compute
DBMS Node
Compute
Compute
DBMS Node
Compute
Single DB Instance

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Introducing Hadoop - Cheap, Flexible Storage + Compute
•A new approach to data processing and data storage
•Rather than a small number of large, powerful servers, it spreads processing over 
large numbers of small, cheap, redundant servers
•Spreads the data you’re processing over  
lots of distributed nodes
•Has scheduling/workload process that sends  
parts of a job to each of the nodes
•And does the processing where the data sits
•Shared-nothing architecture
•Low-cost and highly horizontal scalable
Job Tracker
Task Tracker Task Tracker Task Tracker Task Tracker
Data Node Data Node Task Tracker Task Tracker

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Introducing Hadoop - Cheap, Flexible Storage + Compute
•Hadoop & NoSQL better suited to exploratory analysis
of newly-arrived data
‣Flexible schema - applied by user rather than ETL
‣Cheap expandable storage for detail-level data
‣Better native support for machine-learning and 
data discovery tools and processes
‣Potentially a great fit for our new and emerging 
customer 360 datasets, and great platform for analysis

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Hadoop Designed for Real-Time Storage of Raw Data Feeds
$50k
Hadoop
Node
Voice + Chat
Transcripts
Call Center LogsChat Logs iBeacon Logs Website Logs
Real-time Feeds
Raw Data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Supplement with Batch + API Loads of ERP + 3rd Party Data
$50k
Hadoop
Node
Voice + Chat
Transcripts
Call Center LogsChat Logs iBeacon Logs Website Logs
Real-time Feeds
CRM Data Transactions Social FeedsDemographics
Batch Loads APIs, Web Service Calls
Raw Data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
$50k
Hadoop
Node
Voice + Chat
Transcripts
Raw Data
Customer 360 Apps
Predictive  
Models
SQL-on-Hadoop
Business analytics
Real-time Feeds, 
batch and API

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Hadoop
Node
Hadoop
Node
Hadoop
Node
Hadoop
Node
Voice + Chat
Transcripts
Real-time Feeds, 
batch and API
Hadoop
Node
Compute
Hadoop
Node
Compute ComputeCompute
$5k
Compute Compute
$50k
Hadoop
Node
Raw Data across Cluster Filesystem
Compute

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Hadoop-Based Storage & Compute : A Better Logical Fit
Voice + Chat
Transcripts
Real-time Feeds, 
batch and API
$50k
Hadoop
Node
$50k
Hadoop
Node
$50k
Hadoop
Node
Hadoop
Node
Hadoop
Node
$50k$50k
Hadoop
Node
Hadoop
Node
$50k
Enriched  
Customer
Profile Modeling
Scoring
Hadoop Data
Reservoir 
Raw customer data stored at detail 
Enriched and processed for insights
$50k

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Oracle Engineered system for big data processing and analysis
•Start with Oracle Big Data Appliance Starter Rack - expand up to 18 nodes per rack
•Cluster racks together for horizontal scale-out using enterprise-quality infrastructure
Oracle Big Data Appliance
Starter Rack + Expansion
• Cloudera CDH + Oracle software
• 18 High-spec Hadoop Nodes with
InfiniBand switches for internal
Hadoop traffic, optimised for network
throughput
• 1 Cisco Management Switch
• Single place for support for H/W + S/
W 
Deployed on Oracle Big Data Appliance Engineered System
Oracle Big Data Appliance
Starter Rack + Expansion
• Cloudera CDH + Oracle software
• 18 High-spec Hadoop Nodes with
InfiniBand switches for internal
Hadoop traffic, optimised for network
throughput
• 1 Cisco Management Switch
• Single place for support for H/W + S/
W 
Enriched  
Customer Profile
Modeling
Scoring
Infiniband

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Flexible, Low-Cost Resilient Storage : Hadoop Distributed FS
•The filesystem behind Hadoop, used to store data for Hadoop analysis
‣Unix-like, uses commands such as ls, mkdir, chown, chmod
•Fault-tolerant, with rapid fault detection and recovery
•High-throughput, with streaming data access and large block sizes
•Designed for data-locality, placing data closed to where it is processed
•Accessed from the command-line, via internet (hdfs://), GUI tools etc
[oracle@bigdatalite mapreduce]$ hadoop fs -mkdir /user/oracle/my_stuff
[oracle@bigdatalite mapreduce]$ hadoop fs -ls /user/oracle
Found 5 items
drwx------ - oracle hadoop 0 2013-04-27 16:48 /user/oracle/.staging
drwxrwxrwx - oracle hadoop 0 2012-09-18 17:02 /user/oracle/moviedemo
drwxrwxrwx - oracle hadoop 0 2012-10-17 15:58 /user/oracle/moviework
drwxrwxrwx - oracle hadoop 0 2013-05-03 17:49 /user/oracle/my_stuff
drwxrwxrwx - oracle hadoop 0 2012-08-10 16:08 /user/oracle/stage

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Apache Hive : SQL Access + Table Metadata Over HDFS
•Apache Hive provides a SQL layer over Hadoop, once we understand the structure (schema)
of the data we’re working with
•Exposes HDFS and other Hadoop data as tables and columns
•Provides a simple SQL dialect for queries called HiveQL
•SQL queries are turned into MapReduce jobs under-the-covers
•JDBC and ODBC drivers provide 
access to BI and ETL tools
•Hive metastore (data dictionary) 
leveraged by many other Hadoop tools
‣Apache Pig
‣Cloudera Impala
‣etc
SELECT a, sum(b) 
FROM myTable 
WHERE a<100 
GROUP BY a
Map 
Task
Map 
Task
Map 
Task
Reduce 
Task
Reduce 
Task
Result

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
NoSQL Databases
•Family of database types that reject tabular storage,  
SQL access and ACID compliance
•Focus is on scalability, speed and schema-on-read
‣Oracle NoSQL Database - speed and scalability
‣Apache HBase - speed, scalability and Hadoop
‣MongoDB - native storage of JSON documents
•May or may not run on Hadoop, but associated with it
•Great choice for high-velocity data capture
•CRUD approach vs write-once/read many in HDFS

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Best Data Store for Customer 360 Data : Hadoop & NoSQL
•Data for customer 360 system typically landed into a Hadoop & NoSQL-based
•Applies aggregation, joining and machine-learning processes to extract insights
Data Factory
Data Reservoir
Business
Intelligence Tools
Hadoop Platform
File Based
Integration
Stream
Based
Integration
Data streams
environment
Data sets and
samples
Models and
programs
Marketing /
Sales Applications
Models
Machine
Learning
Segments
Operational Data
Transactions
Customer
Master ata
Unstructured Data
Voice + Chat
Transcripts
ETL Based
Integration
Raw
Customer Data
Data stored in
the original
format (usually
files) such as
SS7, ASN.1,
JSON etc.
Mapped
Customer Data
Data sets
produced by
mapping and
transforming
raw data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Interfaces with CRM Tools, and Accessible
‣CRM can be a consumer of data from the Hadoop-based Customer 360
‣And provide key customer attributes and sales events from CRM activity
‣Allows CRM tools to focus on their core strengths
‣With ability to interface with the Customer 360 system as appropriate
Data Reservoir
 
Business
Intelligence
Tools
CRM System
Models
Machine 
Learning
Segments
Raw  
Customer
Data
Data stored in
the original
format
(usually files)
such as SS7,
ASN.1, JSON
etc.
Mapped
Customer
Data
Data sets
produced by
mapping and
transforming
raw data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Combine with DW for Big Data Management Platform

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Combining Oracle RDBMS with Hadoop + NoSQL
•High-value, high-density data goes into Oracle RDBMS
•Better support for fast queries, summaries, referential integrity etc
•Lower-value, lower-density data goes into Hadoop + NoSQL
‣Also provides flexible schema, more agile development
•Successful next-generation BI+DW projects combine both - neither on their own is sufficient

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Today’s Oracle Information Management Ref Architecture
Actionable 
Events
Event Engine Data  
Reservoir
Data Factory Enterprise
Information Store
Reporting
Discovery Lab
Actionable
Information
Actionable 
Insights
Input
Events
Execution
Innovation
Discovery
Output
Events  
& Data
Structured 
Enterprise
Data
Other
Data

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Next-Generation Layered Data Warehouse Architecture
Virtualization& 
QueryFederation
Enterprise
Performance
Management
Pre-built &  
Ad-hoc  
BI Assets
Information 
Services
Data Ingestion
Information Interpretation
Access & Performance Layer
Foundation Data Layer
Raw Data Reservoir
Data  
Science
Data Engines &  
Poly-structured  
sources
Content
Docs Web & Social Media
SMS
Structured
Data 
Sources
•Operational Data
•COTS Data
•Master & Ref. Data
•Streaming & BAM
Immutable raw data reservoir
Raw data at rest is not interpreted
Immutable modelled data. Business
Process Neutral form. Abstracted from
business process changes
Past, current and future interpretation of
enterprise data. Structured to support agile
access & navigation
Discovery Lab Sandboxes Rapid Development Sandboxes
Project based data stores to
support specific discovery
objectives
Project based data stored to
facilitate rapid content /
presentation delivery
Data Sources

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Oracle Big Data Discovery for Cataloging Customer 360 Data
•Access a rich, interactive catalog of all  
data in Customer 360 data reservoir
•Familiar search and guided navigation  
for ease of use
•See data set summaries, user annotation  
and recommendations
•Add personal and enterprise data to  
Customer 360 datasets via self-service
‣Make sense of the wider customer 
data now loaded into data reservoir

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
The Data Integration Challenge
•Data from all the sources will need to be integrated to create the single customer view
‣Hadoop technologies (Flume, Kafka, Storm) can be used to ingest events, log data
‣Files can be loaded “as is” into the HDFS filesystem
‣Oracle/DB data can be bulk-loaded using Sqoop
‣GoldenGate for trickle-feeding transactional data
•But nature of new data sources brings challenges
•May be semi-structured or unknown schema
‣Joining schema-free datasets
•Need to consider quality and resolve incorrect,  
incomplete, and inconsistent customer data
Voice + Chat
Transcripts
Batch Load 
from files,
DB: 
Easy
Stream from 
APIs, HTTP: 
Moderate
Load raw text
from files: 
Easy
Data Reservoir
Raw  
Customer
Data
Data stored in
the original
format
(usually files)
such as SS7,
ASN.1, JSON
etc.
Mapped
Customer
Data
Data sets
produced by
mapping and
transforming
raw data
Cleanse, 
enrich and 
obfuscate raw
files: 
Lots of work..!
Join
structured+
semi/
unstructured: 
How..?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Landing raw data is easy; then the real work needs to be done - can be > 90% of project
•Four main tasks to take raw data and apply schema and combine together
1. Apply Schema to Raw and Semi-Structured Data
2. Remove Sensitive Data from Any Input Files
3. Identify joins, further enrichments and transforms
4. Store as “mapped” data in data reservoir
Ingesting Raw Customer Data : Two Key Challenges

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Data enrichment tool aimed at domain experts, not programmers
•Uses machine-learning to automate  
data classification + profiling steps
•Automatically highlight sensitive data, 
and offer to redact or obfuscate
•Dramatically reduce the time required 
to onboard new data sources
•Hosted in Oracle Cloud for zero-install
‣File upload and download from browser
‣Automate for production data loads
Raw Data
Data stored in the
original format (usually
files) such as SS7, ASN.
1, JSON etc.
Mapped Data
Data sets produced by
mapping and
transforming raw data
Voice + Chat
Transcripts
Oracle Big Data Preparation Cloud Service

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Step 1: Apply Schema to Raw and Semi-Structured Data
NLP
Embedded Information
in 
Entities
Embedded Information 
No reliable patterns
Invalid and missing data 
Sensitive data
Invalid 
emails
Stream from 
APIs, HTTP: 
Moderate
Batch Load 
from files, DB: 
Easy
Load raw text
from blog
entries, 
reviews

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Step 2: Remove Sensitive Data from Any Input Files
•Automatically profile and analyse datasets
•Use Machine Learning to spot and obfuscate sensitive data automatically

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Step 3 : Identify Common Keys and Joins using BDD
•Data ingest process automatically applies some enrichments - geocoding etc
•Can apply others from Transformation page - simple transformations & Groovy expressions

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Step 4 : Process Joined and Enriched Data Back to Hadoop
•Data joined and transformed within Big Data Discovery can be saved back to Hadoop
•Export to HDFS, register with Hive (optional)
•Supports creation of
Data Reservoir
Raw  
Customer Data
Data stored in the
original format
(usually files) such
as SS7, ASN.1,
JSON etc.
Mapped Customer
Data
Data sets produced
by mapping and
transforming raw
data
Cleanse, 
enrich and 
obfuscate raw
files: 
Lots of work..!
Join structured+
semi/unstructured: 
How..?

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Further Transforming & Managing Customer 360 Data
•Oracle Data Integration Suite offers a wider set of products for managing Customer 360 data
‣Oracle GoldenGate
‣Oracle Enterprise Data Quality
‣Oracle Data Integrator
‣Oracle Enterprise Metadata  
Management
-All Hadoop enabled
-Works across Big Data, 
Relational and Cloud

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Solution Component Logical View
•Data Factory for file, event and DB transaction batch and streaming ingestion
•Big Data Management Platform for combined Hadoop + RDBMS data storage
•Discovery Labs for innovation and sandboxing

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Discovery Labs and Sandboxes : Key to Innovation
•For Customer 360 systems to innovate beyond the initial roll-out, these are critical
•Allows innovation and new subject areas to be developed separate from IT governance
‣Then migrated to production when appropriate
• Support agile development of BI
• Demand management through BICC
• Automated provisioning / de-provisioning
• Data sourced from any data layer or off-platform
• Standardised use of BI tooling
• Careful governance required once work is
complete
• Measure: Engineering backlog and size of
shadow-IT
• Support agile discovery in data
• Demand management through Analytical CC
• Automated provisioning / de-provisioning
• Data sourced from any data layer or off-platform
• Broad range of analytical tools
• Governance step required to operationalise
insights
• Measure: value and rate of new insights to
business

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
And now the clever bit…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
I’m too sexy…

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
• Complete view of customers
• Micro-segmented customer profiles
• Predictive models

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Customer 360-Degree View as Driver of Digital Marketing
 Environmental
User Journey
Offer Feedback
Enriched  
Customer Profile
Correlatin
g
Modeling
Machine 
Learning
Scoring
Real Time 
Offers &
Suggestions
 
Up-Sell / Cross-
Sell
Decisioning
Service
Self-Learning 
Predictive
Models

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Customer 360 systems can use machine learning across all data to build predictive models
•Decision engines (for example, Oracle RTD) can use its data as source
•Wider range of attributes and faster velocity
•Aim to process data and not just to store it
-Identify customers likely to defect,  
work proactively to retain
-foster enhanced engagement
•increase revenue
•pass events and contextual data to  
real-time decisioning engines
Use Machine Learning and Real-Time Decisions to add Value
Real Time
 
Offers & Suggestions
 
Up-Sell / Cross-Sell
Decisioning Service
Self-Learning 
Predictive
Models
 Environmental
User Journey
Offer Feedback
Enriched  
Customer Profile
Correlatin
g
Modeling
Machine 
Learning
Scoring
Operational Data
Transactions
Customer 
Master Data
Unstructured
Data
Voice + Chat
Transcripts
Touch Points
Store
Web
Service

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Real-time decisioning engine from Oracle, part of Oracle BI product family
•Predictive and Personalised Real-Time Recommendations
‣Behaviour-based models that take into 
account activity across all channels
‣At very low levels of granularity
‣Micro-segmentation to individual customer
‣Multi-contextual recommendations  
based on predicted customer needs
•Real-time offers, delivered to any channel
•Feedback loop to improve recommendation
Oracle Real-Time Decisions

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Decision engines and business processes use models to predict customer behavoir
•Traditional CRM-driven decision engines only consider what happened
•Big Data + real-time feeds can dramatically improve model performance
•Model with “Big Data” and potentially thousands of input variables:
•Customer sentient data
•Competitors data
•Environmental data
•Spatial location data
•Long term vs. recent historical behavior
•Sensor data
More Data + Variety Data -> Better Predictive Models
1980s 1990s 2000s 2010s
Empowered
Employees
Digital is
Humanized
Knowledge
Everywhere
Internet of
Things
Mobile as
Primary Channel
Cross-Channel
Service
WHAT’S NEXT

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Event-Driven Personalised Marketing & Offers
•Vastly increased size, time-relevance and scope of customer data into decision models
Touch Points
Store
Web
Service
Operational Data
Events &
Workflow
Transactions
Customer 
Master Data
Profile-Based (Demographic)
Prioritization
 
 Guided Search Guided Search
Pricing Marketing
Engagement
Real Time
 
 
Up-Sell / Cross-Sell
Decisioning Service
Self-Learning 
Predictive
Models
Enriched  
Customer Profile
Correlatin
g
Modeling
Machine 
Learning
Scoring
 Environmental
User Journey
Offer Feedback
Relevant 
Personalized 
Experiences
•Then close the loop with real-time context back into decision engine

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Connect the Silos, Understand Customers, Drive Decisions
execute smarter
listen better
consumption logs,
clickstream & devices
demographic, user and
credit data
customer contacts and
service cases
transactions and
subscriptions
content metadata,
ratings, comments
marketing campaign
response
social media 
activity
programmatic 
advertising
audience
acquisition, retention
multi-channel 
marketing
targeted  
promotions
next best 
offer
personalized
content
product & service 
strategy
content acquisition
Enriched  
Customer Profile
Correlatin
g
Modeling
Machine 
Learning
Scoring

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
•Start with pilot for area of the business that needs a single view of customers
•Then, over time, iterate and build out the Customer 360-degree view
Delivering a Successful Customer 360-Degree View
Start with a business
area that 
needs a single  
customer view
Obtain clear
understanding of
customer online &
offline behaviour
Build out  
Predictive Models 
and Decision Engines 
to deliver value now
Build out Hadoop Data
Reservoir, Feeds 
and link to DW + CRM
Iterate and Build-out, 
add new integrations, 
incrementally building 
capability
Develop and Implement Strategy, Deliver Business
Value
Build DevOps Capability
Pilot & Quick Win
Create Full Production InfrastructurePilot (Virtualised / Commodity) Hadoop Infrastructure

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Packaged Customer 360 Applications
•Packaged Customer 360 applications seem a good way to start…?
•Risks around degree of fit and integration limits, but can be useful for mature projects
See everything
together –
comparisons with a
Set defined by you,
and evolving trend
scores for each
customer
From Data to DNA
– 1000s of metrics
determine
individual DNA –
common, industry
and customer
metrics

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Rittman Mead Customer 360 + Real-Time Decisions Projects
Start with a business
area that 
needs a single  
customer view
Obtain clear
understanding of
customer online &
offline behaviour
Build out  
Predictive Models 
and Decision Engines 
to deliver value now
Build out Hadoop Data
Reservoir, Feeds 
and link to DW + CRM
Iterate and Build-out, 
add new integrations, 
incrementally building 
capability

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Adding a Data Reservoir to Your Oracle Data Warehouse for
Customer 360-Degree Analysis 
Mark Rittman, CTO, Rittman Mead
UKOUG Tech’15, Birmingham, December 2015

Adding a Data Reservoir to your Oracle Data Warehouse for Customer 360-Degree Analysis

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Adding a Data Reservoir to your Oracle Data Warehouse for Customer 360-Degree Analysis

Similar to Adding a Data Reservoir to your Oracle Data Warehouse for Customer 360-Degree Analysis (20)

More from Mark Rittman

More from Mark Rittman (16)

Recently uploaded

Recently uploaded (20)

Adding a Data Reservoir to your Oracle Data Warehouse for Customer 360-Degree Analysis