Big Data & Analytics Architecture

Advanced Analytics Platform Deep Dive
Components, Patterns, Architecture Decisions
ISA-3637 (Tue Nov 5 11:15 AM – 12:15 AM)
Dr. Arvind Sathi asathi@us.ibm.com
Richard Harken rharken@us.ibm.com
Tommy Eunice teunice@us.ibm.com
Mathews Thomas Mathews@us.ibm.com

© 2013 IBM Corporation

Please note
IBM’s statements regarding its plans, directions, and intent are subject to
change or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be
incorporated into any contract. The development, release, and timing of any
future features or functionality described for our products remains at our sole
discretion.

Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job
stream, the I/O configuration, the storage configuration, and the workload
processed. Therefore, no assurance can be given that an individual user will
achieve results similar to those stated here.

Acknowledgements and Disclaimers
Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in
which IBM operates.
The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for
informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant.
While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without
warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this
presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or
representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use
of IBM software.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have
achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended
to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other
results.

© Copyright IBM Corporation 2013. All rights reserved.

•U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM Corp.

•Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere, DB2,
Maximo, Clearcase, Lotus, etc

IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered
trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM
trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols
indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may
also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at
“Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml
If you have mentioned trademarks that are not from IBM, please update and add the following lines:
[Insert any special 3rd party trademark names/attributions here]
Other company, product, or service names may be trademarks or service marks of others.

Content
• Use cases to support Business Architecture
• Components to support Application Architecture
• Data Integration
• Privacy Management & Archiving
• Location & Lifestyle Analytics
• Adaptive Analytics

• Momentum and Conclusions

AAP – Telecommunications Use Cases
Industry
Imperatives
MAJOR use
cases

Create & Deliver
Smarter Services

Transform
Operations

Location Based Services

IT Infrastructure Transformation
(Traditional to Big Data)

Cross Industry Solutions

Voice & Data Fraud

Build Smarter
Networks

Personalize
Customer
Engagements

Network Analytics

Pro Active Call Center

Network Infrastructure Planning
(Performance, Capacity, Usage)

Customer Data/Location
Monetization

Product Knowledge Hub

Smarter Campaigns
Customer Knowledge Hub
Social Media Insight

Emerging
Use Cases

 Smarter Advertising

 Customized Customer
Marketing
 3rd Party API’s
 Cloud services for
SMEs, enterprises
 Contactless services
(payments and banking)
 M2M (smart cars, eHealth)
 Tiered Services

 Big Data Scale
 Investment Decisions
 Lower storage
requirements
 Smarter Returns
 Analyze data before it
lands – then store only
what you need
 New analytic models
 Share critical
information across the
enterprise vs. deliver
multiple copies of the
data
 Traditional
Infrastructure
Optimization
 Product Knowledge Hub

 Content Network
Distribution
 Proactive Device
Management
 Network Fault Prevention
 ICTO (Energy Savings)
 Real Time Traffic
Optimization
 Network Abuse from
excessive data users
 Discrete on-line charging
for quality of experience
 Real time automated
capacity management for
dropped calls
 SON Capacity
Management for special
events (traffic offload)
 Service Migration

 Social Advocacy
 Cross Offering
Transparency
 Smarter Customer
Interaction &
Engagement
 Real-time Customer
Experience Insight
 Smarter Campaigns
 Customer Retention
 Micro Segmentation
Marketing
 Next Best Offer
 Retail cross Channel
optimization

How to turn streaming noisy Telco Location data into meaningful location, then
discover customer insights
Location Pattern Analytics
Stream data
Call Detail
Records
SMS Voice

GPS Tracking

Wifi off load

Reference Data
Cell Tower
Wifi AP Maps

GIS, POI

Special Service
Numbers
e.g bank, 1-800

Big Data
Integration

Mobile Location Data
Processing: Map mapping,
Business rules et.

Spatio-Temporal Event
Association Analysis

Analyzable Location Event Meaningful Location
Data
Who, when, where and what subscriberId:
home:
subscriberId:
Work:
Timestamp:
POIs & period …
Position: latitude +
Sequence of
longitude
meaningful
Precision: 0~2 km
Locations…
Direction: nullable
Commute means:
Speed: nullable
car/subway/bus
Activity : nullable
Micro segmentaton
Business traveler
Regular commuter
Heavy driver
Social Butterfly
Mom
…..

Location Patterns on
Individual and Group
level
Every Sunday
noon, Bob goes to
xxx mall to shopping
and has lunch
Every Thursday
afternoon, Bob goes
to customer site at
XXX
…..

Mobile Couponing Use Case
1) Contacts Offertel Communications to
run campaign for a new store next to a
movie theater

2) Opts-in to receive mobile
coupons from the Telco

7) Posts on twitter,
Facebook public fan
page for Cuppa
Heaven

Telco
Customer Profile

Campaign Delivery
System

6A) Receives mobile
6A) Receives
coupon for new
6B) Deliver
mobile coupon
Cuppa Heaven store Coupons to
for new Cuppa
mobile opt-out
Heaven store
clients via
email & web
site

7) Monitor
Campaign
Performance

5) Priority list
transferred to
conduct
campaign

Advanced
Analytics
Platform

Customer Action

Telco clients who have
opted out of Mobile
Cuppa Heaven/Offertel Action
coupons

3) Use Social media to establish
“Opinion Leaders”, potential
coffee drinkers, movie goers

4) Driving habits, coffee
preference, & opinion leaders
used to prioritize customer target
list

AAP – Media and Entertainment Use Cases

Organizational
FOCUS areas

Create differentiated customer experiences
“Connected Consumer”

Build an agile digital supply chain
“Smarter Media”

Audience & Marketing Optimization

Industry Team
use cases

Operations Analysis & Optimization

Multi-Channel Enablement

Business Process Transformation
Infrastructure Mgmt & Security

Digital Commerce Optimization

(sales play)

360o View of the Customer

Customer & Market Insight

MAJOR use cases

Advertising Optimization

Media, Metadata & Optimization.

•Social Profiling/ Sentiment Analysis
•Churn Optimization
•Customer Care Optimization
•Audience/ Viewing Duplication
•Audience Composition Index
•Multi-Platform Ad Performance
•Advertiser Revenue Analysis
•Real Time Audience Targeting
•CRM Optimization

•Real-time ad targeting
•Ad inventory Optimization
•Real-time ad reporting
•Search engine optimization
•Campaign optimization (in-flight)
•Marketing campaign effectiveness
•Network & infrastructure optimization
•Network Demand Forecasting
•Content optimization
•Content demand forecasting
•IP Rights Optimization

AAP for Real-time Bidding of Advertisements
Telco Website

Content
Provider

Turn

Telco
Flex Tag

TURN DMP

Location
Events /
xDR

Telco
Data

Usage
Data
Integration

Campaign
Feedback
Customer

Predictive
Models

TURN DSP

Campaign
Mgmt

Advanced Analytics Platform
Real-time
Scoring

Bid Req

Customer
Data

Campaign
Details

Analytics
Visualization
Additional data (e.g. Offer acceptance, location)

Offer &
Response

Bid Req

Offer &
Response

Content



New Architecture to Leverage All Data and Analytics

Streams

Data in
Motion

Information
Ingestion
and
Operational
Information
 Stream Processing
 Data Integration
 Master Data

Data at
Rest

Intelligence
Analysis

Real-time
Analytics





Video/Audio
Network/Sensor
Entity Analytics
Predictive

Landing
Area,
Analytics
Zone
and Archive

Exploration
,
Integrated
Warehouse
,
and Mart
Zones

Decision
Management

BI and Predictive
Analytics

Navigation
and Discovery
Data in
Many Forms

Information Governance, Security and Business Continuity

AAP Capabilities

IBM Big Data Advanced Analytics
Platform (AAP) Architecture

Continuous Feed
Sources
Data Repositories

External
Data

3rd party

Visualize, explore, investigate, search
and report

High Volume
Data for
Historical
Analysis

Model
Creation

Capture
Changes

Event
Execution

Open API

Discovery Analytics
Take action on
analytics

Campaign Mgmt.
Pro-active
Customer
Experience
Management
Pro-active
Network Mgmt

Deploy Model
Policy
Mgmt

Real time Scoring
& Decision Mgmt.
Policy
Management

...

B

D

In Database Mining

Database Server
Batch
Data

A

Semi
Structured
Data

Analytics
Engine

UnStructured
E Data

Structured
Data

Hadoop Enterprise Data
Warehouse

Search, Pattern Matching, Quantitative, Qualitative
F

Insight

Create & Deliver
Smarter Services

G

High Performance
Unstructured Data
analysis

Actions

Deduplicate

Data Integration
ETL

F

C

Prediction / Policy Engine

Standardize

Identity
Resolution

Outcome
Optimization

E

Customer
Activities

Historical
Data
Models
Deploy Model

High Velocity

Social

Real-time scoring, classification,
detection and action

Streaming Engine

Network
Policies

Customer
Data

Model Based Predictive Analytics

Sense, Categorize,
Score,
Identify,
Count,
Decide
Align
Focus

Streaming Data

XDR

Application
& Usage
Data

B

D

Network
Topology
Data

High Performance Historical analysis

C

Network
Events

A

Transform Operations

Build Smarter
Networks

Customer Care
Reports &
Dashboards

Ad-hoc
Queries
Simulation

Marketing

Reports

Network
Planning

Dashboards

...
NOC/SOC

Geo/Sem
antic
Mapping

Information
Interaction

Users

Personalize Customer
Engagements

G

AAP Capabilities


Continuous Feed
Sources
Data Repositories

External
Data

3rd party

F
G

High Performance
Unstructured Data
analysis

Discovery Analytics
Take action on
analytics
Customer
Activities

Event
Execution

Streaming Engine
Historical
Data
Models
Deploy Model

High Velocity

Social

and report

Sense, Categorize,
Score,
Identify,
Count,
Decide
Align
Focus

Streaming Data

Network
Policies

Customer
Data


E

InfoSphere Streams

XDR

Application
& Usage
Data

B

D

Network
Topology
Data


C

Network
Events

A

High Volume
DataSPSS
for
Historical
Analysis

Model
Creation

Capture
Changes

BPM

WODM, Optim

Pro-active
Network Mgmt

Deploy Model

Policy
Mgmt

Real time Scoring
& Decision Mgmt.

WODM

B

In Database Mining

Database Server
Batch
Data

PDA
Semi
A

Structured
Data

Analytics
Engine

UnStructured
Structured
Data
Data
E InfoSphere

PDOA
BigInsights
Warehouse

Search, Pattern
InfoSphere
Social Media Matching, Quantitative, Qualitative
F
Insight
Data Explorer
Analytics

Create & Deliver
Smarter Services

Pro-active
Customer
Experience
Management

Policy
Management

...

Actions

Deduplicate

Data Integration
ETL

Open API

C


Data Stage
Quality Stage
Standardize
MDM

Identity
Resolution

Outcome
Optimization

IBM
(Unica)
Campaign Mgmt.
Campaign


Build Smarter
Networks

D

Cognos

Customer Care
Reports &
Dashboards

Ad-hoc
Queries

SPSS
Simulation
Reports
Dashboards

Marketing
Network
Planning
...
NOC/SOC

Geo/Sem
antic
Mapping

Information
Interaction

Users

Engagements

G

AAP Capabilities
Capabilities Overview

Capability

Streaming Engine

Prediction /
Policy Engine

Database
Server

Insight

Information
Interaction

Capability Description

 Align diverse streams of data, identify customers, align to IDs, sense data importance
 Categorize incoming data, use window counts to aggregate atomic data or threshold vioilations,
focus attention on monitored situations abstracted from raw events
 Use scoring models developed by prediction engine to score observations, activities, customers,
etc. in real time
 Make data ready for execution of events – e.g., designing campaign messages based on
information available.
 Includes TEDA and geo-spatial accelerators







Create models using historical data sources
Optimize outcomes by promoting best model for a particular treatment (Champion / Challenger)
Manage policies associated with decisions – e.g., WODM decision rules, Optim data policies, etc.
Includes SPSS Deployment Server
Includes SPSS location analytics






Provide capabilities for storage of structured, unstructured and semi-structured data
Provide capabilities for analytics using DB functions (e.g., SPSS model development)
Provide capabilities for data archival using archival policies
Includes Optim / DS for archival policy execution

 Deep analysis of consumer behavior is performed to mine data for model creation
 Includes unstructured search, pattern matching using arbitrarily defined patterns, qualitative
analytics, quantification of data (e.g., sentiment analysis)
 Includes Big Insights accelerators
 Perform Ad hoc queries, standard reports, dash board
 Run simulation models, what-if analysis
 Geo-spatial and semantic viewing of data

Mature Organizations are Looking for Instantaneous
Insight from Data
Speed to insight

Respondents were asked how quickly business users
require data to be available for analysis or within
processes. Box placement reflects the prevalence of
that requirements within each a stage.
Total respondents n = 973

16

Stream Computing Represents a Paradigm Shift
Traditional Computing

Stream Computing

Historical fact finding

Current fact finding

Find and analyze information stored on disk

Analyze data in motion – before it is stored

Batch paradigm, pull model

Low latency paradigm, push model

Query-driven: submits queries to static data

Data driven – bring data to the analytics

Real-time
Analytics
17

Massively scalable stream analytics
Deployments

Linear Scalability
• Clustered deployments –
unlimited scalability

Source
Adapters

Automated Deployment
• Automatically optimize
operator deployment
across nodes
Performance Optimization
• Parallel & pipeline
operations
• Efficient multi-threading
Analytics on Streaming Data
• Analytic accelerators for a
variety of data types
• Optimized for real-time
performance
18

Analytic
Operators

Sink
Adapters

Streams Studio IDE
Automated and
Optimized
Deployment
Streaming Data
Sources

Streams Runtime

Visualization

Big Data in Real Time with InfoSphere Streams

Filter / Sample
Modify

Analyze

Fuse
Classify

Score

19

Windowed
Aggregates

Annotate

AAP Capabilities


Continuous Feed
Sources
Data Repositories

External
Data

3rd party

F
G

High Performance
Unstructured Data
analysis

Discovery Analytics
Take action on
analytics
Customer
Activities

Event
Execution

Streaming Engine
Historical
Data
Models
Deploy Model

High Velocity

Social

and report

Sense, Categorize,
Score,
Identify,
Count,
Decide
Align
Focus

Streaming Data

Network
Policies

Customer
Data


E

InfoSphere Streams

XDR

Application
& Usage
Data

B

D

Network
Topology
Data


C

Network
Events

A

High Volume
DataSPSS
for
Historical
Analysis

Model
Creation

Capture
Changes

BPM

WODM, Optim

Pro-active
Network Mgmt

Deploy Model

Policy
Mgmt

Real time Scoring
& Decision Mgmt.

WODM

B

In Database Mining

Database Server
Batch
Data

PDA
Semi
A

Structured
Data

Analytics
Engine

UnStructured
Structured
Data
Data
E InfoSphere

PDOA
BigInsights
Warehouse

Search, Pattern
InfoSphere
Social Media Matching, Quantitative, Qualitative
F
Insight
Data Explorer
Analytics

Create & Deliver
Smarter Services

Pro-active
Customer
Experience
Management

Policy
Management

...

Actions

Deduplicate

Data Integration
ETL

Open API

C


Standardize

Identity
Resolution

Outcome
Optimization

IBM
(Unica)
Campaign Mgmt.
Campaign


Build Smarter
Networks

D

Cognos

Customer Care
Reports &
Dashboards

Ad-hoc
Queries

SPSS
Simulation
Reports
Dashboards

Marketing
Network
Planning
...
NOC/SOC

Geo/Sem
antic
Mapping

Information
Interaction

Users

Engagements

G

Content




What is Sensitive Data

Personally Sensitive
• Information that can be misused to harm a person in financial,
employment or social way. (Names, Social Security Number, Credit Card,
etc.)

Network Sensitive
• Information that can be misused to breech or disable critical
network communication (Circuit Identifiers, IP Addresses, etc.)
Corporate Sensitive
• Information that can misused to compromise the competitive
position of a company (Operational Metrics, etc.)

6 steps that work together to achieve an acceptable and
manageable level of data security
Assess Risk

Audit
Define process
Processes &
Information assets

Manage

Implement
Controls

Data masking requires a combination of
process, templates and tools
Our approach brings together data masking infrastructure using DataStage and
ProfileStage, combining with Masking on Demand plug-in using Optim
technology.

Reusable Processes
Identify

Select

Verify

Implement

Validate

Templates
Masking Utilities

Data Definitions

- Incremental Autogen
- Swap
- Relational Group Swap
- String Replacement
- Universal Random

- Customer ID
- Name
- Address
- Credit Card No
- Social Sec No
- Etc.

Tools
InfoSphere Analyzer

Optim, DataStage

Content




Buddies, Hangouts, Globtrotters
10 Top Hangouts

Areas of mobility analytics

 Individual Lifestyle and Usage profiles
 Popular Locations with specific profiles

 Who are the Buddies
 Predicting where people go

Who Are You?
Homebody
Daily Grinder
Delivering the Goods
Globetrotter
Sofa Surfer

Mobile ID

Buddy Rank

2702

1

1256

2

8786

3

4792

4

8950

5

© 2012 IBM Corporation

What are Profiles
• Lifestyle Profiles are defined by marketing analysts for specific
use cases or marketing programs
• Usage Profiles are created using data mining algorithms and
define how a person uses services during the day
• Location Affinity is created with algorithms and determines
preferred locations for individuals throughout the day and
week
• Together these uniquely define a person with relation to how
the retailer or marketer might want to market to them

Creating Groups of Mobility Profiles Enables
Better Prediction for Certain Groups


profiles breakdown like this


Homebody, doesn't visit too many unique locations



Daily Grinder, back and forth to work, quiet weekends, makes
stops along the way



Norm Peterson, inside the lines, no deviations



Delivering the goods, no predictable patterns, many different
locales during the day



Globe Trotter, either not in town, or keeps their phone turned off



Rover Wanderer, spends evenings at various location (sofa
surfers www.couchsurfing.org)



“Other”, is a group hard to categorize

By Profile, when is it easy or difficult to predict
where they will be?
Profile

Day

Time

Predictability

Daily Grinder

Thursday

Dinner

Highest

Daily Grinder

Friday

Afternoon

Lowest

Homebody

Saturday

Night

Highest

Homebody

Wednesday

Morning

Lowest

These are the 2 most predictable profiles, yet there is diversity in their predictability.
To best communicate with Daily Grinders, contact them on Thursday Afternoons just before dinner

Preferred Locations of by profile type at
Lunchtime Weekdays (Central Stockholm)

Daily
Grinders
Night
Shifters
Delivering
the Goods

What analysis is available (Anonymous Data)
From the mobility profiles, summarized, anonymous analysis is
available


Summarized to ensure anonymity, analysis of popular locations
by time of day and profile of subscribers is possible






For retailers this information can help understand what types
of people are nearby at lunch time

What types of people prefer which areas. Some obvious
results are Globe Trotters go to airports, Daily Grinders go to
office buildings. Other non-obvious results show up also.
Are there predictable patterns that we can use to target
certain groups in the future?

What Makes this Possible?




Using the power of Netezza and modeling capabilities of SPSS we
can literally throw all the data at data mining algorithms and create
discrete clusters of subscribers by activity, mobility
Apply the data mining outputs to the entire subscriber base by
creating detailed specific analyses for each subscriber refined by the
mobility profiles

Content



Real-time Adaptive Analytics

High Velocity
Sensor

Analytics Engine

High Volume

Scorer

Predictive Modeler

Adaptive Analytics
• Collaboration across tools

• SPSS and iLOG to manage models and rules
• PDA to do query processing for the models
• Streams to run the model
• PMML to flow models from SPSS / iLOG to Streams

Marketing Assets
Resource

Link

IBM Big Data Hub – Telco Home
Page

http://www-01.ibm.com/software/data/bigdata/industrytelco.html

IBM Big Data Hub Cross-industry

http://www.ibmbigdatahub.com/

Light Reading Webinar – “Big Data
dramatically changes the Telco
Game Plan”

http://www.lightreading.com/webinar.asp?webinar_id=300
92&webinar_promo=1000000332

Big Data Analytics (e-book)

http://ibm.co/Zw0jRW

Big Data Analytics for
Communications Service
Providers (whitepaper)

http://bitly.com/RJHbhj

Telco Industry Blog on IBM Big
Data Hub (Author - Gaurav
Deshpande)

http://www.ibmbigdatahub.com/blog/author/gauravdeshpande

Videos

http://www.youtube.com/watch?v=FIUFYyz03u8
http://www.youtube.com/watch?v=eg8KSLAZ2HM
http://pro.gigaom.com/webinars/netezza-making-bigdata-analytics-pay/
http://youtu.be/bdJu1Pt374g

IBM Big Data / Advanced Analytics Value Proposition

All Telco Data

Combine Network Data (usage, performance,
capacity), Billing Call Detail Records, Subscriber,
Channel, Policy, Device, Social etc.

At Scale

Ability to manage the stored Petabytes of data and
incoming billions of records per day

At Speed of
Business

Ability to process data and analytics in real time and
close to point of origination to support emerging use
cases such as Location Based Services (LBS) and
Machine to Machine (M2M)

Only IBM

Only IBM can deliver the complete end to end
technology and skills to capture quickly the new ERA
value of Telco Big Data

Communities
• On-line communities, User Groups, Technical Forums, Blogs, Social
networks, and more
o Find the community that interests you …
• Information Management bit.ly/InfoMgmtCommunity

• Business Analytics bit.ly/AnalyticsCommunity
• Enterprise Content Management bit.ly/ECMCommunity

• IBM Champions
o Recognizing individuals who have made the most outstanding contributions to
Information Management, Business Analytics, and Enterprise Content
Management communities
•

ibm.com/champion

Thank You
Your feedback is important!
• Access the Conference Agenda Builder to
complete your session surveys
o Any web or mobile browser at
http://iod13surveys.com/surveys.html

o Any Agenda Builder kiosk onsite

Big Data & Analytics Architecture

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (11)

Similaire à Big Data & Analytics Architecture

Similaire à Big Data & Analytics Architecture (20)

Plus de Arvind Sathi

Plus de Arvind Sathi (8)

Dernier

Dernier (20)

Big Data & Analytics Architecture