Bringing Analytical Models in to The Business Field

MODEL DEPLOYMENT:
THE MOMENT OF TRUTH
Presenter: Robin Way IIA Lead Faculty
and President, Corios

MODEL DEPLOYMENT:
THE MOMENT OF TRUTH
Presenter: Robin Way
IIA Lead Faculty
President, Corios
Inquiries about IIA services should be directed to sales@iianalytics.com 2 | 2013 © IIA All Rights Reserved

About IIA
IIA is an independent research firm for organizations
committed to accelerating their business through the power
of analytics. We believe that in the new data economy only
those who compete on analytics win. We know analytics
inside and out - it’s what we do. IIA works across a breadth
of industries to uncover actionable insights gleaned directly
from our network of analytics practitioners, industry experts
and faculty. The result? Our clients learn how best to
leverage the power of analytics for greater success in the
new data economy.
3

BIG IDEAS
• Many companies are not realizing the full economic
potential of their analytic model assets due to lack of
adoption.
• A common language for understanding analytic models
and using them in the business needs to be shared with
IT and the field.
• The financial benefits of proper execution and
deployment of analytic models is exceptionally
compelling.

BACKGROUND
• Corios has conducted over 50 model development and
deployment projects with North American businesses.
• Unlike in model development, few common processes
exist in the world of model deployment.
• This leads to long project cycles, increased technical
and business risk, reduced quality, and reduced
adoption of model-based insights by the field.
Copyright© 2013 IIA All Rights Reserved

WHAT NEEDS TO CHANGE?
1. Establish explicit connections between model scores
and business decisions.
2. Data structures and systems used in development
and deployment are different, and are changing even
faster.
3. Changes in models and their performance over time
need to be easily interpretable, assessed and
catalogued.
4. To deploy and execute models in production
requires practices that are not commonly adopted by
model developers.

THE FIVE D’S OF MODEL DEPLOYMENT
Develop: Model development and packaging
Decisions: Tie operational business decisions to model
scores
Data: Operationalizing analytic model deployment in a
specific data architecture
Delta: Monitor the workflow and performance of models in
the field
Deploy: Implement analytic models via a software
development life cycle

#1: MODEL DEVELOPMENT

DEVELOP: LESSONS LEARNED
• Understand the capabilities and limitations of the IT
production facilities and standards up front.
• Educate IT about the analytics process.
• The tools you use and the models you build need to be
robust to changing field conditions.

#2: DECISIONS
• Scores are not decisions. A decision is the proactive
response of the business to the prospective customer
behavior, involving the expenditure of resources.
• An evidence-based, virtuous cycle needs to be
constructed for making the best decisions.
• Test and learn practices are the best option for
businesses willing to commit to longer-term testing
strategies that maximize the value of information.

CASE STUDY:
NEXT BEST OFFER STRATEGY
The challenge
• Client: The retail marketing group for one of the top 50 banks in the world.
• The bank issues hundreds of offers every month via 30+ campaigns.
• Marketing leadership knew that ranks and scores weren’t enough to make the final decision
about which offer to assign to each customer. They also had to balance product sales goals,
cross-product halo effects, over-time contact strategies and offer cost and margin
contribution alongside customer likelihood to respond.
• In short, which offer should they give each customer in order to grow profitably?
The solution
• The bank implemented a mathematical campaign optimization routine to allocate the margin-
maximizing offer per client for each monthly campaign wave.
• After only four months of design and implementation, the bank released their first optimized
campaigns out the door.
The results
• Comparing year-over-year campaign results, the optimized campaigns executed over a two-
month period in summer 2012 produced a net increase of $22 in gross sales per customer.
In the aggregate, profitable sales growth averaged $3.5 million in monthly incremental
campaign-driven income.

DECISIONS: LESSONS LEARNED
• Don’t only implement business rules based on judgment
and experience; also build prescriptive rules optimizing
for trade-offs between alternatives in pure dollars and
cents terms.
• The best performing offers and treatments are not the
ones you will issue tomorrow; they will be the offers that
your organization has refined over several waves of
disciplined, rigorous trials, paired with conscientious
measurement.

#3: DATA ARCHITECTURES
• Batch scoring in the warehouse worked fine 10 years
ago.
• Larger pools of models, more rapid refresh, and
innovative model scoring methods now dominate
attention.
In-database analytics In-memory analytics
Over-time analytics On-demand analytics

CASE STUDY:
IN-DATABASE ANALYTICS
The challenge
• Client: The data sciences team for a large investment brokerage.
• The brokerage has built response scoring models for their customers, used to
advise financial advisors on a periodic basis.
• The brokerage needed to dramatically reduce the cycle time needed to develop
new models and to refresh their scores, so that financial advisors could have daily
updates on the best treatment for their customers, and hence grow their most
profitable relationships.
• The solution
• Corios designed and implemented massively-parallel model training and model
scoring routines, and trained the client team to modernize the remainder.
• The results
• Compute tasks that previously required days to run, now run in a few minutes or
less, on billions of transactions.
• This also returned lifts on models of 10-20% by taking more transactional
predictors into account, not previously possible, due to computational scale limits.

DATA: LESSONS LEARNED
• Batch warehouse updates still work well if your refresh
timelines are monthly and customers don’t migrate
rapidly.
• Otherwise:
• Consider in-database analytics when transaction-level trends
are important for predictions, and you’ve already made a commit
to data appliances or distributed file systems.
• Consider in-memory analytics when stakeholders need to make
detail-sensitive decisions with intense, interactive visual input.
• Consider over-time analytics for advanced trending, such as
stress testing, customer survival, and new product adoption.
• Consider on-demand scoring for deployment on individual
customers, given a high degree of confidence in your models’
reliability, strong believability in the field, and patience for
testing. Copyright© 2013 IIA All Rights Reserved

#4: DELTA: MONITORING MODEL
PERFORMANCE
• Document the scope, ownership, quality assurance,
approval and signoff, and lifecycle of a model asset.
• Understand the model’s insights, including the
interpretation, defensibility, and identification of the best
business actions to take based on a model score.

EXAMPLE: CREDIT RISK MODEL
CALIBRATION

DELTA LESSONS LEARNED
• Lack of visibility into model inventories is a source of
regulatory pressure that needs to be met now, rather
than “a few years from now”.
• Workflows for tracking the model asset through its
stages of development, review, approval and
retirement, and commit the appropriate staffing to
support.
• Models are worth hundreds of times their development
cost, deserving prudence comparable to managing
assets such as human resources or physical plant.

#5: DEPLOY
• Deploying models in production needs to be a
structured, disciplined activity.
• The software development life cycle (“SDLC”) is an apt
model, with some minor adaptations.

THE SDLC APPLIED TO MODEL
DEPLOYMENT

DEPLOYMENT: LESSONS LEARNED
• Select a single platform for model development and
deployment in order to reduce the quality assurance
challenges that arise when translating from one
platform’s code base to another.
• Quality assurance can be improved by using peer
review, version and change management practices, and
standards for model documentation and packaging.
• Model performance should be tracked both on accounts
that get scored on-demand during the day, as well as
on all accounts on a periodic basis, for tracking model
stability.

JOIN US FOR THE NEXT IIA WEBCASTS
July 24, 2013 Analytics 3.0: Opportunities for Healthcare
Thomas H. Davenport and Jack Phillips
Register at iianalytics.com

Bringing Analytical Models in to The Business Field

Recommandé

Recommandé

Contenu connexe

Dernier

Dernier (20)

En vedette

En vedette (20)

Bringing Analytical Models in to The Business Field

Notes de l'éditeur