Learn how CARFAX utilized the power of Control-M to help drive big data processing via Cloudera. See why it was a no-brainer to choose Control-M to help manage workflows through Hadoop, some of the challenges faced, and the benefits the business received by using an existing, enterprise-wide workload management system instead of choosing “yet another tool.”

1. © Copyright 4/7/2015 BMC Software, Inc1
Robert Stinnett (@robertstinnett)
CARFAX Automation Analyst
October 14, 2014
We Came, We Saw, We
Processed

2. © Copyright 4/7/2015 BMC Software, Inc2
How #BigData and
#Hadoop integrated into
@BMCControlM at
CARFAX
CARFAX helps millions of people buy, sell
and service their used cars better.

3. © Copyright 4/7/2015 BMC Software, Inc3
Agenda
1. Workload Automation at CARFAX
2. Big Data and Hadoop Initiative
3. Reduce, Reuse, Recycle
4. Batches, Unite!

4. © Copyright 4/7/2015 BMC Software, Inc4
Support traditional batch,
data transfer and SLA/SLE
management across
various datacenters.
Everything under one
roof. Integrate with other
software packages to
create an Enterprise wide
workload management
system.
Capacity on demand.
React to the hyper-
growing business.
Workload Automation at CARFAX
1
Manage Integrate Scale

5. © Copyright 4/7/2015 BMC Software, Inc5
• 129,000+ processes a
day
• 350 batch nodes
• 5 Different
Datacenters
• 1 Unified Workload
Management Platform

6. © Copyright 4/7/2015 BMC Software, Inc6
Data can be loaded and
live less than 30 minutes
after receipt
1 record to process today,
1 million records to
process tomorrow
Data is what makes
CARFAX who we are. We
can’t afford to “hope we
got it right”.
It’s All About The Data
#Fast #Dynamic #Reliable

7. © Copyright 4/7/2015 BMC Software, Inc7
Where Does it Come From?
34,000 Data Sources
13 billion records in our VHDB
Data comes in many formats,
even pictures and PDFs!
CARFAX receives data in “any format,
any time, any method”. We process
more data in an hour than many
businesses do in an entire month.

8. © Copyright 4/7/2015 BMC Software, Inc8
Our business is all about
data. We’ve been doing
Big Data long before it was
“cool”.
It had to integrate with
our currently business
processes. It is a vital part
of our data services and
will provide and consume
data from many other
applications.
We see Hadoop and other
Big Data initiatives
replacing many of our
legacy data processing
systems.
Big Data & Hadoop
2
#BigData #Integration #Future

9. © Copyright 4/7/2015 BMC Software, Inc9
Hadoop is still evolving
and maturing. Thousands
of pilot projects out
there, but very few
production installations.
We learned we were one
of the pioneers when it
came to integration.

10. © Copyright 4/7/2015 BMC Software, Inc10
Nobody wanted to
reinvent the wheel. We
didn’t need another
scheduling system.
Hadoop team wanted to
hit he ground running.
Reuse what they were
already familiar with.
Integrate with our existing
DevOps practices across
CARFAX. No silos!
Reuse, Reduce, Recycle
3
#DoMoreWithLess #TimetoMarket #DevOps

11. © Copyright 4/7/2015 BMC Software, Inc11
Agile Development +
Agile Operations
Increased usage of data
center automation and
configuration
management tools
#DevOps

12. © Copyright 4/7/2015 BMC Software, Inc12
We’ve managed to free up
our personnel to do
awesome things, and let
the automation handle
the routine stuff.
Integration, more agile
operations, compliance
and remediation. These
are what we see on the
horizon.
This wasn’t a one person,
or one team project. It
was a whole company
initiative. It has been an
amazing journey, yet
we’ve only just begun.
Batches, Unite!
4
#Today #Tomorrow #AmazingJourney

13. © Copyright 4/7/2015 BMC Software, Inc13
Overall Integration Strategy
Basic Batch
File Transfer Under
Control-M Control
Database Processing,
Web Services
Bladelogic,
Java
Informatica
Start “Run this script”SUMMARY Today “Manage our workloads” Future “Make it all just happen”
20102008 20132004 2012 Beyond
Hadoop,
SAS, ServiceNOW,
“Write Our Own”

14. © Copyright 4/7/2015 BMC Software, Inc14
#KeyTakeaways
1. Hadoop isn’t a silo, it’s part of other IT
processes.
2. Significantly reduce your learning
curve by using what you already have.
3. Integrations reduce management
headaches.
4. Evolution from batch to workload

15. © Copyright 4/7/2015 BMC Software, Inc15
@robertstinnett
robertstinnett@
Carfax.com