1.
Innovation in the Data Warehouse
Kit Menke, Software Architect & Scott Shaw, Sr. Solutions Engineer
DataWorks / Hadoop Summit 2017
July 2017

2.
Agenda
1. Moving to Lambda
2. Forming a Center of Excellence
3. Pain Points

3.
Enterprise Holdings, Inc.
 Our Business
• 9,600 locations
• 90 countries
• 97 thousand employees
• 1.9 million vehicles
 Our Data Warehouse
• Streaming and batch data feeds from over 50 internal
systems & external sources
• 100+ databases and 22+ thousand tables
• Around 1 billion queries executed per month
• 5+ million report executions every month.
• Statistical Modeling & Advanced Analytics - 40+ Projects
Implemented for Predictive & Diagnostic Analytics

4.
Data Warehouse - Present

5.
Data Warehouse Growth
0
20
40
60
80
100
120
140
Terabytes
Space Usage Max Usable Disk Space Total Disk Space

6.
A Next Generation Architecture
 Our considerations…
• Design for streaming
• Scalability & isolation
• Right tool for the right job!
 Decision points
1. Use cases and data gravity
2. Workload
3. Cloud vs on premises

7.
Use cases and data gravity
 Where are your data sources?
 Network
 Consumers

8.
Workload

9.
Cloud vs on premises

10.
Other considerations
 Cloud
• Different design considerations
• Disaster recovery
• Security

11.
Lambda

12.
Serving Layers

13.
Serving Layers

14.
Future Architecture

15.
Implementing an Architecture
 Who?
 “Tip of the spear”
 Learn quickly and adapt
 Out of the day to day support of current
systems

16.
Evaluation
Biz Value
Awareness
& Interest
Evaluation
Technical
Enterprise
Deployment
Enterprise
Production
Modern
Data
Architecture
Point
Deployment
Point
Production
1 – 2 months
2-6 months
9-15 months
18-36 months
1 2 3 4Potential Operational Strategic Data-Driven
Data Lake
Industry
Leadership
Hortonworks
Services
Client
Self Mgmt.
Center Of Excellence Deployment
COE with
HWX
Services
Support
COE with
Reduced
HWX
Services
Self
Sustaining
COE with
On Demand
HWX
Services
Journey to Data Driven Org via Center of Excellence

17.
Hadoop
Resources
Existing
Resources
(internal or vendor provided)
Architect
Training
Data Architect Hadoop Architect
Hive
Training
Business Analyst Hadoop Analyst
Developer
Training
Developer
(Java/Scripting, Python, Ruby, etc…)
Hadoop Developer
Admin
Training
Linux/Windows
Administrator
Cluster Ops
& Admin
1. People : Training and Knowledge Transfer for Existing Staff

18.
DATA SCIENTIST
DATA ANALYST HDF DEVELOPERADMINSTRATORJAVA DEVELOPER
HADOOP APPLICATION
DEVELOPER
BI END-USERS
MANAGERS
EXECUTIVES
APACHE PIG & HIVE (4 DAYS)
HORTONWORKS CERTIFIED PROFESSIONAL DEVELOPER
HORTONWORKS CERTIFIED
ADMINISTRATOR
HADOOP ADMIN I (4d)
HADOOP ADMIN II (4d)
SECURITY (3d)
HORTONWORKS CERTIFIED
SPARK DEVELOPER
ENTERPRISE SPARK I (4d) +
SPARK DATA SCIENCE (4d)
DATA SCIENCE (3d)
HORTONWORKS CERTIFIED
JAVA DEVELOPER
HBASE ESSENTIALS (2d)
JAVA DEVELOPER (4d) +
STORM & TRIDENT (2d)
ONLINE REFERENCE LIBRARY (self-service)
HBASE ESSENTIALS (2d)
HORTONWORKS
DATA FLOW (3d)
HORTONWORKS CERTIFIED
SPARK DEVELOPER
ENTERPRISE SPARK I (4d) +
HORTONWORKS CERTIFIED
SPARK DEVELOPER
ENTERPRISE SPARK I (4d) +
Hortonworks
Certification
Hortonworks
Course
Legend
LEARNING & CERTIFICATION
PATHS

19.
People1
Empower teams and individuals
through successes and failures
 Identify and build individuals
with potential
 Design and implement career
paths and other solutions to
drive individual growth
 Establish strong internal and
external community presence
 Develop training curriculum
focused on forming leaders
Process2 Platform3
Leverage processes and
Configuration Items to enable
excellence
 Establish foundational mission and
directional artifacts
 Design and implement project
methodologies and core
competencies aligned with business
strategy
 Establish knowledge sharing and
collaboration repositories
Leverage the right technology to
address your needs
 Establish architecture and design
principles focused on leveraging
technology to address business
needs
 Deliver technical solutions capable
of supporting innovation in a
governed environment
 Develop enterprise integration
solutions focused on scalability
 Deploy systems and controls
focused on improving quality.
Transferrable skills with training Processes Integration with refinementPlatform that powers Future of Data
Three P’s Are Central to Your Center of Excellence

20.
Our Experience
1. Skill sets
2. Lessons Learned
3. Governance
4. Agile

21.
Skill sets
 Hiring
• Problem: High demand,
complicated ecosystem, low overall experience
• Workaround: hire those who learn quickly and
have skills that transfer easily
 Culture
• Change is hard
• Software Engineer vs
Java Developer

22.
Lessons Learned
 Workload isolation
• Multi-tenancy is possible but difficult
 Debugging / development is hard
• Lots of moving pieces
• Logs spread out across many machines
• Development environments require a lot of
software
• Distributed systems just work differently
 Technologies
• Like: Hive, HBase, Spark
• Dislike: Oozie, Sqoop

23.
Governance
 Open source
• Huge number of projects
• Licenses
 Schema management
• Multiple different databases
• Evolution
 Data
• Different data types, files, no relational data
• Certification process

24.
Agile / Project Management
 Planning
 Dependencies
 Deliverables
 Aggressive timelines
 A lot to learn, so fail fast!

25.
Conclusion
1. Moving to Lambda
2. Forming a Center of Excellence
3. Our Experiences
4. Enjoy the journey!