SlideShare une entreprise Scribd logo
1  sur  17
Capacity Model of an ETL system
Ashok Bhatla
Email – ASHOK.BHATLA.WRITER@GMAIL.COM
What is Business Intelligence?
Business Intelligence (BI) is a combination of tools, processes and
software which help a company to transform data into actionable
knowledge, thereby allowing them to take faster and informed decisions in
order to achieve their strategic goals.
It’s all about providing right information to the management at the right
time with the lowest possible cost.

As we are drowning in data, but
starving for knowledge,
Business Intelligence has
become the No. 1 priority for IT
Managers today.
What is ETL?
ETL stands for Extract, Transform and Load. A transactional system is meant
to be a high performance system so that users can get their work faster.
Running some reports from a Transactional system makes it slower. Therefore,
the concept of ETL gained popularity.

In computing, Extract, Transform, and Load
(ETL) refers to a process in database usage
which involves the following steps
Extracts data from outside sources.
Transforms it to fit operational needs,
which can include joining/reformatting
some tables.
Loads it into the end target (database,
more specifically, operational data
store, data mart, or data warehouse)
Example of ETL
OLTP Systems
Cost
Accounting
System

Payroll
Data
ETL – Joins,
Transforms,
Deletes etc.

Load Data

Sales
Data
Staged
Data
Purchasing
Data

EDW /
Reporting Data
What is Capacity Planning?
 Capacity Planning is the process of identifying the current
computing needs of a business application and to forecast the
future computing needs based on the business plans.
 In other words, it means what computing resources are needed to
meet an application’s service level objectives over a period of time.
 In today’s economic climate, business requirements can change
rapidly depending upon an organization’s strategy and goals.
 Therefore properly managed capacity plans should be able to take
unforeseen requirements into account.
 Capacity Planning can be either done in a very casual manner or
very organized and disciplined methodologies can be used.
 More data driven the capacity planning is, more accurate the
results.
Capacity Planning of an IT System
Capacity planning needs to
ensure that all Hardware (Disks,
Memory, CPU, and Network),
Software resources (User
Licenses) and facilities are
optimally used.

Software Licenses,
No. of Users
Servers, Storage,
Networking, CPU
Data Center Space,
Power, Cooling
Capacity Planning
We cannot manage
something which we
cannot measure.

Avoid
downtimes by
reducing no of
Incidents

Achieve
Performance
Objectives
established by
business

If no corrective action is
taken based on measured
data, then Capacity
Planning is of no use

Proactive
Capacity
Planning

Reduce TCO for
the ETL System

Achieve optimal
utilization of
computing
Resources
Capacity Planning Steps
Identify Service Level Objectives – know
the requirements in business terms

Analyze Current Capacity – Gather data
about resource consumption, ideal times
and peak usage
Know the future business needs and plan
for future capacity needs – How the IT
systems will be able to handle increased
load
Strike a Balance
As per Moore’s Law, IT is getting cheaper
and faster every 18 months. But
organizations cannot wait for next
generation of technology to be available –
as they need to take care of business.

Performance

Utilization
Supply

Demand
Cost
As per Parkinson’s Law, if you give
more resources to customers, they will
find ways to use more resources. IT
managers cannot keep on giving
unlimited resources to users.

Resources
Capacity Challenges for ETL Systems
ETL jobs are of different types
(Full Refresh and some Delta
Refresh), process varying
amounts of data and are
scheduled at different
frequencies. Therefore, there
are always spikes and valleys
of workload.

SQL queries are simple and do
not require parallelism. On
the other hand in an ETL
system, very large datasets
and processed and Workloads
are random in nature and not
easy to predict. This makes it
difficult to predict the
resource requirement.

An enterprise ETL system
processes thousands of
batch jobs on a daily basis.
These Systems connect to
large no. of data sources
which reside on different
platforms and may be on
different networks across
the WAN

Different types of users have
different peak usage
requirements. They have
different needs for
Transaction times, Elapsed
Times and Response Times
Disks Capacity Issues – Engineers spending lots of time cleaning
old stale data
Over Capacity – Paid for extra compute Capacity, but not
utilizing it
Network Slowness Problems – Batch Jobs running slow
sometimes.
No. of User Licenses reaching limits.
Analyse the Complete Picture
User Needs
Transaction Time
Response Time
Elapsed Time
Throughput Time
Data Usage Patterns

Data Complexity

(Type of SQL Queries or ETL Transformations)
(Financial, Marketing or Factory Data)
Business Terms
Volume and Frequency of Data Loads

User Profile

(No. of Batch Jobs and GB of data processed)

(Simple User or Advanced Data Miner)

Storage
( SAN / NAS / Local Disks,)

Processing Power(CPU, No. of Cores )
Technical Terms

Network Bandwidth
(Transfer Rate, Bytes Tx/Rx)

Memory (Physical, Cache, Swap)
Capacity Planning Tools
Vectors of Measurement
Availability
Performance
Throughput
Utilization
Quality
Efficiency

Simulation
Accurate, but needs
lots of time for setup

Testing
Costly, as another
environment similar
to Production is
needed.

Trending
Can be done using
Excel. Simple, but
does not take non
linear behavior into
account

Analytical Modeling
More advanced,
Faster and Accurate
Data Collection
No. of Subject
Period ( WW or Month) Areas

No. of ETL
No. of Projects Batch Jobs

Storage
Consumption

CPU

Network

Disk I/O

Tx/Rx Bytes

How do we collect Performance / Capacity Data?
OS monitoring tools – even freeware like Nagios, kSar, SQLMon. PerfMon
Data collected in SQL tables
Data collected by Software used by the Storage Frames – gives Utilization, Capacity
and Performance Data
Capacity Model for ETL System ??
Examples of some metrics which can be developed
o Average Run time for a Batch job
o Average CPU for a Batch job
o CPU Utilization /Subject Areas /Week
o CPU Utilization / Project / Week
o No. of Batch Jobs / GB of Storage
o No. of Batch Jobs / X Amount of CPU
Dashboard / Indicators
Phase I
Develop a Trending Model in the beginning

Dashboards can be developed using Share Point BI if the Capacity Data is captured
in an Excel Pivot Table or SQL Databases

Phase II
Can we develop a Predictive Model???
Capacity Management of an ETL System

Contenu connexe

Tendances

Asegurando la calidad del dato en mi entorno de business intelligence
Asegurando la calidad del dato en mi entorno de business intelligenceAsegurando la calidad del dato en mi entorno de business intelligence
Asegurando la calidad del dato en mi entorno de business intelligenceMary Arcia
 
Gobierno de datos con Power BI
Gobierno de datos con Power BIGobierno de datos con Power BI
Gobierno de datos con Power BIdbLearner
 
Gobierno de Datos (Data Governance) Lighting Talks
Gobierno de Datos (Data Governance)  Lighting TalksGobierno de Datos (Data Governance)  Lighting Talks
Gobierno de Datos (Data Governance) Lighting Talksproteo5
 
逻辑数据编织 – 构建先进的现代企业数据架构
逻辑数据编织 – 构建先进的现代企业数据架构逻辑数据编织 – 构建先进的现代企业数据架构
逻辑数据编织 – 构建先进的现代企业数据架构Denodo
 
‏‏Chapter 8: Reference and Master Data Management
‏‏Chapter 8: Reference and Master Data Management ‏‏Chapter 8: Reference and Master Data Management
‏‏Chapter 8: Reference and Master Data Management Ahmed Alorage
 
Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kin...
Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kin...Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kin...
Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kin...Amazon Web Services
 
Application Portfolio Rationalization - Whitepaper
Application Portfolio Rationalization - WhitepaperApplication Portfolio Rationalization - Whitepaper
Application Portfolio Rationalization - WhitepaperNIIT Technologies
 
Master Data Management - MDM - Pasos para implementar MDM
Master Data Management - MDM - Pasos para implementar MDMMaster Data Management - MDM - Pasos para implementar MDM
Master Data Management - MDM - Pasos para implementar MDMJose Pla
 
ITIL Foundation Training
ITIL Foundation TrainingITIL Foundation Training
ITIL Foundation Trainingstefanhenry
 
Data Modeling is Data Governance
Data Modeling is Data GovernanceData Modeling is Data Governance
Data Modeling is Data GovernanceDATAVERSITY
 
Sustaining Data Governance and Adding Value for the Long Term
Sustaining Data Governance and Adding Value for the Long TermSustaining Data Governance and Adding Value for the Long Term
Sustaining Data Governance and Adding Value for the Long TermFirst San Francisco Partners
 
Hybrid Cloud & Data Fabric for Dummies
Hybrid Cloud & Data Fabric for DummiesHybrid Cloud & Data Fabric for Dummies
Hybrid Cloud & Data Fabric for DummiesLiberteks
 
Session découverte de la Logical Data Fabric soutenue par la Data Virtualization
Session découverte de la Logical Data Fabric soutenue par la Data VirtualizationSession découverte de la Logical Data Fabric soutenue par la Data Virtualization
Session découverte de la Logical Data Fabric soutenue par la Data VirtualizationDenodo
 
EDW2012_LexisNexis
EDW2012_LexisNexisEDW2012_LexisNexis
EDW2012_LexisNexisJayne Dutra
 
Business Drivers Behind Data Governance
Business Drivers Behind Data GovernanceBusiness Drivers Behind Data Governance
Business Drivers Behind Data GovernancePrecisely
 
RWDG Slides: Governing Your Data Catalog, Business Glossary, and Data Dictionary
RWDG Slides: Governing Your Data Catalog, Business Glossary, and Data DictionaryRWDG Slides: Governing Your Data Catalog, Business Glossary, and Data Dictionary
RWDG Slides: Governing Your Data Catalog, Business Glossary, and Data DictionaryDATAVERSITY
 
La Gouvernance des Données
La Gouvernance des DonnéesLa Gouvernance des Données
La Gouvernance des DonnéesSoft Computing
 

Tendances (20)

Asegurando la calidad del dato en mi entorno de business intelligence
Asegurando la calidad del dato en mi entorno de business intelligenceAsegurando la calidad del dato en mi entorno de business intelligence
Asegurando la calidad del dato en mi entorno de business intelligence
 
UML for Data Architects
UML for Data ArchitectsUML for Data Architects
UML for Data Architects
 
Gobierno de datos con Power BI
Gobierno de datos con Power BIGobierno de datos con Power BI
Gobierno de datos con Power BI
 
Gobierno de Datos (Data Governance) Lighting Talks
Gobierno de Datos (Data Governance)  Lighting TalksGobierno de Datos (Data Governance)  Lighting Talks
Gobierno de Datos (Data Governance) Lighting Talks
 
ITIL - introduction to ITIL
ITIL - introduction to ITILITIL - introduction to ITIL
ITIL - introduction to ITIL
 
逻辑数据编织 – 构建先进的现代企业数据架构
逻辑数据编织 – 构建先进的现代企业数据架构逻辑数据编织 – 构建先进的现代企业数据架构
逻辑数据编织 – 构建先进的现代企业数据架构
 
‏‏Chapter 8: Reference and Master Data Management
‏‏Chapter 8: Reference and Master Data Management ‏‏Chapter 8: Reference and Master Data Management
‏‏Chapter 8: Reference and Master Data Management
 
Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kin...
Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kin...Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kin...
Power Big Data Analytics with Informatica Cloud Integration for Redshift, Kin...
 
Application Portfolio Rationalization - Whitepaper
Application Portfolio Rationalization - WhitepaperApplication Portfolio Rationalization - Whitepaper
Application Portfolio Rationalization - Whitepaper
 
Master Data Management - MDM - Pasos para implementar MDM
Master Data Management - MDM - Pasos para implementar MDMMaster Data Management - MDM - Pasos para implementar MDM
Master Data Management - MDM - Pasos para implementar MDM
 
ITIL Foundation Training
ITIL Foundation TrainingITIL Foundation Training
ITIL Foundation Training
 
Data Modeling is Data Governance
Data Modeling is Data GovernanceData Modeling is Data Governance
Data Modeling is Data Governance
 
Sustaining Data Governance and Adding Value for the Long Term
Sustaining Data Governance and Adding Value for the Long TermSustaining Data Governance and Adding Value for the Long Term
Sustaining Data Governance and Adding Value for the Long Term
 
Hybrid Cloud & Data Fabric for Dummies
Hybrid Cloud & Data Fabric for DummiesHybrid Cloud & Data Fabric for Dummies
Hybrid Cloud & Data Fabric for Dummies
 
Session découverte de la Logical Data Fabric soutenue par la Data Virtualization
Session découverte de la Logical Data Fabric soutenue par la Data VirtualizationSession découverte de la Logical Data Fabric soutenue par la Data Virtualization
Session découverte de la Logical Data Fabric soutenue par la Data Virtualization
 
EDW2012_LexisNexis
EDW2012_LexisNexisEDW2012_LexisNexis
EDW2012_LexisNexis
 
Business Drivers Behind Data Governance
Business Drivers Behind Data GovernanceBusiness Drivers Behind Data Governance
Business Drivers Behind Data Governance
 
Pemeringkatan Indeks KAMI 2014_Intan Rahayu
Pemeringkatan Indeks KAMI 2014_Intan RahayuPemeringkatan Indeks KAMI 2014_Intan Rahayu
Pemeringkatan Indeks KAMI 2014_Intan Rahayu
 
RWDG Slides: Governing Your Data Catalog, Business Glossary, and Data Dictionary
RWDG Slides: Governing Your Data Catalog, Business Glossary, and Data DictionaryRWDG Slides: Governing Your Data Catalog, Business Glossary, and Data Dictionary
RWDG Slides: Governing Your Data Catalog, Business Glossary, and Data Dictionary
 
La Gouvernance des Données
La Gouvernance des DonnéesLa Gouvernance des Données
La Gouvernance des Données
 

En vedette

Capacity management for ETL System
Capacity management for ETL SystemCapacity management for ETL System
Capacity management for ETL SystemASHOK BHATLA
 
Multiple resources for multiple intelligences
Multiple resources for multiple intelligencesMultiple resources for multiple intelligences
Multiple resources for multiple intelligencesXavier Pradheep Singh
 
Data flow in Extraction of ETL data warehousing
Data flow in Extraction of ETL data warehousingData flow in Extraction of ETL data warehousing
Data flow in Extraction of ETL data warehousingDr. Dipti Patil
 
Taming the ETL beast: How LinkedIn uses metadata to run complex ETL flows rel...
Taming the ETL beast: How LinkedIn uses metadata to run complex ETL flows rel...Taming the ETL beast: How LinkedIn uses metadata to run complex ETL flows rel...
Taming the ETL beast: How LinkedIn uses metadata to run complex ETL flows rel...rajappaiyer
 
ETL Validator: Flat File Validation
ETL Validator: Flat File ValidationETL Validator: Flat File Validation
ETL Validator: Flat File ValidationDatagaps Inc
 
Managing users & tables using Oracle Enterprise Manage
Managing users & tables using Oracle Enterprise ManageManaging users & tables using Oracle Enterprise Manage
Managing users & tables using Oracle Enterprise ManageNR Computer Learning Center
 
Capacity planning ppt @ bec doms
Capacity planning ppt @ bec domsCapacity planning ppt @ bec doms
Capacity planning ppt @ bec domsBabasab Patil
 
ETL Validator: Creating Data Model
ETL Validator: Creating Data ModelETL Validator: Creating Data Model
ETL Validator: Creating Data ModelDatagaps Inc
 
Open Source ETL vs Commercial ETL
Open Source ETL vs Commercial ETLOpen Source ETL vs Commercial ETL
Open Source ETL vs Commercial ETLJonathan Levin
 
Crossref webinar - Maintaining your metadata - latest
Crossref webinar - Maintaining your metadata - latestCrossref webinar - Maintaining your metadata - latest
Crossref webinar - Maintaining your metadata - latestCrossref
 
Supply and demand management in services
Supply and demand management in servicesSupply and demand management in services
Supply and demand management in servicesShwetanshu Gupta
 
(BDT303) Construct Your ETL Pipeline with AWS Data Pipeline, Amazon EMR, and ...
(BDT303) Construct Your ETL Pipeline with AWS Data Pipeline, Amazon EMR, and ...(BDT303) Construct Your ETL Pipeline with AWS Data Pipeline, Amazon EMR, and ...
(BDT303) Construct Your ETL Pipeline with AWS Data Pipeline, Amazon EMR, and ...Amazon Web Services
 
Overview sap bo girona nib efimatica
Overview sap bo girona nib efimaticaOverview sap bo girona nib efimatica
Overview sap bo girona nib efimaticaEfimatica
 
Strategic capacity planning for products and services
Strategic capacity planning for products and servicesStrategic capacity planning for products and services
Strategic capacity planning for products and servicesgerlyn bonus
 
Seven building blocks for MDM
Seven building blocks for MDMSeven building blocks for MDM
Seven building blocks for MDMKousik Mukherjee
 
Waiting Line Management
Waiting Line Management Waiting Line Management
Waiting Line Management Joshua Miranda
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsKhalid Salama
 

En vedette (20)

Capacity management for ETL System
Capacity management for ETL SystemCapacity management for ETL System
Capacity management for ETL System
 
Multiple resources for multiple intelligences
Multiple resources for multiple intelligencesMultiple resources for multiple intelligences
Multiple resources for multiple intelligences
 
Manage users & tables in Oracle Database
Manage users & tables in Oracle DatabaseManage users & tables in Oracle Database
Manage users & tables in Oracle Database
 
Data flow in Extraction of ETL data warehousing
Data flow in Extraction of ETL data warehousingData flow in Extraction of ETL data warehousing
Data flow in Extraction of ETL data warehousing
 
Taming the ETL beast: How LinkedIn uses metadata to run complex ETL flows rel...
Taming the ETL beast: How LinkedIn uses metadata to run complex ETL flows rel...Taming the ETL beast: How LinkedIn uses metadata to run complex ETL flows rel...
Taming the ETL beast: How LinkedIn uses metadata to run complex ETL flows rel...
 
ETL Validator: Flat File Validation
ETL Validator: Flat File ValidationETL Validator: Flat File Validation
ETL Validator: Flat File Validation
 
Managing users & tables using Oracle Enterprise Manage
Managing users & tables using Oracle Enterprise ManageManaging users & tables using Oracle Enterprise Manage
Managing users & tables using Oracle Enterprise Manage
 
Capacity planning ppt @ bec doms
Capacity planning ppt @ bec domsCapacity planning ppt @ bec doms
Capacity planning ppt @ bec doms
 
Oracle Tablespace - Basic
Oracle Tablespace - BasicOracle Tablespace - Basic
Oracle Tablespace - Basic
 
ETL Validator: Creating Data Model
ETL Validator: Creating Data ModelETL Validator: Creating Data Model
ETL Validator: Creating Data Model
 
Open Source ETL vs Commercial ETL
Open Source ETL vs Commercial ETLOpen Source ETL vs Commercial ETL
Open Source ETL vs Commercial ETL
 
Crossref webinar - Maintaining your metadata - latest
Crossref webinar - Maintaining your metadata - latestCrossref webinar - Maintaining your metadata - latest
Crossref webinar - Maintaining your metadata - latest
 
Supply and demand management in services
Supply and demand management in servicesSupply and demand management in services
Supply and demand management in services
 
(BDT303) Construct Your ETL Pipeline with AWS Data Pipeline, Amazon EMR, and ...
(BDT303) Construct Your ETL Pipeline with AWS Data Pipeline, Amazon EMR, and ...(BDT303) Construct Your ETL Pipeline with AWS Data Pipeline, Amazon EMR, and ...
(BDT303) Construct Your ETL Pipeline with AWS Data Pipeline, Amazon EMR, and ...
 
Overview sap bo girona nib efimatica
Overview sap bo girona nib efimaticaOverview sap bo girona nib efimatica
Overview sap bo girona nib efimatica
 
Strategic capacity planning for products and services
Strategic capacity planning for products and servicesStrategic capacity planning for products and services
Strategic capacity planning for products and services
 
Seven building blocks for MDM
Seven building blocks for MDMSeven building blocks for MDM
Seven building blocks for MDM
 
Waiting Line Management
Waiting Line Management Waiting Line Management
Waiting Line Management
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake Analytics
 
Capacity planning
Capacity planning Capacity planning
Capacity planning
 

Similaire à Capacity Management of an ETL System

Datawarehousing & DSS
Datawarehousing & DSSDatawarehousing & DSS
Datawarehousing & DSSDeepali Raut
 
Data warehouse concepts
Data warehouse conceptsData warehouse concepts
Data warehouse conceptsobieefans
 
Informatica and datawarehouse Material
Informatica and datawarehouse MaterialInformatica and datawarehouse Material
Informatica and datawarehouse Materialobieefans
 
Should ETL Become Obsolete
Should ETL Become ObsoleteShould ETL Become Obsolete
Should ETL Become ObsoleteJerald Burget
 
Scalable scheduling of updates in streaming data warehouses
Scalable scheduling of updates in streaming data warehousesScalable scheduling of updates in streaming data warehouses
Scalable scheduling of updates in streaming data warehousesFinalyear Projects
 
REAL TIME PROJECTS IEEE BASED PROJECTS EMBEDDED SYSTEMS PAPER PUBLICATIONS M...
REAL TIME PROJECTS  IEEE BASED PROJECTS EMBEDDED SYSTEMS PAPER PUBLICATIONS M...REAL TIME PROJECTS  IEEE BASED PROJECTS EMBEDDED SYSTEMS PAPER PUBLICATIONS M...
REAL TIME PROJECTS IEEE BASED PROJECTS EMBEDDED SYSTEMS PAPER PUBLICATIONS M...Finalyear Projects
 
BI Masterclass slides (Reference Architecture v3)
BI Masterclass slides (Reference Architecture v3)BI Masterclass slides (Reference Architecture v3)
BI Masterclass slides (Reference Architecture v3)Syaifuddin Ismail
 
Real-Time Data Warehouse Loading Methodology Ricardo Jorge S.docx
Real-Time Data Warehouse Loading Methodology Ricardo Jorge S.docxReal-Time Data Warehouse Loading Methodology Ricardo Jorge S.docx
Real-Time Data Warehouse Loading Methodology Ricardo Jorge S.docxsodhi3
 
DATAWAREHOUSE MAIn under data mining for
DATAWAREHOUSE MAIn under data mining forDATAWAREHOUSE MAIn under data mining for
DATAWAREHOUSE MAIn under data mining forAyushMeraki1
 
Эволюция Big Data и Information Management. Reference Architecture.
Эволюция Big Data и Information Management. Reference Architecture.Эволюция Big Data и Information Management. Reference Architecture.
Эволюция Big Data и Information Management. Reference Architecture.Andrey Akulov
 
An Integrated ERP With Web Portal
An Integrated ERP With Web PortalAn Integrated ERP With Web Portal
An Integrated ERP With Web PortalTracy Morgan
 
ETL VS ELT.pdf
ETL VS ELT.pdfETL VS ELT.pdf
ETL VS ELT.pdfBOSupport
 
An Integrated ERP with Web Portal
An Integrated ERP with Web Portal An Integrated ERP with Web Portal
An Integrated ERP with Web Portal acijjournal
 
Designing modern dw and data lake
Designing modern dw and data lakeDesigning modern dw and data lake
Designing modern dw and data lakepunedevscom
 

Similaire à Capacity Management of an ETL System (20)

Datawarehousing & DSS
Datawarehousing & DSSDatawarehousing & DSS
Datawarehousing & DSS
 
Data warehouse concepts
Data warehouse conceptsData warehouse concepts
Data warehouse concepts
 
Informatica and datawarehouse Material
Informatica and datawarehouse MaterialInformatica and datawarehouse Material
Informatica and datawarehouse Material
 
Should ETL Become Obsolete
Should ETL Become ObsoleteShould ETL Become Obsolete
Should ETL Become Obsolete
 
Gowthami_Resume
Gowthami_ResumeGowthami_Resume
Gowthami_Resume
 
Scalable scheduling of updates in streaming data warehouses
Scalable scheduling of updates in streaming data warehousesScalable scheduling of updates in streaming data warehouses
Scalable scheduling of updates in streaming data warehouses
 
REAL TIME PROJECTS IEEE BASED PROJECTS EMBEDDED SYSTEMS PAPER PUBLICATIONS M...
REAL TIME PROJECTS  IEEE BASED PROJECTS EMBEDDED SYSTEMS PAPER PUBLICATIONS M...REAL TIME PROJECTS  IEEE BASED PROJECTS EMBEDDED SYSTEMS PAPER PUBLICATIONS M...
REAL TIME PROJECTS IEEE BASED PROJECTS EMBEDDED SYSTEMS PAPER PUBLICATIONS M...
 
BI Masterclass slides (Reference Architecture v3)
BI Masterclass slides (Reference Architecture v3)BI Masterclass slides (Reference Architecture v3)
BI Masterclass slides (Reference Architecture v3)
 
Real-Time Data Warehouse Loading Methodology Ricardo Jorge S.docx
Real-Time Data Warehouse Loading Methodology Ricardo Jorge S.docxReal-Time Data Warehouse Loading Methodology Ricardo Jorge S.docx
Real-Time Data Warehouse Loading Methodology Ricardo Jorge S.docx
 
DW 101
DW 101DW 101
DW 101
 
DATAWAREHOUSE MAIn under data mining for
DATAWAREHOUSE MAIn under data mining forDATAWAREHOUSE MAIn under data mining for
DATAWAREHOUSE MAIn under data mining for
 
Эволюция Big Data и Information Management. Reference Architecture.
Эволюция Big Data и Information Management. Reference Architecture.Эволюция Big Data и Information Management. Reference Architecture.
Эволюция Big Data и Information Management. Reference Architecture.
 
An Integrated ERP With Web Portal
An Integrated ERP With Web PortalAn Integrated ERP With Web Portal
An Integrated ERP With Web Portal
 
Copy of sec d (2)
Copy of sec d (2)Copy of sec d (2)
Copy of sec d (2)
 
Copy of sec d (2)
Copy of sec d (2)Copy of sec d (2)
Copy of sec d (2)
 
ETL VS ELT.pdf
ETL VS ELT.pdfETL VS ELT.pdf
ETL VS ELT.pdf
 
An Integrated ERP with Web Portal
An Integrated ERP with Web Portal An Integrated ERP with Web Portal
An Integrated ERP with Web Portal
 
Designing modern dw and data lake
Designing modern dw and data lakeDesigning modern dw and data lake
Designing modern dw and data lake
 
E05WAREH1.PPT
E05WAREH1.PPTE05WAREH1.PPT
E05WAREH1.PPT
 
ETL Technologies.pptx
ETL Technologies.pptxETL Technologies.pptx
ETL Technologies.pptx
 

Plus de ASHOK BHATLA

Smart Electric Meters - Role of Govt. in Technology Management
Smart Electric Meters - Role of Govt. in Technology ManagementSmart Electric Meters - Role of Govt. in Technology Management
Smart Electric Meters - Role of Govt. in Technology ManagementASHOK BHATLA
 
World innovation - Knowledge Competitiveness Index
World innovation - Knowledge Competitiveness IndexWorld innovation - Knowledge Competitiveness Index
World innovation - Knowledge Competitiveness IndexASHOK BHATLA
 
R&d management trending between india, china and us
R&d management   trending between india, china and usR&d management   trending between india, china and us
R&d management trending between india, china and usASHOK BHATLA
 
Data centers site selection mathematical model - may 2012
Data centers site selection   mathematical model - may 2012Data centers site selection   mathematical model - may 2012
Data centers site selection mathematical model - may 2012ASHOK BHATLA
 
Dc energy efficiency presentation for psu lecture - ashok bhatla - final
Dc energy efficiency presentation   for psu lecture - ashok bhatla - finalDc energy efficiency presentation   for psu lecture - ashok bhatla - final
Dc energy efficiency presentation for psu lecture - ashok bhatla - finalASHOK BHATLA
 
Solar lantern technology adoption model for indian villages - final
Solar lantern   technology adoption model for indian villages - finalSolar lantern   technology adoption model for indian villages - final
Solar lantern technology adoption model for indian villages - finalASHOK BHATLA
 
Emerging Technology Products for Indian Villages
Emerging Technology Products for Indian VillagesEmerging Technology Products for Indian Villages
Emerging Technology Products for Indian VillagesASHOK BHATLA
 

Plus de ASHOK BHATLA (8)

Smart Electric Meters - Role of Govt. in Technology Management
Smart Electric Meters - Role of Govt. in Technology ManagementSmart Electric Meters - Role of Govt. in Technology Management
Smart Electric Meters - Role of Govt. in Technology Management
 
World innovation - Knowledge Competitiveness Index
World innovation - Knowledge Competitiveness IndexWorld innovation - Knowledge Competitiveness Index
World innovation - Knowledge Competitiveness Index
 
R&d management trending between india, china and us
R&d management   trending between india, china and usR&d management   trending between india, china and us
R&d management trending between india, china and us
 
Ashok career map
Ashok career mapAshok career map
Ashok career map
 
Data centers site selection mathematical model - may 2012
Data centers site selection   mathematical model - may 2012Data centers site selection   mathematical model - may 2012
Data centers site selection mathematical model - may 2012
 
Dc energy efficiency presentation for psu lecture - ashok bhatla - final
Dc energy efficiency presentation   for psu lecture - ashok bhatla - finalDc energy efficiency presentation   for psu lecture - ashok bhatla - final
Dc energy efficiency presentation for psu lecture - ashok bhatla - final
 
Solar lantern technology adoption model for indian villages - final
Solar lantern   technology adoption model for indian villages - finalSolar lantern   technology adoption model for indian villages - final
Solar lantern technology adoption model for indian villages - final
 
Emerging Technology Products for Indian Villages
Emerging Technology Products for Indian VillagesEmerging Technology Products for Indian Villages
Emerging Technology Products for Indian Villages
 

Dernier

Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 

Dernier (20)

Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 

Capacity Management of an ETL System

  • 1. Capacity Model of an ETL system Ashok Bhatla Email – ASHOK.BHATLA.WRITER@GMAIL.COM
  • 2. What is Business Intelligence? Business Intelligence (BI) is a combination of tools, processes and software which help a company to transform data into actionable knowledge, thereby allowing them to take faster and informed decisions in order to achieve their strategic goals. It’s all about providing right information to the management at the right time with the lowest possible cost. As we are drowning in data, but starving for knowledge, Business Intelligence has become the No. 1 priority for IT Managers today.
  • 3. What is ETL? ETL stands for Extract, Transform and Load. A transactional system is meant to be a high performance system so that users can get their work faster. Running some reports from a Transactional system makes it slower. Therefore, the concept of ETL gained popularity. In computing, Extract, Transform, and Load (ETL) refers to a process in database usage which involves the following steps Extracts data from outside sources. Transforms it to fit operational needs, which can include joining/reformatting some tables. Loads it into the end target (database, more specifically, operational data store, data mart, or data warehouse)
  • 4. Example of ETL OLTP Systems Cost Accounting System Payroll Data ETL – Joins, Transforms, Deletes etc. Load Data Sales Data Staged Data Purchasing Data EDW / Reporting Data
  • 5. What is Capacity Planning?  Capacity Planning is the process of identifying the current computing needs of a business application and to forecast the future computing needs based on the business plans.  In other words, it means what computing resources are needed to meet an application’s service level objectives over a period of time.  In today’s economic climate, business requirements can change rapidly depending upon an organization’s strategy and goals.  Therefore properly managed capacity plans should be able to take unforeseen requirements into account.  Capacity Planning can be either done in a very casual manner or very organized and disciplined methodologies can be used.  More data driven the capacity planning is, more accurate the results.
  • 6. Capacity Planning of an IT System Capacity planning needs to ensure that all Hardware (Disks, Memory, CPU, and Network), Software resources (User Licenses) and facilities are optimally used. Software Licenses, No. of Users Servers, Storage, Networking, CPU Data Center Space, Power, Cooling
  • 7. Capacity Planning We cannot manage something which we cannot measure. Avoid downtimes by reducing no of Incidents Achieve Performance Objectives established by business If no corrective action is taken based on measured data, then Capacity Planning is of no use Proactive Capacity Planning Reduce TCO for the ETL System Achieve optimal utilization of computing Resources
  • 8. Capacity Planning Steps Identify Service Level Objectives – know the requirements in business terms Analyze Current Capacity – Gather data about resource consumption, ideal times and peak usage Know the future business needs and plan for future capacity needs – How the IT systems will be able to handle increased load
  • 9. Strike a Balance As per Moore’s Law, IT is getting cheaper and faster every 18 months. But organizations cannot wait for next generation of technology to be available – as they need to take care of business. Performance Utilization Supply Demand Cost As per Parkinson’s Law, if you give more resources to customers, they will find ways to use more resources. IT managers cannot keep on giving unlimited resources to users. Resources
  • 10. Capacity Challenges for ETL Systems ETL jobs are of different types (Full Refresh and some Delta Refresh), process varying amounts of data and are scheduled at different frequencies. Therefore, there are always spikes and valleys of workload. SQL queries are simple and do not require parallelism. On the other hand in an ETL system, very large datasets and processed and Workloads are random in nature and not easy to predict. This makes it difficult to predict the resource requirement. An enterprise ETL system processes thousands of batch jobs on a daily basis. These Systems connect to large no. of data sources which reside on different platforms and may be on different networks across the WAN Different types of users have different peak usage requirements. They have different needs for Transaction times, Elapsed Times and Response Times
  • 11. Disks Capacity Issues – Engineers spending lots of time cleaning old stale data Over Capacity – Paid for extra compute Capacity, but not utilizing it Network Slowness Problems – Batch Jobs running slow sometimes. No. of User Licenses reaching limits.
  • 12. Analyse the Complete Picture User Needs Transaction Time Response Time Elapsed Time Throughput Time Data Usage Patterns Data Complexity (Type of SQL Queries or ETL Transformations) (Financial, Marketing or Factory Data) Business Terms Volume and Frequency of Data Loads User Profile (No. of Batch Jobs and GB of data processed) (Simple User or Advanced Data Miner) Storage ( SAN / NAS / Local Disks,) Processing Power(CPU, No. of Cores ) Technical Terms Network Bandwidth (Transfer Rate, Bytes Tx/Rx) Memory (Physical, Cache, Swap)
  • 13. Capacity Planning Tools Vectors of Measurement Availability Performance Throughput Utilization Quality Efficiency Simulation Accurate, but needs lots of time for setup Testing Costly, as another environment similar to Production is needed. Trending Can be done using Excel. Simple, but does not take non linear behavior into account Analytical Modeling More advanced, Faster and Accurate
  • 14. Data Collection No. of Subject Period ( WW or Month) Areas No. of ETL No. of Projects Batch Jobs Storage Consumption CPU Network Disk I/O Tx/Rx Bytes How do we collect Performance / Capacity Data? OS monitoring tools – even freeware like Nagios, kSar, SQLMon. PerfMon Data collected in SQL tables Data collected by Software used by the Storage Frames – gives Utilization, Capacity and Performance Data
  • 15. Capacity Model for ETL System ?? Examples of some metrics which can be developed o Average Run time for a Batch job o Average CPU for a Batch job o CPU Utilization /Subject Areas /Week o CPU Utilization / Project / Week o No. of Batch Jobs / GB of Storage o No. of Batch Jobs / X Amount of CPU
  • 16. Dashboard / Indicators Phase I Develop a Trending Model in the beginning Dashboards can be developed using Share Point BI if the Capacity Data is captured in an Excel Pivot Table or SQL Databases Phase II Can we develop a Predictive Model???