Bridging the gap: achieving fast data synchronization from SAP HANA by leveraging Hadoop
American Water will share the success story of American Water’s production use case of leveraging Hadoop and Streaming to ingest and supply de-normalized data from the source transactional systems to end-user applications. It covers the end-to-end flow and the challenges faced. The data is de-normalized into single subject views at the source to eliminate complex join logic during ingestion into the data lake. Within the views, only timestamps on highly volatile tables have been exposed to give visibility to updates and inserts that have occurred on a table. NiFi ingests the data with a new processor and then stores it in ACID tables in Hive. The custom processor polls the timestamp columns, which generates paginated queries that consists of the delta. American Water’s use case: Our field employees are our front line with our customers and in the past have felt unable to help customers effectively with our past technologies. One of the largest initiatives is to enable our field employees with accurate and up-to-date information via a new application so they can provide a great customer experience. Speaker John Kuchmek, American Water, Sr. Technologist Adam Michalsky, American Water, Senior Technologist
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Bridging the gap: achieving fast data synchronization from SAP HANA by leveraging Hadoop
Similaire à Bridging the gap: achieving fast data synchronization from SAP HANA by leveraging Hadoop (20)
Plus de DataWorks Summit
Plus de DataWorks Summit (20)
Dernier
Dernier (20)
Bridging the gap: achieving fast data synchronization from SAP HANA by leveraging Hadoop
- 1. Bridging the Gap John “Colonel” Kuchmek Adam “Chief” Michalsky
- 2. WHO WE ARE We serve a broad national footprint and a strong local presence. We provide services to approximately 15 million people in 46 states and Ontario, Canada. We employ 6,900 dedicated and active employees and support ongoing community support and corporate responsibility. We treat and deliver more than one billion gallons of water daily. We are the largest and most geographically diverse publicly traded water and wastewater service provider in the Unites States.
- 3. Problem Statement Achieve fast change data capture from SAP while providing de- normalized data sets to end consumers without impacting the source transactional systems. Hana table replication maintains source system normalization which can be a problem for business logic design in application use No Hana change data capture existed using denormalized table structures
- 4. Environment 4 Management Nodes: (32 Cores x 78 GB) 8 Compute Nodes (32 Cores x 128 GB) 2 Management Nodes: (6 Cores x 16 GB) 5 NiFi Nodes (16 Cores x 64 GB)
- 5. Data Ingestion - Architecture Runtime SLT SOURCE INGEST STORAGE ANALYTICS UI/UX
- 6. Adding Timestamp to SLT
- 8. CDC Process – High Level Staging Delta Base
- 10. Metrics (Average Merge Time) maintenancenotificati onacts meterataglance crmlongtext interactionrecords ecclongtext maintenanceordersta tus contractaccountdoch eaderfb records 2764 15248 19958 18970 20235 8183 175433 base table 36220184 13589752 18753324 74356523 143224450 166172977 398561392 seconds 141 121 178 99 139 152 449 1 10 100 1000 1 10 100 1000 10000 100000 1000000 10000000 100000000 1E+09 #OFRECORDSLOGBASE10 Average Concurrent Merges per Table on LLAP
- 11. Metrics (Max, Min & Mean) min(simultaneous) average(simultaneous) max(simultaneous) records 18970 37255.85714 175433 base table 74356523 121554086 398561392 seconds 99 182.7142857 449 1 10 100 1000 10000 100000 1000000 10000000 100000000 1E+09 #OFRECORDSLOGBASE10 Min,Avg, Max Time for 7 Concurrent Merges on LLAP
- 12. Source System Load DURATION (Seconds) 0 50,000 100,000 150,000 200,000 250,000 300,000 5/18/1813:41 5/18/1813:40 5/18/1813:40 5/18/1813:36 5/18/1813:36 5/18/1813:35 5/18/1813:35 5/18/1813:35 5/18/1813:35 5/18/1813:35 5/18/1813:02 5/18/1813:00 5/18/1812:58 5/18/1812:56 5/18/1812:56 5/18/1812:55 5/18/1812:54 5/18/1812:54 5/18/1812:54 5/18/1810:10 5/18/1810:08 5/18/1810:07 5/18/1810:04 5/18/1810:04 5/18/1810:03 5/18/1810:03 5/18/1810:03 5/18/1810:03 5/18/1810:03 #OFRECORDS TIME Source System Load (snapshot) 250,000-300,000 200,000-250,000 150,000-200,000 100,000-150,000 50,000-100,000 0-50,000
- 13. Average CPU Utilization 0 10 20 30 40 50 60 %UTILIZATION TIME Average CPU Usage Accross 8 Node Cluster Average of Minimum CPU Load Average of Average CPU Load Average of Peak CPU Load
- 14. Average Memory Used (hourly) 0 10 20 30 40 50 60 70 80 90 100 MEMORYINGB TIME Average Memory Used Across 8 Node Cluster Average of Minimum Memory Used Average of Average Memory Used Average of Peak Memory Used
- 15. THANK YOU
Notes de l'éditeur
- In HANA studio we can de-normalize the datasets.
- The end result in HANA will look like this. UPDATE_TS is our timestamp field. Special Notes: A timestamp will only be updated once a change occurs. After initial replication timestamps will be null or 0. If you want to add a timestamp on a table that already exists on SLT then it needs to be re-replicated.
- In HANA studio we can de-normalize the datasets.