How much time and effort (and budget) do organizations spend moving data around the enterprise? Unfortunately, quite a lot. These days, ETL developers are tasked with performing the Extract (E) and Load (L), and spending less time on their craft, building Transformations (T). This changes in the new world of data integration. By offloading data from the RDBMS to Hadoop, with the ability to present it back to the relational database, data can be seamlessly integrated between different source and target systems. Transformations occur on data offloaded to Hadoop, using the latest ETL technologies, or in the target database, with a standard ETL-on-RDBMS tool. In this session, we’ll discuss how the new world of data integration will provide focus on transforming data into insightful information by simplifying the data movement process. Presented at Enkitec E4 2017.

1. Offload, Transform, and Present - the New World of Data Integration
Michael Rainey | E4 2017

2. • Michael Rainey - Technical Advisor
• Spreading the good word about gluent
products with the world
• Data Integration expertise
• Oracle ACE Director
• @mRainey
2
Introduction
we liberate enterprise data

3. 3
About Gluent
Globally distributed team with a deep
background in building high performance
enterprise applications and systems …
2nd place in Strata+HadoopWorld
Startup Showcase 2017.
(Demo video at gluent.com)
Now on a mission to
liberate enterprise data.

4. 4
Gluent Recognized by Gartner Inc’s “Cool Vendors in Data
Management, 2017” report:
Cool Vendors in Data Management, 2017
Adam M. Ronthal, Ted Friedman, Mark A. Beyer, Guido De Simoni, Merv
Adrian, Svetlana Sicular, Ehtisham Zaidi, Roxane Edjlali, 28 April 2017
The Gartner Cool Vendor Logo is a trademark and service mark of Gartner, Inc.,
and/or its affiliates, and is used herein with permission. All rights reserved.
Gartner does not endorse any vendor, product or service depicted in its research
publications, and does not advise technology users to select only those vendors
with the highest ratings or other designation. Gartner research publications
consist of the opinions of Gartner's research organization and should not be
construed as statements of fact. Gartner disclaims all warranties, expressed or
implied, with respect to this research, including any warranties of merchantability
or fitness for a particular purpose.
Download (no registration needed):
https://gluent.com/cool-vendor-2017/

5. 5
Extract Transform Load (ETL)
Source 1
Source 2
Source X
EDW
Transform
Extract
Load

6. 6
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training

7. 6
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training
Employee

8. 6
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training
Employee

9. 7
ETL Examples - IoT from set-top boxes
IoT

10. 7
ETL Examples - IoT from set-top boxes
IoT
EDW
CRM
CRM

11. 7
ETL Examples - IoT from set-top boxes
IoT
EDW
CRM
CRM
Enhanced
customer
data

12. 7
ETL Examples - IoT from set-top boxes
IoT
EDW
CRM
CRM
Enhanced
customer
data

13. 8
ETL Examples - data scientist pipelines

14. 8
ETL Examples - data scientist pipelines

15. 8
ETL Examples - data scientist pipelines

16. 8
ETL Examples - data scientist pipelines

17. 8
ETL Examples - data scientist pipelines

18. 9
Typical ETL development - too much time spent on “E” and “L”
reader = new TransformingReader(reader)
.add(new SetCalculatedField("AvailableCredit",
"parseDouble(CreditLimit) - parseDouble(Balance)"))
.add(new ExcludeFields("CreditLimit",
"Balance"));
DataWriter writer = new
JdbcWriter(getJdbcConnection(), "dp_credit_balance")
.setAutoCloseConnection(true);
JobTemplate.DEFAULT.transfer(reader, writer);

19. 9
Typical ETL development - too much time spent on “E” and “L”
ETL Developer / Data Engineer
should spend their time on
Transformations!
reader = new TransformingReader(reader)
.add(new SetCalculatedField("AvailableCredit",
"parseDouble(CreditLimit) - parseDouble(Balance)"))
.add(new ExcludeFields("CreditLimit",
"Balance"));
DataWriter writer = new
JdbcWriter(getJdbcConnection(), "dp_credit_balance")
.setAutoCloseConnection(true);
JobTemplate.DEFAULT.transfer(reader, writer);

20. 10
We still need to move data!

21. • Data lake - enterprise data stored in raw form
• Data hub - centralized data store with standards, governance, and quality
11
Data lake or data hub?

22. • Sharing data without moving data
• Metadata is stored about each “source” database
• Queries are passed through a metadata
layer, which provides info about where the
data lives and how to translate the query
• But…
• We must recode applications!
• Data remains in RDBMS silos and are
limited to CPU/storage constraints
12
What about data federation?
Source 1
Source 2
Source X
Source 1
Source 2
Source X
App 1
App 2
App 1
App 2

23. 13
Three things we need the ability to do:
1. Offload data from Oracle to Hadoop
2. Load data from Hadoop to Oracle
3. Query Hadoop data in Oracle

24. • “Offload” - copy or move data from the relational database to Hadoop
• Why offload?
• Store data in a centralized location for access and sharing
• Use the distributed, parallel processing power of Hadoop for transformations
• Enable the use of “new world” technologies (Spark, Impala, etc)
• Why Hadoop?
• We can now afford to keep a copy of all enterprise data for data sharing reasons!
14
Offload, not Extract

25. 15
How to offload? There are many options
Tool Offload Data
Sqoop Yes
Oracle Loader for Hadoop
Oracle SQL Connector for HDFS
ODBC Gateway
Big Data SQL
Gluent Data Platform Yes

26. • Command line client used to bulk copy data from a relational database to
HDFS over JDBC connection
• Also works in reverse, HDFS to RDBMS
• Sqoop generates MapReduce jobs for the work
16
Bulk data load with Sqoop
sqoop import --connect jdbc:oracle:thin:@myserver:1521/MYDB1
--username myuser
--null-string ''
--null-non-string ''
--target-dir=/user/hive/warehouse/ssh/sales
--append -m1 --fetch-size=5000
--fields-terminated-by '','' --lines-terminated-by ''n''
--optionally-enclosed-by ''"'' --escaped-by ''"''
--split-by TIME_ID
--query ""SELECT * FROM SSH.SALES WHERE TIME_ID < DATE'1998-01-01'""

27. • Gluent Advisor helps determine which tables
and/or partitions are candidates for offload
• Offload options:
• Move or copy 100% of data
• Move “cold” partitions only
• Data can be kept in-sync with an incremental
offload
• Example: when a partition goes inactive (cold), it
can be offloaded to Hadoop
• Data can be updated from RDBMS to Hadoop
17
Offload data with Gluent Data Platform
gluent.
80% - 20%
offload -x -t EDW.BALANCE_DETAIL_MONTHLY --less-than-value=2017-05-01

28. • Gluent Advisor helps determine which tables
and/or partitions are candidates for offload
• Offload options:
• Move or copy 100% of data
• Move “cold” partitions only
• Data can be kept in-sync with an incremental
offload
• Example: when a partition goes inactive (cold), it
can be offloaded to Hadoop
• Data can be updated from RDBMS to Hadoop
17
Offload data with Gluent Data Platform
gluent.
80% - 20%
offload -x -t EDW.BALANCE_DETAIL_MONTHLY --less-than-value=2017-05-01

29. 18
Three things we need the ability to do:
1. Offload data from Oracle to Hadoop
2. Load data from Hadoop to Oracle
3. Query Hadoop data in Oracle

30. 18
Three things we need the ability to do:
1. Offload data from Oracle to Hadoop
2. Load data from Hadoop to Oracle
3. Query Hadoop data in Oracle
Present

31. • Sqoop in reverse
• Reads HDFS files, converts to JDBC arrays and insert (or update) statements
19
The “L” of ETL - Loading the data
RDBMS

32. 20
Forget the “L”oad. Present the data to the RDBMS
RDBMS

33. 21
Load the data from Hadoop to RDBMS?
Tool Offload Data Load Data
Sqoop Yes Yes
Oracle Loader for Hadoop Yes
Oracle SQL Connector for HDFS Yes
ODBC Gateway Yes
Big Data SQL Yes

34. 21
Load the data from Hadoop to RDBMS?
Tool Offload Data Load Data
Sqoop Yes Yes
Oracle Loader for Hadoop Yes
Oracle SQL Connector for HDFS Yes
ODBC Gateway Yes
Big Data SQL Yes
Gluent Data Platform Yes
Present

35. 21
Load the data from Hadoop to RDBMS?
Tool Offload Data Load Data
Sqoop Yes Yes
Oracle Loader for Hadoop Yes
Oracle SQL Connector for HDFS Yes
ODBC Gateway Yes
Big Data SQL Yes
Gluent Data Platform Yes
Present

36. • Oracle SQL Connector for HDFS
• Uses Oracle External Table to access delimited file or Hive table in Hadoop
• When queried, all data is read from Hadoop to Oracle, then processed
• Oracle to Hadoop over database links
• Create a database link from Oracle to Hadoop using the Impala (or Hive) ODBC
gateway
• Pushes filters to Hadoop but not grouping aggregates
• Big Data SQL
• Access Hive tables via an Oracle external table
• Predicate pushdown
22
Oracle SQL Connector for HDFS

37. 23
Gluent Present
gluent.

38. 23
Gluent Present
gluent.

39. 23
Gluent Present
gluent.

40. 23
Gluent Present
gluent.
present -t "SH.SALES" -xv --target-name="SH.SALES_DIVISION_X"

41. 23
Gluent Present
gluent.
present -t "SH.SALES" -xv --target-name="SH.SALES_DIVISION_X"

42. 24
Three things we need the ability to do:
1. Offload data from Oracle to Hadoop
2. Load data from Hadoop to Oracle
3. Query Hadoop data in Oracle
Present

43. 25
How to query the data? There are many options
Tool Offload Data Load Data Allow Query
Data
Offload
Query
Parallel
Execution
Sqoop Yes Yes Yes
Oracle Loader for Hadoop Yes Yes
Oracle SQL Connector for HDFS Yes Yes Yes
ODBC Gateway Yes Yes Yes
Big Data SQL Yes Yes Yes Yes
Gluent Data Platform Yes Yes Yes Yes
Present

44. 26
Present Data From Anywhere To Anywhere
gluent.

45. 26
Present Data From Anywhere To Anywhere
gluent.

46. 26
Present Data From Anywhere To Anywhere
gluent.

47. 26
Present Data From Anywhere To Anywhere
gluent.

48. 26
Present Data From Anywhere To Anywhere
gluent.

49. • A new approach…a new acronym?
27
Offload Transform Present (OTP?)

50. EDW
28
ETL Examples - employee data
EDW
HR
Network
Facilities
Security
Training
Employee

51. 29
Now using OTP - employee data
EDW
Employee

52. 29
Now using OTP - employee data
EDW
Employee

53. 29
Now using OTP - employee data
EDW
Employee

54. 29
Now using OTP - employee data
EDW
Employee

55. 29
Now using OTP - employee data
EDW
Employee

56. 30
ETL Examples - IoT from set-top boxes
IoT
EDW
CRM
CRM
Enhanced
customer
data

57. EDW
CRM
CRM
IoT
Enhanced
customer
data
30
ETL Examples - IoT from set-top boxes

58. 31
ETL Examples - data scientist pipelines

59. 31
ETL Examples - data scientist pipelines

60. 31
ETL Examples - data scientist pipelines

61. • No upfront massive project cost to move data around.
• Can happen in minutes / hours, not weeks / months
• Large corporations can move as fast as startups
• A new approach to data sharing across enterprise
• Offload all data for simple access
• Takes a different mindset
32
OTP simplifies the process of data movement!

62. 33
we liberate enterprise data
thank you!