Oracle is the most powerful DBMS in the world. However, Oracle's expensive and complex replication makes it difficult to build highly available applications or move data in real-time to data warehouses and popular databases like MySQL. In this webinar you will learn how Continuent Tungsten solves problems with Oracle replication at a fraction of the cost of other solutions and with less management overhead too -- think "Oracle GoldenGate without the price tag!" We will show you how to replicate data in real-time from Oracle to Oracle, from Oracle to MySQL, and loading a data warehouse.

1. ©Continuent 2014
Replicate from Oracle to Oracle,
Oracle to MySQL
and Oracle to Analytics
MC Brown, Dir. of Product Management
Linas Virbalas, Senior Software Engineer

2. ©Continuent 2014
Introducing Continuent
2
• The leading provider of clustering and
replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement

3. ©Continuent 2014
Quick Continuent Facts
• Largest Tungsten installation processes over
700 million transactions daily on 225
terabytes of data
• Tungsten Replicator was application of the
year at the 2011 MySQL User Conference
• Wide variety of topologies including MySQL,
Oracle, Vertica, and MongoDB are in
production now
• MySQL to Hadoop deployments are now in
progress with multiple customers
3

4. ©Continuent 2014©Continuent 2014
Continuent Tungsten Customers
4

5. ©Continuent 2014
Tungsten Replicator
5
Tungsten Replicator is a fast,
open source, database
replication engine
!
Designed for speed and flexibility
GPLV2 license
100% open source
Annual support subscription available from Continuent

6. ©Continuent 2014
Tungsten Master/Slave in Action
6
Master
(Transactions + Metadata)
Slave
THL
DBMS
Logs
Replicator
(Transactions + Metadata)
THLReplicator
Download
transactions
via network
Apply using JDBC

7. ©Continuent 2014
Master Replication Service
7
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Pipeline
MySQL
Master
Transaction
History Log
In-Memory
Queue
Slave
ReplicatorsBinlog
tcp/ip

8. ©Continuent 2014
Slave Replication Service
8
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Pipeline
Transaction
History Log
In-Memory
Queue
Slave
DBMS
Master
Replicator
tcp/ip

9. ©Continuent 2014
Multiple Services per Replicator
Service
frommysql
Service
fromoracle
frommysql
fromoracle
Replicator
Replicator
Replicator
Aggregated

10. ©Continuent 2014
star
master-slave Heterogeneous
fan-in slave all-masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL MySQL

11. ©Continuent 2014
Replicating from
Oracle to Oracle

12. ©Continuent 2014©Continuent 2013
Steps to Homogeneous Replication
1. Restore backup on the slave
2. Set up replication
3. Continue real-time replication
Tungsten Replicator
tpmsetupCDC &

13. ©Continuent 2014
Preparing CDC
13
Tables
Source schema
setupCDC
Oracle CDC
(Synchronous or
Async Hotlog)
Publisher schema
Change tables
Capture started at
<setupCDC_SCN>

14. ©Continuent 2014
CDC in Action
14
Source table X
Source schema
Oracle CDC
(Synchronous or
Async Hotlog)
Publisher schema
Change table X_CT
INSERT
UPDATE
DELETEx
SCN++

15. ©Continuent 2014©Continuent 2013
Installation
15
./tools/tpm configure fromora
--enable-heterogenous-service=true
--user=oracle
--install-directory=/opt/fromora/continuent
--members=alhpa,beta
--master=alpha
!
./tools/tpm configure fromora --hosts=alpha
--datasource-type=oracle
--user=oracle
--datasource-oracle-service=ORCL
--replication-user=DEMO_PUB
--replication-password=DEMO_PUB
--svc-table-engine=CDCASYNC
--property=replicator.global.extract.db.user=tungsten
--property=replicator.global.extract.db.password=secret
!
./tools/tpm configure fromora --hosts=beta
--datasource-type=oracle
--datasource-oracle-service=ORCL
--replication-user=DEMO
--replication-password=DEMO
!
./tools/tpm install fromora
./tools/tpm start fromora

16. ©Continuent 2014
Deployment
16
Service
fromora
Tungsten Master
Replicator
Service
fromora
Tungsten Slave
Replicator
demo
schema
demo_pub
schema
Oracle CDC
(Synchronous or
Async Hotlog)
Tungsten Slave
Replicator
OracleApplierOracleCDCExtractor

17. ©Continuent 2014©Continuent 2013
Heterogeneous
Replication

18. ©Continuent 2014
Use Case: Web Content Publishing
Real-Time Publication
Backend Office Web-Based Catalog

19. ©Continuent 2014©Continuent 2013
Steps to Heterogeneous Replication
1. Set up replication
2. Prepare (translate) schema for
the slave DBMS
3. Provision initial data
4. Continue real-time replication
ddlscan
Tungsten Replicator
Parallel ApplyParallel Extract &
tpmsetupCDC &

20. ©Continuent 2014©Continuent 2013
1. Set Up Replication

21. ©Continuent 2014
Preparing CDC
21
Tables
Source schema
setupCDC
Oracle CDC
(Synchronous or
Async Hotlog)
Publisher schema
Change tables
Capture started at
<setupCDC_SCN>
SCN++

22. ©Continuent 2014©Continuent 2013 22
./tools/tpm configure fromora
--enable-heterogenous-service=true
--user=oracle
--install-directory=/opt/fromora/continuent
--members=alhpa,beta
--master=alpha
!
./tools/tpm configure fromora --hosts=alpha
--datasource-type=oracle
--user=oracle
--datasource-oracle-service=ORCL
--replication-user=DEMO_PUB
--replication-password=DEMO_PUB
--svc-table-engine=CDCASYNC
--property=replicator.global.extract.db.user=tungsten
--property=replicator.global.extract.db.password=secret
--property=replicator.extractor.parallel-extractor.
ChunkDefinitionFile=/opt/fromora/chunks.csv
Oracle Master Part

23. ©Continuent 2014©Continuent 2013
MySQL Slave Part
23
!
!
./tools/tpm configure fromora --hosts=beta
--user=tungsten
--replication-user=tungsten
--replication-password=secret
--svc-applier-filters=CDC,casetransform,rename
--property=replicator.filter.CDC.from=DEMO_PUB.HEARTBEAT
--property=replicator.filter.CDC.to=tungsten_fromora.heartbeat
--property=replicator.filter.casetransform.to_upper_case=false
--property=replicator.filter.rename.definitionsFile=
/opt/frommysql/rename.csv
!
./tools/tpm install fromora
#./tools/tpm start fromora

24. ©Continuent 2014
Deployment
24
Service
fromoracle
Tungsten Master
Replicator
OracleCDCExtractor
No Special Filters
Service
fromoracle
Tungsten Slave
Replicator
Special Filters
•Map names to lower case
•Heartbeat table renaming
demo
schema
demo_pub
schema
Oracle CDC
(Synchronous or
Async Hotlog)
MySQLApplierOracleCDCExtractor

25. ©Continuent 2014©Continuent 2013
2.Translating schema
for the slave

26. ©Continuent 2014©Continuent 2013
Translating Schema
26
• Beginning - how to convert tables?
empty
Tables
•Data types?
•Column lengths?
•Naming conventions?
•Reserved words?
ddlscan

27. ©Continuent 2014©Continuent 2013
ddlscan
27
• Part of Tungsten Replicator, GPL v2
• Translates schema with replication in mind
• Provides errors and warnings
• Can rename schema/tables/columns

28. ©Continuent 2014©Continuent 2013
Usage (Oracle to MySQL Example)
28
$ source continuent/share/env.sh
!
$ ddlscan
-db DEMO
-template ddl-oracle-mysql.vm

29. ©Continuent 2014©Continuent 2013
Translating Schema
29
• ddlscan looks into source schema
Tables
empty

30. ©Continuent 2014©Continuent 2013
Translating Schema
30
• ddlscan translates and renders DDL commands
mysql-ddl.sql
Tables
empty

31. ©Continuent 2014©Continuent 2013
Result of ddlscan
31
DROP TABLE IF EXISTS demo.test;
CREATE TABLE demo.test
(
id1 INT /* NUMBER(10, ?) */ NOT NULL,
id2 INT /* NUMBER(10, ?) */ NOT NULL,
val TINYINT /* NUMBER(3, ?) */,
owner VARCHAR(30) /* VARCHAR2(30) */ NOT NULL,
created DATETIME /* DATE */ NOT NULL,
PRIMARY KEY (id1, id2) /* WARN: no PK found, using
suitable unique index instead: UQ_UK1 */
) ENGINE=InnoDB;
!
CREATE TABLE talks
...

32. ©Continuent 2014©Continuent 2013
Translating Schema
32
• You run resulting SQL file on Oracle
mysql-ddl.sql
Tables
empty

33. ©Continuent 2014©Continuent 2013
Translating Schema
33
• Tables are ready!
Translated tables
(no rows)
Tables

34. ©Continuent 2014©Continuent 2013
Connecting the Dots
34
• Preparation completed
Translated tables
(no rows)
Tables Change
tables
…

35. ©Continuent 2014©Continuent 2013
3. Provisioning &
Replication

36. ©Continuent 2014©Continuent 2013
Parallel Extractor
36
• replicator start # slave
• replicator start offline # master
• trepctl online -provision <setupCDC_SCN>
Tables
Service
fromoracle
Tungsten Master
Replicator
THL
ParallelExtractor
Thread 1
Thread 2
Thread 3
SELECT … AS OF
<setupCDC_SCN>

37. ©Continuent 2014
Accelerate the Slave Provisioning
37
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Stage
Pipeline
Remote
Master
Transaction
History Log
Parallel
Queue
Slave
DBMS
Extract Filter Apply
Extract Filter Apply
Extract Filter Apply
(Assign
Shard ID)

38. ©Continuent 2014©Continuent 2013
Real-time Replication
38
• Automatically switches to real-time extraction
Service
fromoracle
Tungsten Master
Replicator
THL
Oracle CDC
(Synchronous or
Async Hotlog)
Change tables
>=$setupCDC_SCN
OracleCDCExtractor
Tables

39. ©Continuent 2014
Replication to
Vertica and
Hadoop

40. ©Continuent 2014
The Data Warehouse Impedance
Mismatch
40
Replication
CSV
Files
CSV
Files
Buffered
Transactions
Dump/load
Single
Transactions
Batches

41. ©Continuent 2014
Column Store--Real-Time Batches
MySQL/Oracle Tungsten Master
Replicator
Service ora2vr
Special Filters
* pkey - Fill in pkey info
* colnames - Fill in names
* replicate - Ignore tables
Tungsten Slave
Replicator
Service ora2vr
CSV
Files
CSV
Files
CSV
Files
CSV
Files
CSV
Files
Large transaction
batches to leverage
load parallelization

42. ©Continuent 2014
Batch Loading--The Gory Details
Replicator
Service ora2vr
Transactions
from master
CSV
Files
CSV
Files
CSV
Files
Staging
Tables
Staging
Tables
Staging
Tables
Base
Tables
Base
Tables
Base
Tables
Merge
Script
(or)
COPY
directly to
base tables
COPY to
stage tables SELECT to
base tables

43. ©Continuent 2014
Basic Hadoop Loading
MySQL/Oracle Tungsten Master
Replicator
hadoop
Master-Side Filtering
* pkey - Fill in pkey info
* colnames - Fill in names
* replicate - Subset tables to
be replicated
Tungsten Slave
Replicator
hadoop
CSV
Files
CSV
Files
CSV
Files
CSV
Files
CSV
Files
Hadoop
Cluster
Extract from
source DBMS
Load raw CSV to HDFS
(e.g., via LOAD DATA to
Hive)
Access via Hive

44. ©Continuent 2012
Provisioning plus Replication
44
MySQL/Oracle
Tungsten 3.0 Master
hadoop
Tungsten 3.0 Slave
hadoop
CSV
Files
CSV
Files
CSV
Files
CSV
FilesCSV
Apache Sqoop/ETL
Fast data filtering
Buffered
CSV
Programmable
load scripts
Parallel apply
Parallel table
dumps
Low impact
extraction

45. ©Continuent 2014
How the Materialization Works
45
Op
Seqn
o
ID Msg
I 1 1 Hello World!
I 2 2 Meet MC
D 3 1
I 3 1
Goodbye
World
Op Se
qn
ID Msg
I 2 2 Meet
MCI 3 1 Goodby
e World

46. ©Continuent 2014
Tungsten Replicator 3.0 & Hadoop
46
• Extract from MySQL or Oracle
• Hadoop Support
• Provision using Sqoop or parallel extraction
• Schema generation for Hive
• Tools for generating materialized views
• Parallel CSV file loading
• Partition loaded data by commit time
• Schema Change Notification

47. ©Continuent 2014
Continuent Hadoop Tools Roadmap:
Point in Time Table Generation
47
1 2 3 4 5 6 7 8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
1
7
1
8
1
9
2
0
2
1
2
2
2
3
2
4
2
5
2
6
2
7
2
8
2
9
3
0
3
1
3
2
3
3
3
4
3
5
3
6
3
7
3
8
3
9
4
0
4
1
4
2
4
3
4
4
4
5
Monday Wednesday Friday

48. ©Continuent 2014
Continuent Hadoop Tools Roadmap:
Time Series Generation
48
Op Seq
no
ID Date Msg
I 1 1 1/6/1
4
Hello World!
I 2 2 2/6/1
4
Meet MC
I 3 1 2/6/1
4
Goodbye
WorldI 4 1 3/6/1
4
Hello
TuesdayI 4 2 3/6/1
4
Ruby
WednesdayI 5 1 4/6/1
4
Final Count
ID Date Msg
1 1/6/14 Hello World!
1 2/6/14 Goodbye World
1 3/6/14 Hello Tuesday
1 4/6/14 Final Count

49. ©Continuent 2014
Getting Started!
49
• Tungsten Replicator builds are available on
code.google.com
http://code.google.com/p/tungsten-replicator/
• Replicator documentation is available on Continuent
website
http://docs.continuent.com/tungsten-replicator-3.0/
deployment-hadoop.html
• Tungsten Hadoop tools are available on GitHub
https://github.com/continuent/continuent-tools-hadoop
Contact Continuent for support

50. ©Continuent 2014
www.continuent.com
Follow us on Twitter @continuent
!
Tungsten Replicator:
http://code.google.com/p/tungsten-replicator
Our Blogs:
http://scale-out-blog.blogspot.com
http://datacharmer.org/blog
http://www.continuent.com/news/blogs
http://flyingclusters.blogspot.com/
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com