Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
MariaDB Platform for
hybrid transactional/analytical
workloads
Shane K Johnson
Senior Director of Product Marketing
MariaD...
Agenda
1. Hybrid workloads and use cases
2. MariaDB Platform X3
3. Scalability
4. Real-world use cases and hybrid cloud
5....
Hybrid workloads
Database workloads
Transactional
Current data
Range queries
Known queries
Historical data
Aggregate queries
Unkown queries...
Database workloads
Transactional
Current data
Range queries
Known queries
Row-based storage
Indexes
Clustered/Replicated
H...
MariaDB TX
MariaDB Server
MariaDB MaxScale
InnoDB/MyRocks
MariaDB AX
MariaDB Server
MariaDB MaxScale
ColumnStore
App/dev
M...
Database (OLTP) Data warehouse (OLAP)
Application development BI/reporting + data science
Data warehouse
(OLAP)
Database
(OLTP)
Hybrid workloads: the problem
Transactions
App/dev
Analytics
BI/reporting + data sci...
Data warehouse
(OLAP)
Database
(Hybrid)
Hybrid workloads: the solution
Transactions Analytics
App/dev
Analytics
BI/reporti...
MariaDB Platform X3
MariaDB TX 3.0
MariaDB Server 10.3
MariaDB MaxScale 2.2
InnoDB/MyRocks
MariaDB AX 2.0
MariaDB Server 10.2
MariaDB MaxScale...
The database proxy inspects queries and routes them to transactional
and/or analytical database instances.
The change-data...
Applications
Containers
MariaDB Platform X3
MariaDB MaxScale 2.3
CDC
MariaDB Server 10.3
InnoDB/MyRocks
MariaDB Server 10....
Scalability
Hybrid workloads: why scalability is needed
Applications have transactional
and analytical queries
1. Constrained by limit...
Hybrid workloads: options
● Oracle
○ In-Memory Column Store
● Microsoft SQL Server
○ ColumnStore Indexes (Clustered or Non...
Hybrid workloads: full comparison
* IBM Db2 Shadow Tables, ** Oracle IM column store can read from on-disk row storage if ...
Application
Connection 1
MariaDB
Server
Row storage
MariaDB Server
Columnar
storage
MariaDB
MaxScale
Transactional Analyti...
MariaDB Server
(Spider)
Application
Connection 1
MariaDB
Server 1
Row storage
MariaDB
Server 2
Row storage
MariaDB
Server ...
MariaDB Server
(Spider)
Application
Connection 1
MariaDB Server
(ColumnStore)
Node 1
Columnar
storage
Node 2
Columnar
stor...
MariaDB Server
(Spider)
Application
Connection 1
MariaDB
Server 1
Row storage
MariaDB
Server 2
Row storage
MariaDB
Server ...
Real-world use cases
Retail (market research)
Database
(Hybrid)
Transactions Analytics
Transactional
Capture daily product prices
Update produc...
Telecommunications (IP telephony – SaaS)
Database
(Hybrid)
Transactions Analytics
Transactional
Capture call detail record...
Finance (trading)
Database
(Hybrid)
Transactions Analytics
Transactional
Capture trades
Analytical
Analyze trading data
Ar...
Hybrid workloads: perfect for hybrid cloud
● MariaDB Platform separates and isolates different workloads
○ Run different w...
MariaDB Platform
MariaDB Server
MariaDB Server
MariaDB Server
MariaDB Server
MariaDB Server
InnoDB/MyRocks
MariaDB Server
...
Implementation:
change-data-capture
MariaDB Server
(primary)
InnoDB/MyRocks
MariaDB MaxScale
Binlog
Binlog router
MariaDB Server
(secondary)
InnoDB/MyRocks
Ma...
App/Service/Script
(backend)
Bulk data adapter
MariaDB Server
(transactions)
InnoDB/MyRocks
MariaDB MaxScale
Binlog
Binlog...
ColumnStore
(storage)
CDC data adapter
MariaDB Server
(analytics)
ColumnStore
(interface)
MariaDB MaxScale
Binlog routerMy...
MariaDB MaxScale
[replication-router]
type = service
router = binlogrouter
user = repl
password = pass
server_id = 5
maste...
Implementation:
workload-based query routing
Database
Database
MariaDB MaxScale
Server
(server -> IP)
Server
(server -> IP)
Listener
(port -> service)
Service
(request...
Server 1
(transactional)
[sw-db1]
type = server
address = <MariaDB-Server-1-IP>
port = 3306
protocol = MariaDBBackend
Serv...
Server 4
(analytical)
[sw-mcs-um2]
type = server
address = <MariaDB-UM-1-IP>
port = 3306
protocol = MariaDBBackend
Server ...
Listener 2
(analytical)
Listener 1
(transactional)
[MDB-Listener]
type = listener
service = MDB-Service
protocol = MariaDB...
Service 3
(HTAP)
[HTAP-Service]
type = service
router = schemarouter
ignore_databases_regex = .*
servers = MDB-Service-as-...
Database instance
(transactional)
Database instance
(transactional)
Database instance
(analytical)
Database instance
(tran...
Database instance
(transactional)
Database instance
(transactional)
Database instance
(analytical)
Database instance
(tran...
Database instance
(analytical)
Database instance
(transactional)
Database instance
(transactional)
Database instance
(tran...
Database instance
(transactional)
Database instance
(transactional)
Database instance
(analytical)
Database instance
(tran...
Getting started with
Docker
MariaDB Platform X3 in a container
● MariaDB Platform “in-a-box”
● Fully-configured and ready to use out of the box
● Inte...
MariaDB Platform X3 in a container
THANK YOU!
Prochain SlideShare
Chargement dans…5
×

MariaDB Platform for hybrid transactional/analytical workloads

192 vues

Publié le

OpenWorks 2019 Session
In order to provide data-driven customers with more historical data and real-time analytics, MariaDB Platform can be configured for hybrid transactional/analytical workloads by leveraging row storage for current data transactions and columnar storage for historical data and analytics. In this session Shane Johnson, Senior Director of Product Marketing at MariaDB, shows how change-data-capture and query routing, both available out of the box, can be used to bring scalable analytics to customer-facing applications without changing their code – and without depending on a separate data warehouse.

Publié dans : Logiciels
  • Soyez le premier à commenter

MariaDB Platform for hybrid transactional/analytical workloads

  1. 1. MariaDB Platform for hybrid transactional/analytical workloads Shane K Johnson Senior Director of Product Marketing MariaDB Corporation
  2. 2. Agenda 1. Hybrid workloads and use cases 2. MariaDB Platform X3 3. Scalability 4. Real-world use cases and hybrid cloud 5. Implementation: change-data-capture 6. Implementation: workload-based query routing 7. Getting started with Docker
  3. 3. Hybrid workloads
  4. 4. Database workloads Transactional Current data Range queries Known queries Historical data Aggregate queries Unkown queries Analytical
  5. 5. Database workloads Transactional Current data Range queries Known queries Row-based storage Indexes Clustered/Replicated Historical data Aggregate queries Unkown queries Columnar storage No indexes Distributed Analytical
  6. 6. MariaDB TX MariaDB Server MariaDB MaxScale InnoDB/MyRocks MariaDB AX MariaDB Server MariaDB MaxScale ColumnStore App/dev MariaDB TX BI/data science MariaDB AX
  7. 7. Database (OLTP) Data warehouse (OLAP) Application development BI/reporting + data science
  8. 8. Data warehouse (OLAP) Database (OLTP) Hybrid workloads: the problem Transactions App/dev Analytics BI/reporting + data science
  9. 9. Data warehouse (OLAP) Database (Hybrid) Hybrid workloads: the solution Transactions Analytics App/dev Analytics BI/reporting + data science
  10. 10. MariaDB Platform X3
  11. 11. MariaDB TX 3.0 MariaDB Server 10.3 MariaDB MaxScale 2.2 InnoDB/MyRocks MariaDB AX 2.0 MariaDB Server 10.2 MariaDB MaxScale 2.2 ColumnStore 1.2 MariaDB Platform X3 MariaDB MaxScale 2.3 MariaDB Server 10.3 InnoDB/MyRocks MariaDB Server 10.3 ColumnStore 1.3
  12. 12. The database proxy inspects queries and routes them to transactional and/or analytical database instances. The change-data-capture stream replicates all writes from transactional databases to analytical databases within microbatches. MariaDB Platform X3 MariaDB MaxScale 2.3 CDC MariaDB Server 10.3 InnoDB/MyRocks MariaDB Server 10.3 ColumnStore 1.3 Transactional Analytical
  13. 13. Applications Containers MariaDB Platform X3 MariaDB MaxScale 2.3 CDC MariaDB Server 10.3 InnoDB/MyRocks MariaDB Server 10.3 ColumnStore 1.3 Transactional Analytical Kubernetes (Helm) Docker (Compose) C JDBC ODBC Node.js Ingest streaming data Kafka connector Administration SQL Diagnostic Manager SQLyog MariaDB Backup MariaDB Flashback Import bulk data Spark connector C/Java/Python API
  14. 14. Scalability
  15. 15. Hybrid workloads: why scalability is needed Applications have transactional and analytical queries 1. Constrained by limited, lightweight analytics 2. Need full analytics to create competitive features Outgrowing OLTP Applications with lots of customers, lots of transactions 1. Limited to current or recent transaction data (months) 2. Need access to all historical data (years) Using historical data SaaS customers are becoming data-driven organizations 1. They don’t have access to their own data 2. They need to analyze it in unknown/unexpected ways Exposing analytics
  16. 16. Hybrid workloads: options ● Oracle ○ In-Memory Column Store ● Microsoft SQL Server ○ ColumnStore Indexes (Clustered or Nonclustered) ● IBM Db2 ○ Shadow Tables ● MySQL Enterprise ○ None ● EnterpriseDB Postgres Platform ○ None
  17. 17. Hybrid workloads: full comparison * IBM Db2 Shadow Tables, ** Oracle IM column store can read from on-disk row storage if needed Oracle Microsoft IBM * MariaDB MySQL Postgres Row storage (OLTP) Yes Yes Yes Yes Yes Yes Sharded Yes No Yes Yes No No Columnar storage (OLAP) Yes Yes Yes Yes No No Disk-based No ** Yes Yes Yes - - Distributed Yes (RAC) No No Yes - -
  18. 18. Application Connection 1 MariaDB Server Row storage MariaDB Server Columnar storage MariaDB MaxScale Transactional Analytical
  19. 19. MariaDB Server (Spider) Application Connection 1 MariaDB Server 1 Row storage MariaDB Server 2 Row storage MariaDB Server n Row storage MariaDB Server (ColumnStore) Sharding MariaDB MaxScale Transactional Analytical MariaDB Server Columnar storage
  20. 20. MariaDB Server (Spider) Application Connection 1 MariaDB Server (ColumnStore) Node 1 Columnar storage Node 2 Columnar storage Node n Columnar storage Distributed storage MariaDB MaxScale Transactional Analytical MariaDB Server Row storage
  21. 21. MariaDB Server (Spider) Application Connection 1 MariaDB Server 1 Row storage MariaDB Server 2 Row storage MariaDB Server n Row storage MariaDB Server (ColumnStore) CS node 1 Columnar storage CS node 2 Columnar storage CS node n Columnar storage Sharding Distributed storage MariaDB MaxScale Transactional Analytical
  22. 22. Real-world use cases
  23. 23. Retail (market research) Database (Hybrid) Transactions Analytics Transactional Capture daily product prices Update product information Show current prices Store current pricing data Analytical Show prices over time Self-service analytics Store historical pricing data
  24. 24. Telecommunications (IP telephony – SaaS) Database (Hybrid) Transactions Analytics Transactional Capture call detail records Charge by call/message Generate bills Analytical Monitor usage Identify peak periods Estimate costs Self-service analytics
  25. 25. Finance (trading) Database (Hybrid) Transactions Analytics Transactional Capture trades Analytical Analyze trading data Archiving for regulator compliance (historical data) Self-service analytics
  26. 26. Hybrid workloads: perfect for hybrid cloud ● MariaDB Platform separates and isolates different workloads ○ Run different workloads on different infrastructure ○ Place different workloads closer to different users ○ Scale different workloads on different hardware
  27. 27. MariaDB Platform MariaDB Server MariaDB Server MariaDB Server MariaDB Server MariaDB Server InnoDB/MyRocks MariaDB Server ColumnStore On premises for transactions Keep transactions close to the business Public cloud for analytics Incremental path to the cloud, places analytics close to customers
  28. 28. Implementation: change-data-capture
  29. 29. MariaDB Server (primary) InnoDB/MyRocks MariaDB MaxScale Binlog Binlog router MariaDB Server (secondary) InnoDB/MyRocks MariaDB Server (secondary) InnoDB/MyRocks Binlog server
  30. 30. App/Service/Script (backend) Bulk data adapter MariaDB Server (transactions) InnoDB/MyRocks MariaDB MaxScale Binlog Binlog router Avro router CDC protocol CDC client ColumnStore (storage) Write engine MariaDB Server (analytics) ColumnStore (interface) CDC server
  31. 31. ColumnStore (storage) CDC data adapter MariaDB Server (analytics) ColumnStore (interface) MariaDB MaxScale Binlog routerMySQL protocol Avro router CDC protocol MariaDB Server (transactions) InnoDB/MyRocks
  32. 32. MariaDB MaxScale [replication-router] type = service router = binlogrouter user = repl password = pass server_id = 5 master_id = 1 Binlogdir = /var/lib/maxscale Mariadb10-compatibility = 1 filestem = mariadb-bin [avro-router] type = service router = avrorouter source = replication-router avrodir = /var/lib/maxscale [replication-listener] type = listener service = replication-router protocol = MySQLClient port = 6603 Binlog router (service) MySQL protocol (listener) Avro router (service) CDC protocol (listener) [avro-listener] type = listener service = avro-router protocol = cdc port = 4001 mxs_adapter -c /etc/ColumnStore.xml -u {user} -p {password} -h {host} -P {port} -r 2 -d -n -z {schema} {table}
  33. 33. Implementation: workload-based query routing
  34. 34. Database Database MariaDB MaxScale Server (server -> IP) Server (server -> IP) Listener (port -> service) Service (request -> server) Server (server -> IP) Filter(s)Protocol Monitor Applications Database
  35. 35. Server 1 (transactional) [sw-db1] type = server address = <MariaDB-Server-1-IP> port = 3306 protocol = MariaDBBackend Server 2 (transactional) [sw-db2] type = server address = <MariaDB-Server-2-IP> port = 3306 protocol = MariaDBBackend Server 3 (transactional) [sw-db3] type = server address = <MariaDB-Server-3-IP> port = 3306 protocol = MariaDBBackend Service 1 (transactional) [MDB-Service] type = service router = readwritesplit servers = sw-db1,sw-db2, sw-db3 user = maxuser password = maxpwd Listener 1 (transactional) [MDB-Listener] type = listener service = MDB-Service protocol = MariaDBClient port = 4009
  36. 36. Server 4 (analytical) [sw-mcs-um2] type = server address = <MariaDB-UM-1-IP> port = 3306 protocol = MariaDBBackend Server 5 (analytical) [sw-mcs-um2] type = server address = <MariaDB-UM-2-IP> port = 3306 protocol = MariaDBBackend Service 2 (analytical) [CS-Service] type = service router = readconnroute router_options = running servers = sw-mcs-um1, sw-mcs-um2 user = maxuser password = maxpwd Listener 2 (analytical) [CS-Listener] type = listener service = CS-Service protocol = MariaDBClient port = 4010
  37. 37. Listener 2 (analytical) Listener 1 (transactional) [MDB-Listener] type = listener service = MDB-Service protocol = MariaDBClient port = 4009 [CS-Listener] type = listener service = CS-Service protocol = MariaDBClient port = 4010 Server 6 (transactional) Server 7 (analytical) [CS-Service-as-server] type = server address = 127.0.0.1 port = 4010 protocol = MariaDBBackend [MDB-Service-as-server] type = server address = 127.0.0.1 port = 4009 protocol = MariaDBBackend Service 3 (HTAP) Listener 3 (HTAP) [HTAP-Service] type = service router = schemarouter ignore_databases_regex = .* servers = MDB-Service-as-server,CS-Service-as-server Preferred_server = MDB-Service-as-server user = maxuser password = maxpwd filters = target-selector [HTAP-Listener] type = listener service = HTAP-Service protocol = MariaDBClient port = 4011
  38. 38. Service 3 (HTAP) [HTAP-Service] type = service router = schemarouter ignore_databases_regex = .* servers = MDB-Service-as-server,CS-Service-as-server Preferred_server = MDB-Service-as-server user = maxuser password = maxpwd filters = target-selector [target-selector] type = filter module = namedserverfilter match01 = (?i)SELECT.*loan target01 = CS-Service-as-server match02 = .* target02 = MDB-Service-as-server Filter 1 Regex filter (by query syntax) Schema router (by table/view) Named server filter (by query syntax) Hint filter Options
  39. 39. Database instance (transactional) Database instance (transactional) Database instance (analytical) Database instance (transactional) Database instance (analytical) select avg(total) from tbl_purchases where customer=1 select avg(total) from tbl_purchases --maxscale route to analytics MariaDB MaxScale Query routing: hint filter
  40. 40. Database instance (transactional) Database instance (transactional) Database instance (analytical) Database instance (transactional) Database instance (analytical) select avg(total) from tbl_purchases where customer=1 select avg(total) from tbl_purchases --maxscale route to analytics MariaDB MaxScale Query routing: hint filter
  41. 41. Database instance (analytical) Database instance (transactional) Database instance (transactional) Database instance (transactional) Database instance (analytical) select avg(total) from tbl_purchases where customer=1 select avg(total) from tbl_purchases_history MariaDB MaxScale Query routing: schema router tbl_purchases view_purchase_history tbl_purchases
  42. 42. Database instance (transactional) Database instance (transactional) Database instance (analytical) Database instance (transactional) Database instance (analytical) select avg(total) from tbl_purchases where customer=1 select avg(total) from tbl_purchases_history MariaDB MaxScale Query routing: combo 1. Named server filter (attach hint) select avg(total) from tbl_purchases 2. Regex filter
  43. 43. Getting started with Docker
  44. 44. MariaDB Platform X3 in a container ● MariaDB Platform “in-a-box” ● Fully-configured and ready to use out of the box ● Intended as a quickstart for development ● Launched with a Docker command ● Runs on a laptop https://github.com/mariadb-corporation/mariadb-platform-docker/tree/master/single-container
  45. 45. MariaDB Platform X3 in a container
  46. 46. THANK YOU!

×