Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
hosted by
AntsDB
MySQL Compatibility for HBase
August 17,2018
hosted by
Content
01
02
04
03
About Us
Scenario
Architecture
Complementary to
HBase
hosted by
About Us01
hosted by
Water Guo
 Founder of AntsDB
 Love Hadoop and HBase but a heavy RDB user in the past
 Love programming for ex...
hosted by
AntsDB
 Open source project at https://github.com/waterguo/antsdb
 Problem to resolve: NoSQL is taking over th...
hosted by
Scenario02
hosted by
Virtual Telephone Number
 Telecom industry
 Rigid latency requirement: 20ms max
 Data is growing fast
 Built...
hosted by
Solution Architecture
User
Application
Batch Analytics
AntsDB HBase
Spark SQL
hosted by
Lessons Learned
 Linear scalability
 Hadoop ecosystem
 Good random access
performance
 Latency spike
 Trans...
hosted by
Why don’t cache hot data
 Operational data = hot data
 Historical data = majority of big data
 Daily hot data...
hosted by
Silver bullet - caching
✧ Large amount local caching using SSD – 500 GB
✧ Local data fetch -> sub-millisecond la...
hosted by
AntsDB Architecture03
hosted by
Layers
1
2
3
4
SQL Parser/Executor
Caching
MySQL Network
Transaction
MVCC, row locking
HBase Connectivity
MySQL ...
hosted by
Architecture
Subtitle Text
AntsDB
Application MySQL Driver
HBase
Cluster
HBase Connector
Transaction
Manager
SQL...
hosted by
Fight the GC
- Java Unsafe - skip list, cache, WAL, row locking, cursor
- Atomic operation for row locks
- Small...
hosted by
Performance - YCSB write
hosted by
Performance - TPCC
hosted by
Complementary to HBase04
hosted by
MySQL Compatibility
 MySQL protocol – JDBC, ODBC, PHP, Python, Native
 Core SQL syntax
 MySQL data types
 My...
hosted by
MySQL Behaviour Simulation
- Locking
- Commit and rollback
- MVCC
- Quirks such as 0000-00-00 00:00:00 datetime,...
hosted by
Compatible applications
 MySQL command line utilities
 MySQL Workebench
 DBeaver
 SquirrelSQL
 MediaWiki
 ...
hosted by
High availability
Cluster
Low latency CP system
Next step
hosted by
Thanks
Prochain SlideShare
Chargement dans…5
×

HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Database Virtualization

315 vues

Publié le

Water Guo

Publié dans : Internet
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Database Virtualization

  1. 1. hosted by AntsDB MySQL Compatibility for HBase August 17,2018
  2. 2. hosted by Content 01 02 04 03 About Us Scenario Architecture Complementary to HBase
  3. 3. hosted by About Us01
  4. 4. hosted by Water Guo  Founder of AntsDB  Love Hadoop and HBase but a heavy RDB user in the past  Love programming for extreme performance  Previously Software Architecture at Informatica, DataMirror, founder of BDI Systems.
  5. 5. hosted by AntsDB  Open source project at https://github.com/waterguo/antsdb  Problem to resolve: NoSQL is taking over the world but majority of the data applications are still using RDB  Database virtualization software  Attempt to bring MySQL compatibility to HBase
  6. 6. hosted by Scenario02
  7. 7. hosted by Virtual Telephone Number  Telecom industry  Rigid latency requirement: 20ms max  Data is growing fast  Built on top of relational database - MySQL
  8. 8. hosted by Solution Architecture User Application Batch Analytics AntsDB HBase Spark SQL
  9. 9. hosted by Lessons Learned  Linear scalability  Hadoop ecosystem  Good random access performance  Latency spike  Transaction  Join performance What we like HBase What we were struggling
  10. 10. hosted by Why don’t cache hot data  Operational data = hot data  Historical data = majority of big data  Daily hot data = 1-5%  Weekly hot data = 5-10%  Monthly hot data = 10-15%
  11. 11. hosted by Silver bullet - caching ✧ Large amount local caching using SSD – 500 GB ✧ Local data fetch -> sub-millisecond latency ✧ Distributed transaction -> local transaction ✧ Distributed joins -> local joins
  12. 12. hosted by AntsDB Architecture03
  13. 13. hosted by Layers 1 2 3 4 SQL Parser/Executor Caching MySQL Network Transaction MVCC, row locking HBase Connectivity MySQL syntax,functions, data types JDBC, ODBC, PHP, Python, Native Data in and out from HBase
  14. 14. hosted by Architecture Subtitle Text AntsDB Application MySQL Driver HBase Cluster HBase Connector Transaction Manager SQL Parser Local CacheTransactions Query Result Query Result Query Executor
  15. 15. hosted by Fight the GC - Java Unsafe - skip list, cache, WAL, row locking, cursor - Atomic operation for row locks - Small heap size - less than 4G - Escape analysis - Scan performance : 1 million rows / thread / second - Insert performance: 200K inserts / thread / second
  16. 16. hosted by Performance - YCSB write
  17. 17. hosted by Performance - TPCC
  18. 18. hosted by Complementary to HBase04
  19. 19. hosted by MySQL Compatibility  MySQL protocol – JDBC, ODBC, PHP, Python, Native  Core SQL syntax  MySQL data types  MySQL functions – 45  Low latency transactions  Low latency joins  Indexes - Unique, non-unique, full-text
  20. 20. hosted by MySQL Behaviour Simulation - Locking - Commit and rollback - MVCC - Quirks such as 0000-00-00 00:00:00 datetime, zerofill integers - Millisecond latency
  21. 21. hosted by Compatible applications  MySQL command line utilities  MySQL Workebench  DBeaver  SquirrelSQL  MediaWiki  SonarQube  PHPMyAdmin  BechmarkSQL – TPC-C benchmarking  YCSB
  22. 22. hosted by High availability Cluster Low latency CP system Next step
  23. 23. hosted by Thanks

×