Learn how to manage high performance data using VMware vFabric SQLFire

1. Managing High Performance Data with vFabric SQLFire

4. Confidential Common themes in next-gen DB architectures NoSQL, Data Grids, Data Fabrics, NewSQL “ Shared nothing” commodity clusters focus shifts to memory, distributing data and clustering Scale by partitioning the data and move behavior to data nodes HA within cluster and across data centers Add capacity to scale dynamically

6. What is our take with SQLFire?

8. Show me a picture

12. Flexible Deployment Topologies Confidential Java Application cluster can host an embedded clustered database by just changing the URL jdbc:sqlfire:;mcast-port=33666;host-data=true

13. Flexible Deployment Topologies Confidential

14. Partitioning & Replication

15. Explore features through example Assume, thousands of flight rows, millions of flightavailability records

16. Creating Tables Table CREATE TABLE AIRLINES ( AIRLINE CHAR(2) NOT NULL PRIMARY KEY, AIRLINE_FULL VARCHAR(24), BASIC_RATE DOUBLE PRECISION, DISTANCE_DISCOUNT DOUBLE PRECISION,…. ); SQLF SQLF SQLF

17. Replicated Tables CREATE TABLE AIRLINES ( AIRLINE CHAR(2) NOT NULL PRIMARY KEY, AIRLINE_FULL VARCHAR(24), BASIC_RATE DOUBLE PRECISION, DISTANCE_DISCOUNT DOUBLE PRECISION,…. ) REPLICATE ; Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF

18. Partitioned Tables CREATE TABLE FLIGHTS ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEST_AIRPORT CHAR(3) DEPART_TIME TIME, FLIGHT_MILES INTEGER NOT NULL ) PARTITION BY COLUMN( FLIGHT_ID ); Table Partitioned Table Partitioned Table Partitioned Table Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF

19. Partition Redundancy CREATE TABLE FLIGHTS ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , ORIG_AIRPORT CHAR(3), DEST_AIRPORT CHAR(3) DEPART_TIME TIME, FLIGHT_MILES INTEGER NOT NULL) PARTITION BY COLUMN (FLIGHT_ID) REDUNDANCY 1 ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF

20. Partition Colocation CREATE TABLE FLIGHTAVAILABILITY ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER DEFAULT 0, …) PARTITION BY COLUMN (FLIGHT_ID) COLOCATE WITH (FLIGHTS) ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF Colocated Partition Colocated Partition Colocated Partition Redundant Partition Redundant Partition Redundant Partition

21. Persistent Tables CREATE TABLE FLIGHTAVAILABILITY ( FLIGHT_ID CHAR(6) NOT NULL , SEGMENT_NUMBER INTEGER NOT NULL , FLIGHT_DATE DATE NOT NULL , ECONOMY_SEATS_TAKEN INTEGER DEFAULT 0, …) PARTITION BY COLUMN (FLIGHT_ID) COLOCATE WITH (FLIGHTS) PERSISTENT persistentStore ASYNCHRONOUS ; Table Partitioned Table Redundant Partition Partitioned Table Redundant Partition Partitioned Table Redundant Partition Replicated Table Replicated Table Replicated Table SQLF SQLF SQLF Colocated Partition Colocated Partition Colocated Partition Redundant Partition Redundant Partition Redundant Partition sqlf backup /export/fileServerDirectory/sqlfireBackupLocation Data dictionary is always persisted in each server

22. Demo default partitioned tables, colocation, persistent tables

23. Scaling data with Partitioned tables

24. Hash partitioning for linear scaling Key Hashing provides single hop access to its partition But, what if the access is not based on the key … say, joins are involved

28. Partition aware DB design Entity Groups Table FlightAvailability partitioned by FlightID colocated with Flights FlightID is the entity group Key

30. Scaling Application logic with Parallel “Data Aware procedures”

31. Procedures Java Stored Procedures may be created according to the SQL Standard SQLFabric also supports the JDBC type Types.JAVA_OBJECT. A parameter of type JAVA_OBJECT supports an arbitrary Serializable Java object. In this case, the procedure will be executed on the server to which a client is connected (or locally for Peer Clients) CREATE PROCEDURE getOverBookedFlights (IN argument OBJECT, OUT result OBJECT) LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA DYNAMIC RESULT SETS 1 EXTERNAL NAME com.acme.OverBookedFLights;

32. Data Aware Procedures Parallelize procedure and prune to nodes with required data Extend the procedure call with the following syntax: Hint the data the procedure depends on CALL getOverBookedFlights( <bind arguments> ON TABLE FLIGHTAVAILABILITY WHERE FLIGHTID = <SomeFLIGHTID> ; If table is partitioned by columns in the where clause the procedure execution is pruned to nodes with the data (node with <someFLIGHTID> in this case) CALL [PROCEDURE] procedure_name ( [ expression [, expression ]* ] ) [ WITH RESULT PROCESSOR processor_name ] [ { ON TABLE table_name [ WHERE whereClause ] } | { ON {ALL | SERVER GROUPS (server_group_name [, server_group_name ]*) }} ] Fabric Server 2 Fabric Server 1 Client

33. Parallelize procedure then aggregate (reduce) Fabric Server 2 Fabric Server 1 Client Fabric Server 3 CALL SQLF.CreateResultProcessor( processor_name, processor_class_name); register a Java Result Processor (optional in some cases) : CALL [PROCEDURE] procedure_name ( [ expression [, expression ]* ] ) [ WITH RESULT PROCESSOR processor_name ] [ { ON TABLE table_name [ WHERE whereClause ] } | { ON {ALL | SERVER GROUPS (server_group_name [, server_group_name ]*) }} ]

34. Consistency model

37. Scaling disk access with shared nothing disk files and a “journaling” store design

39. Performance benchmark

44. Built using GemFire object data fabric + Derby