DataStax 6 and Beyond

DataStax 6 and Beyond
David Gilardi, Developer Advocate

What’s new in
DataStax 6?
So much AWESOME!

DSE Advanced Performance
Go faster, a lot faster - just upgrade to DSE 6.x

© DataStax, All Rights Reserved.ConfidentialConfidential © DataStax, All Rights Reserved.
> 2X Writes
~ 2X Reads JONATHAN ELLIS
This is a ‘once every
seven years’ rewrite.
Advanced Performance

Is It Really 2x?

2x latency improvements reads/writes
40k fixed throughput test, 3 nodes, RF=3, CL=QUORUM

© DataStax, All Rights Reserved.Confidential
Storage engine optimization

22% faster compaction

TPC (Thread per Core)

SEDA
Stage
Threads
Cores
Stage
Threads
Stage
Threads
Stage
Threads
TPCSEDA
Task
(read/write)

SEDA
TPCSEDA
Stage
Thread
Thread
Thread
Thread
Task
(read/write)

Cores
SEDA
TPCSEDA
Stage
Stage
Stage
Stage
Task
(read/write)

Thread Per Core
TPCSEDA
Task
(read/write)
Core
Thread
Cores
Core
Thread
Core
Thread
Core
Thread
Stage
Stage
Stage
Stage
Task
(read/write)

Thread Per Core - Token ranges
TPCSEDA
Task
(read/write)
Core
Thread
Core
Thread
0 - 10
Cores
Core
Thread
31- 40
11 -
20
Core
Thread
21 - 30
Stage
Stage
Stage
Stage
Task
(read/write)

Thread Per Core - Write
TPCSEDA
Task
(read/write)
Core
Thread
Core
Thread
write (token 1) 0 - 10
Cores
Core
Thread
31- 40
11 -
20
Core
Thread
21 - 30
Stage
Stage
Stage
Stage
Task
(read/write)

Thread Per Core - Write
TPCSEDA
Task
(read/write)
Core
Thread
Core
Thread
Core
Thread
Core
Thread
write (token 1)
write (token 32)
0 - 10
31- 40
Cores
11 -
20
21 - 30
Stage
Stage
Stage
Stage
Task
(read/write)

Thread Per Core - Read
TPCSEDA
Task
(read/write)
Core
Thread
11 -
20
read
(token 25)
Core
Thread
0 - 10
Cores
Core
Thread
21 - 30
Core
Thread
31- 40
Stage
Stage
Stage
Stage
Task
(read/write)

Thread Per Core - Linear scale
TPCSEDA
Task
(read/write)
Core
Thread
Cores
Core
Thread
Core
Thread
Core
Thread
Stage
Stage
Stage
Stage
Task
(read/write)

Thread Per Core - Linear scale
TPCSEDA
Task
(read/write)
Core
Thread
Core
Thread
Core
Thread
Core
Thread
Core
Thread
Cores
Core
Thread
Core
Thread
Core
Thread
Stage
Stage
Stage
Stage
Task
(read/write)

Anti-entropy with NodeSync
“Wait, I have to manually repair my data?” - No, not anymore

Classic repair
Single-process bulk operation =
If failure occurs begin again
The longer classic repair runs,
the more failure prone it is
Manual process

DSE
NodeSync
It’s Automatic!

NodeSync = Hands Off

Classic repair vs NodeSync
Classic repair
NodeSync
Failure
Failure

Classic repair
Creates merkle trees
Streams between nodes =
CPU and network intensive

DSE
NodeSync
No more
streaming merkle trees =
Much LESS computational
and network overhead

Classic repair vs NodeSync
NodeSyncClassic Repair
● if nodesync enabled will NOT
repair
Mutually
Exclusive

DSE Bulk Loader
Loading and unloading your data ALL of it at a time

Bulk Loader
Moves Cassandra data to/from
files in the file system
Uses both CSV or JSON
Command-line interface Data
Data

~ 4X Load
> 4X Unload
DataStax Bulk Loader
DATASTAX CUSTOMER
Please God, give me
a faster and easier
way to load data.

DSE Search
Full text and adhoc queries
or for those times when you really feel like using AllowFiltering (<-DON’T)

DSE Search
CQL Query
Predicates
● Equality and Inequality
● Range
● Contains
● In
● Like
SELECT * FROM killrvideo.users WHERE email = 'eboyeri5@aol.com';
SELECT * FROM killrvideo.users WHERE email != 'eboyeri5@aol.com';
SELECT * FROM killrvideo.users
WHERE created_date >= '2018-04-01' AND created_date < '2018-05-01';
WHERE phone_number CONTAINS '650-389-6000';
WHERE firstname IN ('Beauregard','Muffin');
SELECT * FROM killrvideo.users WHERE lastname LIKE 'McD%';
CREATE TABLE killrvideo.users (
userid UUID,
created_date TIMESTAMP,
email TEXT,
firstname TEXT,
lastname TEXT,
phone_number SET<TEXT>
PRIMARY KEY ((userid))
);

DSE Search
CQL Query
Order By
● Sort by any column
● Ascending (default) or descending order
SELECT email, added_date, lastname, firstname
FROM killrvideo.users
ORDER BY added_date DESC, lastname ASC, firstname ASC;
CREATE TABLE killrvideo.users (
userid UUID,
created_date TIMESTAMP,
email TEXT,
firstname TEXT,
lastname TEXT,
phone_number SET<TEXT>
PRIMARY KEY ((userid))
);

GeoSpatial
Native GeoSpatial Search
CQL based Heatmaps
https://github.com/DataStax-Academy/datastax-vehicle-demo
Native GeoSpatial
Search
● Point
● LineString
● Polygon
CQL based Heatmaps
SELECT * FROM test
WHERE solr_query=
'{"q":"id:*", "facet":
{"heatmap": "point"}}';

solr_query
Don’t worry
solr_query hasn’t changed
for all of your advanced Lucene query needs

NodeSync +
DSE Search =

DSE Graph
When g.V().repeat(constant("relationships are key"))
.emit().times(3)

DSE Advanced
Performance +
DSE Graph =
one-hop traversal performance
increased almost 50%
(g.V().out())
two-hop traversal performance
increased almost 60%
(g.V().out().out())

DSE Graph Enhancements
Increased unification with DSE Analytics
Smart analytics query routing
● Transactional analytics much faster with DSE Graph Frames vs
Gremlin OLAP
● Automatically routes Gremlin OLAP traversals to Graph Frames or
Gremlin OLAP based off best performance
● Users no longer have to choose
● Just execute your Gremlin OLAP traversal

DSE Analytics
When you need to analyze ALL the data

AlwaysON SQL
AlwaysOn SQL support for SparkSQL
● Production ready
− Highly available
− ODBC/JDBC support
− Supports external BI/ETL/other tool connectivity
(Tableau, excel, etc…)
− Unified Authentication with DSE
Developer experience MUCH simpler
● Focus on removing barriers!

DSE Spark Connector
Smarter, Stronger, Faster
Increased performance and optimizer enhancements
● Custom join strategies with Direct Join
− Previously performed full table scans
− Takes advantage of how Cassandra lays out data
− Now automatically uses direct joins where
applicable
Implicit use of DSE Search
DSE Analytics support for Structured Streaming

DSE OpsCenter
Manage, monitor, and deploy your cluster
all while having cool visuals to show your friends

Backup Service
Improvements
Accelerated restores (restores 3x - 6x faster)
● Restore times nearly equal download time
Datacenter level backups
Expanded backup destinations
● S3-compatible storage
− If it uses the S3 API, you can use it
● Azure Blob storage
Clone backups between heterogeneous clusters

Backwards compatibility for Agents

NodeSync Configuration and Status

OpsCenter Enhancements
Upgrade service
● Perform minor version upgrades of DSE clusters
● Upgrades applied to maintain cluster availability
Improved repair service
● incremental and subrange optimizations
− increased performance at scale for large clusters
DSE 6* new metrics (Advanced Performance, new DSE Graph, etc.)
DSE 6* new feature support (AlwaysON SQL, NodeSync, etc…)
Performance and stability improvements

DSE Advanced Security
Lock it down, lock it ALL down!

DSE Advanced Security Enhancements
Separation of Duties
● Privatized schemas
− Protects against unauthorized viewing of database schemas
● Secured administrator
− Allows administration work without data access

DSE Advanced Security Enhancements
KMIP (Key Management Interoperability
protocol)
● Encryption keys stored on external
server
RLAC (Row Level Access Control)
Audit by role
● Reduces audit trail
● Focus on specific human activity

DataStax Studio
Make your development flow with Studio

DataStax Studio
Enhancements
Export/Import DataStax Studio notebooks
Change tracking
● Track notebook changes
● Tag and preview notebook revisions
● Revert to previous checkpoints

DataStax
Studio
Enhancements
DSE Analytics SparkSQL
Visually explore DSE Graphs
● Expand vertex
neighborhoods
● Remove vertices/edges
from view

DataStax Metrics Collector
Functional, agentless, and darn pretty

Metrics Collector
First phase of DSE Insights
DSE Server based data and
metric collection system
Data Collection via sidecar
collectd process
Interfaces with third-party
monitoring software (prometheus,
grafana, etc…)
Docker demo getting started for
easy setup
https://bit.ly/2Eay7QP

Docker and Kafka
Containers and streaming FTW!

Official production-ready DSE Docker images
Docker DSE images now certified for production use
● DataStax Enterprise 5.1.5+
● Studio 2.0.0+
● OpsCenter 6.1.4+
● DDAC (DataStax Distribution of Apache Cassandra) 5.1.11+
Why are these special?
● Default environment variable auto-config
− Get up and going fast
● Ability to disable auto-config for custom config
− Allows easy transition to your specific environment
● Config volume for extensive custom config
− Makes dropping your environment config simple to repeat
● Extensively tested for production use
− All images wrung through same certification tests as all other supported platforms

DataStax Apache Kafka™ Connector
Connector
ConnectorConnector
Apache Kafka™ Connector
● Reads from topics and automatically saves events to DSE
● Supported in DSE 5.0+

Beyond
DataStax 6
More AWESOMER!
*work currently in progress

DataStax Graph
Upcoming stuff that will knock your traversals into the next step!

Index Recommendationsdo I need an edge
index, no, maybe
secondary...

Index Recommendations
Analyze a traversal, receive an index recommendation

Recommendations are intelligent

Recommendations are actionable

DataStax One Model
DSE’s Multi Model DSE’s One Model
Graph Specific Ingest DSE Unified Ingest

DataStax One Model in Action

More Performance!
● Batch performance
− 1,000% throughput increase
− many TB’s of data
● Traversal performance
− single and multi-hop traversals
between 50% and 1,000% faster
respectively
● Global edge traversals (g.E()) now
possible

BigNode
Horizontal scaling is awesome
Now vertically scale your nodes Godzilla size

BigNode

1TB
1TB
1TB 1TB
1TB
Current density
~5TB+
~5TB+
~5TB+
~5TB+ ~5TB+
*features still in development
BigNode density

Add/remove
nodes up to
5x faster
~5TB+
~5TB+
~5TB+
~5TB+
~5TB+
~5TB+
5x
5x
5x

DataStax Labs
● Not for production
● Feedback provided via forums
● Available at downloads.datastax.com
● Initial projects include:
‒ DSE Desktop
‒ DataStax Graph (availability date TBD)
More information: downloads.datastax.com

DataStax 6 and Beyond

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à DataStax 6 and Beyond

Similaire à DataStax 6 and Beyond (20)

Dernier

Dernier (20)

DataStax 6 and Beyond

Notes de l'éditeur