SlideShare une entreprise Scribd logo
1  sur  73
Télécharger pour lire hors ligne
1 million writes per sec. on 60 nodes with Cassandra and EBS
© 2015. All Rights Reserved.
1 Million Writes Per Second w/60
nodes. !
EBS and C*!
Jim Plush - Sr Director of Engineering, CrowdStrike!
Dennis Opacki - Sr Cloud Systems Architect!
An Introduction
to CrowdStrike
We Are CyberSecurity Technology Company
We Detect, Prevent And Respond To All Attack Types In Real Time,
Protecting Organizations From Catastrophic Breaches
We Provide Next Generation Endpoint Protection, Threat Intelligence & Pre &Post
IR Services
NEXT-GEN
ENDPOINT
INCIDENT
RESPONSE
THREAT
INTEL
http://www.crowdstrike.com/introduction-to-crowdstrike-falcon-host/
CrowdStrike Scale
•  Cloud based endpoint protection
•  Single customer can generate > 2TB daily
•  500K+ Events Per Second
•  Multi PetaBytes of managed data
© 2015. All Rights Reserved.
Truisms???
•  HTTPs is too slow to run everywhere
•  All you need is anti-virus
•  Never run Cassandra on EBS
© 2015. All Rights Reserved.
© 2015. All Rights Reserved.
What is EBS?
EBS Data Volume
EBS Data Volume
/mnt/foo
/mnt/bar
EC2 Instance
§  Network Mounted Hard Drive
§  Ability to snapshot data
§  Data encryption at rest & in flight
Existing EBS Assumptions
•  Jittery I/O aka: Noisy neighbors
•  Single Point of Failure in a Region
•  Cost is too damn high
•  Bad Volumes (dd and destroy)
© 2015. All Rights Reserved.
A recent project: initial requirements
•  1PB of incoming event data from millions of devices
•  Modeled as a graph
•  1 million writes per second (burst)
•  Age data out after x days
•  95% write 5% read
© 2015. All Rights Reserved.
We Tried
•  Cassandra + Titan
•  Sharding?
•  Neo4J
•  PostgreSQL, MySQL, SQLite
•  LevelDB/RocksDB
© 2015. All Rights Reserved.
We have to make this work
•  Cassandra had the properties we needed
•  Time for a new approach?
© 2015. All Rights Reserved. http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html
Number of Machines for 1PB
© 2015. All Rights Reserved.
0.
450.
900.
1350.
1800.
2250.
I2.xlarge c4.2XL EBS
Yearly Cost for 1PB Cluster
© 2015. All Rights Reserved.
0.
4.
8.
12.
16.
I2.xlarge-on demand I2.xlarge-reserved c4.2xl - on demand c4.2xl - reserved
Millionsof$
With EBS
Initial Launch
Date Tiered Compaction
© 2015. All Rights Reserved.
…more details by Jeff Jirsa, CrowdStrike
Cassandra Summit 2015 - DTCS
Initial Launch
•  Cassandra 2.0.12 (DSE)
•  m3.2xlarge 8 core
•  Single 4TB EBS GP2 ~10,000 IOPS
•  Default tunings
© 2015. All Rights Reserved.
Performance was terrible
•  12 node cluster
•  ~60K writes per second RF2
•  ~10K writes per 8 core box
•  We went to the experts
© 2015. All Rights Reserved.
© 2015. All Rights Reserved.
Cassandra Summit 2014
Family Search asked the
same question:
Where’s the bottleneck?
https://www.youtube.com/watch?v=Qfzg7gcSK-g
IOPS Available
© 2015. All Rights Reserved.
0.
12500.
25000.
37500.
50000.
I2.xlarge c4.2xlarge
© 2015. All Rights Reserved.
1.3K IOPS?
© 2015. All Rights Reserved.
IOPS
I see you there,
but I can’t reach you!
© 2015. All Rights Reserved.
The magic gates
opened…
We hit 1 million
writes per second
RF3 on 60 nodes
© 2015. All Rights Reserved.
Testing Setup!
Testing Methodology
•  Each test run
•  clean C* instances
•  old test keyspaces dropped
•  13+TBs of data loaded during read testing
•  20 C4.4XL Stress Writers each with their own 1BB sequence
© 2015. All Rights Reserved.
Cluster Topology
© 2015. All Rights Reserved.
Stress Node
10 Instances
AZ: 1A
Stress Nodes
10 Instances
AZ: 1B
20 C* Nodes
AZ: 1A
20 C* Nodes
AZ: 1B
20 C* Nodes
AZ: 1C
OpsCenter
EBS
© 2015. All Rights Reserved.
Cassandra Stress 2.1.x
© 2015. All Rights Reserved.
bin/cassandra-stress user duration=100000m cl=ONE profile=/home/ubuntu/summit_stress.yaml ops
(insert=1) no-warmup -pop seq=1..1000000000 -mode native cql3 -node 10.10.10.XX -rate
threads=1000 -errors ignore!
© 2015. All Rights Reserved.
PCSTAT - Al Tobey
http://www.datastax.com/dev/blog/compaction-improvements-in-cassandra-21
https://github.com/tobert/pcstat
© 2015. All Rights Reserved.
Netflix Test - What is C* capable of?
Netflix Test
© 2015. All Rights Reserved.
1+ Million Writes Per second RF:3 3+ Million Local Writes Per second
NICE!
Netflix Test
© 2015. All Rights Reserved.
Netflix Test
© 2015. All Rights Reserved.
No Dropped Mutations, system healthy at 1.1M after 50 mins
Netflix Test
© 2015. All Rights Reserved.
I/O Util is not peggedCommit Disk = Steady!
Netflix Test
© 2015. All Rights Reserved.
Low IO Wait
Netflix Test
© 2015. All Rights Reserved.
95th Latency = Reasonable
Netflix Test - Read Fail
© 2015. All Rights Reserved.
compression={'chunk_length_kb': '64', 'sstable_compression': 'LZ4Compressor'}
https://issues.apache.org/jira/browse/CASSANDRA-10249
https://issues.apache.org/jira/browse/CASSANDRA-8894
Data Drive Pegged L
Reading Data
•  24 hour read test
•  over 10TBs of data in the CF
•  sustained > 350K reads per
second over 24 hours
•  1M reads/per sec peak
•  CL ONE
•  12 C4.4XL stress boxes
© 2015. All Rights Reserved.
Reading Data
© 2015. All Rights Reserved.
Reading Data
© 2015. All Rights Reserved.
Reading Data
© 2015. All Rights Reserved.
Not Pegged J
Reading Data
© 2015. All Rights Reserved.
7.2ms 95th latency
Netflix Test resource usage
•  180 Less Cores (45 less i2.xlarge instances)
•  24 hour test (sans data transfer cost)
–  Netflix cluster/stress
•  Cost: ~$6300
•  285 i2.xlarge $0.85 per hour
–  CrowdStrike cluster/stress with EBS cost
•  Cost: ~$2600
•  60 C4.4XL $0.88 per hour
Read Notes with EBS
•  Our test was a single 10K IOPS volume
•  More/Bigger Reads?
–  PIOPS gives you as much throughput as you need
–  RAID0 multiple EBS volumes
/mnt/data
EBS Vol1 EBS Vol2
© 2015. All Rights Reserved.
What Unlocked Performance!
Major Tweaks
•  Ubuntu HVM types
•  Enhanced Networking
•  now faster than PVM
•  Ubuntu distro tuned for cloud workloads
•  XFS Filesystem
© 2015. All Rights Reserved.
Major Tweaks
•  Major Tweaks
•  Cassandra 2.1
•  Java 8
•  G1 Garbage Collector - cassandra-env
© 2015. All Rights Reserved.
https://issues.apache.org/jira/browse/CASSANDRA-7486
Major Tweaks
•  C4.4XL 16 core, EBS Optimized
•  4TB, 10,000 IOPS EBS GP2 Encrypted Data Drive
–  160MB/s throughput
•  1TB 3000 IOPS EBS GP2 Encrypted Commit Log Drive
© 2015. All Rights Reserved.
Major Tweaks
•  cassandra-env.sh
•  MAX_HEAP_SIZE=8G
•  JVM_OPTS=“$JVM_OPTS —XX:+UseG1GC”
•  Lots of other minor tweaks
© 2015. All Rights Reserved.
cassandra-env.sh
© 2015. All Rights Reserved.
Put PID in batch mode
Mask CPU0 from the process to reduce context switching
Magic From Al Tobey
YAML Settings
•  cassandra.yaml (based on 16 core)
•  concurrent_reads: 32
•  concurrent_writes: 64
•  memtable_flush_writers: 8
•  trickle_fsync: true
•  trickle_fsync_interval_in_kb: 1000
•  native_transport_max_threads: 256
•  concurrent_compactors: 4
© 2015. All Rights Reserved.
cassandra.yaml
© 2015. All Rights Reserved.
We found a good portion of the CPU load was
being used for internode compression which
reduced write throughput
internode_compression: none
Lessons Learned
•  EBS was never the bottleneck during testing, GP2 is legit
•  If you’re doing batching, write to the same rowkey in the batch
•  Builtin types like list and map come at a performance penalty
•  30% hit on our writes using Map type
•  DTCS is very young (see Jeff Jirsa’s talk)
•  2.1 Stress Tool is tricky but great for modeling workloads
•  How will compression affect your read path?
© 2015. All Rights Reserved.
© 2015. All Rights Reserved.
Test your own!
https://github.com/CrowdStrike/cassandra-tools
It’s just python
•  launch 20 nodes in us-east1
•  python launch.py launch --nodes=20 —config=c4-ebs-hvm
—az=us-east-1a
•  bootstrap the new nodes with C*, RAID/Format disks, etc…
•  fab -u ubuntu bootstrapcass21:config=c4-highperf
•  run arbitrary commands
•  fab -u ubuntu cmd:config=c4-highperf,cmd="sudo rm -rf /
mnt/cassandra/data/summit_stress"
© 2015. All Rights Reserved.
Run custom stress profiles… multi-node support
ubuntu@ip-10-10-10.XX:~$ python runstress.py --profile=stress10 —seednode=10.10.10.XX —-threads=50!
!
!
Going to run: /home/ubuntu/apache-cassandra-2.1.5/tools/bin/cassandra-stress user duration=100000m
cl=ONE profile=/home/ubuntu/summit_stress.yaml ops(insert=1,simple=9) no-warmup -pop
seq=1..1000000000 -mode native cql3 -node 10.10.10.XX -rate threads=50 -errors ignore!
© 2015. All Rights Reserved.
ubuntu@ip-10-10-10.XX:~$ python runstress.py --profile=stress10 --seednode=10.10.10.XX --threads=50!
!
Going to run: /home/ubuntu/apache-cassandra-2.1.5/tools/bin/cassandra-stress user duration=100000m
cl=ONE profile=/home/ubuntu/summit_stress.yaml ops(insert=1,simple=9) no-warmup -pop
seq=1000000001..2000000000 -mode native cql3 -node 10.10.10.XX -rate threads=50 -errors ignore!
export NODENUM=1!
export NODENUM=2!
Where are we today?
•  ~3 months on our EBS based cluster
•  Hundreds of TBs of graph data and growing in C*
•  Billions of vertices/edges
•  Changing perceptions?
Special thanks to
© 2015. All Rights Reserved.
•  Leif Jackson
•  Marcus King
•  Alan Hannan
•  Jeff Jirsa
•  Al Tobey
•  Nick Panahi
•  J.B. Langston
•  Marcus Eriksson
•  Iian Finlayson
•  Dani Traphagen
EBS heading into 2016
© 2015. All Rights Reserved.
4TB	
  (10k	
  IOPS)	
  GP2	
  
IO	
  Hit?	
  Not	
  enough	
  to	
  phase	
  C*	
  
© 2015. All Rights Reserved.
	
  
	
  
So	
  why	
  the	
  hate	
  for	
  EBS?	
  
© 2015. All Rights Reserved.
Following	
  the	
  Crowd	
  –	
  Trust	
  Issues	
  
	
  
•  Used	
  instance-­‐store	
  image	
  and	
  ephemeral	
  
drives	
  
•  Painful	
  to	
  stop/start	
  instances,	
  resize	
  
•  Couldn’t	
  avoid	
  scheduled	
  maintenance	
  (i.e.	
  
Reboot-­‐a-­‐palooza)	
  
•  EncrypUon	
  required	
  shenanigans	
  
© 2015. All Rights Reserved.
Guess	
  What?	
  
•  We	
  sUll	
  had	
  failures	
  
•  Now	
  we	
  get	
  to	
  rebuild	
  from	
  scratch	
  
© 2015. All Rights Reserved.
What	
  do	
  you	
  mean	
  my	
  volume	
  is	
  “stuck”?	
  
	
  
•  April	
  2011	
  –	
  Ne[lix,	
  Reddit	
  and	
  Quora	
  
•  October	
  2012	
  –	
  Reddit,	
  Imgur,	
  Heroku	
  
•  August	
  2013	
  –	
  Vine,	
  AirBNB	
  
EBS’s	
  Troubled	
  Childhood	
  
© 2015. All Rights Reserved.
h`p://techblog.ne[lix.com/2011/04/lessons-­‐
ne[lix-­‐learned-­‐from-­‐aws-­‐outage.html	
  
	
  
•  Spread	
  services	
  across	
  mulUple	
  regions	
  
•  Test	
  failure	
  scenarios	
  regularly	
  (Chaos	
  Monkey)	
  
•  Make	
  Cassandra	
  databases	
  more	
  resilient	
  by	
  avoiding	
  
EBS	
  
Kiss	
  of	
  Death	
  
© 2015. All Rights Reserved.
Amazon	
  moves	
  quickly	
  and	
  quietly:	
  
	
  
•  March	
  2011	
  –	
  New	
  EBS	
  GM	
  
•  July	
  2012	
  –	
  Provisioned	
  IOPs	
  
•  May	
  2014	
  –	
  NaUve	
  EncrypUon	
  
•  Jun	
  2014	
  –	
  GP2	
  (game	
  changer)	
  
•  Mar	
  2015	
  –	
  16TB	
  /	
  10K	
  GP2/	
  20K	
  PIOPS	
  
	
  
	
  
RedempUon	
  
© 2015. All Rights Reserved.
•  PrioriUzed	
  EBS	
  availability	
  and	
  consistency	
  beyond	
  features	
  and	
  
funcUonality	
  
•  Compartmentalized	
  the	
  control	
  plane	
  -­‐	
  broke	
  cross-­‐AZ	
  dependencies	
  
for	
  running	
  volumes	
  
•  Simplified	
  workflows	
  to	
  favor	
  sustained	
  operaUon	
  
•  Tested	
  and	
  simulated	
  via	
  TLA+/PlusCal	
  -­‐	
  be`er	
  understood	
  corner	
  cases	
  
•  Dedicated	
  a	
  large	
  fracUon	
  of	
  engineering	
  resources	
  to	
  reliability	
  and	
  
performance	
  
	
  
RedempUon	
  
© 2015. All Rights Reserved.
Reliability	
  
	
  
EBS	
  Team	
  targets	
  99.999%	
  availability	
  
	
  
	
  exceeding	
  expectaUons	
  
© 2015. All Rights Reserved.
Crowdstrike	
  Today	
  
In	
  past	
  12	
  months,	
  zero	
  EBS-­‐related	
  failures	
  
•  Thousands	
  of	
  GP2	
  data	
  volumes	
  (~2PB	
  data)	
  
•  TransiUoning	
  all	
  systems	
  to	
  EBS	
  root	
  drives	
  
•  Moved	
  all	
  data	
  stores	
  to	
  EBS	
  (C*,	
  Kapa,	
  
ElasUcsearch,	
  Postgres,	
  etc)	
  
© 2015. All Rights Reserved.
Staying	
  Safe	
  -­‐	
  Architecture	
  
•  Select	
  a	
  region	
  with	
  >2	
  AZs	
  (e.g	
  us-­‐east-­‐1	
  or	
  us-­‐
west-­‐2)	
  
	
  
•  Use	
  EBS	
  GP2	
  or	
  PIOPs	
  storage	
  
•  Separate	
  volumes	
  for	
  data	
  and	
  commit	
  logs	
  
© 2015. All Rights Reserved.
Staying	
  Safe	
  -­‐	
  Ops	
  
•  Use	
  EBS	
  volume	
  monitoring	
  
•  Pre-­‐warm	
  EBS	
  volumes?	
  
•  Schedule	
  snapshots	
  for	
  consistent	
  backups	
  
© 2015. All Rights Reserved.
Most	
  Importantly	
  
•  Challenge	
  assumpUons	
  
•  Stay	
  current	
  on	
  AWS	
  blog	
  
•  Talk	
  with	
  your	
  peers	
  
Thank you
@jimplush
@opacki

Contenu connexe

Tendances

RedisConf17- Using Redis at scale @ Twitter
RedisConf17- Using Redis at scale @ TwitterRedisConf17- Using Redis at scale @ Twitter
RedisConf17- Using Redis at scale @ TwitterRedis Labs
 
How to size up an Apache Cassandra cluster (Training)
How to size up an Apache Cassandra cluster (Training)How to size up an Apache Cassandra cluster (Training)
How to size up an Apache Cassandra cluster (Training)DataStax Academy
 
Deep Dive and Best Practices for Real Time Streaming Applications
Deep Dive and Best Practices for Real Time Streaming ApplicationsDeep Dive and Best Practices for Real Time Streaming Applications
Deep Dive and Best Practices for Real Time Streaming ApplicationsAmazon Web Services
 
Apache Spark on K8S and HDFS Security with Ilan Flonenko
Apache Spark on K8S and HDFS Security with Ilan FlonenkoApache Spark on K8S and HDFS Security with Ilan Flonenko
Apache Spark on K8S and HDFS Security with Ilan FlonenkoDatabricks
 
Cassandra presentation at NoSQL
Cassandra presentation at NoSQLCassandra presentation at NoSQL
Cassandra presentation at NoSQLEvan Weaver
 
Introduction to Kafka Cruise Control
Introduction to Kafka Cruise ControlIntroduction to Kafka Cruise Control
Introduction to Kafka Cruise ControlJiangjie Qin
 
10 Tips for AIX Security
10 Tips for AIX Security10 Tips for AIX Security
10 Tips for AIX SecurityHelpSystems
 
Singer, Pinterest's Logging Infrastructure
Singer, Pinterest's Logging InfrastructureSinger, Pinterest's Logging Infrastructure
Singer, Pinterest's Logging InfrastructureDiscover Pinterest
 
RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compactionMIJIN AN
 
Velocity 2015 linux perf tools
Velocity 2015 linux perf toolsVelocity 2015 linux perf tools
Velocity 2015 linux perf toolsBrendan Gregg
 
Spark and S3 with Ryan Blue
Spark and S3 with Ryan BlueSpark and S3 with Ryan Blue
Spark and S3 with Ryan BlueDatabricks
 
Transaction Management on Cassandra
Transaction Management on CassandraTransaction Management on Cassandra
Transaction Management on CassandraScalar, Inc.
 
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache FlinkFlexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
Hadoop File system (HDFS)
Hadoop File system (HDFS)Hadoop File system (HDFS)
Hadoop File system (HDFS)Prashant Gupta
 
iceberg introduction.pptx
iceberg introduction.pptxiceberg introduction.pptx
iceberg introduction.pptxDori Waldman
 
Dual write strategies for microservices
Dual write strategies for microservicesDual write strategies for microservices
Dual write strategies for microservicesBilgin Ibryam
 
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Ryan Blue
 
Hive+Tez: A performance deep dive
Hive+Tez: A performance deep diveHive+Tez: A performance deep dive
Hive+Tez: A performance deep divet3rmin4t0r
 

Tendances (20)

RedisConf17- Using Redis at scale @ Twitter
RedisConf17- Using Redis at scale @ TwitterRedisConf17- Using Redis at scale @ Twitter
RedisConf17- Using Redis at scale @ Twitter
 
How to size up an Apache Cassandra cluster (Training)
How to size up an Apache Cassandra cluster (Training)How to size up an Apache Cassandra cluster (Training)
How to size up an Apache Cassandra cluster (Training)
 
Deep Dive and Best Practices for Real Time Streaming Applications
Deep Dive and Best Practices for Real Time Streaming ApplicationsDeep Dive and Best Practices for Real Time Streaming Applications
Deep Dive and Best Practices for Real Time Streaming Applications
 
Apache Spark on K8S and HDFS Security with Ilan Flonenko
Apache Spark on K8S and HDFS Security with Ilan FlonenkoApache Spark on K8S and HDFS Security with Ilan Flonenko
Apache Spark on K8S and HDFS Security with Ilan Flonenko
 
Cassandra presentation at NoSQL
Cassandra presentation at NoSQLCassandra presentation at NoSQL
Cassandra presentation at NoSQL
 
Introduction to Kafka Cruise Control
Introduction to Kafka Cruise ControlIntroduction to Kafka Cruise Control
Introduction to Kafka Cruise Control
 
10 Tips for AIX Security
10 Tips for AIX Security10 Tips for AIX Security
10 Tips for AIX Security
 
Singer, Pinterest's Logging Infrastructure
Singer, Pinterest's Logging InfrastructureSinger, Pinterest's Logging Infrastructure
Singer, Pinterest's Logging Infrastructure
 
RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compaction
 
Hands-on DNSSEC Deployment
Hands-on DNSSEC DeploymentHands-on DNSSEC Deployment
Hands-on DNSSEC Deployment
 
Velocity 2015 linux perf tools
Velocity 2015 linux perf toolsVelocity 2015 linux perf tools
Velocity 2015 linux perf tools
 
Spark and S3 with Ryan Blue
Spark and S3 with Ryan BlueSpark and S3 with Ryan Blue
Spark and S3 with Ryan Blue
 
Transaction Management on Cassandra
Transaction Management on CassandraTransaction Management on Cassandra
Transaction Management on Cassandra
 
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache FlinkFlexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache Flink
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Hadoop File system (HDFS)
Hadoop File system (HDFS)Hadoop File system (HDFS)
Hadoop File system (HDFS)
 
iceberg introduction.pptx
iceberg introduction.pptxiceberg introduction.pptx
iceberg introduction.pptx
 
Dual write strategies for microservices
Dual write strategies for microservicesDual write strategies for microservices
Dual write strategies for microservices
 
Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)Iceberg: A modern table format for big data (Strata NY 2018)
Iceberg: A modern table format for big data (Strata NY 2018)
 
Hive+Tez: A performance deep dive
Hive+Tez: A performance deep diveHive+Tez: A performance deep dive
Hive+Tez: A performance deep dive
 

En vedette

The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...DataStax
 
Running Cassandra on Amazon EC2
Running Cassandra on Amazon EC2Running Cassandra on Amazon EC2
Running Cassandra on Amazon EC2Dave Gardner
 
Deep Dive: Maximizing EC2 and EBS Performance
Deep Dive: Maximizing EC2 and EBS PerformanceDeep Dive: Maximizing EC2 and EBS Performance
Deep Dive: Maximizing EC2 and EBS PerformanceAmazon Web Services
 
How to run an Enterprise PHP Shop
How to run an Enterprise PHP ShopHow to run an Enterprise PHP Shop
How to run an Enterprise PHP ShopJim Plush
 
Event-Stream Processing with Kafka
Event-Stream Processing with KafkaEvent-Stream Processing with Kafka
Event-Stream Processing with KafkaTim Lossen
 
PostgreSQL and CockroachDB SQL
PostgreSQL and CockroachDB SQLPostgreSQL and CockroachDB SQL
PostgreSQL and CockroachDB SQLCockroachDB
 
Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...
Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...
Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...Amazon Web Services
 
RocksDB detail
RocksDB detailRocksDB detail
RocksDB detailMIJIN AN
 
AWS - an introduction to bursting (GP2 - T2)
AWS - an introduction to bursting (GP2 - T2)AWS - an introduction to bursting (GP2 - T2)
AWS - an introduction to bursting (GP2 - T2)Rasmus Ekman
 
AWS re:Invent 2016: Case Study: Librato's Experience Running Cassandra Using ...
AWS re:Invent 2016: Case Study: Librato's Experience Running Cassandra Using ...AWS re:Invent 2016: Case Study: Librato's Experience Running Cassandra Using ...
AWS re:Invent 2016: Case Study: Librato's Experience Running Cassandra Using ...Amazon Web Services
 
Maximizing EC2 and Elastic Block Store Disk Performance
Maximizing EC2 and Elastic Block Store Disk PerformanceMaximizing EC2 and Elastic Block Store Disk Performance
Maximizing EC2 and Elastic Block Store Disk PerformanceAmazon Web Services
 
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...DataStax
 
Load Testing Cassandra Applications (Ben Slater, Instaclustr) | C* Summit 2016
Load Testing Cassandra Applications (Ben Slater, Instaclustr) | C* Summit 2016Load Testing Cassandra Applications (Ben Slater, Instaclustr) | C* Summit 2016
Load Testing Cassandra Applications (Ben Slater, Instaclustr) | C* Summit 2016DataStax
 
Cassandra Summit 2014: Performance Tuning Cassandra in AWS
Cassandra Summit 2014: Performance Tuning Cassandra in AWSCassandra Summit 2014: Performance Tuning Cassandra in AWS
Cassandra Summit 2014: Performance Tuning Cassandra in AWSDataStax Academy
 
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookTech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
 
Cassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSCassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSAdrian Cockcroft
 
DataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax Academy
 
What no one tells you about writing a streaming app
What no one tells you about writing a streaming appWhat no one tells you about writing a streaming app
What no one tells you about writing a streaming apphadooparchbook
 

En vedette (20)

The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
The Best and Worst of Cassandra-stress Tool (Christopher Batey, The Last Pick...
 
Running Cassandra on Amazon EC2
Running Cassandra on Amazon EC2Running Cassandra on Amazon EC2
Running Cassandra on Amazon EC2
 
Running Cassandra in AWS
Running Cassandra in AWSRunning Cassandra in AWS
Running Cassandra in AWS
 
Deep Dive: Maximizing EC2 and EBS Performance
Deep Dive: Maximizing EC2 and EBS PerformanceDeep Dive: Maximizing EC2 and EBS Performance
Deep Dive: Maximizing EC2 and EBS Performance
 
How to run an Enterprise PHP Shop
How to run an Enterprise PHP ShopHow to run an Enterprise PHP Shop
How to run an Enterprise PHP Shop
 
Event-Stream Processing with Kafka
Event-Stream Processing with KafkaEvent-Stream Processing with Kafka
Event-Stream Processing with Kafka
 
PostgreSQL and CockroachDB SQL
PostgreSQL and CockroachDB SQLPostgreSQL and CockroachDB SQL
PostgreSQL and CockroachDB SQL
 
Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...
Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...
Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...
 
RocksDB detail
RocksDB detailRocksDB detail
RocksDB detail
 
AWS - an introduction to bursting (GP2 - T2)
AWS - an introduction to bursting (GP2 - T2)AWS - an introduction to bursting (GP2 - T2)
AWS - an introduction to bursting (GP2 - T2)
 
AWS re:Invent 2016: Case Study: Librato's Experience Running Cassandra Using ...
AWS re:Invent 2016: Case Study: Librato's Experience Running Cassandra Using ...AWS re:Invent 2016: Case Study: Librato's Experience Running Cassandra Using ...
AWS re:Invent 2016: Case Study: Librato's Experience Running Cassandra Using ...
 
Maximizing EC2 and Elastic Block Store Disk Performance
Maximizing EC2 and Elastic Block Store Disk PerformanceMaximizing EC2 and Elastic Block Store Disk Performance
Maximizing EC2 and Elastic Block Store Disk Performance
 
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
 
Load Testing Cassandra Applications (Ben Slater, Instaclustr) | C* Summit 2016
Load Testing Cassandra Applications (Ben Slater, Instaclustr) | C* Summit 2016Load Testing Cassandra Applications (Ben Slater, Instaclustr) | C* Summit 2016
Load Testing Cassandra Applications (Ben Slater, Instaclustr) | C* Summit 2016
 
Cassandra Summit 2014: Performance Tuning Cassandra in AWS
Cassandra Summit 2014: Performance Tuning Cassandra in AWSCassandra Summit 2014: Performance Tuning Cassandra in AWS
Cassandra Summit 2014: Performance Tuning Cassandra in AWS
 
MyRocks Deep Dive
MyRocks Deep DiveMyRocks Deep Dive
MyRocks Deep Dive
 
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookTech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
 
Cassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSCassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWS
 
DataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The SequelDataStax: Extreme Cassandra Optimization: The Sequel
DataStax: Extreme Cassandra Optimization: The Sequel
 
What no one tells you about writing a streaming app
What no one tells you about writing a streaming appWhat no one tells you about writing a streaming app
What no one tells you about writing a streaming app
 

Similaire à 1 Million Writes per second on 60 nodes with Cassandra and EBS

Approaching hyperconvergedopenstack
Approaching hyperconvergedopenstackApproaching hyperconvergedopenstack
Approaching hyperconvergedopenstackIkuo Kumagai
 
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014Amazon Web Services
 
Your Linux AMI: Optimization and Performance (CPN302) | AWS re:Invent 2013
Your Linux AMI: Optimization and Performance (CPN302) | AWS re:Invent 2013Your Linux AMI: Optimization and Performance (CPN302) | AWS re:Invent 2013
Your Linux AMI: Optimization and Performance (CPN302) | AWS re:Invent 2013Amazon Web Services
 
Tuning the Kernel for Varnish Cache
Tuning the Kernel for Varnish CacheTuning the Kernel for Varnish Cache
Tuning the Kernel for Varnish CachePer Buer
 
CPN302 your-linux-ami-optimization-and-performance
CPN302 your-linux-ami-optimization-and-performanceCPN302 your-linux-ami-optimization-and-performance
CPN302 your-linux-ami-optimization-and-performanceCoburn Watson
 
Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...Fwdays
 
Trying and evaluating the new features of GlusterFS 3.5
Trying and evaluating the new features of GlusterFS 3.5Trying and evaluating the new features of GlusterFS 3.5
Trying and evaluating the new features of GlusterFS 3.5Keisuke Takahashi
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2aspyker
 
To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…Sergey Dzyuban
 
Network Stack in Userspace (NUSE)
Network Stack in Userspace (NUSE)Network Stack in Userspace (NUSE)
Network Stack in Userspace (NUSE)Hajime Tazaki
 
Ceph Day Beijing - Ceph all-flash array design based on NUMA architecture
Ceph Day Beijing - Ceph all-flash array design based on NUMA architectureCeph Day Beijing - Ceph all-flash array design based on NUMA architecture
Ceph Day Beijing - Ceph all-flash array design based on NUMA architectureCeph Community
 
Ceph Day Beijing - Ceph All-Flash Array Design Based on NUMA Architecture
Ceph Day Beijing - Ceph All-Flash Array Design Based on NUMA ArchitectureCeph Day Beijing - Ceph All-Flash Array Design Based on NUMA Architecture
Ceph Day Beijing - Ceph All-Flash Array Design Based on NUMA ArchitectureDanielle Womboldt
 
Ceph in the GRNET cloud stack
Ceph in the GRNET cloud stackCeph in the GRNET cloud stack
Ceph in the GRNET cloud stackNikos Kormpakis
 
Considerations when implementing_ha_in_dmf
Considerations when implementing_ha_in_dmfConsiderations when implementing_ha_in_dmf
Considerations when implementing_ha_in_dmfhik_lhz
 
Memory, Big Data, NoSQL and Virtualization
Memory, Big Data, NoSQL and VirtualizationMemory, Big Data, NoSQL and Virtualization
Memory, Big Data, NoSQL and VirtualizationBigstep
 
OpenStack and NetApp - Chen Reuven - OpenStack Day Israel 2017
OpenStack and NetApp - Chen Reuven - OpenStack Day Israel 2017OpenStack and NetApp - Chen Reuven - OpenStack Day Israel 2017
OpenStack and NetApp - Chen Reuven - OpenStack Day Israel 2017Cloud Native Day Tel Aviv
 
Ceph Day Melbourne - Ceph on All-Flash Storage - Breaking Performance Barriers
Ceph Day Melbourne - Ceph on All-Flash Storage - Breaking Performance BarriersCeph Day Melbourne - Ceph on All-Flash Storage - Breaking Performance Barriers
Ceph Day Melbourne - Ceph on All-Flash Storage - Breaking Performance BarriersCeph Community
 
Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017Dave Holland
 

Similaire à 1 Million Writes per second on 60 nodes with Cassandra and EBS (20)

Approaching hyperconvergedopenstack
Approaching hyperconvergedopenstackApproaching hyperconvergedopenstack
Approaching hyperconvergedopenstack
 
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014
 
Your Linux AMI: Optimization and Performance (CPN302) | AWS re:Invent 2013
Your Linux AMI: Optimization and Performance (CPN302) | AWS re:Invent 2013Your Linux AMI: Optimization and Performance (CPN302) | AWS re:Invent 2013
Your Linux AMI: Optimization and Performance (CPN302) | AWS re:Invent 2013
 
Tuning the Kernel for Varnish Cache
Tuning the Kernel for Varnish CacheTuning the Kernel for Varnish Cache
Tuning the Kernel for Varnish Cache
 
CPN302 your-linux-ami-optimization-and-performance
CPN302 your-linux-ami-optimization-and-performanceCPN302 your-linux-ami-optimization-and-performance
CPN302 your-linux-ami-optimization-and-performance
 
Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...Anton Moldovan "Building an efficient replication system for thousands of ter...
Anton Moldovan "Building an efficient replication system for thousands of ter...
 
Trying and evaluating the new features of GlusterFS 3.5
Trying and evaluating the new features of GlusterFS 3.5Trying and evaluating the new features of GlusterFS 3.5
Trying and evaluating the new features of GlusterFS 3.5
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
 
To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…
 
Network Stack in Userspace (NUSE)
Network Stack in Userspace (NUSE)Network Stack in Userspace (NUSE)
Network Stack in Userspace (NUSE)
 
Ceph Day Beijing - Ceph all-flash array design based on NUMA architecture
Ceph Day Beijing - Ceph all-flash array design based on NUMA architectureCeph Day Beijing - Ceph all-flash array design based on NUMA architecture
Ceph Day Beijing - Ceph all-flash array design based on NUMA architecture
 
Ceph Day Beijing - Ceph All-Flash Array Design Based on NUMA Architecture
Ceph Day Beijing - Ceph All-Flash Array Design Based on NUMA ArchitectureCeph Day Beijing - Ceph All-Flash Array Design Based on NUMA Architecture
Ceph Day Beijing - Ceph All-Flash Array Design Based on NUMA Architecture
 
Ceph in the GRNET cloud stack
Ceph in the GRNET cloud stackCeph in the GRNET cloud stack
Ceph in the GRNET cloud stack
 
Considerations when implementing_ha_in_dmf
Considerations when implementing_ha_in_dmfConsiderations when implementing_ha_in_dmf
Considerations when implementing_ha_in_dmf
 
Tuning Linux for MongoDB
Tuning Linux for MongoDBTuning Linux for MongoDB
Tuning Linux for MongoDB
 
Memory, Big Data, NoSQL and Virtualization
Memory, Big Data, NoSQL and VirtualizationMemory, Big Data, NoSQL and Virtualization
Memory, Big Data, NoSQL and Virtualization
 
OpenStack and NetApp - Chen Reuven - OpenStack Day Israel 2017
OpenStack and NetApp - Chen Reuven - OpenStack Day Israel 2017OpenStack and NetApp - Chen Reuven - OpenStack Day Israel 2017
OpenStack and NetApp - Chen Reuven - OpenStack Day Israel 2017
 
HPC in the Cloud
HPC in the CloudHPC in the Cloud
HPC in the Cloud
 
Ceph Day Melbourne - Ceph on All-Flash Storage - Breaking Performance Barriers
Ceph Day Melbourne - Ceph on All-Flash Storage - Breaking Performance BarriersCeph Day Melbourne - Ceph on All-Flash Storage - Breaking Performance Barriers
Ceph Day Melbourne - Ceph on All-Flash Storage - Breaking Performance Barriers
 
Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017Sanger OpenStack presentation March 2017
Sanger OpenStack presentation March 2017
 

Dernier

Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...MyIntelliSource, Inc.
 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providermohitmore19
 
How To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.jsHow To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.jsAndolasoft Inc
 
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AISyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AIABDERRAOUF MEHENNI
 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Modelsaagamshah0812
 
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVOptimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVshikhaohhpro
 
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.comHR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.comFatema Valibhai
 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxbodapatigopi8531
 
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsUnveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsAlberto González Trastoy
 
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female serviceCALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female serviceanilsa9823
 
A Secure and Reliable Document Management System is Essential.docx
A Secure and Reliable Document Management System is Essential.docxA Secure and Reliable Document Management System is Essential.docx
A Secure and Reliable Document Management System is Essential.docxComplianceQuest1
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfkalichargn70th171
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...Health
 
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...kellynguyen01
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...ICS
 
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...OnePlan Solutions
 
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️anilsa9823
 
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsRight Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsJhone kinadey
 

Dernier (20)

Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
 
How To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.jsHow To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.js
 
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AISyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Models
 
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVOptimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTV
 
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.comHR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.com
 
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS LiveVip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptx
 
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsUnveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
 
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female serviceCALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
 
A Secure and Reliable Document Management System is Essential.docx
A Secure and Reliable Document Management System is Essential.docxA Secure and Reliable Document Management System is Essential.docx
A Secure and Reliable Document Management System is Essential.docx
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
 
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
 
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
 
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsRight Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
 

1 Million Writes per second on 60 nodes with Cassandra and EBS

  • 1. 1 million writes per sec. on 60 nodes with Cassandra and EBS
  • 2. © 2015. All Rights Reserved. 1 Million Writes Per Second w/60 nodes. ! EBS and C*! Jim Plush - Sr Director of Engineering, CrowdStrike! Dennis Opacki - Sr Cloud Systems Architect!
  • 3. An Introduction to CrowdStrike We Are CyberSecurity Technology Company We Detect, Prevent And Respond To All Attack Types In Real Time, Protecting Organizations From Catastrophic Breaches We Provide Next Generation Endpoint Protection, Threat Intelligence & Pre &Post IR Services NEXT-GEN ENDPOINT INCIDENT RESPONSE THREAT INTEL http://www.crowdstrike.com/introduction-to-crowdstrike-falcon-host/
  • 4. CrowdStrike Scale •  Cloud based endpoint protection •  Single customer can generate > 2TB daily •  500K+ Events Per Second •  Multi PetaBytes of managed data © 2015. All Rights Reserved.
  • 5. Truisms??? •  HTTPs is too slow to run everywhere •  All you need is anti-virus •  Never run Cassandra on EBS © 2015. All Rights Reserved.
  • 6. © 2015. All Rights Reserved. What is EBS? EBS Data Volume EBS Data Volume /mnt/foo /mnt/bar EC2 Instance §  Network Mounted Hard Drive §  Ability to snapshot data §  Data encryption at rest & in flight
  • 7. Existing EBS Assumptions •  Jittery I/O aka: Noisy neighbors •  Single Point of Failure in a Region •  Cost is too damn high •  Bad Volumes (dd and destroy) © 2015. All Rights Reserved.
  • 8. A recent project: initial requirements •  1PB of incoming event data from millions of devices •  Modeled as a graph •  1 million writes per second (burst) •  Age data out after x days •  95% write 5% read © 2015. All Rights Reserved.
  • 9. We Tried •  Cassandra + Titan •  Sharding? •  Neo4J •  PostgreSQL, MySQL, SQLite •  LevelDB/RocksDB © 2015. All Rights Reserved.
  • 10. We have to make this work •  Cassandra had the properties we needed •  Time for a new approach? © 2015. All Rights Reserved. http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html
  • 11. Number of Machines for 1PB © 2015. All Rights Reserved. 0. 450. 900. 1350. 1800. 2250. I2.xlarge c4.2XL EBS
  • 12. Yearly Cost for 1PB Cluster © 2015. All Rights Reserved. 0. 4. 8. 12. 16. I2.xlarge-on demand I2.xlarge-reserved c4.2xl - on demand c4.2xl - reserved Millionsof$ With EBS
  • 13. Initial Launch Date Tiered Compaction © 2015. All Rights Reserved. …more details by Jeff Jirsa, CrowdStrike Cassandra Summit 2015 - DTCS
  • 14. Initial Launch •  Cassandra 2.0.12 (DSE) •  m3.2xlarge 8 core •  Single 4TB EBS GP2 ~10,000 IOPS •  Default tunings © 2015. All Rights Reserved.
  • 15. Performance was terrible •  12 node cluster •  ~60K writes per second RF2 •  ~10K writes per 8 core box •  We went to the experts © 2015. All Rights Reserved.
  • 16. © 2015. All Rights Reserved. Cassandra Summit 2014 Family Search asked the same question: Where’s the bottleneck? https://www.youtube.com/watch?v=Qfzg7gcSK-g
  • 17. IOPS Available © 2015. All Rights Reserved. 0. 12500. 25000. 37500. 50000. I2.xlarge c4.2xlarge
  • 18. © 2015. All Rights Reserved. 1.3K IOPS?
  • 19. © 2015. All Rights Reserved. IOPS I see you there, but I can’t reach you!
  • 20.
  • 21. © 2015. All Rights Reserved. The magic gates opened… We hit 1 million writes per second RF3 on 60 nodes
  • 22. © 2015. All Rights Reserved. Testing Setup!
  • 23. Testing Methodology •  Each test run •  clean C* instances •  old test keyspaces dropped •  13+TBs of data loaded during read testing •  20 C4.4XL Stress Writers each with their own 1BB sequence © 2015. All Rights Reserved.
  • 24. Cluster Topology © 2015. All Rights Reserved. Stress Node 10 Instances AZ: 1A Stress Nodes 10 Instances AZ: 1B 20 C* Nodes AZ: 1A 20 C* Nodes AZ: 1B 20 C* Nodes AZ: 1C OpsCenter
  • 25. EBS © 2015. All Rights Reserved.
  • 26. Cassandra Stress 2.1.x © 2015. All Rights Reserved. bin/cassandra-stress user duration=100000m cl=ONE profile=/home/ubuntu/summit_stress.yaml ops (insert=1) no-warmup -pop seq=1..1000000000 -mode native cql3 -node 10.10.10.XX -rate threads=1000 -errors ignore!
  • 27. © 2015. All Rights Reserved. PCSTAT - Al Tobey http://www.datastax.com/dev/blog/compaction-improvements-in-cassandra-21 https://github.com/tobert/pcstat
  • 28. © 2015. All Rights Reserved. Netflix Test - What is C* capable of?
  • 29. Netflix Test © 2015. All Rights Reserved. 1+ Million Writes Per second RF:3 3+ Million Local Writes Per second NICE!
  • 30. Netflix Test © 2015. All Rights Reserved.
  • 31. Netflix Test © 2015. All Rights Reserved. No Dropped Mutations, system healthy at 1.1M after 50 mins
  • 32. Netflix Test © 2015. All Rights Reserved. I/O Util is not peggedCommit Disk = Steady!
  • 33. Netflix Test © 2015. All Rights Reserved. Low IO Wait
  • 34. Netflix Test © 2015. All Rights Reserved. 95th Latency = Reasonable
  • 35. Netflix Test - Read Fail © 2015. All Rights Reserved. compression={'chunk_length_kb': '64', 'sstable_compression': 'LZ4Compressor'} https://issues.apache.org/jira/browse/CASSANDRA-10249 https://issues.apache.org/jira/browse/CASSANDRA-8894 Data Drive Pegged L
  • 36. Reading Data •  24 hour read test •  over 10TBs of data in the CF •  sustained > 350K reads per second over 24 hours •  1M reads/per sec peak •  CL ONE •  12 C4.4XL stress boxes © 2015. All Rights Reserved.
  • 37. Reading Data © 2015. All Rights Reserved.
  • 38. Reading Data © 2015. All Rights Reserved.
  • 39. Reading Data © 2015. All Rights Reserved. Not Pegged J
  • 40. Reading Data © 2015. All Rights Reserved. 7.2ms 95th latency
  • 41. Netflix Test resource usage •  180 Less Cores (45 less i2.xlarge instances) •  24 hour test (sans data transfer cost) –  Netflix cluster/stress •  Cost: ~$6300 •  285 i2.xlarge $0.85 per hour –  CrowdStrike cluster/stress with EBS cost •  Cost: ~$2600 •  60 C4.4XL $0.88 per hour
  • 42. Read Notes with EBS •  Our test was a single 10K IOPS volume •  More/Bigger Reads? –  PIOPS gives you as much throughput as you need –  RAID0 multiple EBS volumes /mnt/data EBS Vol1 EBS Vol2
  • 43. © 2015. All Rights Reserved. What Unlocked Performance!
  • 44. Major Tweaks •  Ubuntu HVM types •  Enhanced Networking •  now faster than PVM •  Ubuntu distro tuned for cloud workloads •  XFS Filesystem © 2015. All Rights Reserved.
  • 45. Major Tweaks •  Major Tweaks •  Cassandra 2.1 •  Java 8 •  G1 Garbage Collector - cassandra-env © 2015. All Rights Reserved. https://issues.apache.org/jira/browse/CASSANDRA-7486
  • 46. Major Tweaks •  C4.4XL 16 core, EBS Optimized •  4TB, 10,000 IOPS EBS GP2 Encrypted Data Drive –  160MB/s throughput •  1TB 3000 IOPS EBS GP2 Encrypted Commit Log Drive © 2015. All Rights Reserved.
  • 47. Major Tweaks •  cassandra-env.sh •  MAX_HEAP_SIZE=8G •  JVM_OPTS=“$JVM_OPTS —XX:+UseG1GC” •  Lots of other minor tweaks © 2015. All Rights Reserved.
  • 48. cassandra-env.sh © 2015. All Rights Reserved. Put PID in batch mode Mask CPU0 from the process to reduce context switching Magic From Al Tobey
  • 49. YAML Settings •  cassandra.yaml (based on 16 core) •  concurrent_reads: 32 •  concurrent_writes: 64 •  memtable_flush_writers: 8 •  trickle_fsync: true •  trickle_fsync_interval_in_kb: 1000 •  native_transport_max_threads: 256 •  concurrent_compactors: 4 © 2015. All Rights Reserved.
  • 50. cassandra.yaml © 2015. All Rights Reserved. We found a good portion of the CPU load was being used for internode compression which reduced write throughput internode_compression: none
  • 51. Lessons Learned •  EBS was never the bottleneck during testing, GP2 is legit •  If you’re doing batching, write to the same rowkey in the batch •  Builtin types like list and map come at a performance penalty •  30% hit on our writes using Map type •  DTCS is very young (see Jeff Jirsa’s talk) •  2.1 Stress Tool is tricky but great for modeling workloads •  How will compression affect your read path? © 2015. All Rights Reserved.
  • 52. © 2015. All Rights Reserved. Test your own! https://github.com/CrowdStrike/cassandra-tools
  • 53. It’s just python •  launch 20 nodes in us-east1 •  python launch.py launch --nodes=20 —config=c4-ebs-hvm —az=us-east-1a •  bootstrap the new nodes with C*, RAID/Format disks, etc… •  fab -u ubuntu bootstrapcass21:config=c4-highperf •  run arbitrary commands •  fab -u ubuntu cmd:config=c4-highperf,cmd="sudo rm -rf / mnt/cassandra/data/summit_stress" © 2015. All Rights Reserved.
  • 54. Run custom stress profiles… multi-node support ubuntu@ip-10-10-10.XX:~$ python runstress.py --profile=stress10 —seednode=10.10.10.XX —-threads=50! ! ! Going to run: /home/ubuntu/apache-cassandra-2.1.5/tools/bin/cassandra-stress user duration=100000m cl=ONE profile=/home/ubuntu/summit_stress.yaml ops(insert=1,simple=9) no-warmup -pop seq=1..1000000000 -mode native cql3 -node 10.10.10.XX -rate threads=50 -errors ignore! © 2015. All Rights Reserved. ubuntu@ip-10-10-10.XX:~$ python runstress.py --profile=stress10 --seednode=10.10.10.XX --threads=50! ! Going to run: /home/ubuntu/apache-cassandra-2.1.5/tools/bin/cassandra-stress user duration=100000m cl=ONE profile=/home/ubuntu/summit_stress.yaml ops(insert=1,simple=9) no-warmup -pop seq=1000000001..2000000000 -mode native cql3 -node 10.10.10.XX -rate threads=50 -errors ignore! export NODENUM=1! export NODENUM=2!
  • 55. Where are we today? •  ~3 months on our EBS based cluster •  Hundreds of TBs of graph data and growing in C* •  Billions of vertices/edges •  Changing perceptions?
  • 56. Special thanks to © 2015. All Rights Reserved. •  Leif Jackson •  Marcus King •  Alan Hannan •  Jeff Jirsa •  Al Tobey •  Nick Panahi •  J.B. Langston •  Marcus Eriksson •  Iian Finlayson •  Dani Traphagen
  • 57. EBS heading into 2016 © 2015. All Rights Reserved.
  • 58. 4TB  (10k  IOPS)  GP2   IO  Hit?  Not  enough  to  phase  C*  
  • 59. © 2015. All Rights Reserved.     So  why  the  hate  for  EBS?  
  • 60. © 2015. All Rights Reserved. Following  the  Crowd  –  Trust  Issues     •  Used  instance-­‐store  image  and  ephemeral   drives   •  Painful  to  stop/start  instances,  resize   •  Couldn’t  avoid  scheduled  maintenance  (i.e.   Reboot-­‐a-­‐palooza)   •  EncrypUon  required  shenanigans  
  • 61. © 2015. All Rights Reserved. Guess  What?   •  We  sUll  had  failures   •  Now  we  get  to  rebuild  from  scratch  
  • 62. © 2015. All Rights Reserved. What  do  you  mean  my  volume  is  “stuck”?     •  April  2011  –  Ne[lix,  Reddit  and  Quora   •  October  2012  –  Reddit,  Imgur,  Heroku   •  August  2013  –  Vine,  AirBNB   EBS’s  Troubled  Childhood  
  • 63. © 2015. All Rights Reserved. h`p://techblog.ne[lix.com/2011/04/lessons-­‐ ne[lix-­‐learned-­‐from-­‐aws-­‐outage.html     •  Spread  services  across  mulUple  regions   •  Test  failure  scenarios  regularly  (Chaos  Monkey)   •  Make  Cassandra  databases  more  resilient  by  avoiding   EBS   Kiss  of  Death  
  • 64. © 2015. All Rights Reserved. Amazon  moves  quickly  and  quietly:     •  March  2011  –  New  EBS  GM   •  July  2012  –  Provisioned  IOPs   •  May  2014  –  NaUve  EncrypUon   •  Jun  2014  –  GP2  (game  changer)   •  Mar  2015  –  16TB  /  10K  GP2/  20K  PIOPS       RedempUon  
  • 65. © 2015. All Rights Reserved. •  PrioriUzed  EBS  availability  and  consistency  beyond  features  and   funcUonality   •  Compartmentalized  the  control  plane  -­‐  broke  cross-­‐AZ  dependencies   for  running  volumes   •  Simplified  workflows  to  favor  sustained  operaUon   •  Tested  and  simulated  via  TLA+/PlusCal  -­‐  be`er  understood  corner  cases   •  Dedicated  a  large  fracUon  of  engineering  resources  to  reliability  and   performance     RedempUon  
  • 66. © 2015. All Rights Reserved. Reliability     EBS  Team  targets  99.999%  availability      exceeding  expectaUons  
  • 67. © 2015. All Rights Reserved. Crowdstrike  Today   In  past  12  months,  zero  EBS-­‐related  failures   •  Thousands  of  GP2  data  volumes  (~2PB  data)   •  TransiUoning  all  systems  to  EBS  root  drives   •  Moved  all  data  stores  to  EBS  (C*,  Kapa,   ElasUcsearch,  Postgres,  etc)  
  • 68.
  • 69. © 2015. All Rights Reserved. Staying  Safe  -­‐  Architecture   •  Select  a  region  with  >2  AZs  (e.g  us-­‐east-­‐1  or  us-­‐ west-­‐2)     •  Use  EBS  GP2  or  PIOPs  storage   •  Separate  volumes  for  data  and  commit  logs  
  • 70. © 2015. All Rights Reserved. Staying  Safe  -­‐  Ops   •  Use  EBS  volume  monitoring   •  Pre-­‐warm  EBS  volumes?   •  Schedule  snapshots  for  consistent  backups  
  • 71. © 2015. All Rights Reserved. Most  Importantly   •  Challenge  assumpUons   •  Stay  current  on  AWS  blog   •  Talk  with  your  peers  
  • 72.