GumGum relies heavily on Cassandra for storing different kinds of metadata. Currently GumGum reaches 1 billion unique visitors per month using 3 Cassandra datacenters in Amazon Web Services spread across the globe. This presentation will detail how we scaled out from one local Cassandra datacenter to a multi-datacenter Cassandra cluster and all the problems we encountered and choices we made while implementing it. How did we architect multi-region Cassandra in AWS? What were our experiences in implementing multi-datacenter Cassandra? How did we achieve low latency with multi-region Cassandra and the Datastax Driver? What are the different Cassandra use cases at GumGum? How did we integrate our Cassandra with Spark?

1. CASSANDRA SUMMIT 2015CASSANDRA SUMMIT 2015
Mario Lazaro
September 24th 2015
#CassandraSummit2015
MULTI-REGION CASSANDRA IN AWSMULTI-REGION CASSANDRA IN AWS

2. WHOAMIWHOAMI
Mario Cerdan Lazaro
Big Data Engineer
Born and raised in Spain
Joined GumGum 18 months ago
About a year and a half experience with
Cassandra

3. #5 Ad Platform in the U.S
10B impressions / month
2,000 brand-safe premium
publisher partners
1B+ global unique visitors
Daily inventory Impressions
processed - 213M
Monthly Image Impressions
processed - 2.6B
123 employees in seven offices

4. AGENDAAGENDA
Old cluster
International Expansion
Challenges
Testing
Modus Operandi
Tips
Questions & Answers

5. 25 Classic nodes cluster
1 Region / 1 Rack hosted in
AWS EC2 US East
Version 2.0.8
Datastax CQL driver
GumGum's metadata including
visitors, images, pages, and ad
performance
Usage: realtime data access
and analytics (MR jobs)
OLD C* CLUSTER - MARCH 2015OLD C* CLUSTER - MARCH 2015

6. OLD C* CLUSTER - REALTIME USE CASEOLD C* CLUSTER - REALTIME USE CASE
Billions of rows
Heavy read workload
60/40
TTLs everywhere - Tombstones
Heavy and critical use of counters
RTB - Read Latency constraints
(total execution time ~50 ms)

7. OLD C* CLUSTER - ANALYTICS USE CASEOLD C* CLUSTER - ANALYTICS USE CASE
Daily ETL jobs to extract / join data
from C*
​Hadoop MR jobs
AdHoc queries with Presto

8. INTERNATIONAL EXPANSIONINTERNATIONAL EXPANSION

9. FIRST STEPSFIRST STEPS
Start C* test datacenters in US
East & EU West and test how C*
multi region works in AWS
Run capacity/performance tests.
We expect 3x times more traffic
in 2015 Q4
FIRST THOUGHTSFIRST THOUGHTS
Use AWS Virtual Private
Cloud (VPC)
Cassandra & VPC present
some connectivity
issues challenges
Replicate entire data
with same number of replicas

10. TOO GOOD TO BE TRUE ...TOO GOOD TO BE TRUE ...

11. CHALLENGESCHALLENGES
Problems between Cassandra in EC2
Classic / VPC and Datastax Java
driver
EC2MultiRegionSnitch uses public
IPs. EC2 instances do not have an
interface with public IP address -
Cannot connect between instances in
the same region using Public IPs
/**
* Implementation of {@link AddressTranslater} used by the driver that
* translate external IPs to internal IPs.
* @author Mario <mario@gumgum.com>
*/
public class Ec2ClassicTranslater implements AddressTranslater {
private static final Logger LOGGER = LoggerFactory.getLogger(Ec2Cla
private ClusterService clusterService;
private Cluster cluster;
private List<Instance> publicDnss;
@PostConstruct
public void build() {
publicDnss = clusterService.getInstances(cluster);
}
/**
* Translates a Cassandra {@code rpc_address} to another address if
* <p>
*
* @param address the address of a node as returned by Cassandra.
* @return {@code address} translated IP address of the source.
*/
public InetSocketAddress translate(InetSocketAddress address) {
for (final Instance server : publicDnss) {
if (server.getPublicIpAddress().equals(address.getHostStrin
LOGGER.info("IP address: {} translated to {}", address.
return new InetSocketAddress(server.getPrivateIpAddress
}
}
return null;
}
public void setClusterService(ClusterService clusterService) {
this.clusterService = clusterService;
}
public void setCluster(Cluster cluster) {
this.cluster = cluster;
}
}
Problems between Cassandra in EC2
Classic / VPC and Datastax Java
driver
EC2MultiRegionSnitch uses public
IPs. EC2 instances do not have an
interface with public IP address -
Cannot connect between instances in
the same region using Public IPs
Region to Region connectivity will use
public IPs - Trust those IPs or use
software/hardware VPN
Problems between Cassandra in EC2
Classic / VPC and Datastax Java
driver
EC2MultiRegionSnitch uses public
IPs. EC2 instances do not have an
interface with public IP address -
Cannot connect between instances in
the same region using Public IPs
Region to Region connectivity will use
public IPs - Trust those IPs or use
software/hardware VPN
Your application needs to connect to
C* using private IPs - Custom EC2
translator

12. Datastax Java Driver Load Balancing
Multiple choices
DCAware + TokenAware
Datastax Java Driver Load Balancing
Multiple choices
DCAware + TokenAware + ?
Datastax Java Driver Load Balancing
Multiple choices
CHALLENGES
“ Clients in one AZ attempt to always communicate with C*
nodes in the same AZ. We call this zone-aware connections. This
feature is built into , Netflix’s C* Java client library.Astyanax

13. Zone Aware Connection:
Webapps in 3 different AZs: 1A, 1B, and 1C
C* datacenter spanning 3 AZs with 3 replicas
CHALLENGESCHALLENGES
1A
1B
1C
1B1B

14. We added it! - Rack/AZ awareness to
TokenAware Policy
CHALLENGESCHALLENGES

15. CHALLENGESCHALLENGES
Third Datacenter: Analytics
Do not impact realtime data access
Spark on top of Cassandra
Spark-Cassandra Datastax connector
Replicate specific keyspaces
Less nodes with larger disk space
Settings are different
Ex: Bloom filter chance

16. CHALLENGESCHALLENGES
Third Datacenter: Analytics
Cassandra Only DC
Realtime
Cassandra + Spark DC
Analytics

17. CHALLENGESCHALLENGES
Upgrade from 2.0.8 to 2.1.5
Counters implementation is buggy in
pre-2.1 versions
“ My code never has bugs. It just develops random
unexpected features

18. CHALLENGESCHALLENGES
“ To choose, or not to choose VNodes. That is the question.
(M. Lazaro, 1990 - 2500)
Previous DC using Classic Nodes
Works with MR jobs
Complexity for adding/removing nodes
Manual manage token ranges
New DCs will use VNodes
Apache Spark + Spark Cassandra Datastax
connector
Easy to add/remove new nodes as traffic
increases

19. TESTINGTESTING

20. TESTINGTESTING
Testing requires creating and modifying
many C* nodes
Create and configuring a C* cluster
is time-consuming / repetitive task
Create fully automated process for
creating/modifying/destroying
Cassandra clusters with Ansible
# Ansible settings for provisioning the EC2 instance
---
ec2_instance_type: r3.2xlarge
ec2_count:
- 0 # How many in us-east-1a ?
- 7 # How many in us-east-1b ?
ec2_vpc_subnet:
- undefined
- subnet-c51241b2
- undefined
- subnet-80f085d9
- subnet-f9138cd2
ec2_sg:
- va-ops
- va-cassandra-realtime-private

21. TESTING - PERFORMANCETESTING - PERFORMANCE
Performance tests using new
Cassandra 2.1 Stress Tool:
Recreate GumGum metadata /
schemas
Recreate workload and make
it 3 times bigger
Try to find limits / Saturate
clients
# Keyspace Name
keyspace: stresscql
#keyspace_definition: |
# CREATE KEYSPACE stresscql WITH replication = {'class': #'
### Column Distribution Specifications ###
columnspec:
- name: visitor_id
size: gaussian(32..32) #domain names are relatively
population: uniform(1..999M) #10M possible domains to pic
- name: bidder_code
cluster: fixed(5)
- name: bluekai_category_id
- name: bidder_custom
size: fixed(32)
- name: bidder_id
size: fixed(32)
- name: bluekai_id
size: fixed(32)
- name: dt_pd
- name: rt_exp_dt
- name: rt_opt_out
### Batch Ratio Distribution Specifications ###
insert:
partitions: fixed(1) # Our partition key is the v
select: fixed(1)/5 # We have 5 bidder_code per dom
batchtype: UNLOGGED # Unlogged batches
#
# A list of queries you wish to run against the schema
#

22. TESTING - PERFORMANCETESTING - PERFORMANCE
Main worry:
Latency and replication overseas
Use LOCAL_X consistency levels in your client
Only one C* node will contact only one C* node in a
different DC for sending replicas/mutations

23. TESTING - PERFORMANCETESTING - PERFORMANCE
Main worries:
Latency

24. TESTING - INSTANCE TYPETESTING - INSTANCE TYPE
Test all kind of instance types. We decided to
go with r3.2xlarge machines for our cluster:
60 GB RAM
8 Cores
160GB Ephemeral SSD Storage for commit logs and
saved caches
RAID 0 over 4 SSD EBS Volumes for data
Performance / Cost and GumGum use
case makes r3.2xlarge the best option
Disclosure: I2 instance family is the best if
you can afford it

25. TESTING - UPGRADETESTING - UPGRADE
Upgrade C* Datacenter from 2.0.8 to 2.1.5
Both versions can cohabit in the same DC
New settings and features tried
​DateTieredCompactionStrategy:
Compaction for Time Series Data
Incremental repairs
Counters new architecture

26. MODUS OPERANDIMODUS OPERANDI

27. MODUS OPERANDIMODUS OPERANDI
Sum up
From: One cluster / One DC in US East
To: One cluster / Two DCs in US East and one DC
in EU West

28. MODUS OPERANDIMODUS OPERANDI
First step:
Upgrade old cluster snitch from EC2Snitch to
EC2MultiRegionSnitch
Upgrade clients to handle it (aka translators)
Make sure your clients do not lose connection to upgraded
C* nodes (JIRA DataStax - )JAVA-809

29. MODUS OPERANDIMODUS OPERANDI
Second step:
Upgrade old datacenter from 2.0.8 to 2.1.5
nodetool upgradesstables (multiple nodes at a time)
Not possible to rebuild a 2.1.X C* node from a 2.0.X C*
datacenter.
rebuild
WARN [Thread-12683] 2015-06-17 10:17:22,845 IncomingTcpConnection.java:91 -
UnknownColumnFamilyException reading from socket;
closing org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=XXX

30. MODUS OPERANDIMODUS OPERANDI
Third step:
Start EU West and new US East DCs within the same
cluster
Replication factor in new DCs: 0
Use dc_suffix to differentiate new Virginia DC from old one
Clients do not talk to new DCs. Only C* knows they exist
Replication factor to 3 on all except analytics 1
​​Start receiving new data
Nodetool rebuild <old-datacenter>
​Old data

31. MODUS OPERANDIMODUS OPERANDI
RF 3
Clients
RF 3:0:0:0RF 3:3:3:1
US East Realtime
EU West Realtime
US East Analytics
Rebuild
Rebuild
Rebuild

32. From 39d8f76d9cae11b4db405f5a002e2a4f6f764b1d Mon Sep 17 00:00:00 2001
From: mario <mario@gumgum.com>
Date: Wed, 17 Jun 2015 14:21:32 -0700
Subject: [PATCH] AT-3576 Start using new Cassandra realtime cluster
---
src/main/java/com/gumgum/cassandra/Client.java | 30 ++++------------------
.../com/gumgum/cassandra/Ec2ClassicTranslater.java | 30 ++++++++++++++--------
src/main/java/com/gumgum/cluster/Cluster.java | 3 ++-
.../resources/applicationContext-cassandra.xml | 13 ++++------
src/main/resources/dev.properties | 2 +-
src/main/resources/eu-west-1.prod.properties | 3 +++
src/main/resources/prod.properties | 3 +--
src/main/resources/us-east-1.prod.properties | 3 +++
.../CassandraAdPerformanceDaoImplTest.java | 2 --
.../asset/cassandra/CassandraImageDaoImplTest.java | 2 --
.../CassandraExactDuplicatesDaoTest.java | 2 --
.../com/gumgum/page/CassandraPageDoaImplTest.java | 2 --
.../cassandra/CassandraVisitorDaoImplTest.java | 2 --
13 files changed, 39 insertions(+), 58 deletions(-)
MODUS OPERANDIMODUS OPERANDI
Start using new Cassandra DCs

33. RF 3:3:3:1
MODUS OPERANDIMODUS OPERANDI
Clients
US East Realtime
EU West Realtime
US East Analytics
RF 0:3:3:1

34. MODUS OPERANDIMODUS OPERANDI
Clients
US East Realtime
EU West Realtime
US East Analytics
RF 0:3:3:1
RF 3:3:1Decomission

35. TIPSTIPS

36. TIPS - AUTOMATED MAINTENANCETIPS - AUTOMATED MAINTENANCE
Maintenance in a multi-region C* cluster:
Ansible + Cassandra maintenance keyspace +
email report = zero human intervention!
CREATE TABLE maintenance.history (
dc text,
op text,
ts timestamp,
ip text,
PRIMARY KEY ((dc, op), ts)
) WITH CLUSTERING ORDER BY (ts ASC) AND
bloom_filter_fp_chance=0.010000 AND
caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
comment='' AND
dclocal_read_repair_chance=0.100000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.000000 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
CREATE INDEX history_kscf_idx ON maintenance.history (kscf);
3-133-65:/opt/scripts/production/groovy$ groovy CassandraMaintenanceCheck.groovy -dc us-east-va-realtime -op compaction -e mari

37. TIPS - SPARKTIPS - SPARK
Number of workers above number of total C*
nodes in analytics
Each worker uses:
1/4 number of cores of each instance
1/3 total available RAM of each instance
Cassandra-Spark connector
​SpanBy
.joinWithCassandraTable(:x, :y)
Spark.cassandra.output.batch.size.bytes
Spark.cassandra.output.concurrent.writes

38. val conf = new SparkConf()
.set("spark.cassandra.connection.host", cassandraNodes)
.set("spark.cassandra.connection.local_dc", "us-east-va-analytics")
.set("spark.cassandra.connection.factory", "com.gumgum.spark.bluekai.DirectLinkConnectionFactory")
.set("spark.driver.memory","4g")
.setAppName("Cassandra presidential candidates app")
TIPS - SPARKTIPS - SPARK
Create "translator" if using EC2MultiRegionSnitch
Spark.cassandra.connection.factory

39. SINCE C* IN EU WEST ...SINCE C* IN EU WEST ...
US West Datacenter!
EU West DC US East DC Analytics DC US West DC

40. Q&AQ&A
GumGum is hiring!
http://gumgum.com/careers