Presenter: Roopa Tangirala, Senior Cloud Data Architect at Netflix High availability is an important requirement for any online business and trying to architect around failures and expecting infrastructure to fail, and even then be highly available, is the key to success. One such effort here at Netflix was the Active-Active implementation where we provided region resiliency. This presentation will discuss the brief overview of the active-active implementation and how it leveraged Cassandra’s architecture in the backend to achieve its goal. It will cover our journey through A-A from Cassandra’s perspective, the data validation we did to prove the backend would work without impacting customer experience. The various problems we faced, like long repair times and gc_grace settings, plus lessons learned and what would we do differently next time around, will also be discussed.

2. ABOUT NETFLIX!

3. NETFLIX

5. ACTIVE - ACTIVE!

6. WHAT IS ACTIVE ACTIVE?
Also called dual active, it is a phrase used to describe a
network of independent processing nodes where each node has
access to replicated database. Traffic intended for a failed node
is either passed onto an existing node or load balanced across
the remaining nodes.

7. WHY ACTIVE-ACTIVE ?!
ENTERPISE IT
SOLUTIONS
WEB SCALE
CLOUD
SOLUTIONS
RAPID
SCALING
HIGH
AVAILABILITY

8. DOES AN INSTANCE FAIL?!
• It can, plan for it!
• Bad code / configuration pushes!
• Latent issues!
• Hardware failure!
• Test with Chaos Monkey!

9. DOES A ZONE FAIL?!
• Rarely, but happened before!
• Routing issues!
• DC-specific issues!
• App-specific issues within a zone!
• Test with Chaos Gorilla!

10. DOES A REGION FAIL?!
• Full region – unlikely, very rare!
• Individual Services can fail region-wide!
• Most likely, a region-wide configuration issues!
• Test with Chaos Kong!

11. EVERYTHING FAILS… EVENTUALLY!
• Keep your services running by embracing isolation and
redundancy!
• Construct a highly agile and highly available service
from ephemeral and assumed broken components!

12. ISOLATION!
• Changes in one region should not affect others!
• Regional outage should not affect others!
• Network partitioning between regions should not affect
functionality / operations!

13. REDUNDANCY!
• Make more than one (of pretty much everything)!
• Specifically, distribute services across Availability
Zones and regions!

14. HISTORY: X-MAS EVE 2012!
• Netflix multi-hour outage!
• US-East1 regional Elastic Load Balancing issue!
!
• “...data was deleted by a maintenance process
that was inadvertently run against the
production ELB state data”!

15. ACTIVE-ACTIVE ARCHITECTURE!

18. THE PROCESS!

19. IDENTIFYING CLUSTERS FOR AA
!

20. SNITCH CHANGES!
EC2Snitch! EC2MultiRegionSnitch!
Uses Private IPs! Uses Public IPs!

21. PRIAM.MULTIREGION.ENABLE =TRUE!
storage_port : Using Private IPs!
ssl_storage_port : Using Public IPs!

23. SPIN UP NODES IN NEW REGION!
us-east-1! us-west-2!
APP

24. UPDATE KEYSPACE!
Update keyspace <keyspace> with placement_strategy =
'NetworkTopologyStrategy'!
and strategy_options = {us-east : 3, us-west-2 : 3};!
Existing region and replication factor ! New region and replication factor!

25. REBUILD NEW REGION
Run – nodetool rebuild us-east-1 on all us-west-2 nodes

26. RUN NODETOOL REPAIR

27. VALIDATION!

28. BENCHMARKING GLOBAL CASSANDRA
WRITE INTENSIVE TEST OF CROSS-REGION REPLICATION
CAPACITY
16 X HI1.4XLARGE SSD NODES PER ZONE = 96 TOTAL
192 TB OF SSD IN SIX LOCATIONS UP AND RUNNING
CASSANDRA IN 20 MINUTES!
US-West-2 Region - Oregon
Zone A
Cassandra Replicas
Zone B
Cassandra Replicas
Zone C
Cassandra Replicas
US-East-1 Region - Virginia
Zone A
Cassandra Replicas
Zone B
Cassandra Replicas
Zone C
Cassandra Replicas
Test
Load
Test
Load
Validation
Load
Interzone Traffic 1 Million Writes!
CL.ONE (Wait for One
Replica to ack)!
1 Million Reads!
after 500 ms!
CL.ONE with No!
Data Loss!
Interregional Traffic!
Up to 9Gbits/s, 83ms! 18 TB backups
from S3

29. TEST FOR THUNDERING HERD!

30. TEST FOR RETRIES!
FAILURE
RETRY

31. KEY METRICS USED!
• 99 /95 th Read Latency (Client & C*)!
• Dropped Metrics on C*!
• Exceptions on C*!
• Heap Usage on C*!
• Threads Pending on C*!

32. CONFIGURATION FOR TEST!
• 24 Node C* SSDs!
• 220 Client instances!
• 70+ Jmeter Instances!

33. C* IOPS

34. TOTAL READ IOPS
TOTAL WRITE IOPS

35. 95th LATENCY
99th LATENCY

36. CHECK FOR CEILING!

37. NETWORK PARTITION!
us-east-1 us-west-2

38. TAKEAWAYS!

39. REPAIRS AFTER EXTENSION ARE PAINFUL !!!

40. TIME TO REPAIR DEPENDS ON!
• Number of regions!
• Number of replicas!
• Data size!
• Amount of entropy!
!

41. ADJUST GC_GRACE AFTER
EXTENSION!
• Column Family Setting!
• Defined in seconds!
• Default 10 days!
• Tweak gc_grace settings to
accommodate time taken to repair!
• BEWARE of deleted columns!

42. RUNBOOK!

43. PLAN FOR CAPACITY!

44. CONSISTENCY LEVEL !
• Check the client for consistency level setting!
• In a Multiregional cluster QUORUM <>
LOCAL_QUORUM!
• Recommended consistency levels
LOCAL_ONE (CASSANDRA-6202) for reads
and LOCAL_QUORUM for writes!
• For region resiliency avoid – ALL or
QUORUM calls!

45. CREATE CHAOS!!!
HOW DO WE KNOW IT WORKS?

46. Benchmark …!
!
Time Consuming !
!
But worth it!!