Sprinklr delivers a complete social media management system for the enterprise. It also helps the world’s largest brands do marketing, advertising, care, sales, research, and commerce on Facebook, Twitter, LinkedIn, and 21 other channels on a global level. This is all done on a single integrated platform. In this session, you learn about Sprinklr’s journey to the cloud and discover how to optimize your NoSQL database on AWS for cost, efficiency, and scale. We also do dive deep into best practices and architectural considerations for designing and managing NoSQL databases, such as Apache Cassandra, MongoDB, Apache CouchDB, and Aerospike on Amazon EC2 and Amazon EBS. We share best practices for instance and volume selection, provide performance tuning hints, and describe cost optimization techniques throughout.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Sprinklr Uses Amazon EBS to Maximize
Its NoSQL Deployment
DAT330
N o v e m b e r , 3 0 , 2 0 1 7
J a m a l M a z h a r , H e a d o f I n f r a s t r u c t u r e a n d D e v O p s , S p r i n k l r
G o p a l a K r i s h n a P a d i d a d a k a l a , P l a t f o r m A r c h i t e c t , S p r i n k l r
A n d r e y Z a y c h i k o v , S r . S o l u t i o n s A r c h i t e c t , A W S

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Why and how to choose NoSQL database
• MongoDB overview & various aspects of large geo-distributed cluster
design
• Sprinklr implementation & challenges, including
• Databases Sprinklr use and their scale
• Mongo DB Deployment Topologies and Scale
• Operational challenges
• Disaster Recovery Process
• Mongo / EBS Performance stats
• Cost Savings
• Security

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Intro: What is the reason for NoSQL rise?
Complexity of data
schemes
Data volume Uptime & Latency

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NoSQL ecosystem
• Hadoop / HBase
• Cassandra
• Scylla
• Hypertable
• Accumulo
• Cloudata
• MonetDB
• HPCC
• Apache Flink
• Splice Machine
• eXtremeDB
• ConcourseDB
• Druid
• KUDU
• Elassandra
• ArangoDB
• OrientDB
• gunDB
• MongoDB
• RethinkDB
• Couchbase Server
• CouchDB
• ToroDB
• SequoiaDB
• NosDB
• RavenDB
• MarkLogic Server
• Clusterpoint Server
• JSON ODM
• NeDB
• Terrastore
• AmisaDB:
• JasDB
• RaptorDB
• djondb
• EJDB
• densodb
• SisoDB
• SDB
• NoSQL embedded db
• ThruDB
• iBoxDB
• BergDB
• ReasonDB
• BagriDB
• Riak
• Redis
• Aerospike
• LevelDB
• RocksDB
• Berkeley DB
• GenieDB
• BangDB
• Chordless
• Scalaris
• Tokyo Cabinet / Tyrant
• Scalien
• Voldemort
• Dynomite
• KAI
• MemcacheDB
• Faircom C-Tree
• LSM
• KitaroDB
• upscaledb
• STSdb
• Tarantool/Box
• Chronicle Map
• Maxtable
• quasardb
• Pincaster
• RaptorDB
• allegro-C
• nessDB
• HyperDex
• SharedHashFile
• Symas LMDB
• Sophia
• NCache
• TayzGrid
• PickleDB
• Mnesia
• LightCloud
• Hibari
• OpenLDAP
• Genomu
• BinaryRage
• Elliptics
• DBreeze
• TreodeDB
• BoltDB
• Serenety
• Cachelot
• filejson
• InfinityDB
• SCR Siemens Common
Repository
• Neo4J
• ArangoDB, OrientDB
• Infinite Graph
• Sparksee
• TITAN
• InfoGrid
• HyperGraphDB
• GraphBase
• Trinity
• AllegroGraph
• BrightstarDB
• Meronymy
• WhiteDB
• Onyx Database
• OpenLink Virtuoso
• VertexDB
• FlockDB
• BrightstarDB
• Execom IOG
• Fallen 8

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NoSQL options on AWS
Amazon DynamoDB Amazon S3 + Amazon
Athena
Amazon EC2 +
Amazon EBS
Amazon DynamoDB is a fast and
flexible NoSQL database service for
all applications that need consistent,
single-digit millisecond latency at
any scale. It is a fully managed cloud
database and supports both
document and key-value store
models. Its flexible data model,
reliable performance, and automatic
scaling of throughput capacity,
makes it a great fit for mobile, web
and many other applications.
Amazon Athena is an
interactive query service that
makes it easy to analyze data
in Amazon S3 using standard
SQL. Athena is serverless, so
there is no infrastructure to
manage, and you pay only
for the queries that you run.
This makes it easy for anyone
with SQL skills to quickly
analyze large-scale datasets.
Amazon Elastic Compute Cloud
(Amazon EC2) is a web service that
provides secure, resizable compute
capacity in the cloud. It is designed to
make web-scale cloud computing
easier for developers.
Amazon EC2’s simple web service
interface allows you to obtain and
configure capacity with minimal
friction.

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Do you really need distributed NoSQL
clusters?
• Unstable & High Load
• No crucial requirements for data consistency
• High Uptime
• Simple data scheme without a large number of
dependencies
• No need to build reports on the fly out of the
whole amount of data
• Latency is crucial and customers are widely
distributed across multiple geos
• Close to 100% data consistency is absolute must
• Stable & predicable load without significant
changes for short period of time
• Well-defined and stable data schema
• Applications & clients can tolerate higher latency
& some deviations there
• There is a need to build reports & queries on
particular data model using full functionality of
SQL
Probably YES Definitely NOT

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Choosing NoSQL technology: Overview

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Crash course: MongoDB

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sprinklr Overview

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sprinklr Architecture

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sprinklr Deployment

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sprinklr Platform—Key technologies

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sprinklr backend architecture evolution

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Databases in Sprinklr and their scale
1320 Nodes
947 Nodes
383 Nodes70 Nodes

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Choosing the right geo for deployment

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Designing the deployment architecture

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example: Designing the deployment
architecture

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application interactions with DB

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application interactions with MongoDB
When to use L4 LB
• Geo-distributed mongos processes
• HA is critical and you experience
multiple mongos failures
Issues with configuration
ReplicaSet
Common error handling issues
• Requests throttling
• Driver errors / issues
Common issues with DB requests
• Slow requests
• Schema issues
• Connectivity

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scale of MongoDB deployment
• 900+ servers, 900 EBS volumes
• 200+ TB of Primary data
• 250 replica sets, 20 clusters, 30
Shards on couple of big clusters, 15 TB
to 30 TB of Primary Data on each big
cluster
• ~2M mongo operations/second
across mongo deployment
• Daily 270 data volume snapshots in
less than 30 min for backup purpose.
• ~30 TB of snapshots delta sync
between east region to west region for
Disaster Recovery purpose

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Planning HA & DR
HA perspective
• Failing over on DB level &
application level
• Routing & configuration for cluster
& application
• Network is prepared for event
handling
• Handling requests increase
DR perspective
• Restoring from the backup
• Preserving data in multiple
locations
• Restoring configuration & auth
data

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HA & DR in 2 AZ region
Master elections procedure
• Default configuration
• Election Weights
HA perspective for 2 AZ failure
• 2 major scenarios
• Local read in worst case scenario
• Proper elections weights set-up
Possible options for 2 AZ
deployments
• Classic set-up with predictable
elections
• Master + Hidden
• Remote node in a replica set

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ReplicaSet topology
Vote :1
Primary
us-east-1b
Vote : 1
Secondary
us-east-1e
Vote : 1
Arbiter
us-east-1a
Vote : 0
HS
us-east-1b
Replication
Replication
Heartbeat

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Backups
Mongo Dump
• Incremental dumps are
not supported
• 50 GB/hour
EBS snapshots
• Incremental multiple
consistence backups
• Backup Process time <
10 Min
• Lock/stop : 2+ hours to
< 1 min
• Cost
File system copy by
using rsync
• Backup Process time
2+ hours
• Lock/stop for 2+
hours
• only one backup /
backup corruption

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Node Recovery
Option 1: Copy data
from EBS snapshot to
instance store
Option 2: Mongo
Sync
OR

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration to EBS
Migration to io1 – PIOPS
• Data node recovery reduced from 4 days to 1 - 2 hours
• Volume size
• Instance type scaling
• Oplog size decreased from 40% of volume size to 10% of
volume size.
Migration to gp2

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Disaster Recovery Process

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MongoDB startup—How to optimize?

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Startup time speedup

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Workload—Over 2 million operations / sec

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Write workload

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Write Latency

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Read Latency

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cost Savings
• hi1.4xlarge to r3.2xlarge and now r4.2xlarge servers
• Backups cost reduced because of EBS snapshots
• Oplog disk reduced from 40% of Volume size to 10%
of Volume size
• WiredTiger Compression
• PIOPS to gp2
• Dynamic Volume size changes
• Storage cost reduced from $155k to $53k =~ 66%
savings

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security
EBS encryption – No
noticeable
performance impact

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Magic of cross-region data transfer with
encryption

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Choosing instance and storage type
Database implementation, data schema and access patterns should always be considered.
Compute and storage types should always be adapted to particular situation and can change
during DB lifetime.
Cassandra MongoDB

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Choosing instance and storage type
• If you want to be always cost-
effective and efficient than
deployment is a journey for you
• Consider Amazon EBS as main
option for most of the workloads
• If your performance requirements
are really high and the size of the
dataset is relatively low—consider
Amazon EC2 with ephemerals,
otherwise—go for Amazon EC2 with
Amazon EBS

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Summary—How Amazon EBS helped us
Daily incremental
consistence multiple
backups for 250+
volumes, 200 TB of
data backed up in < 30
min
DR process achieved with the acceptable RPO, RTOs
• Bandwidth capacity increase by AWS team
• Increase in limits of number of parallel snapshots we can take and transfer
• Reduced cross region sync time from 30 hours to less than 8 hours
High Availability SLA
improved
• Single digit/sub ms
Read/Write latencies
66% cost savings on
Mongo storage
• start small and
dynamically increase of
volume size as needed
• use of gp2 volumes
instead of PIOPS

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!