Disney+ Hotstar is the fastest growing branch of Disney+. Join Disney+ Hotstar Architect Vamsi Subhash and senior data engineer Balakrishnan Kaliyamoorthy to learn…
How Disney+ Hotstar architected their systems to handle massive data loads
Why they chose to replace both Redis and Elasticsearch
Their requirements for massively scalable data infrastructure and evolving data models
How they migrated their data to Scylla Cloud, ScyllaDB’s fully managed NoSQL database-as-a-service, without suffering downtime
2. About Disney+ Hotstar
https://en.wikipedia.org/wiki/Hotstar
- Disney+ Hotstar is an Indian subscription video on-demand streaming service
- By 2019, we had over 300 million active users monthly
- World record of concurrent viewership: 25.3 million
8. Pre-Scylla Architecture
Event
processor
API server
Send watch
event
Client
Kafka
Redis Elasticsearch
Process event
Save to Data-stores
Buffer read from
Data-stores
Serve response
(500GB) (20TB)
Key: 30 bytes
Value: 5-10 KBs
21. Spark Scylla Migrator
Scylla
opensource
New single-node
Scylla opensource
Backup snapshot to
S3
Take Scylla backups
and move to S3
Restore the backup
In new scylla
open-source cluster
Read via spark
Migrate data to
Scylla-cloud cluster
Scylla cloud
Write to
Scylla cloud
Unirestore
22. Recommendations
● Create new cluster in order to avoid any down-time / impact during spark based
migration.
● Increase parallelism until destination load reaches 100% and we don’t see any write
timeouts.
● Blog : https://www.scylladb.com/2019/03/12/deep-dive-into-the-scylla-spark-migrator/
23. Future plans
● We are looking to move more use cases to Scylla cloud cluster instead of our existing
DBs.
● Use-cases :
○ Watchlist - bucket where user saves content he likes to watch-later
○ Offline recommendation metadata
24. User Watch Store
● Time Series
● Append only writes
● Aggregate on reads
● Expiration
W R
26. User Watch Store
● Time range query on user’s watch
○ Aggregate duplicated watches (watch time / video position / etc.)
○ Expiration of the watches (in some time window)
● Near real-time serving
○ Volume: 1.5M rpm writes at peak
○ API serving online requests
33. Mitigation
● Avoid Queue Like Anti-pattern
○ Exclude tombstone with time range in query
○ Cache friendly with more static time range
Remember the split point,
Query from the it!
UTC
00:00
34. Optimizations: Compaction
● Use consistency to CL=ALL for deletes
● Set GC_GRACE_SECONDS to 0
● Schedule daily once major compactions during off peak hours -> to reduce
tombstones without latency impact during the day time.
35. United States
2445 Faber St, Suite #200
Palo Alto, CA USA 94303
Israel
Maskit 4
Herzliya, Israel 4673304
www.scylladb.com
@scylladb
Thank You!