We at ShareChat want to describe our journey of migrating our NoSQL use-cases from different databases to cloud agnostic | performant | cost effective ScyllaDB. We had this mammoth task of migrating all the data from existing databases to ScyllaDB with NO downtime. We built a framework which had minimal impact on the application in terms of latency and downtime during the migration process and that has the ability to fallback to the source table in case of any inconsistency during migration. Our framework solved for multiple use-cases including both counter and non-counter use-cases across 3 languages used in ShareChat (NodeJS, GoLang, Java) through shared library. We built recovery and auditing mechanisms to maintain consistency across source and destination tables. There are 3 main components. For the existing data, we have a analytical job based on apache beam which moves the data between source and destination tables. For the live traffic, we have a generic driver which applications use to perform dual writes in both source and destination tables. For any failure scenarios be it in dual writes or any consistency issues, we have a job to re-sync the inconsistent data. During this entire process, we faced many challenges and ended up pushing ScyllaDB to its limit. We tried different consistency levels, compaction strategies, optimised data models and were finally able to move 100 TB of raw data from multiple databases to ScyllaDB with minimal inconsistency.

1. ShareChat's Journey
Migrating 100TB of Data to
ScyllaDB with NO Downtime
Chinmoy Mahapatra - Software Engineer, Platforms
Anuraj Jain - Software Engineer, Platforms

2. ■ Live Migration Framework Overview
■ Dual Writes Deep Dive
■ Handling Conflicts
■ Handling Counters
■ Export Job Overview
■ Export Job Deep Dive
■ Challenges / Learnings
■ Results
Presentation Agenda

3. Live Migration Architecture

4. DB Driver?

5. Dual Writes?

6. Resolving Conflicts

7. Migration Stages
Stage Read Write
Source DB ScyllaDB Source DB ScyllaDB
Dual writes enabled ✅ ❌ ✅ ✅
Export job Running ✅ ❌ ✅ ✅
Audit ✅ ❌ ✅ ✅
Validation ✅ ❌ ✅ ✅
Read Switch ❌ ✅ ✅ ✅
Write Switch ❌ ✅ ❌ ✅

8. What About Counters?

9. What About Counters?

10. Migration Stages (Counters)
Stage Read Write
Source DB ScyllaDB Source DB ScyllaDB
Write to dirty keystore ✅ ❌ ✅ ❌
Export job Running ✅ ❌ ✅ ❌
Dual writes ✅ ❌ ✅ ✔
Migrate dirty keys ✅ ❌ ✅ ✔
Audit ✅ ❌ ✅ ✅
Validation ✅ ❌ ✅ ✅
Read Switch ❌ ✅ ✅ ✅
Write Switch ❌ ✅ ❌ ✅

11. Exporting the Existing Data From
Source to Destination Table

12. Let’s See Some Code
Our job has majorly 3 parts - Read, Transform, Write

13. How To Do a Timestamped Query?
We provide a CustomMapperFactory to our CassandraIO PTransform, which then uses a
CustomObjectMapper (a mapper interface implementation) and writes to ScyllaDB with a
custom timestamped query using a accessor.

14. How To Do a Timestamped Query?

15. Challenges/Learnings
1. Write Consistency | Quorum vs One
2. Partial Data Export with Apache Beam
3. Segregation of base table and materialized
4. Choosing the right compaction strategy | Incremental vs Null
5. Validation for the migrated data from entire source and
destination table
6. Migrating Counters | ensuring consistency

16. ShareChat’s Usage of ScyllaDB
■ We have migrated around 65+ TB of data already. Plan to
onboard additional 50 TB
■ Almost 35-40 services are using ScyllaDB at scale
■ Our biggest cluster size is around 28 TB and our max throughput
for one of our cluster is 1.5 Million Ops/sec

17. Thank You
Stay in Touch
Chinmoy Mahapatra
chinmoymahapatra@sharechat.co
https://github.com/ChinmoyMahapatra
https://www.linkedin.com/in/chinmoy-
s-mahapatra-5458b517b
Anuraj Jain
anurajjain@sharechat.co
https://github.com/anuraj381
https://www.linkedin.com/in/anuraj-jain-3101