Deck from BigData MeetUp - Bangalore. Presents challenges faced in Migrating Hadoop cluster with 200+ TB compressed data from one data centre to another, from baremetals to cloud, from DNS based systems to infra without DNS.

1. DC Migration and Hadoop
Scale For Big Billion Days
Presented By:
Rahul Agarwal, Operation Engineer, Flipkart
Dhiraj Kumar, Software Development Engineer, Flipkart
Chetna Chaudhari, Software Development Engineer, Flipkart

2. ● DC Landscape Before Migration
● Data at Flipkart - FDP Stats
● Infrastructure Challenges
● Hadoop Clusters Iterations
● Data Migration - Challenges
● Data Migration - Utilities Considered and Developed
● Data Migration - Execution
Agenda

3. 1 Gbps
Shared
10 Gbps
Shared
Primary: All User,
Order & FDP
systems
Secondary: Few batch
processing systems (User
Insights, Ads and
Recommendations)
New: All User
and
FDP systems
DC Landscape

4. Flipkart Data Platform in the old DC
● ~340 nodes
● ~1.7 PB storage
● ~30TB RAM
● ~11000 cores

5. Flipkart Data Platform in the new DC
● ~1000 nodes
● ~30 PB storage
● ~75TB RAM
● ~32000 cores

6. Data at Flipkart - FDP Stats
● New data ingested daily
o ~6 TB on a business-as-usual day
o ~30 TB on sale days
● Number of raw data streams ~1000s
● Number of raw events in a day ~ 3 Billion
● Volume of data processed daily ~ 0.6 PB
● Number of Hadoop jobs run each day ~ 10K

7. InfraStructure Challenges
C1: Validation of the Hardware in entirely new
infrastructure
S1: Ripper Utility based on MR framework doing native
writes/reads on all disks on all datanodes.
C2: Ephemeral IP Addresses and Lack of DNS
S2: Managed Config Service:
• Store Key/Value Pairs - HostName/IP pairs
• ConfD implementation on all clients

8. InfraStructure Challenges - Contd.
C3: Ephemeral Disks and Nodes
S3.1: Quorum of 5 for ZK/JN
S3.2: Isolated Deployments for Each Component
C4: Lesser Memory on NameNode - 180mn FS Objects
S4.1: Delete Zero Size Files.
S4.2: Switching To G1 Helped Reduce Pause Times.

9. Hadoop Cluster Iterations - RedPill
• Utility Around IaaS and Ambari:
– Acquire Instance Types as requested
– Setup MySQL, Ambari Server and Agents
– Determine cluster configurations based on Instance Types
– Co-Host master components for Dev/Test deployments and Isolated
components for Prod setup.
– Generate Blueprint and Cluster Templates
– Deploy Cluster with Blueprint and batch of 50 nodes at first
– Horizontally scale cluster by adding further batches to prevent Repo
service related failures
– TAT of ~20 mins for 100 node cluster

10. Data Migration

11. Data Migration Challenges
• Data publishers/consumers not moving together
– Data consumers could move earlier than the
publishers or vice-versa.
• Migrating PBs of data not feasible over network
• Consistency for raw, prepared and reporting data

12. • Moving Disks from one center to another
– Data centers in different states - Legal Challenges
– Live data , 24/7 in use for analytics
• Replicate the data (Copy, Mirror and Regenerate)
– Files being created and deleted continuously
– Build the supporting services for scale
Solutions Considered | Data Migration

13. Data Migration Utilities- DistCp
• Small Files Performance
• Takes long to build index
• Hard to Figure out Corruption/Copy Aborts
• Content Based Data Validation is weak (CRC)

14. Data Migration Utilities - Transporter
• Configurable Batch Sizes
• Compression at Source
• MD5 sum of the content
• HAR to bundle small files
• DistCP HAR in Binary Mode
• UnHAR at Destination
• MR Validation for MD5 sum of the content
• Regenerate Production Hierarchy
• File Counts Verification

15. Data Migration Utilities - BlueShift
• OSS : https://github.com/flipkart-incubator/blueshift
• Features:
– On the fly compression
– Bulk migration with batches of over 10 Million files
– State management options, either HDFS or DB
– Optimized task scheduling
– Capable of using different protocols for source and destination.
– MD5 based checksum to ensure no corruptions
– Time based file filtering
– filesize based filtering
– Option to ignore exceptions and continue processing

16. Data Replication
• Only copied raw data about O(100TB) compressed
• Too many small files
• some files were very large
– All prepared and reports data generated from raw data
• Propagated delta changes using an Apache Kafka mirror
• Verification utilities to check correctness in data in both clusters
• Ran the full data platform stack in both places for over 2 weeks till
all data publishers and consumers move

17. 2 way sync of Kafka Streams
Mirror
Old DC New DC
Kafka A Kafka C
Kafka B

18. HBase Migration | Solutions
• Copy Table
– issues:
• Full table scan - time consuming
• Secure to UnSecure not supported
• HBase Import/ Exports
– issues:
• Full table scan
• Slower than copy table
• Needs manual interventions
• Extra space
• Decompression while export
• Use Blue-Shift and HBase Bulk loader

19. Blue-Shift + HBase Bulk Load
HDFS
FTP
Disks
Transferred
Over Road
- Trucks
HBase
HDFS

20. Blue-Shift + HBase Bulk Load
• Moved snapshots of derived/computed data over wire (relatively
small)
• Used physical disks to move data (stored in HBase)
• Avoided HBase export. Instead transferred HFiles into disks using
blueshift
– knapsack'ed ~50K files into dozens of physical hard disks
• Disks shipped to new DC
• Transferred HFiles into HDFS using Blueshift

21. Learnings
• MD5 checksum - big win
• Should have used workflow
• Automated process is must
• Having isolations per tenant is must
Achievement :)
• Migration without downtime

22. Thank You !!
Questions ??
@cchaudhari11
@rahul67
@dhiraj2kumar