Twitter stores hundreds of petabytes in multiple datacenters, on multiple Hadoop clusters. ViewFileSystem makes the interaction with our HDFS infrastructure as simple as a single namespace spanning all datacenters and clusters. Users no longer need to use any lengthy URI's. Instead they access files using a path as simple as /datacenter/cluster/user/cecily. We greatly extended the functionality behind ViewFileSystem to 1) automatically detect and add new underlying primitive namespaces (such as the local node filesystem and an HDFS namespace) to the global namespace 2) provide a namespace merge functionality allowing us to split existing namespaces transparently when they are full 3) enable synchronous writes to multiple datacenters simultaneously and transparent failover for reads. This effort resulted in the productivity boost for both: hadoop users and the @TwitterHadoop team, due to much greater user experience and higher reliability. It also contributed to reduction of technical debt as we almost eliminated the usage of web-based Hftp filesystem and migrated most use cases to the native HDFS clients with HA enabled. We'll conclude our presentation with our solutions of offering enhanced metrics and different SLAs in cross-datacenter traffic in Twitter's multi-tenant datacenter infrastructure.

3. ○
○
○
○

4. ○
○
○
○

6. Command Line Interface
hadoop —config /etc/hadoop/revenue-dcA fs -ls /
hadoop —config /etc/hadoop/dw-dcB fs -ls /
……
Directory layout on local file system
/etc/hadoop
revenue-dcA
dw-dcB
core-site.xml, hdfs-site.xml
Log processing
MR
YARN
HDFS
Revenue
MR
YARN
HDFS
Data Warehouse
MR
YARN
HDFS
Log processing
MR
YARN
HDFS
DataCenter A DataCenter B

7. ○
○
○

8. ○
○
○
○
○

9. DW @ dcB
HDFS blocks
Client machine: /etc/hadoop/dw-dcB/hdfs-site.xml, core-site.xml

10. DW @ dcB
HDFS blocks
Client machine:
hadoop -config /etc/hadoop/dw-dcB fs -get /user/bob/fileB
Distributed FileSystem
fs.defaultFS -> hdfs://dw-nn-dcB -> NameNode1, NameNode2
/etc/hadoop/dw-dcB/hdfs-site.xml, core-site.xml

11. DW @ dcB

12. App paths HDFS paths
viewfs://dw-nn-dcB/var hdfs://dw-tmp-dcB/var
viewfs://dw-nn-dcB/user hdfs://dw-user-dcB/user
viewfs://dw-nn-dcB/logs hdfs://dw-log-dcB/logs
mountable.xml
FileSystem
ViewFileSystem (viewfs://)
Distributed FileSystem (hdfs://)
S3 FileSystem

13. Client machine:
hadoop -config /etc/hadoop/dw-dcB fs -get /user/bob/fileB
Distributed FileSystem
dw-user-dcB -> dw-user-NN1-dcB, dw-user-NN2-dcB
ViewFileSystem
/user -> hdfs://dw-user-dcB/user
/etc/hadoop/dw-dcB/hdfs-site.xml, core-site.xml, mountable.xml
DW @ dcB

14. ○
○
Revenue
Revenue
MR
YARN
HDFS
Data Warehouse
Revenue
MR
YARN
HDFS …..
DataCenter A DataCenter B

15. hadoop —config /etc/hadoop/revenue-dcA fs -get /user/bob/fileA

16. hadoop —config /etc/hadoop/revenue-dcA fs -get /user/bob/fileA
hdfs --config /etc/hadoop/dw-dcB fsck -fs hdfs://dw-user-dc2 /

17. hadoop —config /etc/hadoop/revenue-dcA fs -get /user/bob/fileA
hdfs --config /etc/hadoop/dw-dcB fsck -fs hdfs://dw-user-dc2 /
// find all “fileC” files on all clusters
for i in `ls /etc/hadoop`;
do hadoop --config /etc/hadoop/$i fs -ls fileC; done

18. ○

19. ○
Description $sourcePath Result
global namespace viewfs://revenue-nn-dcA/logs/dirB /logs/dirB Source path unresolvable

20. ○
Description $sourcePath Result
global namespace viewfs://revenue-nn-dcA/logs/dirB /logs/dirB Source path unresolvable
Active namenode hdfs://revenue-log-nn1-dcA/logs/dirB /logs/dirB not reliable due to hard
coded namenode

21. ○
Description $sourcePath Result
global namespace viewfs://revenue-nn-dcA/logs/dirB /logs/dirB Source path unresolvable
Active namenode hdfs://revenue-log-nn1-dcA/logs/dirB /logs/dirB not reliable due to hard
coded namenode
hftp & DNS alias hftp://revenue-nn-dcA/logs/dirB hftp reliability and efficiency

22. ○
○
○

26. viewfs://cluster-dc/
user tmp log

27. ClusterA
DC1
root /

30. ○
○

31. scalding> fsShell("-ls /dc*/") // list all clusters
Found N items
...

32. // no need to remember copy(From/To)Local aka get/put
scalding> fsShell("-cp /local/user/cecily/file.txt
/user/cecily/hdfs_file.txt")
res6: Int = 0

38. user user
Forward FileSystem API calls to all target filesystems with
certain policies.

41. c@dc1
FileSystem nfly = FileSystem.get(conf);
out = nfly.create(“/nfly/C/user/cecily/file.txt”);
out.write(....);
out.close() BEGIN
out.close() END
In = nfly.open(“/nfly/C/user/cecily/file.txt”);
c@dc2
/user/cecily/_nfly_tmp_file.txt
/user/cecily/_nfly_tmp_file.txt
close, rename to
/user/cecily/file.txt
close, rename to
/user/cecily/file.txt
client@dc1

42. ○
○
○
○