鶴原翔夢 (LINE Corporaion) 【京都】LINE Developer Meetup #33での発表資料です https://line.connpass.com/event/84852/

1. HBase, no trouble
LINE
1

2. • : (Tsuruhara Tomu)
• : LINE LINE 1
• : HBase
•
• HBase at LINE 2017 http://
linedevday.linecorp.com/jp/2017/#A-6
• LINE Platform Development Chronicle
https://engineering.linecorp.com/ja/blog/
detail/59
2

3. Agenda
• HBase ?
• LINE HBase
• LINE HBase
•
3

4. HBase ?
Apache HBase™ is the Hadoop database, a distributed, scalable, big data store.
Use Apache HBase™ when you need random, realtime read/write access to
your Big Data. This project's goal is the hosting of very large tables -- billions of
rows X millions of columns -- atop clusters of commodity hardware. Apache
HBase is an open-source, distributed, versioned, non-relational database
modeled after Google's Bigtable: A Distributed Storage System for Structured
Data by Chang et al. Just as Bigtable leverages the distributed data storage
provided by the Google File System, Apache HBase provides Bigtable-like
capabilities on top of Hadoop and HDFS.
— http://hbase.apache.org/
4

5. HBase
• ! ! !
• BigData
• Google Bigtable
• Hadoop HDFS
• SQL
• Java
5

6. Why not traditional RDBMS?
6

7. LINE MySQL
• slave
• MySQL
•
• ( 1
)
7

8. HBase : DataModel
: LINE
Row Column Family Column Qualifier Timestamp Value
message.getId() 'd' 'm' message.createTime() message
8

9. HBase Sharding
9

10. LINE
10

11. LINE Messaging System Architecture
11

12. LINE
12

13. HBase Clusters
and more
13

14. HBase Clusters
•
•
• HDD vs PCIe-SSD
•
• Dual cluster
•
14

15. HBase Clusters
• 1000
• 100
• 300
15

16. HBase
• HBase Unit: 5
• talk-server
• HBase
• HBase
• (decommission )
16

17. HBase
• Zookeeper
• Namenode
• ZKFC(ZookeeperFailoverController)
• JournalNode
• DataNode
• HMaster
• RegionServer
17

18. OSS tools + inhouse tools
• Provisioning: Chef/Ansible/PMC
• Monitoring: Prometheus/Grafana/Elasticsearch/IMON
• Automation: hbase-auto
18

19. 19

20. HBase
2012
http://www.slideshare.net/sunsuk7tp/hbase-at-line
20

21. HBase
• HBase version up
• http://linedevday.linecorp.com/jp/2017/#A-6
•
21

22. 22

23. jstack $REGIONSERVER_PID
"PriorityRpcServer.handler=18,queue=0,port=16020" #322 daemon prio=5 os_prio=0 tid=0x00007fd422062800 nid=0x19b89 runnable [0x00007fcb8a821000]
java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617)
at java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:499)
at java.lang.ThreadLocal$ThreadLocalMap.access$200(ThreadLocal.java:298)
at java.lang.ThreadLocal.remove(ThreadLocal.java:222)
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:426)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
at com.yammer.metrics.stats.ExponentiallyDecayingSample.unlockForRegularUsage(ExponentiallyDecayingSample.java:196)
at com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:113)
at com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:81)
at org.apache.hadoop.metrics2.lib.MutableHistogram.add(MutableHistogram.java:81)
at org.apache.hadoop.metrics2.lib.MutableRangeHistogram.add(MutableRangeHistogram.java:59)
at org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceImpl.dequeuedCall(MetricsHBaseServerSourceImpl.java:194)
at org.apache.hadoop.hbase.ipc.MetricsHBaseServer.dequeuedCall(MetricsHBaseServer.java:76)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2192)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
23

24. jstack $REGIONSERVER_PID
"PriorityRpcServer.handler=18,queue=0,port=16020" #322 daemon prio=5 os_prio=0 tid=0x00007fd422062800 nid=0x19b89 runnable [0x00007fcb8a821000]
java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617)
at java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:499)
at java.lang.ThreadLocal$ThreadLocalMap.access$200(ThreadLocal.java:298)
at java.lang.ThreadLocal.remove(ThreadLocal.java:222)
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:426)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
at com.yammer.metrics.stats.ExponentiallyDecayingSample.unlockForRegularUsage(ExponentiallyDecayingSample.java:196)
at com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:113)
at com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:81)
at org.apache.hadoop.metrics2.lib.MutableHistogram.add(MutableHistogram.java:81)
at org.apache.hadoop.metrics2.lib.MutableRangeHistogram.add(MutableRangeHistogram.java:59)
at org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceImpl.dequeuedCall(MetricsHBaseServerSourceImpl.java:194)
at org.apache.hadoop.hbase.ipc.MetricsHBaseServer.dequeuedCall(MetricsHBaseServer.java:76)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2192)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
24

25. ThreadLocal.java
ThreadLocal.ThreadLocalMap#expungeSt
aleEntry(int)
614: // Unlike Knuth 6.4 Algorithm R, we must scan until
615: // null because multiple entries could have been stale.
616: while (tab[h] != null)
617: h = nextIndex(h, len);
➡ 1
1
https://en.wikipedia.org/wiki/Hash_table
25

26. 26

27. HeapDump
hash table bucket
27

28. 28

29. 29

30. 30

31. RpcServer.java
public class Connection {
....
private Counter rpcCount = new Counter(); // number of outstanding rpcs
....
}
public class Counter {
...
private final ThreadLocal<IndexHolder> indexHolderThreadLocal =
new ThreadLocal<IndexHolder>() { ... };
...
}
31

32. 32

33. Connection
• client GC
• 1000 1000
live
33

34. G1GC = Garbage First Garbage Collector
34

35. Connection
• Application Server
• Application connection close
connection
=> = region
35

36. • ThreadLocal#remove
• HBASE-16616 Rpc handlers stuck on ThreadLocalMap.expungeStaleEntry
• 1 Counter ThreadLocal
• HBASE-16146 Counters are expensive...
• 2 HFileBlock ThreadLocal
• HBASE-17072 CPU usage starts to climb up to 90-100% when using G1GC; purge
ThreadLocal usage
• HBase 1.4.2 or backport patch
36

37. • HBase =
• LINE
• 300 nearly 0
•
• SLA ...
37

38. Thank you
38