Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
www.vmcd.org 
HBASE Performance test by YCSB 
----------------------------------------------------------------------------...
www.vmcd.org 
<name> 
HBase DB Binding</<repositories> 
<repository> 
<id> 
central</<name> 
Maven Repository Switchboard<...
www.vmcd.org 
</dependency> 
<dependency> 
<groupId> 
org.apache.hadoop</<artifactId>hadoop-hdfs</artifactId> 
<version>2....
www.vmcd.org 
<execution> 
<phase> 
package</<goals> 
<goal>single</goal> 
</goals> 
</execution> 
</executions> 
</plugin...
www.vmcd.org 
Run test scripts: 
$ls -l |grep .sh 
for mode in a b c d e f g h i j k do for thread in `seq 1 128` do /home...
www.vmcd.org 
Workload Summary 
Total 11 modes to run test script on 1~128 threads 
Workload all (ops/sec a-k mode) 
0 
50...
www.vmcd.org 
Workload A AverageLatency (us) : read50% update50% 
0 
5000 
10000 
15000 
20000 
25000 
1 
5 
9 
13 
17 
21...
www.vmcd.org 
Workload B AverageLatency (us) : read95% update5% 
0 
5000 
10000 
15000 
20000 
25000 
30000 
1 
5 
9 
13 
...
www.vmcd.org 
Workload C AverageLatency (us) : read100% 
0 
5000 
10000 
15000 
20000 
25000 
30000 
1 
5 
9 
13 
17 
21 
...
www.vmcd.org 
Workload D AverageLatency (us) : read95% insert5% 
0 
1000 
2000 
3000 
4000 
5000 
6000 
7000 
1 
5 
9 
13 ...
www.vmcd.org 
Workload E AverageLatency (us) : scan95% insert5% 
0 
10000 
20000 
30000 
40000 
50000 
60000 
70000 
80000...
www.vmcd.org 
Workload F AverageLatency (us) : read50% readmodify50% 
0 
2000 
4000 
6000 
8000 
10000 
12000 
1 
5 
9 
13...
www.vmcd.org 
Workload G AverageLatency (us) : read5% update95% 
0 
2000 
4000 
6000 
8000 
10000 
12000 
1 
5 
9 
13 
17 ...
www.vmcd.org 
Workload H AverageLatency (us) : update50% insert50% 
0 
500 
1000 
1500 
2000 
2500 
3000 
1 
5 
9 
13 
17 ...
www.vmcd.org 
Workload I AverageLatency (us) : read30% insert70% 
0 
2000 
4000 
6000 
8000 
10000 
12000 
1 
5 
9 
13 
17...
www.vmcd.org 
Workload J AverageLatency (us) : read20% update30% insert50% 
0 
2000 
4000 
6000 
8000 
10000 
12000 
14000...
www.vmcd.org 
Workload K AverageLatency (us) : update20% insert80% 
0 
1000 
2000 
3000 
4000 
5000 
6000 
7000 
8000 
1 
...
www.vmcd.org 
HBASE ENV: 3 regionservers and 10 hdfs datanodes 
#hbase version 2014-09-03 22:46:34,973 INFO [main] util.Ve...
www.vmcd.org 
Datanode OS system information : #cat /etc/issue CentOS release 6.5 (Final) 
#cat /proc/cpuinfo |grep proces...
www.vmcd.org 
Block device characteristics VPD page (SBC): Nominal rotation rate: 7200 rpm : 3.5 inch 
=== START OF INFORM...
Prochain SlideShare
Chargement dans…5
×

HBASE Performane Test

1 122 vues

Publié le

Hbase performance test by YCSB including latency and ops (operations/s)

Publié dans : Technologie
  • Soyez le premier à commenter

HBASE Performane Test

  1. 1. www.vmcd.org HBASE Performance test by YCSB ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Bulid YCSB From github We use thumbtack modify version https://github.com/thumbtack-technology/ycsb Modify pom.xml – using CDH platform version: <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 maven-v4_0_0.xsd"> <modelVersion> 4.0.0</<parent> <groupId> com.yahoo.ycsb</<artifactId> root</<version> 0.1.4</</parent> <artifactId>hbase-binding</artifactId>
  2. 2. www.vmcd.org <name> HBase DB Binding</<repositories> <repository> <id> central</<name> Maven Repository Switchboard</<url>http://repo1.maven.org/maven2</url> </repository> <repository> <id>maven-hadoop</id> <name> Hadoop Releases</<url>https://repository.cloudera.com/content/repositories/releases/</url> </repository> <repository> <id>cloudera-repos</id> <name> Cloudera Repos</<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> </repository> </repositories> <dependencies> <dependency> <groupId> org.apache.hbase</<artifactId>hbase-client</artifactId> <version>0.96.1.1-cdh5.0.0</version> <!-- <version>0.96.1.1-hadoop2</version> -->
  3. 3. www.vmcd.org </dependency> <dependency> <groupId> org.apache.hadoop</<artifactId>hadoop-hdfs</artifactId> <version>2.3.0-cdh5.0.0</version> </dependency> <dependency> <groupId> com.yahoo.ycsb</<artifactId> core</<version>${project.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId> org.apache.maven.plugins</<artifactId>maven-assembly-plugin</artifactId> <version>${maven.assembly.version}</version> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <appendAssemblyId> false</</configuration> <executions>
  4. 4. www.vmcd.org <execution> <phase> package</<goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> Run mvn clean package – build HBase DB Binding successfully Copy core-site.xml hbase-site.xml hdfs-site.xml to /home/hadoop/ycsb/hbase/src/main/conf Load 50 millions data to usertable: YCSB Client 0.1 Command line: -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=family -p recordcount=50000000 -s -threads 10 -p measurementtype=timeseries -p timeseries.granularity=2000 –load [OVERALL], Reconnections, 0.0 [OVERALL], RunTime(ms), 823765.0 [OVERALL], Operations, 4.9999968E7 [OVERALL], Throughput(ops/sec), 60696.883213052264
  5. 5. www.vmcd.org Run test scripts: $ls -l |grep .sh for mode in a b c d e f g h i j k do for thread in `seq 1 128` do /home/hadoop/ycsb/bin/ycsb run hbase -P workloads/workload"$mode" -p columnfamily=f1 -p table=test2 -s -threads "$thread" |tee workload"$mode"_t"$thread" done sleep 60 done -rw-rw-r-- 1 hadoop 58901 Sep 10 23:27 workloada_t1 … -rw-rw-r-- 1 hadoop 58901 Sep 10 23:27 workloadk_t120 -rw-rw-r-- 1 hadoop 58943 Sep 10 23:28 workloadk_t121 -rw-rw-r-- 1 hadoop 58998 Sep 10 23:28 workloadk_t122 -rw-rw-r-- 1 hadoop 59037 Sep 10 23:28 workloadk_t123 -rw-rw-r-- 1 hadoop 59087 Sep 10 23:29 workloadk_t124 -rw-rw-r-- 1 hadoop 59120 Sep 10 23:29 workloadk_t125 -rw-rw-r-- 1 hadoop 59176 Sep 10 23:29 workloadk_t126 -rw-rw-r-- 1 hadoop 59223 Sep 10 23:30 workloadk_t127 -rw-rw-r-- 1 hadoop 59266 Sep 10 23:30 workloadk_t128 -rw-rw-r-- 1 hadoop 54592 Sep 11 09:43 load50m.data
  6. 6. www.vmcd.org Workload Summary Total 11 modes to run test script on 1~128 threads Workload all (ops/sec a-k mode) 0 50000 100000 150000 200000 250000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 mode_a mode_b mode_c mode_d mode_e mode_f mode_g mode_h mode_i mode_j mode_k
  7. 7. www.vmcd.org Workload A AverageLatency (us) : read50% update50% 0 5000 10000 15000 20000 25000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read_50% update_50%
  8. 8. www.vmcd.org Workload B AverageLatency (us) : read95% update5% 0 5000 10000 15000 20000 25000 30000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read_95% update_5%
  9. 9. www.vmcd.org Workload C AverageLatency (us) : read100% 0 5000 10000 15000 20000 25000 30000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read100% read100%
  10. 10. www.vmcd.org Workload D AverageLatency (us) : read95% insert5% 0 1000 2000 3000 4000 5000 6000 7000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read 95% insert5%
  11. 11. www.vmcd.org Workload E AverageLatency (us) : scan95% insert5% 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 scan95% insert5%
  12. 12. www.vmcd.org Workload F AverageLatency (us) : read50% readmodify50% 0 2000 4000 6000 8000 10000 12000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read50% readmodify50%
  13. 13. www.vmcd.org Workload G AverageLatency (us) : read5% update95% 0 2000 4000 6000 8000 10000 12000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read5% update95%
  14. 14. www.vmcd.org Workload H AverageLatency (us) : update50% insert50% 0 500 1000 1500 2000 2500 3000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 update50% insert50%
  15. 15. www.vmcd.org Workload I AverageLatency (us) : read30% insert70% 0 2000 4000 6000 8000 10000 12000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read30% insert70%
  16. 16. www.vmcd.org Workload J AverageLatency (us) : read20% update30% insert50% 0 2000 4000 6000 8000 10000 12000 14000 16000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 read20% update30% insert50%
  17. 17. www.vmcd.org Workload K AverageLatency (us) : update20% insert80% 0 1000 2000 3000 4000 5000 6000 7000 8000 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 update20% insert80%
  18. 18. www.vmcd.org HBASE ENV: 3 regionservers and 10 hdfs datanodes #hbase version 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: HBase 0.96.1.1-cdh5.0.0 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: Subversion file: ///var/lib/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.0.0-Packaging-HBase-2014-03-27_22-53-27/hbase-0.96.1.1-cdh5.0.0 -r Unknown 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: Compiled by jenkins on Thu Mar 27 23:02:13 PDT 2014 export HBASE_OPTS="-Xloggc:/home/hadoop/logs/gc -XX:+UseConcMarkSweepGC -XX:PermSize=96m -XX:MaxPermSize=256m -Xss1m" export HBASE_REGIONSERVER_OPTS="$OPTS -XX:+HeapDumpOnOutOfMemoryError -Xms32g -Xmx32g -Xmn16g -XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80" #export HBASE_REGIONSERVER_OPTS="$OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -Xms32g -Xmx32g -Xmn12g -XX:SurvivorRatio=10 -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSClassUnloadingEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:CMSFullGCsBeforeCompaction=0 -XX:+DisableExplicitGC -Dsun.net.client.defaultConnectTimeout=10000 -Dsun.net.client.defaultReadTimeout=30000" export HBASE_MASTER_OPTS="-Xms4g -Xmx4g" export HBASE_LOG_DIR=/home/hadoop/logs/hbase export HBASE_MANAGES_ZK=false
  19. 19. www.vmcd.org Datanode OS system information : #cat /etc/issue CentOS release 6.5 (Final) #cat /proc/cpuinfo |grep processor wc -l 24 #cat /proc/meminfo |more MemTotal: 99035868 kB Mount system info: /dev/sdb1 on data1 type ext4 (rw) /dev/sdc1 on data2 type ext4 (rw) /dev/sdd1 on data3 type ext4 (rw) /dev/sde1 on data4 type ext4 (rw) /dev/sdf1 on data5 type ext4 (rw) /dev/sdg1 on data6 type ext4 (rw) /dev/sdh1 on data7 type ext4 (rw) /dev/sdi1 on data8 type ext4 (rw) /dev/sdj1 on data9 type ext4 (rw) /dev/sdk1 on data10 type ext4 (rw) /dev/sdl1 on data11 type ext4 (rw) /dev/sdm1 on data12 type ext4 (rw) #sg_vpd /dev/sde --page=0xb1
  20. 20. www.vmcd.org Block device characteristics VPD page (SBC): Nominal rotation rate: 7200 rpm : 3.5 inch === START OF INFORMATION SECTION === Device Model: ST2000NM0033-9ZM175 Serial Number: Z1X1YV4Q LU WWN Device Id: 5 000c50 06672b0c2 Firmware Version: SN03 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ACS-2 (revision not indicated) Local Time is: Wed Sep 3 15:53:55 2014 CST SMART support is: Available - device has SMART capability. SMART support is: Enabled

×