Using Apache Spark to access data stored in RDBMS & Hadoop.

1. Integrating data stored in
RDBMS and Hadoop
leoricklin@gmail.com

2. Problem:
How to process data
stored in the
RDBMS (master
tables) and Hadoop
(log tables) within
one platform ?

3. Pros:
● 過程簡單, 讀取RDBMS成為RDD
後進行操作
Cons:
● 系統設定複雜, 需開通所有spark
nodes 對 RDBMS的接口
● JdbcRDD僅能使用在範圍查詢語法,
且範圍條件欄位限制為Long型態
=> 當作spark nodes分佈存取的依
據
Solution 1: Spark with JdbcRDD

4. Mysql table schema
mysql> create table calendar (
date_cd int not null
,wd_flag INT not null)
engine=csv;
mysql> select date_cd, wd_flag from calendar where date_cd >= 20130101 and date_cd <=
20141031;
+----------+---------+
| date_cd | wd_flag |
+----------+---------+
| 20141001 | 0 |
| 20141002 | 0 |
....
62 rows in set (0.05 sec)

5. 利用JdbcRDD讀取Mysql
$SPARK_HOME/bin/spark-shell --jars /usr/share/java/mysql-connector-java-5.1.17.jar --
master yarn-client --num-executors 4 --executor-cores 2 --executor-memory 4g --driver-
memory 1g
scala>
import org.apache.spark.rdd.JdbcRDD
import java.sql.{Connection, DriverManager, ResultSet}
val url="jdbc:mysql://mysqlserver.com:3306/db"
val username = "user"
val password = "pwd"
val driverName="com.mysql.jdbc.Driver"
case class CAL(date_cd:String, hd_flag:Int)
val calrdd = new JdbcRDD( sc
, () => DriverManager.getConnection(url,username,password)
, "select date_cd, wd_flag from calendar where date_cd >= ? and date_cd <= ?"
, 20130101, 20141031, 12
, r =>( CAL(r.getString(1),r.getInt(2)) )
)
// calrdd.count: Long = 62
// scheduler.DAGScheduler: Job 0 finished: ... took 1.429489 s

6. 利用HadooopRDD讀取HDFS
case class LOG(req_date:String, operation:String, http_code:Int)
val path = "hdfs://mycluster/tmp/log.CSV"
val logrdd = sc.textFile(path, 12 // ret: RDD[String]
).map( i => Helper.tokenize(i,",",true) // ret: RDD[Array[String]]
).map(ary => Helper.replaceChar(ary,"-","0") // ret: RDD[Array[String]]
).map(ary => toLOG(ary)) // ret: RDD[LOG]
// logrdd.count: Long = 4000000
// scheduler.DAGScheduler: Job 1 finished: ... took 10.404451 s

7. 進行RDD join
import org.apache.spark.SparkContext._
val logkv = logrdd.map( i => (i.req_date.substring(0,8), i) ) // ret: RDD[String, LOG]
val calkv = calrdd.map( i => (i.date_cd, i) ) // ret: RDD[String, CAL]
val joinrdd = logkv.join(calkv)
/*
joinrdd.first: : (String, (LOG, CAL)) = (
20130129,(
LOG(20130129231106,Put,200)
,CAL(20130129,1))
)
scheduler.DAGScheduler: Job 20 finished... took 97.416966 s
*/
.

8. Solution 2: Spark with JDBC Client
Pros:
● 系統設定簡單, 僅開通driver nodes
對 RDBMS的接口
● 支援原生SELECT語法
● SQL查詢結果可自訂為RDD or
Broadcast variable.
Cons:
● 過程略複雜, 須將ResultSet在
driver node轉為RDD(或BV)後進
行操作
● RDBMS資料量受限於driver node
MEM限制

9. ref Mysql table schema

10. 利用Jdbc Driver讀取Mysql
val conn:java.sql.Connection = SqlHelper.getConn(driverName, uri, username, password)
val stmt:java.sql.Statement = SqlHelper.getStmt(conn)
val sql="select date_cd,wd_flag from calendar where date_cd >= 20130101 and date_cd <=
20141031"
val ret:java.sql.ResultSet = SqlHelper.getResult(stmt, sql)
val rows = new scala.collection.mutable.ListBuffer[Tuple2[String,Int]]()
while (ret.next) {
rows += calToTup(ret)
}
// rows.size:Int = 62
// rows(0):(String, Int) = (20141001,0)
// return immediately
val calrdd = sc.parallelize(rows, 12)

11. ref 利用HadooopRDD讀取HDFS
ref 進行RDD join

12. 進行Map-side Join
...
val sqlkv = new scala.collection.mutable.HashMap[String,Int]()
while (ret.next) {
sqlkv += calToTup(ret)
}
val sqlbc = sc.broadcast(sqlkv)
val logkv = logrdd.map( ... ) /* 利用HadooopRDD讀取HDFS並轉換為RDD[String,LOG] */
val result = logkv.mapPartitions( { iter =>
var sqlkv = sqlbc.value
for{
(key, value) <- iter
if(sqlkv.contains(key))
} yield (key, (value, sqlkv.getOrElse(key, () => "")))
})
/*
result.first: (String, (LOG, Any)) = ( 20130129,( LOG(20130129231106,Put,200) ,1 ) )
scheduler.DAGScheduler: Job 0 finished... took 2.051693 s
*/

13. Summary
● JdbcRDD方式
o Pros: 程式撰寫較簡易
o Cons: 系統設定較複雜
● JDBC Clien方式
o Pros: 系統設定較簡易
o Cons: 程式撰寫較複雜(但可進行性能優化)