1. Introduction

2. to

3. Apache

4. Tajo

5.
Bay

6. Area

7. Hadoop

8. User

9. Group

10.
November,

11. 5th,

12. 2013

14. About Me
• Hyunsik

15. Choi

16. (pronounced:

17. “Hyeon-shick

18. Cheh”)

19.
• PhD

20. (Computer

21. Science

23. Engineering,

24. 2013)

25.
• Director

26. of

27. Research,

28. Gruter

29. Corp,

30. Seoul,

31. South

32. Korea

33.
• Open-source

34. Involvements

35.
– Full-time

36. contributor

37. to

38. Apache

39. Tajo

40. (2013.6

41. ~

42. )

43.
– Apache

44. Tajo

45. (incubating)

46. PPMC

47. member

48. and

49.
committer

50. (2013.3

51. ~

52. )

53.
– Apache

54. Giraph

55. PMC

56. member

57. and

58. committer

59. (2011.

60. 8

61. ~

62. )

63.
• Contacts

64.
– Email:

65. hyunsik@apache.org

66.
– Linkedin:

67. http://linkedin.com/in/hyunsikchoi/

69. Gruter
• Big

70. Data

71. infrastructure

72. startup

73.
• Hadoop

74. platforms;

75. Hadoop

76. ecosystem

77. consu
lting;

78. big

79. data

80. analytics

81. layers

84.
• Teheran

85. Rd.

86. tech

87. district,

88. Seoul,

89. South

90. Korea

91.
3

92. Presentation
Apache

93. Tajo

94.
• Project

95. overview

96.
• System

97. architecture

98.
• Distributed

99. processing

100. model

101.
• Query

102. optimization

103. approach

104.
• Project

105. status

106.
• Project

107. roadmap

108.
• Q

110. A

111.
4

112. Introduction to Tajo
• Tajo

113.
– Big

114. Data

115. Warehouse

116. System

117. on

118. Hadoop

119.
– Developed

120. since

121. 2010

122.
– Apache

123. incubation

124. project

125. (entered

126. the

127. ASF

128. in

129. March

130. 2013)

131.
• Features

132.
–
–
–
–
–
–
SQL

133. standard

134. compliance

135.
Fully

136. distributed

137. SQL

138. query

139. processing

140.
HDFS

141. as

142. a

143. primary

144. storage

145.
Relational

146. model

147. (will

148. be

149. extended

150. to

151. nested

152. model

153. in

154. the

155. future)

156.
ETL

157. as

158. well

159. as

160. low-latency

161. relational

162. query

163. processing

164. (100

165. ms

166. ~)

167.
UDF

168. support

169.
• News

170.
– 0.2-incubating:

171. released

172. November

173. 2013

174.
– 0.8-incubating:

175. to

176. be

177. released

178. December

179. 2013

181. Design Principles
• Failed tasks restart mechanism
• QueryMaster per query
Fault
Tolerance
Scalability
High
Throughput
• Flexible DAG framework
• Cost-based optimization
• Extensible rewrite engine
Query
Optimization

182. Architecture
• HDFS

183. (Primary

184. Storage)

185.
• Master-Worker

186. Model

187. +

188. QueryMaster

189. per

190. Query

191.
– RPC

192. Implementation

193. in

194. Java

195. (Protocol

196. Buffer

197. +

198. Netty)

199.
• Tajo

200. Master

201.
–
–
–
–
Always

202. on

203. standby

204. and

205. instant

206. execution

207. of

208. some

209. kinds

210. of

211. queries

212. (DDLs)

213.
Responsible

214. for

215. serving

216. remote

217. APIs

218. to

219. Clients

220.
Query

221. Parser

222. and

223. Coordination

224. of

225. QueryMasters

226.
Embedded

227. CatalogServer

228. (or

229. run

230. independently)

231.
• Query

232. Master

233. (per

234. Query)

235.
– Logical

236. plan

237. transform

238. to

239. a

240. distributed

241. execution

242. plan.

243.
– Control

244. execution

245. blocks

246. (steps

247. in

248. a

249. job)

250.
– Task

251. scheduling

252.
• Tajo

253. Worker

254.
– Storage

255. Manager

256.
– Local

257. Query

258. Engine

260. Architecture

261. Query Planning Process

262. Tajo Distributed Processing Model
• A

263. query

264. =

265. a

266. directed

267. acyclic

268. graph

269.
• A

270. vertex

271. is

272. a

273. processing

274. unit

275. and

276. contains:

277.
– A

278. logical

279. plan

280. (a

281. DAG

282. of

283. logical

284. operators)

285.
– An

286. enforcer

287. (properties

288. to

289. force

290. physical

291. planning)

292.
• Each

293. edge

294. represents

295. a

296. data

297. flow

298. between

299. vertices

301.
and

302. contains:

303.
– Transmission

304. type

305. (Pull

306. and

307. Push)

308.
– Shuffle

309. type

310. (range,

311. hash,

312. and

313. ..)

314.
– The

315. number

316. of

317. partitions

319. Data Shuffle (Edge)
• Shuffle

320. Methods

321.
– Hash

322.
• Hash

323. repartitioning

324. (intermediate

325. data

326. repartitioning

327. via

328. hash

329. keys)

330.
– Range

331.
• Range

332. repartitioning

333. (intermediate

334. data

335. repartitioning

336. to

337. corresponding

338. d
isjoint-range-assigned

339. nodes)

341.
• Transmission

342. Methods

343.
– Pull

344.
• Step

345. 1:

346. Intermediate

347. data

348. local

349. disk

350. materialization

351.
• Step

352. 2:

353. Materialized

354. intermediate

355. data

356. pull

357.
–

358. Push

359. (will

360. be

361. supported

362. in

363. 0.8)

364.
• Intermediate

365. data

366. transmission

367. (no

368. materialization)

370.
•

371. inter-operator

372. pipelining

373. enabled

375. An Example of Distributed Execution Plan
Join-groupby-sort query plan
Distributed query execution plan
select col1, sum(col2) as total, avg(col3) as average from r1, r2
where r1.col1 = r2.col2 group by col1 order by average;

376. Query Optimization
• Cost-based

377. Join

378. Optimization

379. (Greedy

380. Heuristic)

381.
– Best

382. join

383. order

384. guessing

385. eliminated!

386.
• Extensible

387. Rewrite

388. Rule

389. Engine

390.
– Enhanced

391. rewrite

392. rule

393. interface

394. with

395.
• Query

396. block

397. graph

398. for

399. relationships

400. of

401. query

402. blocks

403. in

404. a

405. query

406.
• Join

407. graph

408. for

409. representing

410. join

411. relations

412.
• Other

413. utilities

414. for

415. plan

416. and

417. expressions

419. Query Optimization
• Progressive

420. Query

421. Optimization

422.
– Runtime

423. statistics

424. collection

425.
– Ad

426. hoc

427. range

428. and

429. partition

430. determination

431. according

432. to

434.
operator

435. type

436. (join,

437. aggregation,

438. and

439. sort)

440.
– Query

441. Reoptimization

442. (planned)

443.
• Runtime

444. join

445. order

446. determination

447. and

448. distributed

449. join

450. strategy

451.
• Pull-based

452. or

453. push-based

454. transmission

455. determination

456.
14

457. Current Status
• SQL

458. Support

459.
– Standard:

460. ANSI

461. SQL

462. 2003

463. compliance

464.
– Non-standard:

465. Extensive

466. PostgreSQL

467. support

468.
• Functions

469. (regexp_replace,

470. split_part,…⋯)

471.
– Scheduled:

472.
• OuterJoin

473. (0.8),

474. Exists

475. (1.0),

476. In

477. Subquery

478. (1.0)

479.
• Distributed

480. Join,

481. groupby,

482. sort

483. operators

484. available

485.
• Blocking/Asynchronous

486. Java

487. Client

488. API

489.
• Tajo

490. Catalog

491.
– Derby

492. and

493. MySQL

494. persistent

495. store

496.
– tajo_dump,

497. an

498. utility

499. for

500. backup

501. and

502. restore

504. Current Status
• Various

505. file

506. format

507. supports

508.
–
–
–
–
–
CSV

509. format

510.
RowFile

511. (Tajo’s

512. own

513. row

514. store

515. file

516. format)

517.
RawFile

518. (for

519. local

520. disk/network

521. materialization)

522.
RCFile

523. (Text/Binary

524. (de)serializer

525. and

526. Hive

527. compatible)

528.
Parquet

529. (the

530. next

531. release)

532.
• Scanner/Appender

533. Interface

534. for

535. custom

536. file

537. formats

538.
• Very

539. fast

540. scan

541. performance

542.
– QueryMaster

543. schedules

544. tasks

545. with

546. balancing

547. disk

548. volume

550.
loads

551. for

552. each

553. node

554.
– As

555. a

556. result,

557. disk-bound

558. queries

559. show

560. average

561. scan

562. 60-110

563. MB
/s

564. per

565. disk

566. (SATA2

567. and

568. SAS)

570. Experiments
• Tajo

571. (master

573. branch)

574. vs.

575. Impalad_version

576. 1.1.1

577. vs.

578. Hive

579. 0.10-cdh4

580.
• TPC-H

581. data

582. set

583. 100GB

584.
• Cache

585. dropped

586. for

587. each

588. experiment

589.
• 10G

590. networks

591.
• 6

592. cluster

593. nodes

594.
• Each

595. machine

596. is

597. equipped

598. with

599. :

600.
– Intel

601. Xeon

602. CPU

603. E5

604. 2640

605. 2.50GHz

606. x

607. 4

608.
– 64

609. GB

610. memory

611.
– 6

612. SATA2

613. disks

614.
17

615. Seconds
Experiments
Some of TPC-H Queries on 100GB
18

616. Roadmap
• 0.2

617. release

618. is

619. being

620. voted

621. on

622. incubator-general@a.o

623.
• December

624. 2013:

625. Apache

626. Tajo

627. 0.8

628. Release

629.
–
–
–
–
–
More

630. SQL

631. standard

632. features;

633. more

634. stability

635.
Outer

636. join

637. support

638.
Hadoop

639. 2.2.0-beta

640. porting

641.
Table

642. Partitioning:

643. Hash,

644. Range,

645. List,

646. Column

647. (Hive

648. style)

649.
Catalog

650. access

651. to

652. HCatalog

653.
• Q1

654. 2014:

655. Apache

656. Tajo

657. 1.0-alpha

658. release

659.
–
–
–
–
More

660. powerful

661. rewrite

662. rules

663.
More

664. fault

665. tolerance

666.
Window

667. functions

668.
A

669. prototype

670. of

671. JIT

672. query

673. compilation

675. vectorized

676. engine

678. Get Involved!
• Getting

679. Started

680.
– http://wiki.apache.org/tajo/GettingStarted

681.
• Checkout

682. the

683. development

684. branch

685.
– http://wiki.apache.org/tajo/BuildInstruction

686.
• Jira

687. ­–

688. Issue

689. Tracker

690.
– https://issues.apache.org/jira/browse/TAJO

691.
• Join

692. the

693. mailing

694. list

695.
– tajo-dev-subscribe@incubator.apache.org