Flink Forward 2015

1. Data science lifecycle with Apache Flink
and
Apache Zeppelin (incubating)
Flink Forward
Moon moon@nflabs.com
NFLabs www.nflabs.com

2. Content
1. Data science lifecycle
2. Zeppelin for data science
3. Zeppelin and Flink
4. Project Roadmap

3. Data science lifecycle

4. Data Science: process
https://en.wikipedia.org/wiki/Data_analysis

5. Data Science: tools
MLlib

6. Data Science: people
Engineer Data Scientist
DevOps Business
http://aarondavis.design/

7. Content
1. Data science lifecycle
2. Zeppelin for data science
3. Zeppelin and Flink
4. Project Roadmap

8. Zeppelin for data scientist

9. ProjectTimeline
ASF Incubation12.2014
08.2014 Started getting adoption
http://zeppelin.incubator.apache.org
12.2012 Commercial Product for data analysis
10.2013 Open sourced a single feature

10. Hadoop Landscape
Cloudera-ML
ML-base
MRQL
Shark
?

11. Commercial Product
12.2012

12. Zeppelin
10.2013

13. Zeppelin
10.2013

14. Zeppelin
08.2014

15. Zeppelin
08.2014

16. Third-party Products
10.2014

17. Apache Incubation Proposal
11.2014

18. Acceptance by Incubator
23.12.2014

19. Current Status
1 Release
68 Contributors worldwide
722 Stars on GH
300/900 Emails at users/dev @i.a.o

20. Interactive Notebooks

21. InteractiveVisualization

22. Multiple Backends

23. Zeppelin & Friends
Z-Manager
ZeppelinHub
…⋯
Collaboration/Sharing
Packaging & Deployment Zeppelin + Full stack on a cloud
Packages Backend Integration

24. OnlineViewer

25. Deployment
https://github.com/hortonworks-gallery/ambari-zeppelin-service

26. Deployment

27. As a Service

28. Before
Cloudera-ML
ML-base
MRQL
Shark
?

29. After
Cloudera-ML
ML-base
MRQL
Shark

30. Content
1. Data science lifecycle
2. Zeppelin for data science
3. Zeppelin and Flink
4. Project Roadmap

31. Flink integration
Integrated through Interpreter
Data processing system abstraction in Zeppelin

32. Interpreter
http://zeppelin.incubator.apache.org/docs/development/writingzeppelininterpreter.html

33. Writing an Interpreter
public abstract void open();
public abstract void close();
public abstract InterpreterResult interpret(String st, InterpreterContext context);
public abstract void cancel(InterpreterContext context);
public abstract int getProgress(InterpreterContext context);
public abstract List<String> completion(String buf, int cursor);
public abstract FormType getFormType();
public Scheduler getScheduler();
Must
have
Good
to have
Advanced

34. Flink Interpreter
https://github.com/apache/incubator-zeppelin/blob/master/flink/src/main/java/org/apache/zeppelin/flink/FlinkInterpreter.java
Zeppelin Server
Thrift
Flink Interpreter
Interpreter JVM
process
FlinkILoop
ExecutionEnvironment

35. Using interpreter
Configure Bind use

36. Using interpreter
Use different
interpreters in the
same notebook

37. Display System
Zeppelin Server
Flink Interpreter Other Interpreter
Zeppelin webapp
Websocket, REST
Text Html Table Angular

38. Display System
Select display
system through
output

39. Built in scheduler
Built-in scheduler runs
your notebook with
cron expression.

40. Flexible layout
Flexible layout

41. DEMO

42. Content
1. Data science lifecycle
2. Zeppelin for data science
3. Zeppelin and Flink
4. Project Roadmap

43. Flink Integration
• ZeppelinContext :Access to Zeppelin provided features
• - Dynamic form
• - Angular display system
• Dependency loading
• Auto completion
• Cancel
• Get progress information

44. Thank you
Q & A
Moon
moon@nflabs.com
NFLabs
www.nflabs.com
http://zeppelin.incubator.apache.org/

45. Project roadmap

46. Multi-tenancy
Two approaches
1. Implement authentication,ACL inside of Zeppelin
https://github.com/apache/incubator-zeppelin/pull/53
2. Run Zeppelin on top of Docker
http://github.com/NFLabs/z-manager

47. Zeppelin for organizations

48. An Engineer
engineer by http://aarondavis.design/

49. ATeam
engineer by http://aarondavis.design/

50. An Organization
engineer by http://aarondavis.design/

51. That’s too many!
engineer by http://aarondavis.design/

52. What is the problem?
Too much:
Install
Configure
Cluster resources

53. Solution?
We have containers
+
reverse proxy

55. Z Manager PoC
httpd + mod_php
nginx
Linux box
engineer by http://aarondavis.design/
2 days, bash + php :(

56. Z Manager PoC

57. Z Manager
http://github.com/NFLabs/z-manager
Apache 2.0 Licence
Containerized deployment per user
Reverse proxy
Single binary
Simple web application
Z Manager
SGA to ASF coming *

58. Z Manager
Auto-update
engineer by http://aarondavis.design/
Linux box
go + react :)
Z Manager process

59. Z Manager

60. Helium

61. People do the similar work
with different data
New visualization
Model & Algorithm
Data process pipeline
engineer by http://aarondavis.design/

62. Package and distribute work
New visualization
Model & Algorithm
Data process pipeline
Pkg
Repo
engineer by http://aarondavis.design/

63. Helium
https://s.apache.org/helium
Platform for
on top of Apache Zeppelin
Data Analytics Application

64. Helium Application
= +
View Algorithm
Zeppelin provided Resources

65. Resources
Data
Computing
Any java object

66. -

67. Result

68. of

69. last

70. execution

71. -

72. JDBC

73. connection

74. (from

75. JDBC

76. Interpreter)*

77. -

78. SparkContext

79. (from

80. SparkInterpreter)

81. -

82. Flink

83. environment

84. (from

85. FlinkInterpreter)*
-

86. Provided

87. by

88. user

89. created

90. Interpreter
-

91. Provided

92. by

93. user

94. created

95. Helium

96. application

98. Application Examples
Data
Computing
-

99. ex)

100. get

101. git

102. commit

103. log

104. data

115. https://github.com/Leemoonsoo/zeppelin-gitcommitdata
Visualization

117. -

118. ex)

119. run

120. cpu

121. usage

122. monitoring

123. code

124. across

125. spark

126. cluster,

127. using

128. SparkContext

139. https://github.com/Leemoonsoo/zeppelin-sparkmon
-

140. ex)

141. display

142. result

143. data

144. as

145. a

146. wordcloud

157. https://github.com/Leemoonsoo/zeppelin-wordcloud

158. How it works
Zeppelin

159. Server
Web

160. browser
View
Interpreter

161. Process
Algorithm
Resource

162. pool
Resource

163. pool
Resource

164.
pools

165. are

166.
connected
“Algorithm

167. runs

168. where

169. resource

170. exists”

171. API
class YourApplication extends org.apache.zeppelin.helium.Application {
@Override
public void run(ApplicationArgument arg, InterpreterContext context) {
…..
}
}
Easy

172. API
Just

173. extend

174. helium.Application

175. Application Spec
{
mavenArtifact : groupId:artifactId:version,
className : your.helium.application.Class,
icon : fa fa-cloud,
name : My app name,
description : “some description,
consume : [
org.apache.spark.SparkContext
]
}
Simple
Writing

176. a

177. spec

178. file

179. allow

180. Zeppelin

181. load

182. application

183. Deploy
Public
Repository
Private
Repository
Handy

184.
Private
Public
Packaged

185. to

186. Jar

187. and

189.
Distributed

190. through

191.
Maven
Downloaded

192. on

193. the

194. fly

195.
and

196. run

197. when

198. user

199. selects

200.
it

201. Thank you
Q A
Moon
moon@nflabs.com
NFLabs
www.nflabs.com
http://zeppelin.incubator.apache.org/