Slides from JavaDay Kiev 2015 talk

1. Develop
modern
apps
using
Spring
ecosystem
at
BigData
era

2. LEAD
SOFTWARE
ENGINEER
AT
EPAM
SYSTEMS.
PATIONATE
DEVELOPER,
SPEAKER,
ACTIVE
MEMBER
OF
KIEV
JUG.
PARTICIPATE
IN
DIFFERENT
EDUCATIONAL
INITIATIVES,
ENGINEERING
EVENTS
AND
JCP/ADOPTJSR
PROGRAMS.
OLEG TSAL-TSALKO
ABOUT ME

3. Spring
for
a
long
Mme
no
more
than
just
a
Framework

6. Spring
Boot
• General
purpose
plaRorm
for
Web
app
and
client
app
development
which
makes
complex
apps
development
look
easy
• Rapid
development
of
diff
type
applicaMons
based
on
solid
Spring
foundaMon
• No
boilerplate
code.
You
write
only
needfull/
funcMonal
code.
Spring
knows
beVer
how
make
it
all
work.
• Nevertheless
easy
to
customize
and
extend
• Simple
and
robust
dependency
management

7. [DEMO:]
Tweets
Boot
Data
REST
app
1. Basic
5
mins
Boot
Data
Rest
app
which
simply
works!
2. Lets
add
logging
and
external
properMes
support
3. Lets
add
integraMon
tests
4. Lets
add
Tweet
lifecycle
event
listeners
5. Lets
add
Actuator
monitoring
support
6. Lets
add
much
more
fields
to
Tweet
enMty
7. Lets
add
Tweet
compact
projecMon
8. Lets
add
validaMon
9. Lets
add
addiMonal
stuff
to
Tweet
HATEOUS
json
view
10. Lets
add
more
custom
queries
11. But
beVer
do
not
try
to
make
Java
8
data
and
Mme
types
work))

9. Spring
XD
• General
purpose
plaRorm
for
developing
BigData
apps
for
data
ingesMon,
data
stream
and
batch
processing,
data
export
and
data
analyMcs
• Scalable
and
fault
tolerant
plaRorm
in
your
service
• DeclaraMve
processing
workflow
development,
deployment
and
management
• Huge
number
of
predefined
available
modules
that
can
be
used
straight
away
to
bring
your
processing
pipelines
in
life
in
seconds
• Easy
to
extend
plaRorm
and
pluggable
custom
modules
• Build
on
top
of
rich
and
robust
Spring
foundaMon

15. Real
world
example
for
XD
Raw
data
is
captured
from
mobile
phones
in
JSON
format
by
a
Spring
XD
cluster
where
several
processes
are
performed.
The
data
is
then
stored
in
an
HDFS
cluster
where
Spring
XD
batch
jobs
use
SQL
via
the
HAWQ
interface
to
HDFS
and
store
the
calculated
reports
in
Redis.
Spring
Boot
is
then
used
with
Angular.js
and
D3.js
to
show
analyMcs
to
end
users.

16. [DEMO:]
Spring
XD
toy.
Play
with
me!
1. Show
XD
bootstrap
in
singlenode
mode
and
deploy
simple
streams.
Show
XD
shell
and
XD
UI
usages.
2. Show
XD
distributed
mode
configuraMon
with
external
Zookeeper,
DB
and
Redis.
Show
XD
meta
data
in
Zookeeper.
Deploy
same
simple
streams
on
cluster
and
show
XD
cluster
fault
tolerance
and
recovery.
3. Show
example
of
predefined
XD
module
configuraMon
(e.g.
hdfs)
4. Create
custom
module
and
upload
it
into
XD
5. Show
DEMO
for
stream
reading
tweets
from
TwiVer
and
posMng
them
into
REST
client
app
developed
before
via
HTTP
and
wriMng
them
into
HDFS
in
parallel

18. Spring
XD
(out
of
scope…)
• Show
XD
job
running
over
Hadoop
• Show
RabbitMQ
integraMon
• Show
DEMO
on
running
either
MapReduce
job
using
Cascading
or
Pig
over
HDFS
tweets
data
OR
running
Spark
app
over
HDFS
tweets
data

19. Spring
Boot/XD
enable
twelve
factor
apps
I.
Codebase
One
codebase
tracked
in
revision
control,
many
deploys
II.
Dependencies
Explicitly
declare
and
isolate
dependencies
III.
Config
Store
config
in
the
environment
IV.
Backing
Services
Treat
backing
services
as
aVached
resources
V.
Build,
release,
run
Strictly
separate
build
and
run
stages
VI.
Processes
Execute
the
app
as
one
or
more
stateless
processes
VII.
Port
binding
Export
services
via
port
binding
VIII.
Concurrency
Scale
out
via
the
process
model
IX.
Disposability
Maximize
robustness
with
fast
startup
and
graceful
shutdown
X.
Dev/prod
parity
Keep
development,
staging,
and
producMon
as
similar
as
possible
XI.
Logs
Treat
logs
as
event
streams
XII.
Admin
processes
Run
admin/management
tasks
as
one-­‐off
processes

20. Everything
wasn’t
so
smooth
though…
• SPEL
doesn’t
work
on
filter
with
JSON
elements
access
implicitly
as
it
works
with
transformer
module.
To
make
filter
work
you
need
either
to
convert
JSON
to
Tuple
or
to
use
JSON
Xpath
SPEL
syntax
• When
transform
JSON
into
Tuple
and
back
WARNings
are
thrown
on
Jackson
transformaMon???
• In
order
implicit
converMon
to
Tuple
works,
you
need
to
specify
contentType
MIME
header
in
HTTP
request
otherwise
it
can’t
automaMcally
cast
text/plain
MIME
message
to
Tuple
•
If
you
deploy
hVp
module
on
port
which
is
occupied
it
won’t
fail
on
deploy
but
your
stream
will
simply
silently
not
work
• SomeMmes
container
gets
broken
and
smth
stops
to
work
like
Taps
in
my
case
for
example
and
no
errors
in
logs.
Only
container
restart
helps.
• Spring
data
rest
doesn’t
expose
enMty
IDs
in
JSON
body
by
default
• When
Mongo
enMty
save
operaMon
failing
as
a
result
of
HTTP
call
no
excepMons
visible
in
logs,
just
500
ERROR
response
to
client
without
proper
stack
trace,
however
if
you
call
MongoTemplate
directly
you’ll
get
proper
stack
trace.
• If
HTTP
GET
request
contentType=applicaMon/json
then
response
will
be
rendered
as
applicaMon/x-­‐
spring-­‐data-­‐compact+json,
when
by
default
it
will
be
HAL
document
• By
default
HVp
POST
request
set
contentType=text/plain
which
failed
to
be
transformed
into
enMty
• hVp://wiki.fasterxml.com/JacksonFAQDateHandling
• hVps://github.com/FasterXML/jackson-­‐datatype-­‐jsr310
• Mongo
doesn’t
support
Java
8
Date
and
Time
types
naMvely
• hVps://jira.spring.io/browse/DATAREST-­‐524
• hVp://stackoverflow.com/quesMons/25072967/spring-­‐xd-­‐redis-­‐message-­‐bus-­‐removing-­‐headers-­‐
from-­‐the-­‐message
issue
with
headers
passing
in
Redis
• SpringXD
doesn’t
support
packages
starMng
with
‘java’))!!!

21. Links
• Spring
-­‐
hVp://spring.io
• Zookeeper
-­‐
hVp://zookeeper.apache.org/doc/trunk/zookeeperStarted.html
• Hadoop
-­‐
hVp://hadoop.apache.org/docs/r2.7.1/hadoop-­‐project-­‐dist/hadoop-­‐
common/SingleCluster.html
• XD
-­‐
hVp://docs.spring.io/spring-­‐xd/docs/current-­‐SNAPSHOT/reference/html
• Boot
-­‐
hVp://docs.spring.io/autorepo/docs/spring-­‐boot/1.2.0.M2/reference/
htmlsingle/
• DataRest
-­‐
hVp://docs.spring.io/spring-­‐data/rest/docs/2.3.2.RELEASE/reference/
html/
• Redis
-­‐
hVp://redis.io/documentaMon
• RestTemplate
-­‐
hVps://spring.io/blog/2009/03/27/rest-­‐in-­‐spring-­‐3-­‐resVemplate

22. Thank
you!
TwiVer:
@tsaltsol
Skype:
oleg.tsalko
GitHub:
olegts
Code
shown:
hVps://github.com/olegts/
SpringXDCustomTweetsTransformer
hVps://github.com/olegts/TweetsBootDataRestApp