Java BigData Full Stack Development (version 2.0)

Java BigData Full Stack
Development as is ...
Alexey Zinovyev, Java Trainer in EPAM

About
With IT since 2007
With Java since 2009
With Hadoop since 2012
With EPAM since 2015

3Java Big Data Full Stack Development
Contacts
E-mail : Alexey_Zinovyev@epam.com
Twitter : @zaleslaw @BigDataRussia
vk.com/big_data_russia Big Data Russia
vk.com/java_jvm Java & JVM langs

The Good Old Days

HRs & RMs are looking for Java developers

Is Java Dream Team waiting You?

Required Skills
• Advanced SQL
• Basic Linux
• Core Java & JVM
• Backend Development Experience
• Basic Computer Science Level

REAL WORLD

Let’s just use Javascript in frontend ONLY

In frontend
ONLY?

Cruel world

Do you know ML JS library?

Wild animals everywhere

And what I tell you

It’s Time for Java Superhero, yeah!

Before patterns discovering you should ..
• Select small pieces
• Define default values for missed
data
• Remove strange signals from data
• Merge some tables in one if
required

How it really works
• Share your date with us
• Our magic manipulations
• Building an answering machine
• PROFIT!!!

How to start?

WHAT IS BIG DATA?

Joke about Excel

5V

Every 60 seconds…

From Mobile Devices

From Industry

We started to keep and handle stupid new things!

10^6 rows
in MySQL

GB->TB->PB->?

Is BigData about PBs?

It’s hard to …
• .. store
• .. handle
• .. search in
• .. visualize
• .. send in network

Likes in Classmates: how to count?

Crazy Zoo
2012

Crazy Zoo
2016

What will be
lighted this
training

NOSQL

What’s the problem with RBDMS’s
• Caching
• Master/Slave
• Cluster
• Table Partitioning
• Sharding

Family

Database
party

Spring Data

How to start?

Java MongoDB Driver + Robomongo

BIG DATA TOOL MASTER
VS
DATA SCIENTIST

TRAIN
MODEL

Datasets
• Facebook users, tweets
• Trade transactions
• Government
• Medicine (genomic data)
• Telecommunications

Data Sources
• Relational Databases
• Data warehouses (Historical data)
• Files in CSV or in binary format
• Internet or electronic mails
• Scientific, research (R, Octave,
Matlab)

Hey, man, predict something!

Man or sofa?

Typical questions for DM
• Which loan applicants are high-risk?

• How do we detect phone card fraud?

• What is the revenue prediction for next year?

• What is the revenue prediction for next year?
• Can you recommend music for users?

Green circle is blue square or red
triangle? Let’s ask its neighbors!
kNN (k-nearest neighbor)

Collaborative Filtering

Machine Learning vs Traditional Programming

Data
Science

Can a Java programmer to be a Data Scientist?

Sexy Data Scientist

Real Data Scientist

How to start?

Weka

HADOOP

Hadoop and Data Knights

Hadoop

MapReduce in different languages

MapReduce for WordCount

Hadoop
Jobs

Hadoop frameworks
• Universal (MapReduce, Tez, RDD in Spark)
• Abstract (Pig, Pipeline Spark)
• SQL - like (Hive, Impala, Spark SQL)
• Processing graph (Giraph, GraphX)
• Machine Learning (Mahout, MLib)
• Stream processing (Spark Streaming, Storm)

SPARK

SPARK: the bloody son of MR
• MapReduce in memory
• Up to 50x faster than Hadoop
• RDD is a basic building block
(immutable distributed
collections of objects)
• Pipeline API (no needs in PIG)

Spark
Family

MLlib supports
• Classification and regression
• Collaborative filtering
• Clustering
• Dimensionality reduction
• Optimization

Code sample MLlib (K-Means)
// Cluster the data into two classes using KMeans
int numClusters = 2;
int numIterations = 20;
KMeansModel clusters = KMeans.train(parsedData.rdd(), numClusters, numIterations);
// Evaluate clustering by computing Within Set Sum of Squared Errors
double WSSSE = clusters.computeCost(parsedData.rdd());
System.out.println("Within Set Sum of Squared Errors = " + WSSSE);
// Save and load model
clusters.save(sc.sc(), "myModelPath");
KMeansModel sameModel = KMeansModel.load(sc.sc(), "myModelPath");

MLlib
• .. extends scikit-learn (Python lib) and Mahout
• .. runs fully on Spark and supports Spark’s Pipeline API
• .. dataset is represented by Spark SQL’s SchemaRDD
• .. supports Hive like external data source
• .. is well for large datasets and parallelized algorithms

It solves all problems!

How to start?

HDP Zoo

Ok, Google!

AWS Amazon

Infrastructure issues are waiting YOU!

DEEP LEARNING

Deep Learning help us build NEW FUTURE

HOW TO LEARN?

1. Read books and write ‘pet’ projects
DIFFERENT WAYS

2. Become a mentee in Mentoring Process
DIFFERENT WAYS

3. MOOC
DIFFERENT WAYS

3. MOOC
4. Take a training course
DIFFERENT WAYS

3. MOOC
4. Take a training course
5. Visit conferences
DIFFERENT WAYS

Recommended Books

Contacts
E-mail : Alexey_Zinovyev@epam.com
Twitter : @zaleslaw @BigDataRussia
vk.com/big_data_russia Big Data Russia
vk.com/java_jvm Java & JVM langs

Java BigData Full Stack Development (version 2.0)

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Java BigData Full Stack Development (version 2.0)

Similaire à Java BigData Full Stack Development (version 2.0) (20)

Plus de Alexey Zinoviev

Plus de Alexey Zinoviev (20)

Dernier

Dernier (20)

Java BigData Full Stack Development (version 2.0)