Resource-Efficient Deep Learning Model Selection on Apache Spark

Resource-efficient Deep
Learning Model Selection
on Apache Spark
Yuhao Zhang and Supun Nakandala
ADALab, University of California, San Diego

About us
▪ PHD students from ADALab at UCSD, advised by
Prof. Arun Kumar
▪ Our research mission: democratize data science
▪ More:
Supun Nakandala
https://scnakandala.github.io/
Yuhao Zhang
https://yhzhang.info/
ADALab
https://adalabucsd.github.io/

Introduction
Artiﬁcial Neural Networks (ANNs) are
revolutionizing many domains - “Deep Learning”

Problem: training deep nets is Painful!
Batch size?
8, 16, 64, 256 ...
Model architecture?
3 layer CNN,5 layer
CNN, LSTM…
Learning rate?
0.1, 0.01, 0.001,
0.0001 ...
Regularization?
L2, L1, Dropout,
Batchnorm ...
4 4 4 4
256 Different conﬁgurations !
Model performance = f(model architecture, hyperparameters, ...)
→Trial and error
Need for speed → $$$
(Distributed DL)
→ Better utilization of resources

Outline
1. Background
a. Mini-batch SGD
b. Task Parallelism
c. Data Parallelism
2. Model Hopper Parallelism (MOP)
3. MOP on Apache Spark
a. Implementation
b. APIs
c. Tests

Introduction - mini-batch SGD
Model
Updated Model
η ∇
Learning
rate
Avg. of
gradients
X1 X2 y
1.1 2.3 0
0.9 1.6 1
0.6 1.3 1
... ... ...
... ... ...
... ... ...
... ... ...
... ... ...
... ... ...
One
mini-batch
The most popular algorithm family for
training deep nets

Introduction - mini-batch SGD
X1 X2 y
1.1 2.3 0
0.9 1.6 1
0.6 1.3 1
... ... ...
... ... ...
... ... ...
... ... ...
... ... ...
... ... ...
One epoch
One mini-batch
Sequential

Models (tasks)
Machines with replicated
datasets
Task Parallelism - Problem Setting

(Embarrassing) Task Parallelism
Con: wasted storage

(Embarrassing) Task Parallelism
Con: wasted network
Shared FS or
data repo

Data Parallelism - Problem Setting
Models(tasks)
Partitioned data
High data scalability

Data Parallelism
Queue
Training on one mini-batch
or full partition
● Update only per epoch: bulk synchronous parallelism
(model averaging)
○ Bad convergence
● Update per mini-batch: sync parameter server
○ + Async updates: async parameter server
○ + Decentralized: MPI allreduce (Horovod)
○ High communication cost
Updates

Task Parallelism
+ high throughput
- low data scalability
- memory/storage wastage
Data Parallelism
+ high data scalability
- low throughput
- high communication cost
Model Hopper Parallelism (Cerebro)
+ high throughput
+ high data scalability
+ low communication cost
+ no memory/storage wastage

Model Hopper Parallelism -
Problem Setting
Models (tasks)
Partitioned data

Model Hopper Parallelism
Training on full
local partitions
One
sub-epoch

Training on full
local partitions
Model hopping
& training
One
sub-epoch

Training on full
local partitions
Model hopping
& training
Model hopping
& training
One
sub-epoch

Training on full
local partitions
Model hopping
& training
Model hopping
& trainingOne
epoch
One
sub-epoch

Heterogeneous Tasks
Time
Redundant sync barrier!
Queue

Cerebro -- Data System with MOP

MOP (Cerebro)
on Spark Spark Driver
Cerebro
Scheduler
Spark Worker
Cerebro
Worker
Spark Worker
Cerebro
Worker
Distributed File System (HDFS, NFS)

Implementation Details
▪ Spark DataFrames converted to partitioned Parquet
and locally cached in workers
▪ TensorFlow threads run training on local data
partitions
▪ Model Hopping implemented via shared ﬁle system

Example: Grid Search on
Model Selection + Hyperparameter
Search
▪ Two model architecture: {VGG16, ResNet50}
▪ Two learning rate: {1e-4, 1e-6}
▪ Two batch size: {32, 256}

Initialization
from pyspark.sql import SparkSession
import cerebro
spark = SparkSession.builder.master(...) # initialize spark
spark_backend = cerebro.backend.SparkBackend(
spark_context=spark.sparkContext, num_workers=num_workers
) # initialize cerebro
data_store = cerebro.storage.HDFSStore('hdfs://...') # set the shared data
storage

Deﬁne the Models
params = {'model_arch':['vgg16', 'resnet50'], 'learning_rate':[1e-4, 1e-6], 'batch_size':[32, 256]}
def estimator_gen_fn(params):
'''A model factory that returns an estimator,
given the input hyper-parameters, as well as model architectures'''
if params['model_arch'] == 'resnet50':
model = ... # tf.keras model
elif params['model_arch'] == 'vgg16':
model = ... # tf.keras model
optimizer = tf.keras.optimizers.Adam(lr=params['learning_rate']) # choose optimizer
loss = ... # define loss
estimator = cerebro.keras.SparkEstimator(model=model,
optimizer=optimizer,
loss=loss,
batch_size=params['batch_size'])
return estimator

Run Grid Search
df = ... # read data in as Spark DataFrame
grid_search = cerebro.tune.GridSearch(spark_backend,
data_store,
estimator_gen_fn,
params,
epoch=5,
validation=0.2,
feature_columns=['features'],
label_columns=['labels'])
model = grid_search.fit(df)

Tests - Setups - Hardware
▪ 9-node cluster, 1 master + 8 workers
▪ On each nodes:
▪ Intel Xeon 10-core 2.20 GHz CPU x 2
▪ 192 GB RAM
▪ Nvidia P100 GPU x 1

Tests - Setups - Workload
▪ Model selection + hyperparameter tuning on
ImageNet
▪ Adam optimizer
▪ Grid search space:
▪ Model architecture: {ResNet50, VGG16}
▪ Learning rate: {1e-4, 1e-6}
▪ Batch size: {32, 256}
▪ L2 regularization: {1e-4, 1e-6}

Tests - Results - Learning Curves

Tests - Results - Per Epoch Runtimes
* Horovod uses GPU kernels for communication. Thus, it has high GPU utilization.

Tests - Results - Runtimes
* Horovod uses GPU kernels for communication. Thus, it has high GPU utilization.
System
Runtime (hrs/epoch)
GPU Utili. (%)
Storage
Footprint (GiB)
Train Validation
TF PS - Async 8.6 250
Horovod 92.1 250
Cerebro-Spark 2.63 0.57 42.4 250
TF Model Averaging 1.94 0.03 72.1 250
Celery 1.69 0.03 82.4 2000
Cerebro-Standalone 1.72 0.05 79.8 250

Tests - Cerebro-Spark Gantt Chart
▪ Only overhead: stragglers randomly caused by TF 2.1 Keras Model saving/loading.
Overheads range from 1% to 300%
Stragglers

Tests - Cerebro-Spark Gantt Chart
▪ One epoch of training
▪ (Almost) optimal!

Tests - Cerebro-Standalone Gantt Chart

Other Available Hyperparameter
Tuning Algorithms
▪ PBT
▪ HyperBand
▪ ASHA
▪ Hyperopt

More Features to Come
▪ Grouped learning
▪ API for transfer learning
▪ Model parallelism

References
▪ Cerebro project site
▪ https://adalabucsd.github.io/cerebro-system
▪ Github repo
▪ https://github.com/adalabucsd/cerebro-system
▪ Blog post
▪ https://adalabucsd.github.io/research-blog/cerebro.html
▪ Tech report
▪ https://adalabucsd.github.io/papers/TR_2020_Cerebro.pdf

Feedback
Your feedback is important to us.
Don’t forget to rate and
review the sessions.

Resource-Efficient Deep Learning Model Selection on Apache Spark

Resource-Efficient Deep Learning Model Selection on Apache Spark

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Resource-Efficient Deep Learning Model Selection on Apache Spark

Similaire à Resource-Efficient Deep Learning Model Selection on Apache Spark (20)

Plus de Databricks

Plus de Databricks (20)

Dernier

Dernier (20)

Resource-Efficient Deep Learning Model Selection on Apache Spark