Advanced Techniques to Accelerate Model Tuning | Software for AI Optimization Summit 2021 Technical Session

SigOpt. Confidential.
Advanced Techniques to Accelerate Model Tuning
Michael McCourt
Head of Engineering, SigOpt, an Intel company
June 8, 2021
Software for AI Optimization Summit

Today’s agenda
Advances in tuning AI models
• Implementing black box optimization strategies for tuning at scale
• Balancing multiple competing metrics
• Incorporating model information to accelerate tuning
© Intel Corporation. SigOpt and the SigOpt logo are trademarks of Intel Corporation or its subsidiaries
2

Tuning AI models
Making the right decisions for generalization
Training models finds the parameters which minimize the loss (inaccuracy) of a
model for a given set of training data and fixed set of hyperparameters.
Tuning models finds hyperparameters which allow a model to perform in a
production setting at a level which all the stakeholders find acceptable.
Today, we will discuss how SigOpt guides this tuning process.
3

SigOpt workflow diagram
Your firewall
Training
Data
AI, ML, DL, Simulation
Model
Model Evaluation or
Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale with
your needs
OPTIMIZATION ENGINE
Explore and exploit with a variety
of techniques
REST
API
Parameters or
Hyperparameters
5

Optimally tuning AI models at scale
How do we power and manage this optimization?
At our core, we use black box algorithms to power the optimization.
• Bayesian optimization (a.k.a., model-based optimization)
• Evolutionary algorithms
• Quasi-random sampling (through QMCPy)
The theoretical designs of these algorithms is insufficient for production.
• Limited time for our computations
• Multiple trainings running asynchronously in parallel
• Customer edits to the optimization domain/results
We focus our discussion only on Bayesian optimization, the most suitable for ML.
6

Standard BO computational workflow
Online/Offline computation
7

Asynchronous BO computational workflow
Online/Offline computation
API Request
Report
Results
API Request
Next Point
Retrieve
Next Point
Rerank All
Points
8

Balancing multiple competing metrics
How do we satisfy multiple stakeholders?
Tuning models with scalar optimization is satisfying, but often incomplete.
Real-world circumstances involve multiple competing metrics.
• High accuracy, low inference time (example)
• Maximizing true positives, minimizing false diagnoses (example)
• Minimal incorrect fraud alerts, easily interpretable model (example)
• High accuracy, limited size, low computation (example)
Our goal must change, from finding the answer to understanding our options.
9

Studying the Pareto frontier
Understanding the tradeoff between two metrics
Can we find hyperparameters that maximize both accuracy and inference speed?
Probably not … but we can learn what is possible.
Accuracy Speed
Accuracy
Speed
Pareto
Frontier
All
Possible
Models
10

Finding the Pareto frontier
What strategies exist?
Much literature exists on finding the Pareto frontier.
• Genetic algorithms are by far the most popular (core reference).
• They work best with massive experimentation/compute.
• This may be impractical in ML tuning.
Bayesian optimization has started to address this problem:
• First with linear combinations of metrics,
• Later with analysis of the Pareto frontier hypervolume,
• Also by adaptively enforcing artificial constraints.
These model-based strategies are more efficient than genetic algorithms.
11

Building neural network models for edge devices
Practical limitations on neural networks
Situation: When deploying neural networks to edge devices (e.g., cell phones or televisions) they must
perform well without requiring supercomputers to run them. Simplified form of this blog.
Metrics to be balanced: Validation accuracy, Stability under edge deployment
12 © Intel Corporation. SigOpt and the SigOpt logo are trademarks of Intel Corporation or its subsidiaries

Dealing with more than 2 metrics
What is practical?
Approximating the Pareto frontier with 2 metrics requires many points.
• Our algorithm has proved effective in tests and in practice.
• Moving beyond 2 metrics requires many more points.
• This may be infeasible for ML tuning situations that have long training times.
What can be done to manage this higher number of metrics?
• We could enforce some metrics as constraints.
• We could define success to more easily incorporate more metrics.
13

Incorporating more metrics as constraints
Turning a numeric metric into “good enough”
Customers can pass more than two metrics to us as constraints. (example)
• Example: Maximize accuracy, minimize computation, 0.15M or less network
size
Our algorithm modifies our computations to incorporate this constraint.
14
Grayed out
points
violate the
constraint

Constraint active search for many metrics
Exploring all the “good enough” parameters
If a user were only interested in satisfying constraints for all metrics …
• Every choice of hyperparameters would be only satisfactory/unsatisfactory.
• Model-building is independent of the number of metrics.
No optimization is needed … only a search to find all satisfactory models.
This is the thesis of our latest innovation: constraint active search.
• Explore the parameter space to give users actionable information as efficiently
as possible.
ICML 2021 paper accepted, feature to be released July 1.

Constraint active search for many metrics
Exploring all the “good enough” parameters

Incorporating model information to accelerate tuning
Moving beyond black box optimization
Our algorithms generally do not leverage information about customer models.
• Many customers want that knowledge to live behind their firewall.
Increasingly, customers seem willing to convey certain information to us.
• Example: The type of model, the role of hyperparameters, the size of their data
We want to use this information to give customers a better experience.

Features specific to gradient-boosted methods
What can be done?
To address model-specific problems, we need to consider a class of models.
• We have focused on XGBoost and LightGBM.
• These have well-defined structure and common hyperparameters.
• These are very popular, both broadly and within our customer base.
• These are likely to have both very experienced users and new users.

Automatic hyperparameter domain selection
If we know the model, we
may already know which
hyperparameters to
study.
Quality of life change for
our customers, and we
gain knowledge about
these hyperparameters.

XGBoost/LightGBM specific prior beliefs
Our computations are more effective
with prior beliefs -- knowledge about
which hyperparameter values are
likely to be good.
In this gradient-boosted setting, we
can load these beliefs for our
customers.

Thank you for having me!
We are starting a Q&A session now, or
email mccourt@sigopt.com with questions.
We’re hiring! New roles for
• Front end engineer
• Full stack engineer
• Product marketing manager
• Product design lead
• Senior product manager
Please contact mccourt@sigopt.com!

Try SigOpt for free
Interested in trying our product?
Sign up here or go to
app.sigopt.com/signup
Our free plan includes: 10 users per organization with 1 TB data storage, 500 hyperparameter optimization
experiments per month and 1,000 observations per experiment, unlimited tracked training runs, and email
support with 48-hour expected response time

Notices & Disclaimers
© Intel Corporation. Intel, SigOpt, the Intel logo, the SigOpt logo, and other Intel marks are
trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the
property of others.
No product or component can be absolutely secure.
Intel does not control or audit third-party data. You should consult other sources to evaluate
accuracy.
Your costs and results may vary.
Intel and SigOpt technologies may require enabled hardware, software or service activation.
Performance varies by use, configuration and other factors. Learn more at
https://sigopt.com/resources/ and www.Intel.com/PerformanceIndex.

Advanced Techniques to Accelerate Model Tuning | Software for AI Optimization Summit 2021 Technical Session

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Advanced Techniques to Accelerate Model Tuning | Software for AI Optimization Summit 2021 Technical Session

Similaire à Advanced Techniques to Accelerate Model Tuning | Software for AI Optimization Summit 2021 Technical Session (20)

Plus de Intel® Software

Plus de Intel® Software (16)

Dernier

Dernier (20)

Advanced Techniques to Accelerate Model Tuning | Software for AI Optimization Summit 2021 Technical Session