This presentation introduces Tune and Fugue, frameworks for intuitive and scalable hyperparameter optimization (HPO). Tune supports both non-iterative and iterative HPO problems. For non-iterative problems, Tune supports grid search, random search, and Bayesian optimization. For iterative problems, Tune generalizes algorithms like Hyperband and Asynchronous Successive Halving. Tune allows tuning models both locally and in a distributed manner without code changes. The presentation demonstrates Tune's capabilities through examples tuning Scikit-Learn and Keras models. The goal of Tune and Fugue is to make HPO development easy, testable, and scalable.
5. Questions
● Is parameter tuning a machine learning problem?
● Are there common ways to tune both classical models and deep
learning models?
● Why is it so hard to do distributed parameter tuning?
6. Tuning Problems In General
General Parameter Tuning
Hyperparameter Tuning (for Machine Learning)
Some Classical
Models
Deep Learning Models
Some Classical
Models
Non-Iterative Problems Iterative Problems
7. Distributed Parameter Tuning
● Not everything can be parallelized
● The tuning logic is always complex and tedious
● Popular tuning frameworks are not distributed environment
friendly
● Spark is not suitable for iterative tuning problems
9. Our Goals
● For non-iterative problems:
○ Unify grid and random search, make other plugable
● For iterative problems:
○ Generalize SOTA algos such as Hyperband and ASHA
● For both
○ Tune both locally and distributedly without code change
○ Make tuning development iterable and testable
○ Minimize moving parts
○ Minimize interfaces
12. Random Search
a: Rand(0,1)
b: Choice(“a”,“b”)
c: 3.14
a:0.12, b:”a”, c:3.14
a:0.66, b:”a”, c:3.14
a:0.32, b:”b”, c:3.14
a:0.94, b:”a”, c:3.14
Search Space Candidates
Pros: complexity and distribution are controlled, good for continuous variables
Cons: by luck, not deterministic, large number of samples are normally needed
13. Bayesian Optimization
objective: a^2
a: Rand(-1,1)
-0.66 -> 0.76 -> -0.18
-> 0.75 -> 0.90
-> 0.07 -> 0.00
-> 0.41 -> 0.12 -> 0.66
Search Space Candidates
Pros: less compute to guess the optimal parameters
Cons: sequential operations may require more time
17. Challenges
● Realtime asynchronous communication
● The overhead for checkpointing iterations can be significant
● Single iterative problem can’t be parallelized
● A lot of boilerplate code
24. Let’s Collaborate!
● Create specialized higher level APIs for major tuning cases so
users can do tuning with minimal code and without learning
distributed systems
● Enable advanced users to create fully customized, platform
agnostic and scale agnostic tuning pipelines with tune’s lower
level APIs