This talk explains how you can train and tune efficiently using metric strategy to assign, store, and optimize a variety of metrics, even changing them over time. Tobias Andreassen, who supports a number of our systematic trading customers, explained how he helps customers tune more efficiently with these SigOpt features in real-world scenarios.

1. SigOpt. Confidential.
Talk #3
Training, Tuning, and Metric Strategy
SigOpt Talk Series
Tuning for Systematic Trading
Tobias Andreasen — Machine Learning Engineer
Tuesday, May 19, 2020

2. SigOpt. Confidential.
Abstract
SigOpt provides an extensive set of advanced features,
which help you, the expert, save time while
increasing model performance via experimentation.
Today, we will continue this talk series by discussing
training, tuning, and metric strategies in the context of
SigOpt, and use this to come up with a recipe for how
this can be used for model experimentation.

3. SigOpt. Confidential.
Overview of SigOpt

4. SigOpt. Confidential.
Accelerate and amplify the
impact of modelers everywhere

5. SigOpt. Confidential.
Experiment Insights Optimization Engine
Track, analyze and reproduce any
model to improve the productivity of
your modeling
Enterprise Platform
Automate hyperparameter tuning to
maximize the performance and impact
of your models
Standardize experimentation across
any combination of library,
infrastructure, model or task
On-Premise Hybrid/Multi
Solution: Experiment, optimize and analyze at scale
5

6. SigOpt. Confidential.
SigOpt Features
Enterprise
Platform
Optimization
Engine
Experiment
Insights
Reproducibility
Intuitive web dashboards
Cross-team permissions
and collaboration
Advanced experiment
visualizations
Usage insights
Parameter importance
analysis
Multimetric optimization
Continuous, categorical, or
integer parameters
Constraints and
failure regions
Up to 10k observations,
100 parameters
Multitask optimization and
high parallelism
Training Monitor and
Automated Early Stopping
Infrastructure agnostic
REST API
Parallel Resource Scheduler
Black-Box Interface
Tunes without
accessing any data
Libraries for Python,
Java, R, and MATLAB

7. SigOpt. Confidential.
Experimentation Recipe

8. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
Black Box Optimization
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
8

9. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
9
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
class SIGOPT:
APITOKEN =
'PPPPPOJUZPFSAAAAADSSEYPCPNPGGGGGLNLRYKSRBWYZTTFI'
CLIENTID = 1
PROJECT = 'quant_series'
Creating SigOpt Project
SigOpt Access Class

10. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
10
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Load Data

11. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
11
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
class Data:
X, Y = datasets.load_breast_cancer(return_X_y=True)
def __init__(self, test_size=0.3):
self.x_train, self.x_test, self.y_train, self.y_test = train_test_split(
self.X, self.Y, test_size=test_size)
def train_data(self,):
return [self.x_train, self.y_train]
def test_data(self,):
return [self.x_test, self.y_test]
def num_features(self,):
return self.X.shape[1]
def num_classes(self,):
return len(np.unique(self.Y))
Load Data
Data Loader Class

12. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
12
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Create Evaluation Framework

13. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
13
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
class Evaluation:
METRICS = [{'name': 'accuracy', 'objective': 'maximize', 'strategy': 'optimize'},
{'name': 'f1', 'objective': 'maximize', 'strategy': 'store'},
{'name': 'precision', 'objective': 'maximize', 'strategy': 'store'},
{'name': 'recall', 'objective': 'maximize', 'strategy': 'store'}, ]
@classmethod
def return_object(cls, y):
y_true = y[0]
y_hat = y[1]
# needs to be replaced by data input and score calculations
scores = []
scores.append(accuracy_score(y_true, y_hat))
scores.append(f1_score(y_true, y_hat, average='macro'))
scores.append(precision_score(
y_true, y_hat, average='macro', zero_division=1))
scores.append(recall_score(y_true, y_hat, average='macro'))
return [{'name': m['name'], 'value': s} for m, s in zip(cls.METRICS, scores)]
Create Evaluation Framework
Model Evaluation Class

14. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
14
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Create Model

15. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
15
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
class LogisticRegression:
NAME = 'Logistic Regression'
BUDGET = 10
PARAMETERS = [{'name': 'log_c', 'type': 'double',
'bounds': {'min': -11, 'max': 0}, 'grid': BUDGET}]
def __init__(self, assignment):
c = np.exp(assignment['log_c'])
self.model = LogisticRegression(C=c)
self.y_hat = None
self.y_true = None
def train(self, data):
x = data[0]
y = data[1]
self.model.fit(x, y)
def test(self, data):
x = data[0]
y = data[1]
self.y_hat = self.model.predict(x)
self.y_true = y
def evaluate(self,):
return [self.y_true, self.y_hat]
Create Model
Baseline Model Class

16. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
16
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Optimization Loop

17. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
17
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
conn = Connection(client_token=SIGOPT.APITOKEN)
experiment_meta = {
'type': 'grid',
'project': SIGOPT.PROJECT,
'name': Model.NAME,
'parameters': Model.PARAMETERS,
'metrics': Evaluation.METRICS}
experiment = conn.experiments().create(**experiment_meta)
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
model = Model(suggestion.assignments)
model.train(data.train_data())
model.test(data.test_data())
conn.experiments(experiment.id).observations().create(
suggestion=suggestion.id,
values=Evaluation.return_object(model.evaluate()),)
experiment = conn.experiments(experiment.id).fetch()
Optimization Loop
Experiment Create Call
Optimization Loop

18. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
18
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
conn = Connection(client_token=SIGOPT.APITOKEN)
experiment_meta = {
'type': 'grid',
'project': SIGOPT.PROJECT,
'name': Model.NAME,
'parameters': Model.PARAMETERS,
'metrics': Evaluation.METRICS}
experiment = conn.experiments().create(**experiment_meta)
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
model = Model(suggestion.assignments)
model.train(data.train_data())
model.test(data.test_data())
conn.experiments(experiment.id).observations().create(
suggestion=suggestion.id,
values=Evaluation.return_object(model.evaluate()),)
experiment = conn.experiments(experiment.id).fetch()
Optimization Loop
Experiment Create Call
Optimization Loop

19. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
19
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
conn = Connection(client_token=SIGOPT.APITOKEN)
experiment_meta = {
'type': 'grid',
'project': SIGOPT.PROJECT,
'name': Model.NAME,
'parameters': Model.PARAMETERS,
'metrics': Evaluation.METRICS}
experiment = conn.experiments().create(**experiment_meta)
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
model = Model(suggestion.assignments)
model.train(data.train_data())
model.test(data.test_data())
conn.experiments(experiment.id).observations().create(
suggestion=suggestion.id,
values=Evaluation.return_object(model.evaluate()),)
experiment = conn.experiments(experiment.id).fetch()
Optimization Loop
Experiment Create Call
Optimization Loop

20. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
20
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Metric Strategy
- Storing up to 50 non-optimized
metrics
- Gain insights through
additional model information
Objective
- Storing business specific
metrics to drive the best
decisions
- Hit business targets through
modeling
Adding Metrics

21. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
21
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Adding Metrics

22. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
22
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Investigating Metrics

23. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
23
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Investigating Metrics

24. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
24
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Metric Constraints
- Guardrail your modeling by
applying constraints to
non-optimized metrics
Objective
- Use business requirements
as constraints for your
modeling
- Maximize modeling impact
Adding Metric Constraints

25. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
25
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Adding Metric Constraints

26. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
26
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Create Model

27. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
27
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
class NeuralNetwork:
NAME = 'Neural Network'
BUDGET = 30
EPOCHS = 40
MAXCHECKPOINTS = {'max_checkpoints': EPOCHS, }
NUMFEATURES = None
NUMCLASSES = None
PARAMETERS = [{'name': 'num_nodes', 'type': 'int', 'bounds': {'min': 4, 'max': 40}},
{'name': 'log_learning_rate', 'type': 'double',
'bounds': {'min': -11, 'max': 0}},
{'name': 'batch_size', 'type': 'int', 'bounds': {'min': 4, 'max': 10}}]
def __init__(self, assignment):
self.num_nodes = assignment['num_nodes']
self.learning_rate = np.exp(assignment['log_learning_rate'])
self.batch_size = assignment['batch_size']
self.model = keras.Sequential(
[layers.Dense(self.num_nodes, activation='relu', input_shape=(self.NUMFEATURES,)),
layers.Dense(self.NUMCLASSES, activation='softmax'), ])
opt = keras.optimizers.Adam(learning_rate=self.learning_rate)
self.model.compile(
loss='sparse_categorical_crossentropy', optimizer=opt)
Create Model
Advanced Model Class

28. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
28
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Optimization Loop

29. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
29
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
conn = Connection(client_token=SIGOPT.APITOKEN)
experiment_meta = {
'project': SIGOPT.PROJECT,
'name': Model.NAME,
'parameters': Model.PARAMETERS,
'observation_budget': Model.BUDGET,
'training_monitor': Model.MAXCHECKPOINTS,
'metrics': Evaluation.METRICS,}
experiment = conn.experiments().create(**experiment_meta)
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
model = Model(suggestion.assignments)
training_run = conn.experiments(
experiment.id).training_runs().create(suggestion=suggestion.id)
for _ in range(model.EPOCHS):
model.train(data.train_data())
model.test(data.test_data())
checkpoint = conn.experiments(experiment.id).training_runs(
training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),)
Optimization Loop
Experiment Create Call
Optimization Loop with Training Monitor

30. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
30
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
conn = Connection(client_token=SIGOPT.APITOKEN)
experiment_meta = {
'project': SIGOPT.PROJECT,
'name': Model.NAME,
'parameters': Model.PARAMETERS,
'observation_budget': Model.BUDGET,
'training_monitor': Model.MAXCHECKPOINTS,
'metrics': Evaluation.METRICS,}
experiment = conn.experiments().create(**experiment_meta)
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
model = Model(suggestion.assignments)
training_run = conn.experiments(
experiment.id).training_runs().create(suggestion=suggestion.id)
for _ in range(model.EPOCHS):
model.train(data.train_data())
model.test(data.test_data())
checkpoint = conn.experiments(experiment.id).training_runs(
training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),)
Optimization Loop
Experiment Create Call
Optimization Loop with Training Monitor

31. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
Optimization Loop with Training Monitor
31
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
conn = Connection(client_token=SIGOPT.APITOKEN)
experiment_meta = {
'project': SIGOPT.PROJECT,
'name': Model.NAME,
'parameters': Model.PARAMETERS,
'observation_budget': Model.BUDGET,
'training_monitor': Model.MAXCHECKPOINTS,
'metrics': Evaluation.METRICS,}
experiment = conn.experiments().create(**experiment_meta)
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
model = Model(suggestion.assignments)
training_run = conn.experiments(
experiment.id).training_runs().create(suggestion=suggestion.id)
for _ in range(model.EPOCHS):
model.train(data.train_data())
model.test(data.test_data())
checkpoint = conn.experiments(experiment.id).training_runs(
training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),)
Optimization Loop
Experiment Create Call

32. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
32
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Training Monitor
- Monitor convergence for
better model performance
- Increase generalization by
seeing best convergence
Objective
- Decide on the best model for
production by performance as
well as convergence
Looking at Convergence

33. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
33
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Looking at Convergence

34. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
34
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
model = Model(suggestion.assignments)
training_run = conn.experiments(
experiment.id).training_runs().create(suggestion=suggestion.id)
for _ in range(model.EPOCHS):
model.train(data.train_data())
model.test(data.test_data())
checkpoint = conn.experiments(experiment.id).training_runs(
training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),)
if checkpoint.should_stop:
conn.experiments(experiment.id).observations().create(
training_run=training_run.id)
break
Looking at Convergence
Optimization Loop with Training Monitor and Early Stopping

35. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
35
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
model = Model(suggestion.assignments)
training_run = conn.experiments(
experiment.id).training_runs().create(suggestion=suggestion.id)
for _ in range(model.EPOCHS):
model.train(data.train_data())
model.test(data.test_data())
checkpoint = conn.experiments(experiment.id).training_runs(
training_run.id).checkpoints().create(values=Evaluation.return_object(model.evaluate()),)
if checkpoint.should_stop:
conn.experiments(experiment.id).observations().create(
training_run=training_run.id)
break
Looking at Convergence
Optimization Loop with Training Monitor and Early Stopping

36. SigOpt. Confidential.SigOpt. Confidential.
Key Use Cases
36
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
class SIGOPT:
APITOKEN =
'DBOKLMJUZPFSYYCCEDSSEYPNPNPPLOWHLNLRYKSRBDYZIIFI'
CLIENTID = 1
PROJECT = 'quant_series'
Finishing up Project
SigOpt Access Class

37. SigOpt. Confidential.SigOpt. Confidential.
Key Use Casesclass SIGOPT:
APITOKEN =
'DBOKLMJUZPFSYYCCEDSSEYPNPNPPLOWHLNLRYKSRBDYZIIFI'
CLIENTID = 1
PROJECT = 'quant_series'
37
Experimentation Recipe:
Project Hygiene
Data Handling
Evaluation Strategy
Baseline Modeling
Metric Strategy
Advanced Modeling
Optimization Feedback Loop
- Convergence
- Early Stopping
Project Handoff
Finishing up Project
SigOpt Access Class

38. SigOpt. Confidential.
Tobias Andreasen | tobias@sigopt.com
For more information visit: https://sigopt.com/research/
Questions?