Automatic composition and parametrisation of multicomponent predictive systems (MCPSs) consisting of chains of data transformation steps is a challenging task. In this paper we propose and describe an extension to the Auto-WEKA software which now allows to compose and optimise such flexible MCPSs by using a sequence of WEKA methods. In the experimental analysis we focus on examining the impact of significantly extending the search space by incorporating additional hyperparameters of the models, on the quality of the found solutions. In a range of extensive experiments three different optimisation strategies are used to automatically compose MCPSs on 21 publicly available datasets. A comparison with previous work indicates that extending the search space improves the classification accuracy in the majority of the cases. The diversity of the found MCPSs are also an indication that fully and automatically exploiting different combinations of data cleaning and preprocessing techniques is possible and highly beneficial for different predictive models. This can have a big impact on high quality predictive models development, maintenance and scalability aspects needed in modern application and deployment scenarios.

1. Towards Automatic Composition of
MultiComponent Predictive Systems
Manuel Martin Salvador, Marcin Budka, Bogdan Gabrys
Data Science Institute, Bournemouth University, UK
April 18th, 2016
Seville, Spain

3. Predictive modelling
Labelled
Data
Supervised
Learning
Algorithm
Predictive
Model

4. Data is imperfect
Missing
Values
Noise
High
dimensionality
Outliers
Question Mark: http://commons.wikimedia.org/wiki/File:Question_mark_road_sign,_Australia.jpg
Noise: http://www.flickr.com/photos/benleto/3223155821/
Outliers: http://commons.wikimedia.org/wiki/File:Diagrama_de_caixa_com_outliers_and_whisker.png
3D plot: http://salsahpc.indiana.edu/plotviz/

5. MultiComponent Predictive Systems
Data Postprocessing PredictionsPreprocessing
Predictive
Model

6. MultiComponent Predictive Systems
Preprocessing
Data
Predictive
Model
Postprocessing Predictions
Preprocessing
Preprocessing Predictive
Model
Predictive
Model

7. Algorithm Selection
What are the best
algorithms to
process my data?

8. Hyperparameter Optimisation
How to tune the
hyperparameters to get
the best performance?

9. CASH problem
k-fold cross validation
Combined Algorithm Selection and Hyperparameter configuration problem
Objective function
(e.g. classification error)
HyperparametersAlgorithms
Training dataset
Validation dataset
Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms.
In: Proc. of the 19th ACM SIGKDD. (2013) 847–855

10. Auto-WEKA
WEKA methods as search space
One-click black box
Data + Time Budget → MCPS
Our contribution
Recursive extension of complex
hyperparameters in the search space.
Code available in https://github.com/dsibournemouth/autoweka

11. Search space
Hyperparameters
PREV NEW
756 1186

12. Optimisation strategies
● Grid search: exhaustive exploration of the whole search space. Not feasible in high
dimensional spaces.
● Random search: explores the search space randomly during a given time.
● Bayesian optimisation: assumes that there is a function between the hyperparameters
and the objective and try to explore the most promising parts of the search space.
Hutter, F., Hoos, H. H., & Leyton-
Brown, K. (2011). Sequential
Model-Based Optimization for
General Algorithm Configuration.
Learning and Intelligent
Optimization, 6683 LNCS, 507–
523.

13. Evaluated strategies
1. WEKA-Def: All the predictors and meta-predictors are run using WEKA’s
default hyperparameter values.
2. Random search: The search space is randomly explored.
3. SMAC: Sequential Model-based Algorithm Configuration incrementally
builds a Random Forest as inner model.
4. TPE: Tree-structure Parzen Estimation uses Gaussian Processes to
incrementally build an inner model.
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011). Sequential Model-Based Optimization for General Algorithm Configuration. Learning and Intelligent Optimization,
6683 LNCS, 507–523.
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kegl, Algorithms for Hyper-Parameter Optimization. in Advances in NIPS 24, 2011, pp. 1–9.

14. Experiments
21 datasets (classification problems)
Budget: 30 CPU-hours (per run)
25 runs with different seeds
Timeout: 30 minutes
Memout: 3GB RAM

15. Results
Classification error on test set
● WEKA-Def (best): 1/21
● Random search (mean): 4/21
● SMAC (mean): 10/21
● TPE (mean): 6/21
Search spaces
● NEW > PREV: 52/63

16. Best MCPSs found

17. Conclusion and future work
Automation of composition and optimisation of MCPSs is feasible
Extending the search space has helped to find better solutions
Bayesian optimisation strategies have performed better than random search in
most cases
Future work:
● Still gap for improvement in Bayesian optimisation strategies.
● Multi-objective optimisation (e.g. time and error).
● Adaptive optimisation in changing environments.

18. Thank you!
msalvador@bournemouth.ac.uk
Paper available in https://dx.doi.org/10.1007/978-3-319-32034-2_3
Slides available in http://slideshare.net/draxus