AutoML

News

We held AutoML workshops at both ICML and ECMLPKDD this year. Check the workshop page for further information.
At ECMLPKDD, we also held a tutorial on AutoML; we’ve posted the slides for this on the tutorial webpage.

Overview: What is AutoML?

Machine learning (ML) has achieved considerable successes in recent years and an ever-growing number of disciplines rely on it. However, this success crucially relies on human machine learning experts to perform the following tasks:

  • Preprocess the data
  • Select appropriate features
  • Select an appropriate model family
  • Optimize model hyperparameters
  • Postprocess machine learning models
  • Critically analyze the results obtained.

As the complexity of these tasks is often beyond non-ML-experts, the rapid growth of machine learning applications has created a demand for off-the-shelf machine learning methods that can be used easily and without expert knowledge. We call the resulting research area that targets progressive automation of machine learning AutoML.
Although it focuses on end users without expert ML knowledge, AutoML also offers new tools to machine learning experts, for example to:

  • Perform architecture search over deep representations
  • Analyse the importance of hyperparameters.

Following the paradigm of Programming by Optimization, AutoML advocates the development of flexible software packages that can be instantiated automatically in a data-driven way.

Examples of AutoML

AutoML aims to create software that can be used out-of-the-box by ML novices. Some recent examples showcase what is possible:

  • AutoWEKA is an approach for the simultaneous selection of a machine learning algorithm and its hyperparameters; combined with the WEKA package it automatically yields good models for a wide variety of data sets.
  • Deep neural networks are notoriously dependent on their hyperparameters, and modern optimizers have achived better results in setting them than humans (Bergstra et al, Snoek et al).
  • Making a science of model search: a complex computer vision architecture could automatically be instantiated to yield state-of-the-art results on 3 different tasks: face matching, face identification, and object recognition.

Methods

AutoML draws on many disciplines of machine learning, prominently including

  • Bayesian optimization
  • Regression models for structured data and big data
  • Meta learning
  • Transfer learning, and
  • Combinatorial optimization.

Hyperparameter optimization systems

Several recent systems for the Bayesian optimization of machine learning hyperparameters facilitate AutoML. These include:

We also provide two packages for hyperparameter optimization: