Overview of HPO Tools

Here we will give an overview of commonly used and well-known Hyperparameter Optimization (HPO) tools (where only a few of them were developed by us).  It is by no means complete. If you miss a tool on the list, please let us know.

Bayesian Optimization

Bayesian Optimization (BO) is considered to be a state-of-the-art approach for expensive black-box functions and thus has been widely implemented in different HPO tools.

Spearmint was one of the first successful open source Bayesian Optimization tools for HPO.

SMAC3 is well known for optimizing complex and structured spaces (e.g., for optimizing entire ML pipelines) based on RF-surrogate models.

Hyperopt is a distributed HPO tool implemented with a Tree of Parzen Estimators (TPE).

Scikit-optimize is a BO tool that is built on the top of scikit-learn (sklearn).

Bayesian Optimization is a (constrained) global optimization tool.

HyperMapper is a BO tool that supports working with unknown constraints and multi-objective optimization.

OpenBox is a general framework for black-box optimization, incl. HPO, and supports multi-objective optimization, multi-fidelity, early-stopping, transfer learning and parallel BO.

Dragonfly is an open source python library for scalable BO with multi-fidelity and multi-objective optimization.

GPflowOpt is a python tool for BO using GPflow and Tensorflow.

BoTorch is a tool doing BO with GPyTorch and PyTorch.

TurBO is a scalable BO tool for large scale paralleled HPO problems with many function evaluations.

Evolutionary Algorithms

Evolutionary algorithms (EA) are another popular black-box optimization framework, as EA can easily be parallelized and the time in each iteration does not grow as number of evaluations grow.

DEHB combines multi-fidelity optimization, such as in Hyperband, with differential evaluations.

DEAP is a novel evolutionary computation framework with lots of popular EA implementations

Nevergrad is a gradient-free optimization platform. Nevergrad  implements different EA that meets the requirements of different problems.

Expert-prior supported algorithms

PriorBand replaces random sampling in Hyperband with a collection of sampling strategies, including sampling from user priors.

General HPO Tools

Here we will introduce several tools that implement both BO and EA approaches

Optuna is an automatic HPO framework that allows to dynamically construct the search space.

Oríon is an asynchronous framework for black-box function optimization.

Ray Tune is a scalable framework HPO framework that allows to employ several aforementioned framework as optimizers.