Differentiable ARchiTecture Search (DARTS) was widely appreciated in the NAS community due to its conceptual simplicity and effectiveness. In order to relax the discrete space to be continuous, DARTS linearly combines different operation choices to create a mixed operation. Afterwards, one can apply standard gradient descent to optimize in this relaxed architecture space.
In our ICLR 2020 paper, RobustDARTS, we identify and investigate failure modes of DARTS that lead to degenerate architectures. By computing the Hessian of the validation loss with respect to the architectural parameters, we can use its eigenvalues as indicators to early stop the search or regularize more the one-shot model in DARTS, leading to more robust versions of DARTS.
Notable follow-ups:
– SmoothDARTS (ICML 2020)
– Rethinking Architecture Selection in Differentiable NAS (ICLR 2021)
– DARTS- (ICLR 2021)
– Lambda-DARTS (ICLR 2023)