Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2021
Nunes, Matheus; Fraga, Paulo M; Pappa, Gisele L
Fitness Landscape Analysis of Graph Neural Network Architecture Search Spaces Proceedings Article
In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 876–884, Association for Computing Machinery, Lille, France, 2021, ISBN: 9781450383509.
@inproceedings{10.1145/3449639.3459318,
title = {Fitness Landscape Analysis of Graph Neural Network Architecture Search Spaces},
author = {Matheus Nunes and Paulo M Fraga and Gisele L Pappa},
url = {https://doi.org/10.1145/3449639.3459318},
doi = {10.1145/3449639.3459318},
isbn = {9781450383509},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference},
pages = {876–884},
publisher = {Association for Computing Machinery},
address = {Lille, France},
series = {GECCO '21},
abstract = {Neural Architecture Search (NAS) is the name given to a set of methods designed to
automatically configure the layout of neural networks. Their success on Convolutional
Neural Networks inspired its use on optimizing other types of neural network architectures,
including Graph Neural Networks (GNNs). GNNs have been extensively applied over several
collections of real-world data, achieving state-of-the-art results in tasks such as
circuit design, molecular structure generation and anomaly detection. Many GNN models
have been recently proposed, and choosing the best model for each problem has become
a cumbersome and error-prone task. Aiming to alleviate this problem, recent works
have proposed strategies for applying NAS to GNN models. However, different search
methods converge relatively fast in the search for a good architecture, which raises
questions about the structure of the problem. In this work we use Fitness Landscape
Analysis (FLA) measures to characterize the search space explored by NAS methods for
GNNs. We sample almost 90k different architectures that cover most of the fitness
range, and represent them using both a one-hot encoding and an embedding representation.
Results of the fitness distance correlation and dispersion metrics show the fitness
landscape is easy to be explored, and presents low neutrality.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
automatically configure the layout of neural networks. Their success on Convolutional
Neural Networks inspired its use on optimizing other types of neural network architectures,
including Graph Neural Networks (GNNs). GNNs have been extensively applied over several
collections of real-world data, achieving state-of-the-art results in tasks such as
circuit design, molecular structure generation and anomaly detection. Many GNN models
have been recently proposed, and choosing the best model for each problem has become
a cumbersome and error-prone task. Aiming to alleviate this problem, recent works
have proposed strategies for applying NAS to GNN models. However, different search
methods converge relatively fast in the search for a good architecture, which raises
questions about the structure of the problem. In this work we use Fitness Landscape
Analysis (FLA) measures to characterize the search space explored by NAS methods for
GNNs. We sample almost 90k different architectures that cover most of the fitness
range, and represent them using both a one-hot encoding and an embedding representation.
Results of the fitness distance correlation and dispersion metrics show the fitness
landscape is easy to be explored, and presents low neutrality.
Li, Boyang; Lu, Qing; Jiang, Weiwen; Jung, Taeho; Shi, Yiyu
A mining pool solution for novel proof-of-neural-architecture consensus Proceedings Article
In: 2021 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), pp. 1-3, 2021.
@inproceedings{9461067,
title = {A mining pool solution for novel proof-of-neural-architecture consensus},
author = {Boyang Li and Qing Lu and Weiwen Jiang and Taeho Jung and Yiyu Shi},
doi = {10.1109/ICBC51069.2021.9461067},
year = {2021},
date = {2021-01-01},
booktitle = {2021 IEEE International Conference on Blockchain and Cryptocurrency (ICBC)},
pages = {1-3},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Zhang, Miao; Su, Steven W; Pan, Shirui; Chang, Xiaojun; Abbasnejad, Ehsan M; Haffari, Reza
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients Proceedings Article
In: Meila, Marina; Zhang, Tong (Ed.): Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pp. 12557–12566, PMLR, 2021.
@inproceedings{DBLP:conf/icml/ZhangSPCAH21,
title = {iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients},
author = {Miao Zhang and Steven W Su and Shirui Pan and Xiaojun Chang and Ehsan M Abbasnejad and Reza Haffari},
editor = {Marina Meila and Tong Zhang},
url = {http://proceedings.mlr.press/v139/zhang21s.html},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 38th International Conference on Machine Learning,
ICML 2021, 18-24 July 2021, Virtual Event},
volume = {139},
pages = {12557--12566},
publisher = {PMLR},
series = {Proceedings of Machine Learning Research},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
ZHAO, Huan; YAO, Quanming; TU, Weiwei
Search to aggregate neighborhood for graph neural network Proceedings Article
In: 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 552-563, 2021.
@inproceedings{9458743,
title = {Search to aggregate neighborhood for graph neural network},
author = {Huan ZHAO and Quanming YAO and Weiwei TU},
url = {https://ieeexplore.ieee.org/abstract/document/9458743},
doi = {10.1109/ICDE51399.2021.00054},
year = {2021},
date = {2021-01-01},
booktitle = {2021 IEEE 37th International Conference on Data Engineering (ICDE)},
pages = {552-563},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Deng, Xueqing; Zhu, Yi; Tian, Yuxin; Newsam, Shawn D
AutoAdapt: Automated Segmentation Network Search for Unsupervised Domain Adaptation Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2106-13227,
title = {AutoAdapt: Automated Segmentation Network Search for Unsupervised Domain Adaptation},
author = {Xueqing Deng and Yi Zhu and Yuxin Tian and Shawn D Newsam},
url = {https://arxiv.org/abs/2106.13227},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2106.13227},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Negi, Shubham; Chakraborty, Indranil; Ankit, Aayush; Roy, Kaushik
NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2106-12125,
title = {NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems},
author = {Shubham Negi and Indranil Chakraborty and Aayush Ankit and Kaushik Roy},
url = {https://arxiv.org/abs/2106.12125},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2106.12125},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Wu, Robert; Saxena, Nayan; Jain, Rohan
Poisoning the Search Space in Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2106-14406,
title = {Poisoning the Search Space in Neural Architecture Search},
author = {Robert Wu and Nayan Saxena and Rohan Jain},
url = {https://arxiv.org/abs/2106.14406},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2106.14406},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Su, Xiu; You, Shan; Xie, Jiyang; Zheng, Mingkai; Wang, Fei; Qian, Chen; Zhang, Changshui; Wang, Xiaogang; Xu, Chang
Vision Transformer Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2106-13700,
title = {Vision Transformer Architecture Search},
author = {Xiu Su and Shan You and Jiyang Xie and Mingkai Zheng and Fei Wang and Chen Qian and Changshui Zhang and Xiaogang Wang and Chang Xu},
url = {https://arxiv.org/abs/2106.13700},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2106.13700},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Wang, Xinyi; Xiang, Tiange; Zhang, Chaoyi; Song, Yang; Liu, Dongnan; Huang, Heng; Cai, Weidong
BiX-NAS: Searching Efficient Bi-directional Architecture for Medical Image Segmentation Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2106-14033,
title = {BiX-NAS: Searching Efficient Bi-directional Architecture for Medical Image Segmentation},
author = {Xinyi Wang and Tiange Xiang and Chaoyi Zhang and Yang Song and Dongnan Liu and Heng Huang and Weidong Cai},
url = {https://arxiv.org/abs/2106.14033},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2106.14033},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Ney, Jonas; Loroch, Dominik Marek; Rybalkin, Vladimir; Weber, Nico; ü, Jens Kr; Wehn, Norbert
HALF: Holistic Auto Machine Learning for FPGAs Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2106-14771,
title = {HALF: Holistic Auto Machine Learning for FPGAs},
author = {Jonas Ney and Dominik Marek Loroch and Vladimir Rybalkin and Nico Weber and Jens Kr ü and Norbert Wehn},
url = {https://arxiv.org/abs/2106.14771},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2106.14771},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Niu, Shuaicheng; Wu, Jiaxiang; Xu, Guanghui; Zhang, Yifan; Guo, Yong; Zhao, Peilin; Wang, Peng; Tan, Mingkui
AdaXpert: Adapting Neural Architecture for Growing Data Proceedings Article
In: Meila, Marina; Zhang, Tong (Ed.): Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pp. 8184–8194, PMLR, 2021.
@inproceedings{DBLP:conf/icml/NiuWXZGZWT21,
title = {AdaXpert: Adapting Neural Architecture for Growing Data},
author = {Shuaicheng Niu and Jiaxiang Wu and Guanghui Xu and Yifan Zhang and Yong Guo and Peilin Zhao and Peng Wang and Mingkui Tan},
editor = {Marina Meila and Tong Zhang},
url = {http://proceedings.mlr.press/v139/niu21a.html},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 38th International Conference on Machine Learning,
ICML 2021, 18-24 July 2021, Virtual Event},
volume = {139},
pages = {8184--8194},
publisher = {PMLR},
series = {Proceedings of Machine Learning Research},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Barakbayeva, Togzhan; Demirci, Fatih M
Fully automatic CNN design with inception blocks Proceedings Article
In: Jiang, Xudong; Fujita, Hiroshi (Ed.): Thirteenth International Conference on Digital Image Processing (ICDIP 2021), pp. 275 – 280, International Society for Optics and Photonics SPIE, 2021.
@inproceedings{,
title = {Fully automatic CNN design with inception blocks},
author = {Togzhan Barakbayeva and Fatih M Demirci},
editor = {Xudong Jiang and Hiroshi Fujita},
url = {https://doi.org/10.1117/12.2601117},
year = {2021},
date = {2021-01-01},
booktitle = {Thirteenth International Conference on Digital Image Processing (ICDIP 2021)},
volume = {11878},
pages = {275 -- 280},
publisher = {SPIE},
organization = {International Society for Optics and Photonics},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Wang, Dong-Sheng; Zhao, Cui-Ping; Wang, Qi; Ofori, Kwame Dwomoh; Han, Bin; Liu, Ga-Qiong; Wang, Shi; Wang, Hua-Yu; Zhang, Jing
ANAS: Sentence Similarity Calculation Based on Automatic Neural Architecture Search Proceedings Article
In: Shi, Zhongzhi; Chakraborty, Mihir; Kar, Samarjit (Ed.): Intelligence Science III, pp. 185-195, 2021.
@inproceedings{10.1007/978-3-030-74826-5_16b,
title = {ANAS: Sentence Similarity Calculation Based on Automatic Neural Architecture Search},
author = {Wang, Dong-Sheng and Zhao, Cui-Ping and Wang, Qi and Ofori, Kwame Dwomoh and Han, Bin and Liu, Ga-Qiong and Wang, Shi and Wang, Hua-Yu and Zhang, Jing},
editor = {Shi, Zhongzhi and Chakraborty, Mihir and Kar, Samarjit},
url = {https://link.springer.com/chapter/10.1007%2F978-3-030-74826-5_16},
year = {2021},
date = {2021-01-01},
booktitle = {Intelligence Science III},
pages = {185-195},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Xu, Jingjing; Zhao, Liang; Lin, Junyang; Gao, Rundong; Sun, Xu; Yang, Hongxia
KNAS: Green Neural Architecture Search Proceedings Article
In: Meila, Marina; Zhang, Tong (Ed.): Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pp. 11613–11625, PMLR, 2021.
@inproceedings{DBLP:conf/icml/XuZLG0Y21,
title = {KNAS: Green Neural Architecture Search},
author = {Jingjing Xu and Liang Zhao and Junyang Lin and Rundong Gao and Xu Sun and Hongxia Yang},
editor = {Marina Meila and Tong Zhang},
url = {http://proceedings.mlr.press/v139/xu21m.html},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 38th International Conference on Machine Learning,
ICML 2021, 18-24 July 2021, Virtual Event},
volume = {139},
pages = {11613--11625},
publisher = {PMLR},
series = {Proceedings of Machine Learning Research},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Papież, Bartłomiej W; Yaqub, Mohammad; Jiao, Jianbo; Namburete, Ana I L; Noble, Alison J (Ed.)
Improving Generalization of ENAS-Based CNN Models for Breast Lesion Classification from Ultrasound Images Proceedings Article
In: Papież, Bartłomiej W; Yaqub, Mohammad; Jiao, Jianbo; Namburete, Ana I L; Noble, Alison J (Ed.): Medical Image Understanding and Analysis, pp. 438–453, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-80432-9.
@inproceedings{10.1007/978-3-030-80432-9_33,
title = {Improving Generalization of ENAS-Based CNN Models for Breast Lesion Classification from Ultrasound Images},
editor = {Bart{ł}omiej W Papie{ż} and Mohammad Yaqub and Jianbo Jiao and Ana I L Namburete and Alison J Noble},
url = {https://link.springer.com/chapter/10.1007/978-3-030-80432-9_33},
isbn = {978-3-030-80432-9},
year = {2021},
date = {2021-01-01},
booktitle = {Medical Image Understanding and Analysis},
pages = {438--453},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Neural Architecture Search (NAS) is one of the most recent developments in automating the design process for deep convolutional neural network (DCNN) architectures. NAS and later Efficient NAS (ENAS) based models have been adopted successfully for various applications including ultrasound image classification for breast lesions. Such a data driven approach leads to creation of DCNN models that are more applicable to the data set at hand but with a risk for model overfitting. In this paper, we first investigate the extent of the ENAS model generalization error problem by using different test data sets of ultrasound images of breast lesions. We have observed a significant reduction of overall average accuracy by nearly 10% and even more severe reduction of specificity rate by more than 20%, indicating that model generalization error is a serious issue with ENAS models for breast lesion classification in ultrasound images. To overcome the generalization error, we examined the effectiveness of a range of techniques including reducing model complexity, use of data augmentation, and use of unbalanced training sets. Experimental results show that different methods for the tuned ENAS models achieved different levels of accuracy when they are tested on internal and two external test data sets. The paper demonstrates that ENAS models trained on an unbalanced dataset with more benign cases tend to generalize well on unseen images achieving average accuracies of 85.8%, 82.7%, and 88.1% respectively for the internal and the two external test data sets not only on specificity alone, but also sensitivity. In particular, the generalization of the refined models across internal and external test data is maintained.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Chen, Boyu; Li, Peixia; Li, Chuming; Li, Baopu; Bai, Lei; Lin, Chen; Sun, Ming; Yan, Junjie; Ouyang, Wanli
GLiT: Neural Architecture Search for Global and Local Image Transformer Proceedings Article
In: 2021 {IEEE/CVF} International Conference on Computer Vision, {ICCV} 2021, Montreal, QC, Canada, October 10-17, 2021, 2021.
@inproceedings{DBLP:journals/corr/abs-2107-02960,
title = {GLiT: Neural Architecture Search for Global and Local Image Transformer},
author = {Boyu Chen and Peixia Li and Chuming Li and Baopu Li and Lei Bai and Chen Lin and Ming Sun and Junjie Yan and Wanli Ouyang},
url = {https://arxiv.org/abs/2107.02960},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {2021 {IEEE/CVF} International Conference on Computer Vision, {ICCV}
2021, Montreal, QC, Canada, October 10-17, 2021},
journal = {CoRR},
volume = {abs/2107.02960},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Elsken, Thomas; Staffler, Benedikt; Zela, Arber; Metzen, Jan Hendrik; Hutter, Frank
Bag of Tricks for Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-03719,
title = {Bag of Tricks for Neural Architecture Search},
author = {Thomas Elsken and Benedikt Staffler and Arber Zela and Jan Hendrik Metzen and Frank Hutter},
url = {https://arxiv.org/abs/2107.03719},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.03719},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Lomurno, Eugenio; Samele, Stefano; Matteucci, Matteo; Ardagna, Danilo
Pareto-Optimal Progressive Neural Architecture Search Proceedings Article
In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1726–1734, Association for Computing Machinery, Lille, France, 2021, ISBN: 9781450383516.
@inproceedings{10.1145/3449726.3463146,
title = {Pareto-Optimal Progressive Neural Architecture Search},
author = {Eugenio Lomurno and Stefano Samele and Matteo Matteucci and Danilo Ardagna},
url = {https://doi.org/10.1145/3449726.3463146},
doi = {10.1145/3449726.3463146},
isbn = {9781450383516},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference Companion},
pages = {1726–1734},
publisher = {Association for Computing Machinery},
address = {Lille, France},
series = {GECCO '21},
abstract = {Neural Architecture Search (NAS) is the process of automating architecture engineering,
searching for the best deep learning configuration. One of the main NAS approaches
proposed in the literature, Progressive Neural Architecture Search (PNAS), seeks for
the architectures with a sequential model-based optimization strategy: it defines
a common recursive structure to generate the networks, whose number of building blocks
rises through iterations. However, NAS algorithms are generally designed for an ideal
setting without considering the needs and the technical constraints imposed by practical
applications. In this paper, we propose a new architecture search named Pareto-Optimal
Progressive Neural Architecture Search (POPNAS) that combines the benefits of PNAS
to a time-accuracy Pareto optimization problem. POPNAS adds a new time predictor to
the existing approach to carry out a joint prediction of time and accuracy for each
candidate neural network, searching through the Pareto front. This allows us to reach
a trade-off between accuracy and training time, identifying neural network architectures
with competitive accuracy in the face of a drastically reduced training time.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
searching for the best deep learning configuration. One of the main NAS approaches
proposed in the literature, Progressive Neural Architecture Search (PNAS), seeks for
the architectures with a sequential model-based optimization strategy: it defines
a common recursive structure to generate the networks, whose number of building blocks
rises through iterations. However, NAS algorithms are generally designed for an ideal
setting without considering the needs and the technical constraints imposed by practical
applications. In this paper, we propose a new architecture search named Pareto-Optimal
Progressive Neural Architecture Search (POPNAS) that combines the benefits of PNAS
to a time-accuracy Pareto optimization problem. POPNAS adds a new time predictor to
the existing approach to carry out a joint prediction of time and accuracy for each
candidate neural network, searching through the Pareto front. This allows us to reach
a trade-off between accuracy and training time, identifying neural network architectures
with competitive accuracy in the face of a drastically reduced training time.
-, Seyed Mojtaba Marvasti; Khaghani, Javad; Cheng, Li; -, Hossein Ghanei; Kasaei, Shohreh
CHASE: Robust Visual Tracking via Cell-Level Differentiable Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-03463,
title = {CHASE: Robust Visual Tracking via Cell-Level Differentiable Neural Architecture Search},
author = {Seyed Mojtaba Marvasti - and Javad Khaghani and Li Cheng and Hossein Ghanei - and Shohreh Kasaei},
url = {https://arxiv.org/abs/2107.03463},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.03463},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Li, Jianquan; Liu, Xiaokang; Zhang, Sheng; Yang, Min; Xu, Ruifeng; Qin, Fengqing
Accelerating Neural Architecture Search for Natural Language Processing with Knowledge Distillation and Earth Mover's Distance Proceedings Article
In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2091–2095, Association for Computing Machinery, Virtual Event, Canada, 2021, ISBN: 9781450380379.
@inproceedings{10.1145/3404835.3463017,
title = {Accelerating Neural Architecture Search for Natural Language Processing with Knowledge Distillation and Earth Mover's Distance},
author = {Jianquan Li and Xiaokang Liu and Sheng Zhang and Min Yang and Ruifeng Xu and Fengqing Qin},
url = {https://doi.org/10.1145/3404835.3463017},
doi = {10.1145/3404835.3463017},
isbn = {9781450380379},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {2091–2095},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Canada},
series = {SIGIR '21},
abstract = {Recent AI research has witnessed increasing interests in automatically designing the
architecture of deep neural networks, which is coined as neural architecture search
(NAS). The automatically searched network architectures via NAS methods have outperformed
manually designed architectures on some NLP tasks. However, training a large number
of model configurations for efficient NAS is computationally expensive, creating a
substantial barrier for applying NAS methods in real-life applications. In this paper,
we propose to accelerate neural architecture search for natural language processing
based on knowledge distillation (called KD-NAS). Specifically, instead of searching
the optimal network architecture on the validation set conditioned on the optimal
network weights on the training set, we learn the optimal network by minimizing the
knowledge loss transferred from a pre-trained teacher network to the searching network
based on Earth Mover's Distance (EMD). Experiments on five datasets show that our
method achieves promising performance compared to strong competitors on both accuracy
and searching speed. For reproducibility, we submit the code at: https://github.com/lxk00/KD-NAS-EMD.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
architecture of deep neural networks, which is coined as neural architecture search
(NAS). The automatically searched network architectures via NAS methods have outperformed
manually designed architectures on some NLP tasks. However, training a large number
of model configurations for efficient NAS is computationally expensive, creating a
substantial barrier for applying NAS methods in real-life applications. In this paper,
we propose to accelerate neural architecture search for natural language processing
based on knowledge distillation (called KD-NAS). Specifically, instead of searching
the optimal network architecture on the validation set conditioned on the optimal
network weights on the training set, we learn the optimal network by minimizing the
knowledge loss transferred from a pre-trained teacher network to the searching network
based on Earth Mover's Distance (EMD). Experiments on five datasets show that our
method achieves promising performance compared to strong competitors on both accuracy
and searching speed. For reproducibility, we submit the code at: https://github.com/lxk00/KD-NAS-EMD.
Chen, Jiamin; Gao, Jianliang; Chen, Yibo; Oloulade, Moctard Babatounde; Lyu, Tengfei; Li, Zhao
GraphPAS: Parallel Architecture Search for Graph Neural Networks Proceedings Article
In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2182–2186, Association for Computing Machinery, Virtual Event, Canada, 2021, ISBN: 9781450380379.
@inproceedings{10.1145/3404835.3463007,
title = {GraphPAS: Parallel Architecture Search for Graph Neural Networks},
author = {Jiamin Chen and Jianliang Gao and Yibo Chen and Moctard Babatounde Oloulade and Tengfei Lyu and Zhao Li},
url = {https://doi.org/10.1145/3404835.3463007},
doi = {10.1145/3404835.3463007},
isbn = {9781450380379},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {2182–2186},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Canada},
series = {SIGIR '21},
abstract = {Graph neural architecture search has received a lot of attention as Graph Neural Networks
(GNNs) has been successfully applied on the non-Euclidean data recently. However,
exploring all possible GNNs architectures in the huge search space is too time-consuming
or impossible for big graph data. In this paper, we propose a parallel graph architecture
search (GraphPAS) framework for graph neural networks. In GraphPAS, we explore the
search space in parallel by designing a sharing-based evolution learning, which can
improve the search efficiency without losing the accuracy. Additionally, architecture
information entropy is adopted dynamically for mutation selection probability, which
can reduce space exploration. The experimental result shows that GraphPAS outperforms
state-of-art models with efficiency and accuracy simultaneously.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
(GNNs) has been successfully applied on the non-Euclidean data recently. However,
exploring all possible GNNs architectures in the huge search space is too time-consuming
or impossible for big graph data. In this paper, we propose a parallel graph architecture
search (GraphPAS) framework for graph neural networks. In GraphPAS, we explore the
search space in parallel by designing a sharing-based evolution learning, which can
improve the search efficiency without losing the accuracy. Additionally, architecture
information entropy is adopted dynamically for mutation selection probability, which
can reduce space exploration. The experimental result shows that GraphPAS outperforms
state-of-art models with efficiency and accuracy simultaneously.
Tan, Haoxian; Guo, Sheng; Zhong, Yujie; Huang, Weilin
Mutually-aware Sub-Graphs Differentiable Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-04324,
title = {Mutually-aware Sub-Graphs Differentiable Architecture Search},
author = {Haoxian Tan and Sheng Guo and Yujie Zhong and Weilin Huang},
url = {https://arxiv.org/abs/2107.04324},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.04324},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Narayanan, Ashwin Raaghav; Zela, Arber; Saikia, Tonmoy; Brox, Thomas; Hutter, Frank
Multi-headed Neural Ensemble Search Proceedings Article
In: -, Hsuan (Ed.): ICML 2021 Workshop on Uncertainty and Robust-ness in Deep Learning, 2021.
@inproceedings{DBLP:journals/corr/abs-2107-04369,
title = {Multi-headed Neural Ensemble Search},
author = {Ashwin Raaghav Narayanan and Arber Zela and Tonmoy Saikia and Thomas Brox and Frank Hutter},
editor = {Hsuan -},
url = {https://arxiv.org/abs/2107.04369},
year = {2021},
date = {2021-01-01},
booktitle = { ICML 2021 Workshop on Uncertainty and Robust-ness in Deep Learning},
journal = {CoRR},
volume = {abs/2107.04369},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Li, Victoria R; Zhang, Zijun; Troyanskaya, Olga G
CROTON: an automated and variant-aware deep learning framework for predicting CRISPR/Cas9 editing outcomes Journal Article
In: Bioinformatics, vol. 37, no. Supplement_1, pp. i342-i348, 2021, ISSN: 1367-4803.
@article{10.1093/bioinformatics/btab268,
title = {CROTON: an automated and variant-aware deep learning framework for predicting CRISPR/Cas9 editing outcomes},
author = {Victoria R Li and Zijun Zhang and Olga G Troyanskaya},
url = {https://doi.org/10.1093/bioinformatics/btab268},
doi = {10.1093/bioinformatics/btab268},
issn = {1367-4803},
year = {2021},
date = {2021-01-01},
journal = {Bioinformatics},
volume = {37},
number = {Supplement_1},
pages = {i342-i348},
abstract = {CRISPR/Cas9 is a revolutionary gene-editing technology that has been widely utilized in biology, biotechnology and medicine. CRISPR/Cas9 editing outcomes depend on local DNA sequences at the target site and are thus predictable. However, existing prediction methods are dependent on both feature and model engineering, which restricts their performance to existing knowledge about CRISPR/Cas9 editing.Herein, deep multi-task convolutional neural networks (CNNs) and neural architecture search (NAS) were used to automate both feature and model engineering and create an end-to-end deep-learning framework, CROTON (CRISPR Outcomes Through cONvolutional neural networks). The CROTON model architecture was tuned automatically with NAS on a synthetic large-scale construct-based dataset and then tested on an independent primary T cell genomic editing dataset. CROTON outperformed existing expert-designed models and non-NAS CNNs in predicting 1 base pair insertion and deletion probability as well as deletion and frameshift frequency. Interpretation of CROTON revealed local sequence determinants for diverse editing outcomes. Finally, CROTON was utilized to assess how single nucleotide variants (SNVs) affect the genome editing outcomes of four clinically relevant target genes: the viral receptors ACE2 and CCR5 and the immune checkpoint inhibitors CTLA4 and PDCD1. Large SNV-induced differences in CROTON predictions in these target genes suggest that SNVs should be taken into consideration when designing widely applicable gRNAs.https://github.com/vli31/CROTON.Supplementary data are available at Bioinformatics online.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Wen, Yu-Wei; Peng, Sheng-Hsuan; Ting, Chuan-Kang
Two-Stage Evolutionary Neural Architecture Search for Transfer Learning Journal Article
In: IEEE Transactions on Evolutionary Computation, pp. 1-1, 2021.
@article{9488292,
title = {Two-Stage Evolutionary Neural Architecture Search for Transfer Learning},
author = {Yu-Wei Wen and Sheng-Hsuan Peng and Chuan-Kang Ting},
url = {https://ieeexplore.ieee.org/abstract/document/9488292},
doi = {10.1109/TEVC.2021.3097937},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Evolutionary Computation},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
-hun Shim, Jae; Kong, Kyeongbo; Kang, Suk -Ju
Core-set Sampling for Efficient Neural Architecture Search Proceedings Article
In: ICML 2021 Workshop on Subset Selection in ML, 2021.
@inproceedings{DBLP:journals/corr/abs-2107-06869,
title = {Core-set Sampling for Efficient Neural Architecture Search},
author = {Jae -hun Shim and Kyeongbo Kong and Suk -Ju Kang},
url = {https://arxiv.org/abs/2107.06869},
year = {2021},
date = {2021-01-01},
booktitle = { ICML 2021 Workshop on Subset Selection in ML},
journal = {CoRR},
volume = {abs/2107.06869},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Sinha, Nilotpal; -, Kuan
Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-07266,
title = {Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy},
author = {Nilotpal Sinha and Kuan -},
url = {https://arxiv.org/abs/2107.07266},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.07266},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Chen, Lei; Yuan, Fajie; Yang, Jiaxi; Yang, Min; Li, Chengming
Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-07173,
title = {Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search},
author = {Lei Chen and Fajie Yuan and Jiaxi Yang and Min Yang and Chengming Li},
url = {https://arxiv.org/abs/2107.07173},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.07173},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Liu, Zhu; Ma, Long; Liu, Risheng; Fan, Xin
Learning to Discover a Unified Architecture for Low-Level Vision Journal Article
In: IEEE Signal Processing Letters, vol. 28, pp. 1470-1474, 2021.
@article{9483692,
title = {Learning to Discover a Unified Architecture for Low-Level Vision},
author = {Zhu Liu and Long Ma and Risheng Liu and Xin Fan},
url = {https://ieeexplore.ieee.org/abstract/document/9483692},
doi = {10.1109/LSP.2021.3096456},
year = {2021},
date = {2021-01-01},
journal = {IEEE Signal Processing Letters},
volume = {28},
pages = {1470-1474},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Gao, Jiahui; Xu, Hang; Shi, Han; Ren, Xiaozhe; Yu, Philip L H; Liang, Xiaodan; Jiang, Xin; Li, Zhenguo
AutoBERT-Zero: Evolving BERT Backbone from Scratch Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-07445,
title = {AutoBERT-Zero: Evolving BERT Backbone from Scratch},
author = {Jiahui Gao and Hang Xu and Han Shi and Xiaozhe Ren and Philip L H Yu and Xiaodan Liang and Xin Jiang and Zhenguo Li},
url = {https://arxiv.org/abs/2107.07445},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.07445},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Chrostoforidis, Aristeidis; Kyriakides, George; Margaritis, Konstantinos G
A Novel Evolutionary Algorithm for Hierarchical Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-08484,
title = {A Novel Evolutionary Algorithm for Hierarchical Neural Architecture Search},
author = {Aristeidis Chrostoforidis and George Kyriakides and Konstantinos G Margaritis},
url = {https://arxiv.org/abs/2107.08484},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.08484},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Odema, Mohanad; Rashid, Nafiul; Demirel, Berken Utku; Faruque, Mohammad Abdullah Al
LENS: Layer Distribution Enabled Neural Architecture Search in Edge-Cloud Hierarchies Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-09309,
title = {LENS: Layer Distribution Enabled Neural Architecture Search in Edge-Cloud Hierarchies},
author = {Mohanad Odema and Nafiul Rashid and Berken Utku Demirel and Mohammad Abdullah Al Faruque},
url = {https://arxiv.org/abs/2107.09309},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.09309},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Roth, Holger R; Yang, Dong; Li, Wenqi; Myronenko, Andriy; Zhu, Wentao; Xu, Ziyue; Wang, Xiaosong; Xu, Daguang
Federated Whole Prostate Segmentation in MRI with Personalized Neural Architectures Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-08111,
title = {Federated Whole Prostate Segmentation in MRI with Personalized Neural Architectures},
author = {Holger R Roth and Dong Yang and Wenqi Li and Andriy Myronenko and Wentao Zhu and Ziyue Xu and Xiaosong Wang and Daguang Xu},
url = {https://arxiv.org/abs/2107.08111},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.08111},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Louati, Hassen; Bechikh, Slim; Louati, Ali; Aldaej, Abdulaziz; Said, Lamjed Ben
Evolutionary Optimization of Convolutional Neural Network Architecture Design for Thoracic X-Ray Image Classification Proceedings Article
In: Fujita, Hamido; Selamat, Ali; -, Jerry Chun; Ali, Moonis (Ed.): Advances and Trends in Artificial Intelligence. Artificial Intelligence Practices - 34th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2021, Kuala Lumpur, Malaysia, July 26-29, 2021, Proceedings, Part I, pp. 121–132, Springer, 2021.
@inproceedings{DBLP:conf/ieaaie/LouatiBLAS21,
title = {Evolutionary Optimization of Convolutional Neural Network Architecture Design for Thoracic X-Ray Image Classification},
author = {Hassen Louati and Slim Bechikh and Ali Louati and Abdulaziz Aldaej and Lamjed Ben Said},
editor = {Hamido Fujita and Ali Selamat and Jerry Chun - and Moonis Ali},
url = {https://doi.org/10.1007/978-3-030-79457-6_11},
doi = {10.1007/978-3-030-79457-6_11},
year = {2021},
date = {2021-01-01},
booktitle = {Advances and Trends in Artificial Intelligence. Artificial Intelligence
Practices - 34th International Conference on Industrial, Engineering
and Other Applications of Applied Intelligent Systems, IEA/AIE 2021,
Kuala Lumpur, Malaysia, July 26-29, 2021, Proceedings, Part I},
volume = {12798},
pages = {121--132},
publisher = {Springer},
series = {Lecture Notes in Computer Science},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Thuy, Hang Duong Thi; Minh, Tuan Nguyen; Van, Phi Nguyen; Quoc, Long Tran
Fully Automated Machine Learning Pipeline for Echocardiogram Segmentation Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-08440,
title = {Fully Automated Machine Learning Pipeline for Echocardiogram Segmentation},
author = {Hang Duong Thi Thuy and Tuan Nguyen Minh and Phi Nguyen Van and Long Tran Quoc},
url = {https://arxiv.org/abs/2107.08440},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.08440},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Zhang, Peng; Tang, Jinsong; Zhong, Heping; Ning, Mingqiang; Liu, Dandan; Wu, Ke
Self-Trained Target Detection of Radar and Sonar Images Using Automatic Deep Learning Journal Article
In: IEEE Transactions on Geoscience and Remote Sensing, pp. 1-14, 2021.
@article{9489360,
title = {Self-Trained Target Detection of Radar and Sonar Images Using Automatic Deep Learning},
author = {Peng Zhang and Jinsong Tang and Heping Zhong and Mingqiang Ning and Dandan Liu and Ke Wu},
url = {https://ieeexplore.ieee.org/abstract/document/9489360},
doi = {10.1109/TGRS.2021.3096011},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
pages = {1-14},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Xue, Yu; Wang, Yankang; Liang, Jiayu; Slowik, Adam
A Self-Adaptive Mutation Neural Architecture Search Algorithm Based on Blocks Journal Article
In: IEEE Computational Intelligence Magazine, vol. 16, no. 3, pp. 67-78, 2021.
@article{9492170,
title = {A Self-Adaptive Mutation Neural Architecture Search Algorithm Based on Blocks},
author = {Yu Xue and Yankang Wang and Jiayu Liang and Adam Slowik},
url = {https://ieeexplore.ieee.org/abstract/document/9492170},
doi = {10.1109/MCI.2021.3084435},
year = {2021},
date = {2021-01-01},
journal = {IEEE Computational Intelligence Magazine},
volume = {16},
number = {3},
pages = {67-78},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Phan, Quan Minh; Luong, Ngoc Hoang
Enhancing Multi-objective Evolutionary Neural Architecture Search with Surrogate Models and Potential Point-Guided Local Searches Proceedings Article
In: Fujita, Hamido; Selamat, Ali; Lin, Jerry Chun-Wei; Ali, Moonis (Ed.): pp. 460–472, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-79457-6.
@inproceedings{10.1007/978-3-030-79457-6_39,
title = {Enhancing Multi-objective Evolutionary Neural Architecture Search with Surrogate Models and Potential Point-Guided Local Searches},
author = {Quan Minh Phan and Ngoc Hoang Luong},
editor = {Hamido Fujita and Ali Selamat and Jerry Chun-Wei Lin and Moonis Ali},
url = {https://link.springer.com/chapter/10.1007/978-3-030-79457-6_39},
isbn = {978-3-030-79457-6},
year = {2021},
date = {2021-01-01},
pages = {460--472},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In this paper, we investigate two methods to enhance the efficiency of multi-objective evolutionary algorithms (MOEAs) when solving Neural Architecture Search (NAS) problems. The first method is to use a surrogate model to predict the accuracy of candidate architectures. Only promising architectures with high predicted accuracy values would then be truly trained and evaluated while the ones with low predicted accuracy would be discarded. The second method is to perform local search for potential solutions on the non-dominated front after each MOEA generation. To demonstrate the effectiveness of the proposed methods, we conduct experiments on benchmark datasets of both macro-level (MacroNAS) and micro-level (NAS-Bench-101, NAS-Bench-201) NAS problems. Experimental results exhibit that the proposed methods achieve improvements on the convergence speed of MOEAs toward Pareto-optimal fronts, especially for macro-level NAS problems.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Xue, Song; Chen, Hanlin; Xie, Chunyu; Zhang, Baochang; Gong, Xuan; Doermann, David
Fast and Unsupervised Neural Architecture Evolution for Visual Representation Learning Journal Article
In: IEEE Computational Intelligence Magazine, vol. 16, no. 3, pp. 22-32, 2021.
@article{9492168,
title = {Fast and Unsupervised Neural Architecture Evolution for Visual Representation Learning},
author = {Song Xue and Hanlin Chen and Chunyu Xie and Baochang Zhang and Xuan Gong and David Doermann},
url = {https://ieeexplore.ieee.org/abstract/document/9492168},
doi = {10.1109/MCI.2021.3084394},
year = {2021},
date = {2021-01-01},
journal = {IEEE Computational Intelligence Magazine},
volume = {16},
number = {3},
pages = {22-32},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Do, Tu; Luong, Ngoc Hoang
Insightful and Practical Multi-objective Convolutional Neural Network Architecture Search with Evolutionary Algorithms Proceedings Article
In: Fujita, Hamido; Selamat, Ali; Lin, Jerry Chun-Wei; Ali, Moonis (Ed.): pp. 473–479, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-79457-6.
@inproceedings{10.1007/978-3-030-79457-6_40,
title = {Insightful and Practical Multi-objective Convolutional Neural Network Architecture Search with Evolutionary Algorithms},
author = {Tu Do and Ngoc Hoang Luong},
editor = {Hamido Fujita and Ali Selamat and Jerry Chun-Wei Lin and Moonis Ali},
url = {https://link.springer.com/chapter/10.1007/978-3-030-79457-6_40},
isbn = {978-3-030-79457-6},
year = {2021},
date = {2021-01-01},
pages = {473--479},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {This paper investigates a comprehensive convolutional neural network (CNN) representation that encodes both layer connections, and computational block attributes for neural architecture search (NAS). We formulate NAS as a bi-objective optimization problem, where two competing objectives, i.e., the validation accuracy and the model complexity, need to be considered simultaneously. We employ the well-known multi-objective evolutionary algorithm (MOEA) nondominated sorting genetic algorithm II (NSGA-II) to perform multi-objective NAS experiments on the CIFAR-10 dataset. Our NAS runs obtain trade-off fronts of architectures of much wider ranges and better quality compared to NAS runs with less comprehensive representations. We also transfer promising architectures to other datasets, i.e., CIFAR-100, Street View House Numbers, and Intel Image Classification, to verify their applicability. Experimental results indicate that the architectures on the trade-off front obtained at the end of our NAS runs can be straightforwardly employed out of the box without any further modification.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Dey, Debadeepta; Shah, Shital; Bubeck, Sebastien
Ranking Architectures by Feature Extraction Capabilities Proceedings Article
In: 8th ICML Workshop on Automated Machine Learning (AutoML), 2021.
@inproceedings{<LineBreak>dey2021ranking,
title = {Ranking Architectures by Feature Extraction Capabilities},
author = {Debadeepta Dey and Shital Shah and Sebastien Bubeck},
url = {https://openreview.net/forum?id=z0IHb2AUhE},
year = {2021},
date = {2021-01-01},
booktitle = {8th ICML Workshop on Automated Machine Learning (AutoML)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Ma, Jingchen; He, Ni; Yoon, Jin H; Ha, Richard; Li, Jiao; Ma, Weimei; Meng, Tiebao; Lu, Lin; Schwartz, Lawrence H; Wu, Yaopan; Ye, Zhaoxiang; Wu, Peihong; Zhao, Binsheng; Xie, Chuanmiao
Distinguishing benign and malignant lesions on contrast-enhanced breast cone-beam CT with deep learning neural architecture search Journal Article
In: European Journal of Radiology, vol. 142, pp. 109878, 2021, ISSN: 0720-048X.
@article{MA2021109878,
title = {Distinguishing benign and malignant lesions on contrast-enhanced breast cone-beam CT with deep learning neural architecture search},
author = {Jingchen Ma and Ni He and Jin H Yoon and Richard Ha and Jiao Li and Weimei Ma and Tiebao Meng and Lin Lu and Lawrence H Schwartz and Yaopan Wu and Zhaoxiang Ye and Peihong Wu and Binsheng Zhao and Chuanmiao Xie},
url = {https://www.sciencedirect.com/science/article/pii/S0720048X21003594},
doi = {https://doi.org/10.1016/j.ejrad.2021.109878},
issn = {0720-048X},
year = {2021},
date = {2021-01-01},
journal = {European Journal of Radiology},
volume = {142},
pages = {109878},
abstract = {Purpose
To utilize a neural architecture search (NAS) approach to develop a convolutional neural network (CNN) method for distinguishing benign and malignant lesions on breast cone-beam CT (BCBCT).
Method
165 patients with 114 malignant and 86 benign lesions were collected by two institutions from May 2012 to August 2014. The NAS method autonomously generated a CNN model using one institution’s dataset for training (patients/lesions: 71/91) and validation (patients/lesions: 20/23). The model was externally tested on another institution’s dataset (patients/lesions: 74/87), and its performance was compared with fine-tuned ResNet-50 models and two breast radiologists who independently read the lesions in the testing dataset without knowing lesion diagnosis.
Results
The lesion diameters (mean ± SD) were 18.8 ± 12.9 mm, 22.7 ± 10.5 mm, and 20.0 ± 11.8 mm in the training, validation, and external testing set, respectively. Compared to the best ResNet-50 model, the NAS-generated CNN model performed three times faster and, in the external testing set, achieved a higher (though not statistically different) AUC, with sensitivity (95% CI) and specificity (95% CI) of 0.727, 80% (66–90%), and 60% (42–75%), respectively. Meanwhile, the performances of the NAS-generated CNN and the two radiologists’ visual ratings were not statistically different.
Conclusions
Our preliminary results demonstrated that a CNN autonomously generated by NAS performed comparably to pre-trained ResNet models and radiologists in predicting malignant breast lesions on contrast-enhanced BCBCT. In comparison to ResNet, which must be designed by an expert, the NAS approach may be used to automatically generate a deep learning architecture for medical image analysis.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
To utilize a neural architecture search (NAS) approach to develop a convolutional neural network (CNN) method for distinguishing benign and malignant lesions on breast cone-beam CT (BCBCT).
Method
165 patients with 114 malignant and 86 benign lesions were collected by two institutions from May 2012 to August 2014. The NAS method autonomously generated a CNN model using one institution’s dataset for training (patients/lesions: 71/91) and validation (patients/lesions: 20/23). The model was externally tested on another institution’s dataset (patients/lesions: 74/87), and its performance was compared with fine-tuned ResNet-50 models and two breast radiologists who independently read the lesions in the testing dataset without knowing lesion diagnosis.
Results
The lesion diameters (mean ± SD) were 18.8 ± 12.9 mm, 22.7 ± 10.5 mm, and 20.0 ± 11.8 mm in the training, validation, and external testing set, respectively. Compared to the best ResNet-50 model, the NAS-generated CNN model performed three times faster and, in the external testing set, achieved a higher (though not statistically different) AUC, with sensitivity (95% CI) and specificity (95% CI) of 0.727, 80% (66–90%), and 60% (42–75%), respectively. Meanwhile, the performances of the NAS-generated CNN and the two radiologists’ visual ratings were not statistically different.
Conclusions
Our preliminary results demonstrated that a CNN autonomously generated by NAS performed comparably to pre-trained ResNet models and radiologists in predicting malignant breast lesions on contrast-enhanced BCBCT. In comparison to ResNet, which must be designed by an expert, the NAS approach may be used to automatically generate a deep learning architecture for medical image analysis.
Luo, Xiangzhong; Liu, Di; Huai, Shuo; Kong, Hao; Chen, Hui; Liu, Weichen
Designing Efficient DNNs via Hardware-Aware Neural Architecture Search and Beyond Journal Article
In: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, pp. 1-1, 2021.
@article{9496596,
title = {Designing Efficient DNNs via Hardware-Aware Neural Architecture Search and Beyond},
author = {Xiangzhong Luo and Di Liu and Shuo Huai and Hao Kong and Hui Chen and Weichen Liu},
url = {https://ieeexplore.ieee.org/abstract/document/9496596},
doi = {10.1109/TCAD.2021.3100249},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Gracheva, Ekaterina
Trainless model performance estimation based on random weights initialisations for neural architecture search Journal Article
In: Array, vol. 12, pp. 100082, 2021, ISSN: 2590-0056.
@article{GRACHEVA2021100082,
title = {Trainless model performance estimation based on random weights initialisations for neural architecture search},
author = {Ekaterina Gracheva},
url = {https://www.sciencedirect.com/science/article/pii/S2590005621000308},
doi = {https://doi.org/10.1016/j.array.2021.100082},
issn = {2590-0056},
year = {2021},
date = {2021-01-01},
journal = {Array},
volume = {12},
pages = {100082},
abstract = {Neural architecture search has become an indispensable part of the deep learning field. Modern methods allow to find one of the best performing architectures, or to build one from scratch, but they typically make decisions based on the trained accuracy information. In the present article we explore instead how the architectural component of a neural network affects its prediction power. We focus on relationships between the trained accuracy of an architecture and its accuracy prior to training, by considering statistics over multiple initialisations. We observe that minimising the coefficient of variation of the untrained accuracy, CVU, consistently leads to better performing architectures. We test the CVU as a neural architecture search scoring metric using the NAS-Bench-201 database of trained neural architectures. The architectures with the lowest CVU value have on average an accuracy of 91.90±2.27, 64.08±5.63 and 38.76±6.62 for CIFAR-10, CIFAR-100 and a downscaled version of ImageNet, respectively. Since these values are statistically above the random baseline, we make a conclusion that a good architecture should be stable against weights initialisations. It takes about 190 s for CIFAR-10 and CIFAR-100 and 133.9 s for ImageNet16-120 to process 100 architectures, on a batch of 256 images, with 100 initialisations.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Chakraborty, Biswadeep; Mukhopadhyay, Saibal
textdollar(mu)textdollarDARTS: Model Uncertainty-Aware Differentiable Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-11500,
title = {textdollar(mu)textdollarDARTS: Model Uncertainty-Aware Differentiable Architecture Search},
author = {Biswadeep Chakraborty and Saibal Mukhopadhyay},
url = {https://arxiv.org/abs/2107.11500},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.11500},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
Yin, Yichun; Chen, Cheng; Shang, Lifeng; Jiang, Xin; Chen, Xiao; Liu, Qun
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models Proceedings Article
In: Zong, Chengqing; Xia, Fei; Li, Wenjie; Navigli, Roberto (Ed.): Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 5146–5157, Association for Computational Linguistics, 2021.
@inproceedings{DBLP:conf/acl/YinCSJCL20,
title = {AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models},
author = {Yichun Yin and Cheng Chen and Lifeng Shang and Xin Jiang and Xiao Chen and Qun Liu},
editor = {Chengqing Zong and Fei Xia and Wenjie Li and Roberto Navigli},
url = {https://doi.org/10.18653/v1/2021.acl-long.400},
doi = {10.18653/v1/2021.acl-long.400},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational
Linguistics and the 11th International Joint Conference on Natural
Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual
Event, August 1-6, 2021},
pages = {5146--5157},
publisher = {Association for Computational Linguistics},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Termritthikun, Chakkrit; Jamtsho, Yeshi; Ieamsaard, Jirarat; Muneesawang, Paisarn; Lee, Ivan
EEEA-Net: An Early Exit Evolutionary Neural Architecture Search Journal Article
In: Engineering Applications of Artificial Intelligence, vol. 104, pp. 104397, 2021, ISSN: 0952-1976.
@article{TERMRITTHIKUN2021104397,
title = {EEEA-Net: An Early Exit Evolutionary Neural Architecture Search},
author = {Chakkrit Termritthikun and Yeshi Jamtsho and Jirarat Ieamsaard and Paisarn Muneesawang and Ivan Lee},
url = {https://www.sciencedirect.com/science/article/pii/S0952197621002451},
doi = {https://doi.org/10.1016/j.engappai.2021.104397},
issn = {0952-1976},
year = {2021},
date = {2021-01-01},
journal = {Engineering Applications of Artificial Intelligence},
volume = {104},
pages = {104397},
abstract = {The goals of this research were to search for Convolutional Neural Network (CNN) architectures, suitable for an on-device processor with limited computing resources, performing at substantially lower Network Architecture Search (NAS) costs. A new algorithm entitled an Early Exit Population Initialisation (EE-PI) for Evolutionary Algorithm (EA) was developed to achieve both goals. The EE-PI reduces the total number of parameters in the search process by filtering the models with fewer parameters than the maximum threshold. It will look for a new model to replace those models with parameters more than the threshold. Thereby, reducing the number of parameters, memory usage for model storage and processing time while maintaining the same performance or accuracy. The search time was reduced to 0.52 GPU day. This is a huge and significant achievement compared to the NAS of 4 GPU days achieved using NSGA-Net, 3,150 GPU days by the AmoebaNet model, and the 2,000 GPU days by the NASNet model. As well, Early Exit Evolutionary Algorithm networks (EEEA-Nets) yield network architectures with minimal error and computational cost suitable for a given dataset as a class of network algorithms. Using EEEA-Net on CIFAR-10, CIFAR-100, and ImageNet datasets, our experiments showed that EEEA-Net achieved the lowest error rate among state-of-the-art NAS models, with 2.46% for CIFAR-10, 15.02% for CIFAR-100, and 23.8% for ImageNet dataset. Further, we implemented this image recognition architecture for other tasks, such as object detection, semantic segmentation, and keypoint detection tasks, and, in our experiments, EEEA-Net-C2 outperformed MobileNet-V3 on all of these various tasks. (The algorithm code is available at https://github.com/chakkritte/EEEA-Net).},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Du, Quan; Xu, Nuo; Li, Yinqiao; Xiao, Tong; Zhu, Jingbo
Topology-Sensitive Neural Architecture Search for Language Modeling Journal Article
In: IEEE Access, vol. 9, pp. 107416-107423, 2021.
@article{9502097,
title = {Topology-Sensitive Neural Architecture Search for Language Modeling},
author = {Quan Du and Nuo Xu and Yinqiao Li and Tong Xiao and Jingbo Zhu},
url = {https://ieeexplore.ieee.org/abstract/document/9502097},
doi = {10.1109/ACCESS.2021.3101255},
year = {2021},
date = {2021-01-01},
journal = {IEEE Access},
volume = {9},
pages = {107416-107423},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Liu, Yuqiao; Tang, Yehui; Sun, Yanan
Homogeneous Architecture Augmentation for Neural Predictor Journal Article
In: CoRR, vol. abs/2107.13153, 2021.
@article{DBLP:journals/corr/abs-2107-13153,
title = {Homogeneous Architecture Augmentation for Neural Predictor},
author = {Yuqiao Liu and Yehui Tang and Yanan Sun},
url = {https://arxiv.org/abs/2107.13153},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.13153},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Broni-Bediako, Clifford; Murata, Yuki; Mormille, Luiz H B; Atsumi, Masayasu
Searching for CNN Architectures for Remote Sensing Scene Classification Journal Article
In: IEEE Transactions on Geoscience and Remote Sensing, pp. 1-13, 2021.
@article{9497513,
title = {Searching for CNN Architectures for Remote Sensing Scene Classification},
author = {Clifford Broni-Bediako and Yuki Murata and Luiz H B Mormille and Masayasu Atsumi},
doi = {10.1109/TGRS.2021.3097938},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
pages = {1-13},
keywords = {},
pubstate = {published},
tppubtype = {article}
}