AutoML | Literature on Neural Architecture Search

2248.

Kung, S. Y.

XNAS: A Regressive/Progressive NAS for Deep Learning Journal Article

In: ACM Trans. Sen. Netw., 2022, ISSN: 1550-4859, (Just Accepted).

@article{10.1145/3543669,

title = {XNAS: A Regressive/Progressive NAS for Deep Learning},

author = {S. Y. Kung},

url = {https://doi.org/10.1145/3543669},

doi = {10.1145/3543669},

issn = {1550-4859},

year  = {2022},

date = {2022-05-01},

urldate = {2022-05-01},

journal = {ACM Trans. Sen. Netw.},

publisher = {Association for Computing Machinery},

address = {New York, NY, USA},

abstract = {Deep learning has achieved great and broad breakthroughs in many real-world applications. In particular, the task of training the network parameters has been masterly handled by the Back-Propagation (BP) learning. However, the pursuit on optimal network structures remains largely an art of trial-and-error. This prompts some urgency to explore an architecture engineering process, collectively known as “Neural Architecture Search” (NAS). In general, NAS is a design software system for automating the search of effective neural architecture. This paper proposes a X-learning NAS (XNAS) to automatically train network’s structure and parameters. Our theoretical footing is built upon the subspace and correlation analyses between the input layer, hidden layer, and output layer. The design strategy hinges upon the underlying principle that the network should be coerced to learn how to structurally improve the input/output correlation successively (i.e. layer by layer). It embraces both PNAS and RNAS, short for Progressive and Regressive NAS respectively. For unsupervised RNAS, PCA (Principal Component Analysis) is a classic tool for subspace analyses. By further incorporating teacher’s guidance, PCA can be extended to Regression Component Analysis (RCA) to facilitate supervised NAS design. This allows the machine to extract components most critical to the targeted learning objective. We shall further extend the subspace analysis from MLPs to CNNs, via introduction of Convolutional-PCA (CPCA) or, more simply, Deep-PCA (DPCA). The supervised variant of DPCA will be named Deep-RCA (DRCA). The subspace analyses allow us to compute optimal eigenvectors (resp. eigen-filters) and principal components (resp. eigen-channels) for optimal NAS design of MLPs (resp. CNNs). Based on the theoretical analysis, a X-Learning paradigm is developed to jointly learn the structure and parameters of learning models. The objective is to reduce the network complexity while retaining (and sometimes improving) the performance. With carefully pre-selected baseline models, X-Learning has shown great successes in numerous classification-type and/or regression-type applications. We have applied X-learning to the ImageNet datasets for classification and div2k for image enhancements By applying X-Learning to two types of baseline models, MobileNet and ResNet, both the low-power and high-performance application categories can be supported. Our X-learning experiments include the ImageNet datasets for classification and div2k for image enhancements. Our simulations confirm that X-learning is by and large very competitive relative to the state-of-the-arts approaches.},

note = {Just Accepted},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Close

Deep learning has achieved great and broad breakthroughs in many real-world applications. In particular, the task of training the network parameters has been masterly handled by the Back-Propagation (BP) learning. However, the pursuit on optimal network structures remains largely an art of trial-and-error. This prompts some urgency to explore an architecture engineering process, collectively known as “Neural Architecture Search” (NAS). In general, NAS is a design software system for automating the search of effective neural architecture. This paper proposes a X-learning NAS (XNAS) to automatically train network’s structure and parameters. Our theoretical footing is built upon the subspace and correlation analyses between the input layer, hidden layer, and output layer. The design strategy hinges upon the underlying principle that the network should be coerced to learn how to structurally improve the input/output correlation successively (i.e. layer by layer). It embraces both PNAS and RNAS, short for Progressive and Regressive NAS respectively. For unsupervised RNAS, PCA (Principal Component Analysis) is a classic tool for subspace analyses. By further incorporating teacher’s guidance, PCA can be extended to Regression Component Analysis (RCA) to facilitate supervised NAS design. This allows the machine to extract components most critical to the targeted learning objective. We shall further extend the subspace analysis from MLPs to CNNs, via introduction of Convolutional-PCA (CPCA) or, more simply, Deep-PCA (DPCA). The supervised variant of DPCA will be named Deep-RCA (DRCA). The subspace analyses allow us to compute optimal eigenvectors (resp. eigen-filters) and principal components (resp. eigen-channels) for optimal NAS design of MLPs (resp. CNNs). Based on the theoretical analysis, a X-Learning paradigm is developed to jointly learn the structure and parameters of learning models. The objective is to reduce the network complexity while retaining (and sometimes improving) the performance. With carefully pre-selected baseline models, X-Learning has shown great successes in numerous classification-type and/or regression-type applications. We have applied X-learning to the ImageNet datasets for classification and div2k for image enhancements By applying X-Learning to two types of baseline models, MobileNet and ResNet, both the low-power and high-performance application categories can be supported. Our X-learning experiments include the ImageNet datasets for classification and div2k for image enhancements. Our simulations confirm that X-learning is by and large very competitive relative to the state-of-the-arts approaches.

Close

2247.

AL-Ghamdi, Abdullah S. AL-Malaise; Ragab, Mahmoud; AlGhamdi, Saad Abdulla; Asseri, Amer H.; ans Deepika Koundal, Romany F. Mansour

Cognitive Computing Paradigms for Medical Big Data Processing and its Trends Journal Article

In: Computational Intelligence and Neuroscience, 2022.