Neural Architecture Search Space Design: From Coarse to Fine-Grained

Neural Architecture Search is a technique for finding the optimal architecture for a given task. However, neural architecture algorithms are typically implemented to search within a pre-defined search space designed by human experts. In most cases, this search space contains only operations from a single architecture type. The homogeneous search space might assist the optimizer in finding models comparable to handcrafted architectures. However, this also limits the diversity of architectures in the search space, thereby hindering the optimizer from finding novel architectures beyond handcrafted architectures. Indeed, different architectures are designed with different inductive biases: a convolutional neural network aims to discover local data dependencies; a recurrent neural network considers the dependencies of each token and its previous tokens; and transformers without positional encoding treat each token equally, regardless of its position in the sequence. Applying a single architecture type to all tasks might not guarantee finding the optimal architecture for a given task. In this thesis, we show how to gradually release the constraints from different hierarchical levels in the architecture search space. Starting with time series forecasting tasks, we first introduce AutoPyTorch-TS in Chapter 3. Unlike other AutoML frameworks that build a search space for each model individually, AutoPyTorch-TS provides a unified hierarchical macro-level search space that shares hyperparameters across different model families. This provides a flexible search space that greatly reduces the number of hyperparameters in the search space. AutoPyTorch-TS provides a flexible way for users to add new architecture components to its search space. However, it cannot discover novel architectures from its search space. We introduce DARTS-TS in Chapter 4. DARTS-TS further relaxes the constraints between operations by designing multiple bridge layers to meet the I/O requirements of different architectures in the search space. This adaptation in the micro-level search space allows the combination of any architectural components, greatly expanding the number of architecture types searched. DARTS-TS uses the same operation for all the tokens in the sequence. However, not all the tokens contribute equally to the final output. A network should learn this information to discard tokens that are no longer necessary for subsequent predictions. In Chapter 5, we present Neural Attention Search (NAtS) to solve this. Building on top of a transformer architecture, NAtS searches for the optimal token types for each token. These token types determine how long each token survives in subsequent time steps. Hence, this approach can also be considered as searching for different architecture operations for each token, with each operation having different receptive fields. This brings us to the search space with the finest granularity -- the token-wise granularity, where each token gets a role that fits the token in the sequence best. Overall, this thesis provides a different perspective on the design principle of neural architecture search space. We hope that this provides a future opportunity for the NAS research to contribute even in the era of large language models.