When Neural Architecture Search Meets Hardware Implementation: from Hardware Awareness to Co-Design

Neural Architecture Search (NAS), that automatically identifies the best network architecture, is a promising technique to respond to the ever-growing demand for application-specific Artificial Intelligence (AI). On the other hand, a large number of research efforts have been put on implementing and optimizing AI applications on the hardware. Out of all leading computation platforms, Field Programmable Gate Arrays (FPGAs) stand out due to its flexibility and versatility over ASICs and its efficiency over CPUs and GPUs. To identify the best neural architecture and hardware implementation pair, a number of research works are emerging to involve the awareness of hardware efficiency in the NAS process, which is called "hardware-aware NAS". Unlike the conventional NAS with a mono-criteria of accuracy, hardware-aware NAS is a multi-objective optimization problem, which aims to identify the best network and hardware pair to maximize accuracy with guaranteed hardware efficiency. Most recently, the co-design of neural architecture and hardware has been put forward to further push forward the Pareto frontier between accuracy and efficiency trade-off. This paper will review and discuss the current progress in the neural architecture search and the implementation on hardware.