A multilevel attention network with sub-instructions for continuous vision-and-language navigation

Abstract

The aim of vision-and-language navigation (VLN) is to develop agents that navigate mapless environments via linguistic and visual observations. Continuous VLN, which more accurately mirrors real-world conditions than its discrete counterpart does, faces unique challenges such as real-time execution, complex instruction understanding, and long sequence prediction. In this work, we introduce a multilevel instruction understanding mechanism and propose a multilevel attention network (MLANet) to address these challenges. Initially, we develop a nonlearning-based fast sub-instruction algorithm (FSA) to swiftly generate sub-instructions without the need for annotations, achieving a speed enhancement of 28 times over the previous methods. Subsequently, our multilevel attention (MLA) module dynamically integrates visual features with both high- and low-level linguistic semantics, forming multilevel global semantics to bolster the complex instruction understanding capabilities of the model. Finally, we introduce the peak attention loss (PAL), which enables the flexible and adaptive selection of the current sub-instruction, thereby improving accuracy and stability achieved for long trajectories by focusing on the relevant local semantics. Our experimental findings demonstrate that MLANet significantly outperforms the baselines and is applicable to real-world robots.