University of Twente Student Theses


Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation

Hoftijzer, D.M. (2023) Language-Based Augmentation to Address Shortcut Learning in Object-Goal Navigation.

[img] PDF
Abstract:Deep Reinforcement Learning (DRL) has shown great potential in enabling robots to find certain objects (e.g., ‘find a bed’) in environments like homes or schools. This task is known as Object-Goal Navigation (ObjectNav). Although DRL has shown impressive results, the simulators are key and may be biased or limited. This creates a profound risk of shortcut learning i.e., learning a policy tailored to specific visual details of training environments. Therefore, in this work, we aim to deepen our understanding of shortcut learning in ObjectNav, its implications and propose a solution. We design an experiment for inserting a shortcut bias in the appearance of training environments. As an example, we associate room types to specific wall colors (e.g., bedrooms have green walls), and observe poor generalization of a SOTA ObjectNav method to environments where this is not the case (e.g., bedrooms now have blue walls). Further analysis shows that shortcut learning is the root cause: the agent learns to navigate to target objects, by simply searching for the associated wall color of the target object’s room. To solve this, we propose Language-Based Augmentation (L-B). Our key insight is that we can leverage the multimodal feature space of a Vision-Language (V-L) model to augment visual representations directly at the feature-level, requiring no changes to the simulator, and only an addition of one layer to the model. Where the SOTA ObjectNav method’s success rate drops 69%, our proposal has only a drop of 23%.
Item Type:Essay (Master)
TNO, Den Haag, Netherlands
Faculty:EEMCS: Electrical Engineering, Mathematics and Computer Science
Subject:54 computer science
Programme:Electrical Engineering MSc (60353)
Link to this item:
Export this item as:BibTeX
HTML Citation
Reference Manager


Repository Staff Only: item control page