Incorporating Simulated Spatial Context Information Improves the Effectiveness of Contrastive Learning Models

Lizhen Zhu, James Z. Wang, Wonseuk Lee, Brad Wyble
The Pennsylvania State University, University Park, USA
Abstract:

Visual learning often occurs in a specific context, where an agent acquires skills through exploration and tracking of its location in a consistent environment. The historical spatial context of the agent provides a similarity signal for self-supervised contrastive learning. We present a unique approach, termed environmental spatial similarity (ESS), that complements existing contrastive learning methods. Using images from simulated, photorealistic environments as an experimental setting, we demonstrate that ESS outperforms traditional instance discrimination approaches. Moreover, sampling additional data from the same environment substantially improves accuracy and provides new augmentations. ESS allows remarkable proficiency in room classification and spatial prediction tasks, especially in unfamiliar environments. This learning paradigm has the potential to enable rapid visual learning in agents operating in new environments with unique visual characteristics. Potentially transformative applications span from robotics to space exploration. Our proof of concept demonstrates improved efficiency over methods that rely on extensive, disconnected datasets.

Full Paper
(high-resolution PDF, open access, 5.2MB)

More information


Citation: Lizhen Zhu, James Z. Wang, Wonseuk Lee and Brad Wyble, ``Incorporating Simulated Spatial Context Information Improves the Effectiveness of Contrastive Learning Models,'' Patterns, vol. 5, no. 5, article 100964, pp. 1-15, 2024.

© 2024 The Authors. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the authors.

Last Modified: August 20, 2024
© 2024