Introduction

Multimodality as self-supervision

Cross-modal learning predicts one modality from another, enabling self-supervised representation learning from the agent’s sensory observations.

Multiple sensory modalities provide time-locked streams of information sensed from the same underlying physical world. Cross-modal learning, which learns to predict one modality from another, can therefore build representations without external supervision beyond the sensory modalities themselves.

Applied across a diverse set of modalities and combined with well-designed pre-training objectives, this mechanism can produce a rich representation solely through multimodality. Developmental psychologists have hypothesised that multimodal learning is one of the key drivers of efficient yet effective biological intelligence. Meanwhile, devices such as phones and robots already possess the sensors needed to apply this mechanism in the real world.

Studying cross-modal learning in a controlled sandbox

Play animation

A multisensory agent moves through a physical space, collecting aligned multimodal observations for cross-modal learning.

How far can cross-modal learning go with today’s AI machinery? Existing multimodal approaches largely study this question using large and diverse Internet-scale datasets. We instead construct a controlled sandbox in which a multisensory agent has access to a physical space and collects aligned multimodal observations from it. This lets us control the source and diversity of the agent’s experience.

What if we pre-train and evaluate in the same test space?

Test-Space Training uses unlabelled multimodal observations from the test space for self-supervised pre-training, transfers the learned representation using labelled data from an external space, and evaluates it in the same test space.

Within this sandbox, we restrict the agent’s world to a single physical space, which we call the test space. The agent collects unlabelled multimodal data in this space and uses it for self-supervised cross-modal pre-training, without access to task-specific labels from the test space. The learned representation is then transferred to novel downstream tasks using labelled data from outside the test space, and evaluated in the test space. We refer to this framework as Test-Space Training (TST).

It creates a specialisation setting in which the goal is to build a performant model for a specific test space, irrespective of its ability to generalise elsewhere. It is particularly interesting, since it matches practical deployments such as household robots and AR / VR devices, which often operate within a bounded user space and primarily need to perform well there.

From sensory modalities to richer multimodal supervision

Play animation

TST-MM begins with sensory modalities and expands its modality dictionary using pseudo-labelled modalities generated by pre-trained networks over test-space observations.

We instantiate cross-modal learning using multimodal masked modelling, and refer to the resulting model as TST-MM. A central design choice is the modality dictionary: the included modalities determine the supervision available during pre-training and shape the learned representation.

We begin with modalities captured directly by hardware sensors, such as RGB and depth, along with modalities derived from programmatic transformations of these images. We then expand the modality dictionary with pseudo-labelled modalities derived from pre-trained networks, effectively distilling their representations into a model specialised to the test space.

Many user devices, e.g., a household robot, augmented reality glasses, or an iPhone, are equipped with a rich array of sensors that enable multimodal data collection. They can acquire modalities beyond RGB images, such as depth, motion sensing, and haptic feedback. Such data enables cross-modal learning, i.e., predicting the response of one sensor from another, thereby leveraging multimodality as a signal for self-supervised learning (see Figure 2). There has also been evidence in developmental psychology that suggests that multimodality is employed by biological organisms to bootstrap better representations of their environment.

Figure 3. Many practical applications require vision models to operate within certain test space, for instance a household robot in user home. In such scenarios, we primarily care about the performance of the model in the test space, regardless of its generalization performance elsewhere.

In this work, we are interested in studying the potential of multimodality as a source of self-supervision in pre-training vision models. To do so in a controlled manner, we construct a sandbox setup wherein we reduce the operating space of the user device to a specific physical space, or as we refer to it, the test space. This implies that self-supervised pre-training of visual representations and downstream evaluations both happen in the same user space.

Figure 4. In this work, as opposed to the de-facto approach of building models with large-scale internet-based data, which rely on their generalization abilities, we study whether we can build performant models for the test space, by just collecting modality-rich data in just the test space. This serves as a sandbox to study the potential of multimodality as a source of self-supervision in pre-training vision models, and understand the trade-offs between specialization to the test space and generalization to other spaces.

This sandbox has several desirable properties. As opposed to the de facto setup of learning a generalist model, pre-trained on large and diverse pre-training data (often based on the internet), and relying on its generalization abilities to perform well in the test space, our setup allows us to study the ability of multimodal pre-training to learn a distribution agnostic to its generalization abilities. Additionally, it also represents practical evaluation scenarios such as household robotics, where user devices are expected to be highly performant in their own space, regardless of their generalization abilities elsewhere.

Figure 5. Generalist models (left), which are pre-trained once, on large-scale internet based datasets, and then deployed to various downstream spaces. Test-space training (right), contrary to generalists, pre-trains the models on unlabelled data from the test space itself. We show that test-space training, results in specialist models that outperform generalist models, when evaluated on the test space.

To learn representations under this setup, we propose Test-Space Training (Figure 5, right), which enables multimodal pre-training data collection in the same space that the device would be deployed on. We use this data to perform self-supervised pre-training via cross-modal learning, leading to TST-MM.

Multimodal self-supervision can produce strong test-space specialists

We compare TST-MM with models pre-trained on large-scale Internet data, including CLIP, DINOv2, and 4M-21. Across the evaluated test spaces, TST-MM outperforms these baselines on semantic segmentation and object detection, while remaining competitive on image captioning.

TST-MM uses both sensory and pseudo-labelled modalities. The latter distill knowledge from multiple off-the-shelf networks, applied to test-space observations, into a single specialist model. We later examine how much can be achieved using sensory modalities alone.

Qualitative Results

We also perform qualitative frame-by-frame evaluations on semantic segmentation in Scannet++ and Replica. TST-MM produces more consistent predictions across viewpoints than internet-based generalists such as CLIP, DINOv2, and 4M-21.

Analysing multimodality in TST

I. Multimodality drives the gains

To study the contribution of multimodal self-supervision within TST, we compare RGB-only masked modelling, TST-MAE, with multimodal masked modelling, TST-MM. TST-MAE improves over training from scratch but remains below the internet-pretrained DINOv2. With cross-modal supervision from both sensory and pseudo-labelled modalities, TST-MM achieves substantially better performance and surpasses DINOv2.

II. Sensory modalities only

The previous analysis establishes the benefit of the full modality dictionary, but does not separate supervision available directly from the agent’s onboard sensors and knowledge distilled through pre-trained networks. We therefore construct TST-MM (sensors), a model using RGB and depth, along with surface normals derived from depth and Canny edges derived from RGB, while excluding pseudo-labelled modalities generated by external networks.

TST-MM (sensors). The modality dictionary contains RGB and depth, together with surface normals derived from depth and Canny edges derived from RGB. Pseudo-labelled modalities generated by external networks are excluded.

On ScanNet++, TST-MM (sensors) performs on par with DINOv2 on semantic segmentation and remains competitive, though lower, on object detection. Sensory and sensor-derived modalities therefore provide strong self-supervision on their own, while pseudo-labelled modalities remain important for the best overall performance.

III. How does TST-MM scale with modalities?

We next study how TST-MM scales as the modality dictionary grows. We start with RGB, and incrementally add more modalities. Due to the combinatorial complexity of studying all possible combination, we only sample all possible options for two (RGB+X) and eight (All-X) modalities, where X is the modality added or dropped. For intermediate dictionary sizes, we randomly sample 8 modality subsets and report the average performance. Performance improves as more modalities are added, while variation across subsets decreases. TST-MM therefore scales with a richer modality dictionary and becomes progressively less sensitive to the exact choice of modalities.

IV. Structure vs. scale: scaling data or scaling modalities?

Generalist models commonly improve by scaling data and model size across diverse external sources. In this analysis, we ask whether, when the test space is known, adding structure through modalities can be more effective than collecting more RGB-only data elsewhere.

Here, data scale is increased by adding RGB-only data from external spaces, while structure is increased by adding modalities within the test space. Across the evaluated range, scaling modalities produces larger gains. For a given test space, modality-rich data can therefore be more valuable than additional unimodal data from external spaces.

Measuring Specialisation in TST

We define specialization as high performance in a given test space, even if the same model does not generalize equally well to other spaces. In other words, a model is specialized to Space A if it performs well in Space A, but performs worse when evaluated in different spaces. This section asks whether TST produces this kind of space-specific specialization. We study whether access to the test space matters, and how specialisation in the test space trades off against generalization to held-out spaces.

I. Does TST specialise to the test space?

The goal of TST is to specialise a model to a given test space. But does the resulting model truly specialise to that space, or would pre-training in a similar space work just as well?

We consider three distinct but visually similar spaces, with comparable layouts and object appearance. We pre-train one model in each space and evaluate each of them across all three spaces.

Test-space versus similar-space pre-training. TST uses unlabelled data from the test space itself, while the alternative uses data from a similar but non-identical space.

Performance is highest whenever the pre-training and evaluation spaces match, producing a clear diagonal in the cross-space evaluation matrix. This shows that access to the test space provides a benefit for specialisation beyond what is learned from a similar space.

II. Specialisation-Generalisation Tradeoff

The previous analysis shows that a model performs best when pre-training and evaluation use the same space. We next ask whether a single model can retain this test-space performance while also generalising to unseen spaces.

To isolate the effect of the pre-training data source, we keep the model size and total number of pre-training samples fixed. Starting with data only from the test space, we progressively replace test-space samples with data from an increasing number of similar external spaces. Each model is evaluated in the original test space and across a set of held-out spaces.

This reveals a specialisation-generalisation trade-off: under a fixed model size and pre-training sample budget, distributing samples across a more diverse set of spaces improves generalisation to held-out spaces but weakens specialisation to the test space.

III. How much external data is 1 test space worth?

We find that pre-training on the test space is important for specialization, and cannot be substituted by a similar space. However, we take this one step further, and ask, can we substitute test-space data with data from many similar spaces? How many spaces would we need? We use ProcTHOR to generate a large number of similar houses (IID to the test space) and pre-train models using an increasing number of them.

The curve below shows the performance of each model on the test space not seen during pre-training compared to pre-training on the corresponding test space. We find that even thousands of similar spaces are not enough to substitute pre-training on the exact same space that we deploy in.

Multimodal iSensorKit: Sensory data collection on iOS

To support future work on cross-modal learning with real device sensors, we release Multimodal iSensorKit, an iOS app for collecting aligned multimodal data from commodity devices. The app records synchronised streams such as RGB, LiDAR/depth, IMU, magnetometer, ambient lighting, pressure, and other sensor signals.

Discussion and Future work

Our work presents a controlled sandbox for studying multimodality as self-supervision. In this sandbox, a multisensory agent collects multimodal data within a physical space. We focus on the case where its operating space is restricted to a single physical space, the test space. We call this Test-Space Training (TST): unlabelled multimodal observations from the test space are used for cross-modal masked modelling, and the resulting representations are evaluated on novel tasks in the same space.

Across three indoor datasets and multiple downstream tasks, TST-MM shows that multimodality can provide strong self-supervision for test-space specialisation. Sensory modalities alone provide substantial supervision, while adding pseudo-labelled modalities produces specialists that outperform internet-pretrained generalists. Our analyses also show that adding structure through modalities can be more effective than adding RGB-only data from similar external spaces, and that broader spatial diversity trades off against test-space specialisation.

Our work is a problem setup rather than a complete solution for leveraging deployment-space data for specialisation. Although multimodal masked modelling achieves state-of-the-art test-space performance, a gap to the upper bound remains. We hope future work will explore improved multimodal modelling objectives in the sandbox via additional inductive biases, such as multi-view consistency, and hardware modalities including IMU, gyroscope, and magnetometer signals.

Citation

@inproceedings{singh2026tst,
            title={Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
            },
            author={Kunal Pratap Singh and Ali Garjani and Rishubh Singh and Muhammad Uzair Khattak and Efe Tarhan and Jason Toskov and Andrei Atanov and O{\u{g}}uzhan Fatih Kar and Amir Zamir},
            booktitle={International Conference on Learning Representations (ICLR)},
            year={2026}
        }

Acknowledgements

This work was supported under project ID 43 as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure. This material is based on work that is partially funded by an unrestricted gift from Google. This work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI). We also thank Daniel Filipe Jana and the EPFL SCITAS team for their support. The authors also thank Chandan Yeshwanath for help with the Scannet++ dataset, and Roman Bachmann for useful discussions.