Introduction

Multimodality as self-supervision

Cross-modal learning predicts one modality from another, enabling self-supervised representation learning from the agent’s sensory observations.

Multiple sensory modalities provide time-locked streams of information sensed from the same underlying physical world. Cross-modal learning predicts one modality from another. The sensory streams themselves provide supervision, allowing the model to learn representations without task labels.

Developmental psychologists have hypothesised that multimodal learning is one of the key drivers of efficient yet effective biological intelligence. As many commonly used devices, e.g., iPhones, are already equipped with a rich array of sensors, this setup is readily applicable in the real world and currently underutilized. We believe that cross-modal learning through well-designed pre-training objectives applied across a constellation of diverse modalities can produce a rich representation from those multimodal sensory streams alone.

A controlled sandbox for studying cross-modal learning

Play animation ↻

A multisensory agent moves through a physical space and collects aligned multimodal observations for cross-modal learning.

How far can cross-modal learning go with today's machine learning tools? Existing multimodal approaches largely study this question using pre-collected internet-based datasets. We instead construct a controlled sandbox. In this sandbox, a multisensory agent moves through a physical space and collects aligned multimodal observations. This lets us control the source and diversity of the agent’s experience.

Test-Space Training

Test-Space Training uses unlabelled multimodal observations from the test space for self-supervised pre-training, transfers the learned representation using labelled data from an external space, and evaluates it in the same test space.

Within this sandbox, we restrict the agent’s world to a single physical space: the test space. The agent collects unlabelled multimodal data in this space and uses it for self-supervised cross-modal pre-training. The learned representation is then evaluated on novel downstream tasks in the test space. We refer to this framework as Test-Space Training (TST).

This creates a specialisation setting. The goal is to build a performant model for a specific test space, irrespective of its ability to generalise elsewhere. This setting matches practical deployments such as household robots and AR/VR devices. These systems often operate within a bounded user space and primarily need to perform well there.

From sensory modalities to richer multimodal supervision

TST-MM (sensors) uses the sensor modalities and learning-free transformations shown on the left. TST-MM expands this dictionary with the pseudo-labelled modalities shown on the right.

A central design choice in cross-modal learning is the modality dictionary: the set of modalities used for cross-modal prediction. It directly affects the usefulness of the learned representation.

We begin with RGB and depth captured directly by hardware sensors. We add two learning-free transformations: Canny edges from RGB and surface normals from depth. We call the model trained with this dictionary TST-MM (sensors). We then expand the dictionary with pseudo-labelled modalities produced by pre-trained networks. This can be viewed as distilling their knowledge into a representation specialised to the test space. We call the resulting model TST-MM. Importantly, TST-MM outperforms the individual pre-trained networks used to produce its pseudo-labelled modalities.

Many user devices, e.g., a household robot, augmented reality glasses, or an iPhone, are equipped with a rich array of sensors that enable multimodal data collection. They can acquire modalities beyond RGB images, such as depth, motion sensing, and haptic feedback. Such data enables cross-modal learning, i.e., predicting the response of one sensor from another, thereby leveraging multimodality as a signal for self-supervised learning (see Figure 2). There has also been evidence in developmental psychology that suggests that multimodality is employed by biological organisms to bootstrap better representations of their environment.

Figure 3. Many practical applications require vision models to operate within certain test space, for instance a household robot in user home. In such scenarios, we primarily care about the performance of the model in the test space, regardless of its generalization performance elsewhere.

In this work, we are interested in studying the potential of multimodality as a source of self-supervision in pre-training vision models. To do so in a controlled manner, we construct a sandbox setup wherein we reduce the operating space of the user device to a specific physical space, or as we refer to it, the test space. This implies that self-supervised pre-training of visual representations and downstream evaluations both happen in the same user space.

Figure 4. In this work, as opposed to the de-facto approach of building models with large-scale internet-based data, which rely on their generalization abilities, we study whether we can build performant models for the test space, by just collecting modality-rich data in just the test space. This serves as a sandbox to study the potential of multimodality as a source of self-supervision in pre-training vision models, and understand the trade-offs between specialization to the test space and generalization to other spaces.

This sandbox has several desirable properties. As opposed to the de facto setup of learning a generalist model, pre-trained on large and diverse pre-training data (often based on the internet), and relying on its generalization abilities to perform well in the test space, our setup allows us to study the ability of multimodal pre-training to learn a distribution agnostic to its generalization abilities. Additionally, it also represents practical evaluation scenarios such as household robotics, where user devices are expected to be highly performant in their own space, regardless of their generalization abilities elsewhere.

Figure 5. Generalist models (left), which are pre-trained once, on large-scale internet based datasets, and then deployed to various downstream spaces. Test-space training (right), contrary to generalists, pre-trains the models on unlabelled data from the test space itself. We show that test-space training, results in specialist models that outperform generalist models, when evaluated on the test space.

To learn representations under this setup, we propose Test-Space Training (Figure 5, right), which enables multimodal pre-training data collection in the same space that the device would be deployed on. We use this data to perform self-supervised pre-training via cross-modal learning, leading to TST-MM.

Multimodal self-supervision can produce strong test-space specialists

We compare TST-MM with internet-based generalist models, including CLIP, DINOv2, and 4M-21. The figure below highlights semantic segmentation on ScanNet++; the full results across other datasets and tasks are reported below.

TST-MM combines sensory modalities with pseudo-labelled modalities produced by multiple off-the-shelf networks. Training with these pseudo-labelled modalities distils their knowledge into a single model specialised to the test space. Importantly, TST-MM outperforms the individual pre-trained networks used to produce its pseudo-labelled modalities. We also separately study how much can be achieved using sensory modalities alone.

Qualitative Results

We also perform qualitative frame-by-frame evaluations on semantic segmentation in Scannet++ and Replica. TST-MM produces more consistent predictions across viewpoints than internet-based generalists such as CLIP, DINOv2, and 4M-21.

Analysing multimodality in Test-Space Training

I. Multimodality drives the gains

To study the contribution of multimodal self-supervision within TST, we compare RGB-only masked modelling, TST-MAE, with multimodal masked modelling, TST-MM. TST-MAE improves over training from scratch but remains below the internet-pretrained DINOv2. With cross-modal supervision from both sensory and pseudo-labelled modalities, TST-MM achieves substantially better performance and surpasses DINOv2.

II. Sensory modalities only

III. How does TST-MM scale with modalities?

We next study how TST-MM scales as the modality dictionary grows. We start with RGB, and incrementally add more modalities. Due to the combinatorial complexity of studying all possible combination, we only sample all possible options for two (RGB+X) and eight (All-X) modalities, where X is the modality added or dropped. For intermediate dictionary sizes, we randomly sample 8 modality subsets and report the average performance. Performance improves as more modalities are added, while variation across subsets decreases. TST-MM therefore scales with a richer modality dictionary and becomes progressively less sensitive to the exact choice of modalities.

IV. Scaling data or scaling modalities?

Generalist models often improve by training on more data from diverse external sources. Here, we ask whether, for a known test space, it is more effective to scale the amount of pre-training data or the number of modalities available for cross-modal learning.

We compare models trained on data from 5, 750, or 3,000 spaces using modality dictionaries containing 1, 3, 7, or 9 modalities. Across the evaluated settings, increasing the number of modalities produces larger gains than increasing the number of spaces. This shows that, in this setting, a richer set of modalities in the test space can be more effective than additional data from external spaces.

Measuring Specialisation in Test-Space Training

We define specialization as high performance in a given test space, even if the same model does not generalize equally well to other spaces. In other words, a model is specialized to Space A if it performs well in Space A, but performs worse when evaluated in different spaces. This section asks whether TST produces this kind of space-specific specialization. We study whether access to the test space matters, and how specialisation in the test space trades off against generalization to held-out spaces.

I. Does Test-Space Training specialise to the test space?

The goal of TST is to specialise a model to a given test space. But does the resulting model truly specialise to that space, or would pre-training in a similar space work just as well?

We consider three distinct but visually similar spaces, with comparable layouts and object appearance. We pre-train one model in each space and evaluate each of them across all three spaces.

Test-space versus similar-space pre-training. TST uses unlabelled data from the test space itself, while the alternative uses data from a similar but non-identical space.

Performance is highest along the diagonal of the cross-space evaluation matrix, where the pre-training and evaluation spaces match. This shows that access to the test space provides a benefit for specialisation beyond what is learned from a similar space.

II. Specialisation-Generalisation Tradeoff

The previous analysis shows that a model performs best when pre-training and evaluation use the same space. We next ask whether a single model can retain this test-space performance while also generalising to unseen spaces.

To isolate the effect of the pre-training data source, we keep the model size and total number of pre-training samples fixed. We start with data only from the test space. We then progressively replace test-space samples with data from an increasing number of similar external spaces. Each model is evaluated in the original test space and across a set of held-out spaces.

This reveals a specialisation-generalisation trade-off: under a fixed model size and pre-training sample budget, distributing samples across a more diverse set of spaces improves generalisation to held-out spaces but weakens specialisation to the test space.

III. How much external data is 1 test space worth?

We find that pre-training on the test space is important for specialization, and cannot be substituted by a similar space. However, we take this one step further, and ask, can we substitute test-space data with data from many similar spaces? How many spaces would we need? We use ProcTHOR to generate a large number of similar houses (IID to the test space) and pre-train models using an increasing number of them.

The curve below shows the performance of each model on the test space not seen during pre-training compared to pre-training on the corresponding test space. We find that even thousands of similar spaces are not enough to substitute pre-training on the exact same space that we deploy in.

Multimodal iSensorKit: Sensory data collection on iOS

To support future work on cross-modal learning with hardware sensor modalities, we release Multimodal iSensorKit, an iOS app for collecting synchronised multimodal data using an iPhone or iPad. It records aligned streams including RGB, LiDAR/depth, IMU, magnetometer, ambient light, pressure, GPS, and more, making it easier to collect rich multimodal data directly from real physical spaces.

Discussion and Future work

We study multimodality as a source of self-supervision for representation learning. Unlike prior work that studies it using pre-collected internet-scale datasets, we consider an alternative setup: a controlled sandbox in which a multisensory agent operates in the physical world and collects multimodal data. We perform cross-modal learning on this data to learn self-supervised representations and evaluate them on downstream tasks.

We focus particularly on the case where the agent’s operating space is restricted to a single physical space, the test space. We call this Test-Space Training (TST): unlabeled multimodal observations from the test space are used for cross-modal masked modelling, and the resulting representations are evaluated on downstream tasks in the same space. Across various datasets and tasks, we find that cross-modal learning on rich multimodal data from the test space enables TST to achieve state-of-the-art performance in that space, outperforming the evaluated generalist baselines pre-trained on large-scale internet-based datasets.

Our work establishes a problem setting for cross-modal learning under test-space specialization and provides an initial solution. Despite these results, a substantial gap to the fully supervised upper bound remains. Future work could address this gap through stronger multimodal pre-training objectives and active data-collection policies that select informative views of the test space for pre-training.

Finally, our work asks whether, when the deployment space is known, serving off-the-shelf generalist models trained on internet-scale datasets should remain the de facto approach, or whether space-specific specialists offer a viable alternative.

Citation

@inproceedings{singh2026tst,
            title={Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
            },
            author={Kunal Pratap Singh and Ali Garjani and Rishubh Singh and Muhammad Uzair Khattak and Efe Tarhan and Jason Toskov and Andrei Atanov and O{\u{g}}uzhan Fatih Kar and Amir Zamir},
            booktitle={International Conference on Learning Representations (ICLR)},
            year={2026}
        }

Acknowledgements

This work was supported under project ID 43 as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure. This material is based on work that is partially funded by an unrestricted gift from Google. This work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI). We also thank Daniel Filipe Jana and the EPFL SCITAS team for their support. The authors also thank Chandan Yeshwanath for help with the Scannet++ dataset, and Roman Bachmann for useful discussions.