In Search of Forgotten Domain Generalization
University of Tübingen, MPI-IS, Tübingen AI Center
University of Tübingen, MPI-IS, TÜbingen AI Center
tl;dr: CLIP's high performance on style-centric domain shifts is significantly influenced by the presence of such images in its training set.
News
Feb '25 | Our paper was accepted at ICLR 2025 as a spotlight! |
Oct '24 | The pre-print is now available on arXiv. |
Abstract
Out-of-Domain (OOD) generalization is the ability of a model trained on one or more domains to generalize to unseen domains. In the ImageNet era of computer vision, evaluation sets for measuring a model's OOD performance were designed to be strictly OOD with respect to style. However, the emergence of foundation models and expansive web-scale datasets has obfuscated this evaluation process, as datasets cover a broad range of domains and risk test domain contamination. In search of the forgotten domain generalization, we create large-scale datasets subsampled from LAION---LAION-Natural and LAION-Rendition---that are strictly OOD to corresponding ImageNet and DomainNet test sets in terms of style. Training CLIP models on these datasets reveals that a significant portion of their performance is explained by in-domain examples. This indicates that the OOD generalization challenges from the ImageNet era still prevail and that training on web-scale data merely creates the illusion of OOD generalization. Furthermore, through a systematic exploration of combining natural and rendition datasets in varying proportions, we identify optimal mixing ratios for model generalization across these domains. Our datasets and results re-enable meaningful assessment of OOD robustness at scale---a crucial prerequisite for improving model robustness.
Acknowledgements & Funding
We would like to thank Vishaal Udandarao for helpful discussions and feedback. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A. WB acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. WB is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting PM, TW, RZ, ER, and AJ.
BibTeX
If you find our study helpful, please cite our paper:
title={In Search of Forgotten Domain Generalization},
author={Prasanna Mayilvahanan and Roland Zimmermann and Thaddäus Wiedemer and Evgenia Rusak and Attila Juhos and Matthias Bethge and Wieland Brendel},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=Fk3eod9aaD}
}