DataComp
First rigorous benchmark for advancing multimodal dataset creation.
The goal of DataComp is to better understand how to curate datasets for training foundation models. To this end, the PIs assembled a large collaboration involving more than 30 researchers from industry and academia and created DataComp, the first rigorous benchmark for creating multimodal datasets. The DataComp testbed enables researchers in both academia and industry to investigate new methods for curating training sets for multimodal foundation models.
The testbed itself is centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in the benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running a standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources.
Our baseline experiments show that the DataComp workflow leads to better training sets. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet, outperforming OpenAI's CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute.
