ytaek-oh/eqben-images

Equivariant Similarity for Vision-Language Foundation Models

ICCV 2023

Tan Wang,  Kevin Lin,  Linjie Li,  Chung-Ching Lin,  Zhengyuan Yang,  Hanwang Zhang,  Zicheng Liu,  Lijuan Wang
Nanyang Technological University,   Microsoft Corporation

About

This study explores the concept of equivariance in vision-language foundation models (VLMs), focusing specifically on the multimodal similarity function that is not only the major training objective but also the core delivery to support downstream tasks. Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes. Our key contributions are three-fold:

1. A novel benchmark named EqBen (Equivariant Benchmark) to benchmark VLMs with visual-minimal change samples.
2. A plug-and-play regularization loss EqSim (Equivariant Similarity Learning) to improve the equivariance of current VLMs.
3. Toolkit provides an one-stop evaluation: not only for EqBen, but also for previous related benchmarks (Winoground, VALSE, etc).
   

Data Download

• Download images from huggingface hub: Please check the Files and versions tab above.

• Full-Test Set: the user can download the EqBen raw image data (tar.gz file, ~100G) and annotation (after randomize) (200M) via Google Drive. [UPDATE-2023-09] The original annotation is the annotation after randomize (non-public) for the total fairness. And the users are required to upload the results json/np file to CodaLab for getting the final results. Due to the unstability of CodaLab, we decide to public the whole original annotation. This annotation file formalized similar to Winoground and can be downloaded here.

• Light Full-Test Set: to improve the usability, we also provide a light version of EqBen by converting all the png image to the jpg using convert. Feel free to download here. But please note that you may make some small revisement to the path in the annotation (change the .png to .jpg).

• Sub-Test Set: we also provide a 10% subset (~25K image-text pairs) for the ease of visualization and validation. The label of the EqBen sub-set is opensource and the format follows the winoground style. But please note that the samples in the subset is randomly sorted and not be classified to each category. Please down the raw image data (tar.gz file, ~10G) and annotation via Google Drive.

---

• This is the unofficial distribution of images in the eqben benchmark.
• For the official repository, please visit https://github.com/Wangt-CN/EqBen.
• Some part of this README.md is taken from the official repository