Closing the Modality Gap via Simplex-Constrained Representations· 通过简单约束表示关闭模态差距
While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent \emph{modality gap}. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, $S^{d-1}$, to the probability simplex, $Δ^{d-1}$. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid $\ell_2$ gap by 97--99\% across five multimodal retrieval benchmarks. This confirms that simplicial
利用概率单纯形约束表示减少多模态模型中的模态差距。
- 核心方法
- 将嵌入域从单位球面$S^{d-1}$改为概率单纯形$Δ^{d-1}$，并通过添加轻量级softmax适配器和使用总变异相似度来实现跨模态校准。
- 适合谁读
- 适合多模态对比学习的研究者和技术人员阅读。
- 要解决的问题
- 多模态对比模型（如CLIP和CLAP）在学习共享表示时存在持续的模态差距，即不同模态的嵌入向量间存在显著差异。
- 关键实验
- 在五个多模态检索基准测试上进行了实验，显示质心$l_2$差距显著减少。
- 主要贡献
- 方法显著减少了五个多模态检索基准测试中的质心$l_2$差距，减少比例达到97-99%。
- 意义与局限
- 此方法提供了一种有效的手段来解决多模态对比学习中的模态差距问题，对于提高模型的多模态理解能力具有重要意义；但可能需要进一步验证其在更多任务上的泛化能力。