A research paper on arXiv proposes a smarter way to stop vision-language models from acting more certain than their training data justifies.
The paper, titled "Text as Partial Constraint" (TPC), targets a specific failure mode: image captions are almost always incomplete, yet standard contrastive training treats them as full descriptions. When you swap in a paraphrase or leave out a detail, similarity scores become brittle. The TPC framework addresses this by splitting caption signals into a "core" — the shared semantic agreement across multiple caption views — and a "residual" of unsaid details the model should not lean on. At inference time, a single-view predictor handles one query without needing multiple captions. An uncertainty-aware contrastive objective further damps confidence when caption views disagree.
The practical payoff is measurable. On ImageNet, TPC reaches 81.42 Top-1 clean accuracy and 64.05 robust accuracy — the robust figure is the more telling number, since it reflects performance under adversarial pressure rather than ideal conditions. On a 14-dataset transfer suite, it hits 76.19 clean and 52.03 robust. Downstream, under a LLaVA-1.5-7B stack, it scores 85.16 on the POPE F1 hallucination benchmark and 59.57 accuracy on OKVQA, improving over standard vision-language pretraining on both.
Vision-language alignment has been a known weak point since CLIP-era models started confidently mismatching images and oddly worded queries. TPC does not claim to solve the problem, but the POPE result is worth watching — hallucination in large vision-language models is an active pain point, and a pretraining fix that costs no extra inference overhead is exactly the kind of upstream intervention that tends to compound downstream.