Key takeaway: Adding simple structure at inference-time, through image crops and text segments, is a powerful, training-free way to improve Vision-Language Compositionality performance.

Joint work with @Ander Salaberria @eagirre.bsky.social @gazkune.bsky.social @hitz-zentroa.bsky.social

Our analysis shows that:
1. There is room to improve the quality of extracted text segments.
2. Our method achieves significant performance gains in Winoground's non-trivial instances.
3. Isolated image crops can lose size and quantity information, leaving room for improvement.

Why are image crops crucial? 🤔 We found that simply adding text segments isn't enough. The biggest performance gains come when text segments are paired with image crops, proving the power of serial image computing.

We've evaluated it across three diverse datasets: BiVLC, Winoground (171 instances), and BiSCoR-Ctrl. See the significant improvements by inference-time approach (ITA) on three existing models:

Our approach is straightforward yet effective:
1. Divide the image into smaller crops.
2. Extract text segments capturing objects, attributes and relations.
3. Use the VLM to find image crops that best fit the text segments.
4. Aggregate matching similarities for the final score.