I quickly tested if SigLIP or CLIP embeddings show evidence of attribute binding and they don’t (however n=1 image) - an image of a red cube with a blue sphere compared with texts “red cube next to blue sphere” and “blue cube next to red sphere” doesn’t get a higher similarity score for the correct label than for the wrong one (CLIP, SigLIP).
Thanks for the link and suggestions!
I quickly tested if SigLIP or CLIP embeddings show evidence of attribute binding and they don’t (however n=1 image) - an image of a red cube with a blue sphere compared with texts “red cube next to blue sphere” and “blue cube next to red sphere” doesn’t get a higher similarity score for the correct label than for the wrong one (CLIP, SigLIP).
Nice quick check!
Just to be clear: This is for the actual full models? Or for the ‘model embeddings’ as in you’re doing a comparison right after the embedding layer?
This is for the full models—I simply used both models on replicate and gave one image and two text labels as input: CLIP, SigLIP