7 ms·
Just a fun technical note: CLIP would be possible with any language and vision encoder. It turns out that transformers were the most efficient encoder choices
by dontreact 4y ago
Just a fun technical note:
CLIP would be possible with any language and vision encoder.
It turns out that transformers were the most efficient encoder choices available at the time but most likely the approach would have given interesting results with a resnet and some sort of convolutional language encoder. In fact the paper has a resnet as one model for the vision side.
- ShamelessC 4y agoYeah they trained multiple ResNet models. The ViT's (vision transformers) tended to outperform the ResNet's although they discuss a wide range of datasets and I believe the ResNet ones are better in some cases. They have also been used for visual salience tasks as they may have a better representation of positional information in a scene. I think they do use a modified ResNet with some form of attention in it - but I'm not clear on the specifics and my understanding is that is somewhat common now.