The weights are for the ImageNet category only, just FYI. Which should mean the model is only trained to pair image outputs with the CLIP vector from the very

The weights are for the ImageNet category only, just FYI. Which should mean the model is only trained to pair image outputs with the CLIP vector from the very short ImageNet labels, so the quality of intermediate latent space for long form text input would be limited. If one wanted to use pretrained weights I'd recommend a model trained on LAION-400M instead. The architecture looks cool though, seems to be a smaller model.