7 ms·
Probably not what you're after, but I've considered having a separate small mm-model act as a seeing-eye dog for the bigger more capable one.
by xscott 1mo ago
Probably not what you're after, but I've considered having a separate small mm-model act as a seeing-eye dog for the bigger more capable one.
- cosmez 1mo agoBut how do you prompt this smaller model to give back information? I've tried that in the past but didn't go well. What I do is to send written handoff files between models to pass context around, but only had good results with big vision models as well.
- xscott 1mo agoSo many possibilities for how you could glue it all together. However, when I send Gemma 4 12B in llama.cpp an image with no accompanying text, it assumes I want a description and gives me one. I just tried with an audio file, and it transcribed the lyrics as I hoped. Then it made a bunch of suggestions about what do next, which I wasn't after. I could probably fix that by sending some text to narrow the scope.
- bellowsgulch 1mo agoI've tried this, too, and for whatever reason, simple things like having a subagent with vision capabilities hallucinates responses to the blind models. I don't know why, and it's not consistent, but I don't see this happening when the primary agent is a vision model.