5 ms·
What does your perception look like, are you using raw screenshots? GUI snapshots? Vision is very difficult for these, and snapshots are incomplete, is what I'v
by iiJDSii 2y ago
What does your perception look like, are you using raw screenshots? GUI snapshots? Vision is very difficult for these, and snapshots are incomplete, is what I've found in some earlier experiments.
- mountainriver 2y agoPerception is just 1-2 screenshots. A number of recent VLM models have a lot more pretraining data on GUI interactions, which helps.
- iiJDSii 2y agoSuch as? Are they able to recognize arbitrary GUI elements from various desktop programs, web browsers, etc?
- mountainriver 2y agoQwen2.5-vl seems to be the best right now by our tests. UI-TARS by bytedance also has a good amount of pretraining. Molmo is also very good at coordinates.