5 ms·
Hey there everyone, now that AI can "see" very well with GPT-V, I was wondering if it can interact with a computer like we do, just by looking at it. Well, one
by rchaves 3y ago
Hey there everyone, now that AI can "see" very well with GPT-V, I was wondering if it can interact with a computer like we do, just by looking at it. Well, one of the shortcommings of GPT-V is that it cannot really pinpoint the x,y coordinates of something in the screen very well, but I solved it by combining it with simple OCR, and annotating for GPT-V to tell where it wants to click
Turns out with very few lines of code the results are already impressive, GPT-V can really control my computer super well and I can as it to do whatever tasks by itself, it clicks around, type stuff and press buttons to navigate
Would love to hear your thoughts on it!
- hnuser123456 3y agoHave you already seen OthersideAI self-operating-computer? It sounds like exactly what you're describing: https://www.youtube.com/watch?v=UKRti40U8IA https://www.youtube.com/watch?v=UKRti40U8IA
- rchaves 3y agoyes actually, but I only saw it after I've implemented it, I had actually searched for something like that before but I guess Google is worse and worse those days however, I tried self-operating-computer, and it could not find the right x,y positions on the screen executes the task as effectively
- wills_forward 3y agoI really like the elegant simplicity of tagging the screen elements like that and not obfuscating it away. Nice work too!
- rchaves 3y agothanks! I took my inspiration from Vim browser plugin (https://chromewebstore.google.com/detail/vimium/dbepggeogbaibhgnhhndojpepiihcmeb https://chromewebstore.google.com/detail/vimium/dbepggeogbai...), they have a shortcut F that allows you to choose any element on the website to navigate from thanks vim!