6 ms·

Yeah that one looks like controlled mouse updates. I wonder if they sped up the video?

by simonw 10d ago

tarruda 10d ago
I'm certain the video is sped up. Still possible to script computer use, and I think this is what happens. One of the suggests this: https://x.com/DanielSMatthews/status/2096311244504572177 https://x.com/DanielSMatthews/status/2096311244504572177 Could be hallucination, but I gave this to an LLM and this is what it suggested: " The workflow shown in the video—processing an image and then controlling a computer interface to draw it—is a combination of two well-established fields: Computer Vision and UI Automation. You do not necessarily need a Large Language Model to perform the underlying image processing; standard algorithms can do this deterministically. Step A: Image Processing (The "Brain") You can write a script (using Python libraries like OpenCV or Pillow) to process the reference photo: - Edge Detection: Use filters (like Canny or Sobel) to find the "high spatial frequencies" (outlines). - Color Quantization: Use clustering algorithms (like K-Means) in the HSL space to group millions of pixels into a small palette of distinct colors. - Vectorization: Convert these processed shapes into a set of coordinates (SVG paths) that represent exactly where the mouse needs to move. Step B: UI Automation (The "Hand") Once the image is converted into a set of instructions (coordinates and color codes), you can use automation tools to physically control the computer and draw on Canva. - Browser Automation: Developers have already created projects that use Selenium (a web automation tool) combined with edge detection algorithms to draw images onto HTML canvases. - The script reads the pixel data, calculates the mouse coordinates, and executes the "click-and-drag" actions in the browser. - Computer Use APIs: In the case of GPT-6 Astra, the model uses a "Computer Use" interface. It effectively takes the processed image data (or generates it internally) and outputs high-level commands (e.g., "Move mouse to X,Y," "Click," "Select Hex Color #FF5733"), which the system then executes on the screen. " Seems plausible and easier to believe. Also, feels like a "magic trick" designed to fool the user into believing that the agent is drawing interactively by using its vision, since it could just have written a python script that takes the input image, and produces the exact same result without automating the screen.

Yeah that one looks like controlled mouse updates. I wonder if they sped up the video?

by simonw 10d ago

Yeah that one looks like controlled mouse updates. I wonder if they sped up the video?

I'm certain the video is sped up. Still possible to script computer use, and I think this is what happens. One of the suggests this: https://x.com/DanielSMatthews/status/2096311244504572177 https://x.com/DanielSMatthews/status/2096311244504572177 Could be hallucination, but I gave this to an LLM and this is what it suggested: " The workflow shown in the video—processing an image and then controlling a computer interface to draw it—is a combination of two well-established fields: Computer Vision and UI Automation. You do not necessarily need a Large Language Model to perform the underlying image processing; standard algorithms can do this deterministically. Step A: Image Processing (The "Brain") You can write a script (using Python libraries like OpenCV or Pillow) to process the reference photo: - Edge Detection: Use filters (like Canny or Sobel) to find the "high spatial frequencies" (outlines). - Color Quantization: Use clustering algorithms (like K-Means) in the HSL space to group millions of pixels into a small palette of distinct colors. - Vectorization: Convert these processed shapes into a set of coordinates (SVG paths) that represent exactly where the mouse needs to move. Step B: UI Automation (The "Hand") Once the image is converted into a set of instructions (coordinates and color codes), you can use automation tools to physically control the computer and draw on Canva. - Browser Automation: Developers have already created projects that use Selenium (a web automation tool) combined with edge detection algorithms to draw images onto HTML canvases. - The script reads the pixel data, calculates the mouse coordinates, and executes the "click-and-drag" actions in the browser. - Computer Use APIs: In the case of GPT-6 Astra, the model uses a "Computer Use" interface. It effectively takes the processed image data (or generates it internally) and outputs high-level commands (e.g., "Move mouse to X,Y," "Click," "Select Hex Color #FF5733"), which the system then executes on the screen. " Seems plausible and easier to believe. Also, feels like a "magic trick" designed to fool the user into believing that the agent is drawing interactively by using its vision, since it could just have written a python script that takes the input image, and produces the exact same result without automating the screen.