8 ms·

I haven't tried any of these demos, but I'm not surprised they stop impressing once you go deep. What got be mostly impressed were the demos of Astra doing com

by tarruda 10d ago

I haven't tried any of these demos, but I'm not surprised they stop impressing once you go deep. What got be mostly impressed were the demos of Astra doing computer use. At my job I do some RPA and can appreciate how challenging it can be. Yet they make it look extremely easy to operate a tool like Blender at super human speeds.

simonw 10d ago
I've been using Blender via Astra (and Sol before they) without any UI automation at all - Blender supports writing and executing Python scripts, here's how I'm doing that: https://til.simonwillison.net/llms/blender-coding-agents-macos https://til.simonwillison.net/llms/blender-coding-agents-mac... I'd be surprised if any of the impressive Blender demos doing the rounds at the moment were built by having an agent control the mouse and keyboard against the Blender application.
- tarruda 10d ago
  Interesting, thanks for sharing! A scripting API makes the problem much more approachable, but what about those videos where Astra is drawing people from a photo? Here's one using canva: https://x.com/iam_zachi/status/2095992132620136677 https://x.com/iam_zachi/status/2095992132620136677 Is that also using scripting to batch updates? It does look as if the mouse is moving.
  - simonw 10d ago
    Yeah that one looks like controlled mouse updates. I wonder if they sped up the video?
    - tarruda 10d ago
      I'm certain the video is sped up. Still possible to script computer use, and I think this is what happens. One of the suggests this: https://x.com/DanielSMatthews/status/2096311244504572177 https://x.com/DanielSMatthews/status/2096311244504572177 Could be hallucination, but I gave this to an LLM and this is what it suggested: " The workflow shown in the video—processing an image and then controlling a computer interface to draw it—is a combination of two well-established fields: Computer Vision and UI Automation. You do not necessarily need a Large Language Model to perform the underlying image processing; standard algorithms can do this deterministically. Step A: Image Processing (The "Brain") You can write a script (using Python libraries like OpenCV or Pillow) to process the reference photo: - Edge Detection: Use filters (like Canny or Sobel) to find the "high spatial frequencies" (outlines). - Color Quantization: Use clustering algorithms (like K-Means) in the HSL space to group millions of pixels into a small palette of distinct colors. - Vectorization: Convert these processed shapes into a set of coordinates (SVG paths) that represent exactly where the mouse needs to move. Step B: UI Automation (The "Hand") Once the image is converted into a set of instructions (coordinates and color codes), you can use automation tools to physically control the computer and draw on Canva. - Browser Automation: Developers have already created projects that use Selenium (a web automation tool) combined with edge detection algorithms to draw images onto HTML canvases. - The script reads the pixel data, calculates the mouse coordinates, and executes the "click-and-drag" actions in the browser. - Computer Use APIs: In the case of GPT-6 Astra, the model uses a "Computer Use" interface. It effectively takes the processed image data (or generates it internally) and outputs high-level commands (e.g., "Move mouse to X,Y," "Click," "Select Hex Color #FF5733"), which the system then executes on the screen. " Seems plausible and easier to believe. Also, feels like a "magic trick" designed to fool the user into believing that the agent is drawing interactively by using its vision, since it could just have written a python script that takes the input image, and produces the exact same result without automating the screen.

by tarruda 10d ago

simonw 10d ago

I've been using Blender via Astra (and Sol before they) without any UI automation at all - Blender supports writing and executing Python scripts, here's how I'm doing that: https://til.simonwillison.net/llms/blender-coding-agents-macos https://til.simonwillison.net/llms/blender-coding-agents-mac... I'd be surprised if any of the impressive Blender demos doing the rounds at the moment were built by having an agent control the mouse and keyboard against the Blender application.

tarruda 10d ago
Interesting, thanks for sharing! A scripting API makes the problem much more approachable, but what about those videos where Astra is drawing people from a photo? Here's one using canva: https://x.com/iam_zachi/status/2095992132620136677 https://x.com/iam_zachi/status/2095992132620136677 Is that also using scripting to batch updates? It does look as if the mouse is moving.
- simonw 10d ago
  Yeah that one looks like controlled mouse updates. I wonder if they sped up the video?
  - tarruda 10d ago
    I'm certain the video is sped up. Still possible to script computer use, and I think this is what happens. One of the suggests this: https://x.com/DanielSMatthews/status/2096311244504572177 https://x.com/DanielSMatthews/status/2096311244504572177 Could be hallucination, but I gave this to an LLM and this is what it suggested: " The workflow shown in the video—processing an image and then controlling a computer interface to draw it—is a combination of two well-established fields: Computer Vision and UI Automation. You do not necessarily need a Large Language Model to perform the underlying image processing; standard algorithms can do this deterministically. Step A: Image Processing (The "Brain") You can write a script (using Python libraries like OpenCV or Pillow) to process the reference photo: - Edge Detection: Use filters (like Canny or Sobel) to find the "high spatial frequencies" (outlines). - Color Quantization: Use clustering algorithms (like K-Means) in the HSL space to group millions of pixels into a small palette of distinct colors. - Vectorization: Convert these processed shapes into a set of coordinates (SVG paths) that represent exactly where the mouse needs to move. Step B: UI Automation (The "Hand") Once the image is converted into a set of instructions (coordinates and color codes), you can use automation tools to physically control the computer and draw on Canva. - Browser Automation: Developers have already created projects that use Selenium (a web automation tool) combined with edge detection algorithms to draw images onto HTML canvases. - The script reads the pixel data, calculates the mouse coordinates, and executes the "click-and-drag" actions in the browser. - Computer Use APIs: In the case of GPT-6 Astra, the model uses a "Computer Use" interface. It effectively takes the processed image data (or generates it internally) and outputs high-level commands (e.g., "Move mouse to X,Y," "Click," "Select Hex Color #FF5733"), which the system then executes on the screen. " Seems plausible and easier to believe. Also, feels like a "magic trick" designed to fool the user into believing that the agent is drawing interactively by using its vision, since it could just have written a python script that takes the input image, and produces the exact same result without automating the screen.