There's already companies working on this: https://axiom.ai/ https://axiom.ai/ I would guess that it's just a matter of converting the DOM accessibility tree i

There's already companies working on this: https://axiom.ai/ https://axiom.ai/ I would guess that it's just a matter of converting the DOM accessibility tree into text descriptions, e.g. "There is a button that says 'Start'" And then converting text like "Click the Start button" back into an actual action on the page.