9 ms·

Imagine porting this to a dedicated app that can access the context of the open window and the text on the screen, providing an almost real-time assistant for e

Imagine porting this to a dedicated app that can access the context of the open window and the text on the screen, providing an almost real-time assistant for everything you do on screen.

column 3y ago
Automatically take a screenshot and feed it to https://github.com/vikhyat/moondream https://github.com/vikhyat/moondream or similar? Doable. But while very impressive, the results are a bit of mixed bag (some hallucinations)
- cristyansv 3y ago
  I'm sure something like the accessibility API will have a smaller latency. https://developer.apple.com/library/archive/samplecode/UIElementInspector/Introduction/Intro.html https://developer.apple.com/library/archive/samplecode/UIEle...
- summarity 3y ago
  rewind.ai seems to be moving in this direction
  - cristyansv 3y ago
    this looks equally scary and incredible, especially the "summarize what I worked on today" examples.
    - fragmede 3y ago
      it works really well, and locally too!