7 ms·

Show HN: Screen memory without screenshots, just text to Markdown

62 points by Dramatize 23d ago

It's a macOS menu bar app that reads the text of your focused window every few seconds through the Accessibility API. No screenshots, no video, or OCR. It writes plain markdown, one file per day, into a folder you choose. Point Claude Code (or anything with file access) at the folder and ask "what did I work on Tuesday?" or have it build memory about your projects. The folder contains an AGENTS.md explaining the format to the model. Thought it might be useful for someone so chucked it up on GitHub

joka88xj 23d ago
[flagged]
Dramatize 13d ago
I've released the v 1.0.0 with a bunch of improvements + a signed app. It's the first time I've done this, so still working out exactly what's required: https://github.com/dragthelake/ambient-context https://github.com/dragthelake/ambient-context or ambientcontext.app
purplecats 23d ago
neat. why not screenshot and tesseract (videos/images/viewport/etc)
- latexr 23d ago
  Project reasons aside, macOS has perfectly capable, local, native OCR that you can access programmatically, without having to install something else.
  - hzwanip 23d ago
    OCR starts from screenshots though. This only grabs text via accessibility APIs, before rendering or any screenshots.
    - latexr 23d ago
      Yes, I know. Like I said: “Project reasons aside”. I’m not suggesting OCR for this, I’m imparting the general information to be used in other situations that in macOS you can OCR without requiring third-party tools.
      - hzwanip 23d ago
        Fair! It is actually new and interesting information for me (not into macOS dev)
- Dramatize 23d ago
  Because you then have the macOS orange screen sharing warning/icon. I don't really want to record my screen, just the text is enough.
  - hetsaraiya 23d ago
    Agree
- jasonjmcghee 23d ago
  When I built rem, I spent significant effort getting screenshot -> ocr + screenshot -> ffmpeg loop energy efficient, but it definitely is more expensive than accessibility API. You also save a lot of disk space and writes to disk. That being said, you lose the cool swipe to go back in time and search through history and visually see, features. And situations where accessibility isn't supported. And as others have mentioned, built in ocr is definitely better than tesseract.
etchalon 23d ago
I experimented with this exact same approach earlier this year. It's barely sufficient, because, bluntly, most apps just aren't wired up right. So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out. Will try this app, to see if it improved on my own approach, but man, the hope levels are low.
- Dramatize 23d ago
  Yeah don't get your hope up too much. I need to push through more cleaning up of what's captured. Let me know how it goes, keen for some feedback
- hetsaraiya 23d ago
  I haven't read about how the Codex Appshots work yet, but this can be used to extract text properly. I guess. How does this idea look to you?
  - Dramatize 23d ago
    Appshots is fine for what it is, but not great for building ambient context about your daily usage/work.
    - hetsaraiya 23d ago
      But we can clean that extracted text from Appshots. UPDATE: As i just checked it is also using Accessibility API. So i guess we will have access to the same set to data.
- zahlman 23d ago
  One of the reasons people like TUIs is because the text is always just right there.
  - ShinyLeftPad 23d ago
    Not if it's rendered on GPU, I guess?
Dramatize 23d ago
One app that's using this technique (not exactly sure if it's the same) is Littlebird: https://littlebird.ai/ https://littlebird.ai/ I also saw that HeyClicky started doing something similar but end up removing from the product.
zahlman 23d ago
> reads the text of your focused window every few seconds through the Accessibility API > It writes plain markdown Where are the formatting decisions coming from?
- Dramatize 23d ago
  So the markdown is scaffolding the app generates deterministically. The ## heading is built from the block's timestamps, the app name and the window title. The frontmatter is per-day boilerplate. The file:/url: line is the window's backing document where the app exposes one. The captured text underneath is written exactly as the tree handed it over: plain lines, no reconstruction. That lossiness is also why the file/URL references exist. Trying to rebuild a document's formatting from its accessibility tree is a losing game, so instead each block records where the real document lives, and the LLM reading the file can open the original if the fragments aren't enough. "Plain markdown" was meant as "a markdown file you can open anywhere", not "faithful markdown conversion of what you saw"
alec_builds 23d ago
[dead]
jwr 23d ago
Hmm. What would be really helpful for me would be an enhancement of the TypeWhisper app for dictation that would be able to get the context of what I am dictating into and send it along with the prompt to an LLM. Probably much simpler and much less of a privacy problem (I run my own local LLM for that purpose so that nothing leaves the machine).
jv22222 23d ago
Nice constraint. I've been building in this space for a while now and the lossiness question is a real one. The accessibility tree works for a lot of things but the more I worked on it the more stuff I found needed pixels. YMMV.
cobertos 23d ago
I've liked the focused window tracker in https://activitywatch.net https://activitywatch.net
- Dramatize 22d ago
  Oh interesting, I'll check it out
petesergeant 23d ago
A cool project, but feels like fundamentally the wrong approach when what you do most likely leaves a string of structured digital footprints anyway. I’ve got an agent that fills out my timesheets by looking at git commits, agent history, Slack messages, emails, and time-tracker tickets. I guess I could add relevant web-browsing?
Dramatize 22d ago
Update: I've added a prompt to process the raw output into something more useful: https://github.com/dragthelake/ambient-context/blob/main/docs/day-context-prompt.md https://github.com/dragthelake/ambient-context/blob/main/doc... This is pretty magical tbh Also added MIT license and added some more pruning to reduce initial capture by around 10%.
bickov 18d ago
The per-app breakage is the real problem with text extraction, not accuracy, like etchalon pointed out. Screenshots dodge that since they look the same no matter what is rendering. Cropping to the window and marking it up before it goes anywhere also skips the full screen recording and the orange icon. https://slimsnap.ai https://slimsnap.ai already works that way.

62 points by Dramatize 23d ago

joka88xj 23d ago

[flagged]

Dramatize 13d ago

I've released the v 1.0.0 with a bunch of improvements + a signed app. It's the first time I've done this, so still working out exactly what's required: https://github.com/dragthelake/ambient-context https://github.com/dragthelake/ambient-context or ambientcontext.app

purplecats 23d ago

neat. why not screenshot and tesseract (videos/images/viewport/etc)

latexr 23d ago
Project reasons aside, macOS has perfectly capable, local, native OCR that you can access programmatically, without having to install something else.
- hzwanip 23d ago
  OCR starts from screenshots though. This only grabs text via accessibility APIs, before rendering or any screenshots.
  - latexr 23d ago
    Yes, I know. Like I said: “Project reasons aside”. I’m not suggesting OCR for this, I’m imparting the general information to be used in other situations that in macOS you can OCR without requiring third-party tools.
    - hzwanip 23d ago
      Fair! It is actually new and interesting information for me (not into macOS dev)
Dramatize 23d ago
Because you then have the macOS orange screen sharing warning/icon. I don't really want to record my screen, just the text is enough.
- hetsaraiya 23d ago
  Agree
jasonjmcghee 23d ago
When I built rem, I spent significant effort getting screenshot -> ocr + screenshot -> ffmpeg loop energy efficient, but it definitely is more expensive than accessibility API. You also save a lot of disk space and writes to disk. That being said, you lose the cool swipe to go back in time and search through history and visually see, features. And situations where accessibility isn't supported. And as others have mentioned, built in ocr is definitely better than tesseract.

etchalon 23d ago

I experimented with this exact same approach earlier this year. It's barely sufficient, because, bluntly, most apps just aren't wired up right. So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out. Will try this app, to see if it improved on my own approach, but man, the hope levels are low.

Dramatize 23d ago
Yeah don't get your hope up too much. I need to push through more cleaning up of what's captured. Let me know how it goes, keen for some feedback
hetsaraiya 23d ago
I haven't read about how the Codex Appshots work yet, but this can be used to extract text properly. I guess. How does this idea look to you?
- Dramatize 23d ago
  Appshots is fine for what it is, but not great for building ambient context about your daily usage/work.
  - hetsaraiya 23d ago
    But we can clean that extracted text from Appshots. UPDATE: As i just checked it is also using Accessibility API. So i guess we will have access to the same set to data.
zahlman 23d ago
One of the reasons people like TUIs is because the text is always just right there.
- ShinyLeftPad 23d ago
  Not if it's rendered on GPU, I guess?

Dramatize 23d ago

One app that's using this technique (not exactly sure if it's the same) is Littlebird: https://littlebird.ai/ https://littlebird.ai/ I also saw that HeyClicky started doing something similar but end up removing from the product.

zahlman 23d ago

> reads the text of your focused window every few seconds through the Accessibility API > It writes plain markdown Where are the formatting decisions coming from?

Dramatize 23d ago
So the markdown is scaffolding the app generates deterministically. The ## heading is built from the block's timestamps, the app name and the window title. The frontmatter is per-day boilerplate. The file:/url: line is the window's backing document where the app exposes one. The captured text underneath is written exactly as the tree handed it over: plain lines, no reconstruction. That lossiness is also why the file/URL references exist. Trying to rebuild a document's formatting from its accessibility tree is a losing game, so instead each block records where the real document lives, and the LLM reading the file can open the original if the fragments aren't enough. "Plain markdown" was meant as "a markdown file you can open anywhere", not "faithful markdown conversion of what you saw"

alec_builds 23d ago

[dead]

jwr 23d ago

Hmm. What would be really helpful for me would be an enhancement of the TypeWhisper app for dictation that would be able to get the context of what I am dictating into and send it along with the prompt to an LLM. Probably much simpler and much less of a privacy problem (I run my own local LLM for that purpose so that nothing leaves the machine).

jv22222 23d ago

Nice constraint. I've been building in this space for a while now and the lossiness question is a real one. The accessibility tree works for a lot of things but the more I worked on it the more stuff I found needed pixels. YMMV.

cobertos 23d ago

I've liked the focused window tracker in https://activitywatch.net https://activitywatch.net

Dramatize 22d ago
Oh interesting, I'll check it out

petesergeant 23d ago

A cool project, but feels like fundamentally the wrong approach when what you do most likely leaves a string of structured digital footprints anyway. I’ve got an agent that fills out my timesheets by looking at git commits, agent history, Slack messages, emails, and time-tracker tickets. I guess I could add relevant web-browsing?

Dramatize 22d ago

Update: I've added a prompt to process the raw output into something more useful: https://github.com/dragthelake/ambient-context/blob/main/docs/day-context-prompt.md https://github.com/dragthelake/ambient-context/blob/main/doc... This is pretty magical tbh Also added MIT license and added some more pruning to reduce initial capture by around 10%.

bickov 18d ago

The per-app breakage is the real problem with text extraction, not accuracy, like etchalon pointed out. Screenshots dodge that since they look the same no matter what is rendering. Cropping to the window and marking it up before it goes anywhere also skips the full screen recording and the orange icon. https://slimsnap.ai https://slimsnap.ai already works that way.