5 ms·
Rather than teach the agent where grafana is, how to use a sentry trace id to find a otel traceparent, where my ALB is, I'm what clusters do what, which namespa
by hadlock 9d ago
Rather than teach the agent where grafana is, how to use a sentry trace id to find a otel traceparent, where my ALB is, I'm what clusters do what, which namespace prod is in, what our stack looks like, that thing that looks broken actually isn't, etc etc etc. I just paste in the sentry trace id, a guess of what might be wrong ("i think we overtuned gunicorn again" or "developer bob pushed short sha 123456 and nothing is working") and say "use your investigate skill" then get up and go get a coffee and usually by the time i get back i have an investigate doc filled out from a common template in my notes repo. The agent almost never spends any time spinning it's wheels finding out what the various environments do, where they're located, what our metrics and logging looks like, how to access it etc etc.
To be fair it's the only skill i have/use but I got real tired of explaining the same 12 things over and over. Having it document every incident means I have a dense library of every problem we've run into over the last six months which helps identify recurring problems for RCA
- amoriz 6d ago[flagged]