4 ms·
Anthropic: Demystifying Evals for AI Agents
- dangelosaurus 8mo agoI work on Promptfoo (an open-source eval framework). Appreciate the mention here. This post captures a lot of the hard lessons around agent evals. In particular, task ambiguity and brittle graders are things we run into constantly.