You can. The point of having your agent keep track of it is that it will likely notice things you won't, and it can automate cataloging it with relevant metadat

You can. The point of having your agent keep track of it is that it will likely notice things you won't, and it can automate cataloging it with relevant metadata (prompts, environment, examples, etc.) that make it trivial to automate rerunning those tests when new models launch.