4 ms·
They are told outside the test because if they can't find it when given hints then it's safe to assume it won't find it given no hints. It verifies to test, to
by brigandish 3mo ago
They are told outside the test because if they can't find it when given hints then it's safe to assume it won't find it given no hints. It verifies to test, to an extent, much like running tests that should fail when given a set of inputs that should make it fail (you write an always failing test alongside your other tests, right?;)
- isomorphic_duck 3mo agoNo, the purpose was to create a (automated) test set in the first place. The author builds an LLM judge which can score the LLMs participating during test-time. That would be why the author used the strongest model (Opus 4,7 at the time) as the judge.
- brigandish 3mo agoYes, by giving hints it verifies the test because if it can't find it with hints then it shouldn't find it without hints, else there is some other mechanism at work (which would also be useful to know, like access to other tools).