5 ms·
The ARCC-AGI-3 performance is absolutely incredible. The magnitude of change here is so high that I'm almost incredulous. Is this real? Did the benchmark get ga
by aliljet 14d ago
The ARCC-AGI-3 performance is absolutely incredible. The magnitude of change here is so high that I'm almost incredulous. Is this real? Did the benchmark get gamed?
- ionwake 14d agomy first suspicion is gaming - but i have no idea honestly
- aesthesia 14d agoARC-AGI-3 scoring is constructed in a weird nonlinear way (the level score is the square of the ratio between the AI's number of moves and the human median) so this kind of discontinuous jump is to be expected.
- enraged_camel 14d agoThey used a custom harness. It's not a one-to-one comparison.