> We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n. Thank you. I just changed my opinion on this t

> We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n. Thank you. I just changed my opinion on this thanks to you. I agree now, since we tend to execute LLM tasks once instead of N times anyway. I suspect models like Fable executes the same task N times in parallel and picks best answer or merges them to for a better answer.