5 ms·
It's all scam. ClosedAI trained on the data they were tested on, so no, nothing here is impressive.
by dot1x 2y ago
It's all scam. ClosedAI trained on the data they were tested on, so no, nothing here is impressive.
- Tanjreeve 2y agoThey definitely did or they probably did? Is there any source for that just so I can point It out to people?
- catalypso 2y agoJust a clarification, they tuned on the public training dataset, not the semi-private one. The 87.5% score was on the semi-private eval, which means the model was still able to generalize well. That being said, the fact that this is not a "raw" base model, but one tuned on the ARC-AGI tests distribution takes away from the impressiveness of the result — How much ? — I'm not sure, we'd need the un-tuned base o3 model score for that. In the meantime, comparing this tuned o3 model to other un-tuned base models is unfair (apples-to-oranges kind of comparison).