6 ms·
> weight update during eval: this is a form of test time training and not really cheating. Possibly "not really cheating", but it does make benchmark compariso
by piqufoh 15d ago
> weight update during eval: this is a form of test time training and not really cheating.
Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.
- evilmathkid 15d agono, most models on kaggle are finetuning during test time, (including LLM based approaches) Pure frontier LLMs dont, but thats because nobody knows how to make it work cleanly and at scale. Once someone makes it work, it will be deployed