5 ms·
Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration. On wikitext2 there was no difference. I con
by stared 8d ago
Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration.
On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same.
So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.
- kmike84 8d agoYeah, makes sense; e2e is different and valuable, KLD is not a replacement. As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SFT-Data https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?
- stared 7d agoNope. If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.