The problem is benchmarking. Not everyone has a 500k token workstream of the model they are setting up for the first time to run it against 10 different config

The problem is benchmarking. Not everyone has a 500k token workstream of the model they are setting up for the first time to run it against 10 different config and compare differences. And if you download benchmarks from the net they are likely poisoned by models being trained on them.