6 ms·
But why should I care? If my metrics are speed and cost? How many tokens it takes as a user is arbitrary to some extent.
by prism56 2mo ago
But why should I care? If my metrics are speed and cost? How many tokens it takes as a user is arbitrary to some extent.
- WhitneyLand 2mo agoIf speed is a metric for you, tokens required to solve a problem affects that metric. All else being equal passing triple the amount of tokens through a model to solve the same problem makes it slower. Doesn’t mean this model is bad, and it has to be considered how cheap it is, but it’s a factor.
- rat9988 2mo agoStill, seems useless to measure tokens when you can measure speed.