4 ms·
The weirdness of LLMs is that they're so damn good at so many things but then you see these glaring gaps that instantly make them seem dumb. We desperately need
by timbilt 2y ago
The weirdness of LLMs is that they're so damn good at so many things but then you see these glaring gaps that instantly make them seem dumb.
We desperately need benchmarks and evals that test these kinds of hard to pin down cognitive abilities
- turnsout 2y agoAbsolutely. This is not a new observation, but another thing they struggle with is self-reporting confidence intervals. When I've asked LLMs to classify/tag things along with a confidence metric, the number seems random and has no connection to the quality or difficulty of the classification.