7 ms·
Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like
by losvedir 1mo ago
Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like almost who cares what it produces? No one is providing value if the extent of their contribution is so small.
Much more interesting is providing a million tokens of meaningful input and getting 1000 tokens out (high level critique of a detailed design doc, finding a subtle bug in a big codebase, etc).
- ulrikrasmussen 1mo agoI think both can be interesting, but surely the 100 words will lead to a lot of variability in the exact shape of the result because so much of the spec will be left up to the LLM. However, it is still interesting to judge the quality of the result and how self-consistent it is. Not by looking at the actual webpage, but by looking at the code and whether it is nicely organized, makes proper use of abstractions, and in general doesn't feel like spaghetti.
- maccard 1mo agoGreat - can you show us some examples of these? Every time this topic comes up here it’s full of people saying “you’re using it wrong” and apparently it should just be intuitive. Yet all the material is like this.