5 ms·
If LLMs were good at summarization, this wouldn't be necessary. Turns out a stochastic model of language is not a summary in the way humans think of summaries.
by ttpphd 2y ago
If LLMs were good at summarization, this wouldn't be necessary. Turns out a stochastic model of language is not a summary in the way humans think of summaries. Thus all this extra faff.
- sroussey 2y agoWhat are the good models for summarization? I have found all, particularly local models, to be poor. Is there a leaderboard for summarization somewhere?
- rafaelmn 2y agoHow do you evaluate quality ? Also I suspect the performance between models would varry between datasets. Heck it would vary on same model/source if you included that your mother was being held hostage and will be killed unless you summarize the source correctly :). I think you are still stuck with try if it works for you and hope it generalizes beyond your evaluation.
- mrlongroots 2y agoI think summarization quality can only be a subjective criterion measured using user studies and things like that. The task itself is not very well-defined. You want a lossy representation that preserves the key points -- this may require context that the model does not have. For technical/legal text, seemingly innocuous words can be very load-bearing, and their removal can completely change the semantics of the text, but achieving this reliably requires complete context and reasoning.
- deleted 2y ago[deleted]
- imoreno 2y ago>evaluate quality [information content of summary] / [information content of original] for summaries of a given length cap?
- anon373839 2y agoThere are actually some clever approaches to eval abstractive summarization. Examples: https://eugeneyan.com/writing/evals/#summarization-consistency-relevance-length https://eugeneyan.com/writing/evals/#summarization-consisten...