mrciffa

8 ms·

1.
Show HN: MCP Log Reader – Open-source server to analyze MCP logs faster(github.com)
2 points by mrciffa 1y ago|0 comments
2.
by mrciffa 2y ago
It should work with any type of model, obviously longer chain of thoughts will be more difficult to analyse by the evaluation model, because it will have way more reasoning steps to identify and separate. The quality of the outcome depends
3.
by mrciffa 2y ago
We are currently giving broad suggestions with an insight model that can be chosen during the setup. We will try to update and improve the suggestion prompt/code to make them more granular with new releases
4.
Show HN: Klarity – OS tool to debug LLM reasoning patterns with entropy analysis(github.com)
3 points by mrciffa 2y ago|4 comments
5.
by mrciffa 2y ago
Yes, reasoning models can potentially be optimized with our uncertainty estimations. We are currently testing the library with DeepSeek R1
6.
by mrciffa 2y ago
Unfortunately LLMs are a gigantic monster to understand, we were considering your same approach with sliding window and we will try to keep the library updated with better and more reliable approaches based on new research papers and our in
7.
by mrciffa 2y ago
Apache-2.0 is correct one
8.
by mrciffa 2y ago
Exactly! Uncertainty is critical to correctly evaluate LLM performance and we don't need reasoning models to spend thousands of tokens on simple questions
9.
by mrciffa 2y ago
We want to integrate reasoning models as next steps because we see a lot of value in understanding better CoTs behaviour (DeepSeek R1 & Co)
10.
by mrciffa 2y ago
Oh damn, you are right. It's my first opensource project and I didn't thought about it
11.
by mrciffa 2y ago
In the example I'm using the instruction tuned version of Qwen2.5-7B to generate the insights
12.
Show HN: Klarity – Open-source tool to analyze uncertainty/entropy in LLM output(github.com)
132 points by mrciffa 2y ago|26 comments