nirga

6 ms·

1.
by nirga 8d ago
I think both the iPhone itself and things like AirPods form recent years are truly innovative so I wouldn’t say that
2.
An LLM attacked our lab so we made it work for us(jesta.ai)
2 points by nirga 2mo ago|0 comments
3.
by nirga 10mo ago
Hey! We don’t support an otel collector directly - you have to connect it to some backend. Minimal one can be jaeger for example
4.
by nirga 1y ago
Sorry didn’t know that. It’s not my article so didn’t want to attribute the title to myself
5.
by nirga 1y ago
For example - the fact that the FE environment variables are hardcoded at build time makes it hard to just deploy a container
6.
Why I Won’t Use Next.js(epicweb.dev)
24 points by nirga 1y ago|8 comments
7.
by nirga 2y ago
I think that's the key benefit of using OpenTelemetry - it's pretty efficient and the performance footprint is negligible.
8.
by nirga 2y ago
Thanks for spotting those! We'll fix it asap
9.
by nirga 2y ago
I think you can (pretty) easily set this up with an otel collector and something that replays data from S3 - there's a native implementation that converts otel to clickhouse
10.
by nirga 2y ago
You can do it and it's a good way of doing that - from our experiments that can catch most errors. You don't even need to use different models - even using the same model (I don't mean asking "are you sure?" - just
11.
by nirga 2y ago
Thanks! It can vary greatly between use cases - but we've seen extremely high detection rates for tagged texts (>95%). When switching to production, this gets trickier since you don't know what you don't know (so it's
12.
by nirga 2y ago
You're right. We faced those same issues. So we plan to move those prompts and completions to be sent as log events with some reference to the trace/span and not actually on the span. The span can then only contain the most import
13.
by nirga 2y ago
Thanks! I wasn’t offended or anything, don’t get the wrong impression. What strikes me odd is the fact that an AI that checks AI is an issue. Because AI can mean a lot of things - from a encoder architecture, a neural network, or a simple r
14.
by nirga 2y ago
It has the same logic of saying you dont want to use a computer to monitor or test your code since it will mean that a computer will monitor a computer. AI is a broad term, I agree you can use GPT (or any LLM) to grade an LLM in an accurate
15.
by nirga 2y ago
I replied to you in a different thread, I don't think calling our companies "deceptive" will help you or me get anywhere. While I agree with you that detection will never be hermetic, I don't think this is the goal. By d
16.
by nirga 2y ago
I'm sorry but this is not what we do. We don't use LLMs to grade your LLM calls.
17.
by nirga 2y ago
I think that LLMs are hallucinating by design. I'm not sure we'll ever get to a 0% hallucinations and we should be ok with it (at least for the next coming years?). So getting an alert on hallucination becomes less interesting. Wh
18.
by nirga 2y ago
I think it depends on the use case and how you define hallucinations. We've seen our metrics perform well (=correlates with human feedback) for use cases like summarization, RAG question-answering pipeline, and entity extraction. At th
19.
by nirga 2y ago
Ping me over slack (traceloop.com/slack) or email nir at traceloop dot com
20.
by nirga 2y ago
roger that! I like them though (am I a normie then?)
21.
by nirga 2y ago
I tend to find classic NLP metric more predictable and stable than "LLM as a judge" metrics so I'd try to see if you rely on them more. We've written a couple of blog posts about some of them: https://www.trac
22.
by nirga 2y ago
We trained our own models for some of them, and we combined some well known NLP metrics (like Gruen [1]) to make this work. You're right that it's hard to figure out how to "trust" these metrics. But you shouldn't l
23.
by nirga 2y ago
I know! When we started every time I was googling "traceloop" this was the first result. 2 reasons why we chose it (in this order): 1. traceloop.com was available 2. we work with traces
24.
by nirga 2y ago
I have it internally, I can share it if you want! But to the point of comparison between these and tools like Traceloop - it's interesting to see this space and how each platform takes it's own path and finds its own use cases. La
25.
by nirga 2y ago
Great question and I see you already got a similar answer but I'll add some of my thoughts on this. We are actively promoting OpenLLMetry as a vendor agnostic way of observing LLMs (see some examples [1], [2]). We believe that people
26.
by nirga 2y ago
Thanks so much! I always say that I'm a strong believer in open protocols so I'd love to assist you if you want to use OpenLLMetry as your SDK. We onboarded other startups / competitors like Helicone and Honeyhive and it&#x27
27.
by nirga 2y ago
Thanks! We differentiate in 2 ways: 1. We focus on real-time monitoring. This is where we see the biggest pain with our customers, so we spent a lot of time researching and building the right metrics that can run at scale, fast and at low c
28.
by nirga 2y ago
OpenLLMetry creator here. We’re building the most popular OpenTelemetry instrumentation for LLm providers including OpenAI, Anthropic, Pinecone, Langchain and >10 others since last August [1]. We’re using import hooks (like other otel in
29.
by nirga 2y ago
How does it do that for ruby for example? (which is in the link you provided). OTEL instrumentation for HTTP doesn’t instrument the body so you won’t be able to see token usage, prompts and completions. Or am I missing something?
30.
by nirga 2y ago
But if you have a high variance when calculating a specific score for the same text output - how can it even be useful? Let's say you get score 20 for text A and then score 40 for text B - you can't infer that text A is necessaril

More ›