6 ms·

To me traces (or maybe more specifically spans) are essentially a structured log with a unique ID and a reference to a parent ID. Very open to have someone exp

To me traces (or maybe more specifically spans) are essentially a structured log with a unique ID and a reference to a parent ID. Very open to have someone explain why I'm wrong or why they should be handled separately.

kiitos 1y ago
Traces have a very specific data model, and corresponding limitations, which don't really accommodate log events/messages of arbitrary size. The access model for traces is also fundamentally different vs. that of logs.
- phillipcarter 1y ago
  There are practical limitations mostly with backend analysis tools. OTel does not define a limit on how large a span is. It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.
  - kiitos 1y ago
    > There are practical limitations mostly with backend analysis tools Not just end-of-line analysis tools, but also initiating SDKs, and system agents, and intermediate middle-boxes -- really anything that needs to parse OTel. Spec > SDK > Trace > Span limits: https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-limits https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-lim... Spec > Common > Attribute limits: https://opentelemetry.io/docs/specs/otel/common/#attribute-limits https://opentelemetry.io/docs/specs/otel/common/#attribute-l... I know the spec says the default AttributeValueLengthLimit = infinity, but... > It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example. ...I'd love to learn about any OTel-compatible pipeline/system that supports attribute values of arbitrary size! because I've personally not seen anything that lets you get bigger than O(1MB).
    - phillipcarter 1y ago
      Well yeah, there are practical limits imposed by the fact that these have to run on real systems. But in practice, you find that you're limited by your backend observability system because it was designed for a world of many events with narrow data, not fewer events with wider data (so-called "wide events"). OTel and the standard toolkit you get with it doesn't prevent you from doing wide events.
      - kiitos 1y ago
        "Wide events" describe a structure/schema for incoming data on the "write path" to a system. That's fine. But that data always needs to be transformed, specialized, for use-case specific "read paths" offered by that same system, in order to be efficient. You can "do wide events" on ingest but you always need to transform them to specific (narrow? idk) events/metrics/summarizations/etc. for the read paths, that's the whole challenge of the space.
        phillipcarter 1y ago
        You…don’t? This is why tools like ClickHouse and Honeycomb are starting to grow, you just aggregate what you need at query time, and the cost to query is not usually too expensive. The tradeoff is each event has a higher per-unit cost, but this is often the more favorable tradeoff.
        kiitos 1y ago
        > you just aggregate what you need at query time, and the cost to query is not usually too expensive The entire challenge of observability systems is rooted in the fact that the volume of input data (wide events) on the write path, is astronomically larger than what can ever be directly evaluated by any user-facing system on the read path. Data transformation and specialization and etc. is the whole ball-game. If you can build something directly on top of raw wide-events, and it works for you, that's cool, but it means that you're operating at trivial and non-representative scale.

kiitos 1y ago

Traces have a very specific data model, and corresponding limitations, which don't really accommodate log events/messages of arbitrary size. The access model for traces is also fundamentally different vs. that of logs.

phillipcarter 1y ago
There are practical limitations mostly with backend analysis tools. OTel does not define a limit on how large a span is. It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.
- kiitos 1y ago
  > There are practical limitations mostly with backend analysis tools Not just end-of-line analysis tools, but also initiating SDKs, and system agents, and intermediate middle-boxes -- really anything that needs to parse OTel. Spec > SDK > Trace > Span limits: https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-limits https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-lim... Spec > Common > Attribute limits: https://opentelemetry.io/docs/specs/otel/common/#attribute-limits https://opentelemetry.io/docs/specs/otel/common/#attribute-l... I know the spec says the default AttributeValueLengthLimit = infinity, but... > It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example. ...I'd love to learn about any OTel-compatible pipeline/system that supports attribute values of arbitrary size! because I've personally not seen anything that lets you get bigger than O(1MB).
  - phillipcarter 1y ago
    Well yeah, there are practical limits imposed by the fact that these have to run on real systems. But in practice, you find that you're limited by your backend observability system because it was designed for a world of many events with narrow data, not fewer events with wider data (so-called "wide events"). OTel and the standard toolkit you get with it doesn't prevent you from doing wide events.
    - kiitos 1y ago
      "Wide events" describe a structure/schema for incoming data on the "write path" to a system. That's fine. But that data always needs to be transformed, specialized, for use-case specific "read paths" offered by that same system, in order to be efficient. You can "do wide events" on ingest but you always need to transform them to specific (narrow? idk) events/metrics/summarizations/etc. for the read paths, that's the whole challenge of the space.
      - phillipcarter 1y ago
        You…don’t? This is why tools like ClickHouse and Honeycomb are starting to grow, you just aggregate what you need at query time, and the cost to query is not usually too expensive. The tradeoff is each event has a higher per-unit cost, but this is often the more favorable tradeoff.
        kiitos 1y ago
        > you just aggregate what you need at query time, and the cost to query is not usually too expensive The entire challenge of observability systems is rooted in the fact that the volume of input data (wide events) on the write path, is astronomically larger than what can ever be directly evaluated by any user-facing system on the read path. Data transformation and specialization and etc. is the whole ball-game. If you can build something directly on top of raw wide-events, and it works for you, that's cool, but it means that you're operating at trivial and non-representative scale.