8 ms·

The problem is not the instrumentation but the way everyone of them work. A metric is a point in time. A metric is very small but you have a lot of them. A lo

The problem is not the instrumentation but the way everyone of them work. A metric is a point in time. A metric is very small but you have a lot of them. A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc. A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation. How do you decide when to suddenly do the trace and send it? IF you always do the trace, you have to pay for the overhead of that tracing constantly.

spockz 26d ago
Technically, you can use the same places in the code where you stop/start/fork traces to also be the places where you increment the counters/gauges, etc. Which I think the GP was alluding to when describing the micrometer solution. Similarly, you can derive metrics for log lines without having to emit the actual log lines. Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.
TylerE 26d ago
At that point you almost might as well just log everything. The decision logic is likely about as complex as just doing it. Then I suppose you have a watchdog task that fires off every, say, 15 minutes or an hour or something, looks at the collected data, and either decides to keep it or trash it while recording a tiny "nothing interesting" datapoint.
- Flamkuchlo 26d ago
  Loghandling is quite resource intensive. All the log ingestion systems i have seen were bigger elastic search clusters.
twic 26d ago
Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging. All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0". The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.
- Flamkuchlo 26d ago
  A metric is not event based. You don't have a metric 'person logged in' because you would need to scrape the metric at the moment a person logged in. You have a metric called 'overall people have logged in so far' and you do math on it. The 'person logged in' is an event you log out.
- PunchyHamster 26d ago
  > Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)
  - krab 26d ago
    The app's memory usage is an aggregation of the alloc/free events. I think the original point was that all of the metrics, traces and logs are conceptually the same but for efficiency, we store less data in each place, not the full history. Personally, for the systems I work on, having an easy way to turn logs into metrics and vice versa, without deciding up front, would be a slight benefit.
  - growse 23d ago
    A clock ticking every second is generating an event every second. If you sample the CPU usage at 1Hz, the metric is attached to the tick event.
- jandrewrogers 26d ago
  This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction. Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance. In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure. "Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems. Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.
jaen 26d ago
What? All of this has been solved for a long time. How do you think hyperscalers do this? Search keyword: "Adaptive sampling"
- Flamkuchlo 26d ago
  Adaptive sampling is not tracing, its sampling. Tracing traces a particular event. I'm quite aware of the difference between sampling, tracing and profiling.
  - jaen 18d ago
    No... Adaptive sampling is a family of statistical methods to choose an appropriate decimation strategy for arbitrary events based on real-world occurrence distributions. It can be applied to tracing, metrics, logging ("sampling") and profiling.

spockz 26d ago

Technically, you can use the same places in the code where you stop/start/fork traces to also be the places where you increment the counters/gauges, etc. Which I think the GP was alluding to when describing the micrometer solution. Similarly, you can derive metrics for log lines without having to emit the actual log lines. Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.

TylerE 26d ago

At that point you almost might as well just log everything. The decision logic is likely about as complex as just doing it. Then I suppose you have a watchdog task that fires off every, say, 15 minutes or an hour or something, looks at the collected data, and either decides to keep it or trash it while recording a tiny "nothing interesting" datapoint.

Flamkuchlo 26d ago
Loghandling is quite resource intensive. All the log ingestion systems i have seen were bigger elastic search clusters.

twic 26d ago

Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging. All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0". The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.

Flamkuchlo 26d ago
A metric is not event based. You don't have a metric 'person logged in' because you would need to scrape the metric at the moment a person logged in. You have a metric called 'overall people have logged in so far' and you do math on it. The 'person logged in' is an event you log out.
PunchyHamster 26d ago
> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)
- krab 26d ago
  The app's memory usage is an aggregation of the alloc/free events. I think the original point was that all of the metrics, traces and logs are conceptually the same but for efficiency, we store less data in each place, not the full history. Personally, for the systems I work on, having an easy way to turn logs into metrics and vice versa, without deciding up front, would be a slight benefit.
- growse 23d ago
  A clock ticking every second is generating an event every second. If you sample the CPU usage at 1Hz, the metric is attached to the tick event.
jandrewrogers 26d ago
This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction. Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance. In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure. "Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems. Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.

jaen 26d ago

What? All of this has been solved for a long time. How do you think hyperscalers do this? Search keyword: "Adaptive sampling"

Flamkuchlo 26d ago
Adaptive sampling is not tracing, its sampling. Tracing traces a particular event. I'm quite aware of the difference between sampling, tracing and profiling.
- jaen 18d ago
  No... Adaptive sampling is a family of statistical methods to choose an appropriate decimation strategy for arbitrary events based on real-world occurrence distributions. It can be applied to tracing, metrics, logging ("sampling") and profiling.