6 ms·

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today: 1. Every major vendor is sti

by bilalq 26d ago

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today: 1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time. 2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now. 3. Serverless runtimes pay a heavy penalty for cold starts with OTel. 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage. 5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was. 6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?

cyberax 26d ago
> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage. You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent. App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.
- bilalq 26d ago
  Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads. Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to. In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.
  - clintonb 25d ago
    (I’m not the person you replied to, but have experience here.) I follow the [gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/ https://opentelemetry.io/docs/collector/deploy/gateway/). Everything sends telemetry to our gateway, which exports to ClickHouse (formerly Datadog). We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.
  - cyberax 25d ago
    A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet. > In a runtime like Lambda, how would you avoid the need to run an edge collector? Here's my setup (in Go, very simplified): > // Instantiate a new slog logger > logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger)) > // Use the logger as needed My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.
    - bilalq 25d ago
      In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?
      - Game_Ender 25d ago
        This is what I have done with CLI apps the directly send to the OTEL vendor. It works great.
      - ojkelly 25d ago
        Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block. [0] https://github.com/open-telemetry/opentelemetry-lambda https://github.com/open-telemetry/opentelemetry-lambda
arcanemachiner 26d ago
So what's the alternative then? (Genuine question, not hypothetical snark.)
- bilalq 26d ago
  There isn't really a great alternative without vendor lock-in. If you go all-in on AWS Cloudwatch/X-Ray, it's a really easy setup with low effort. If you go all-in on Datadog, it's pretty easy. But if you want to mix Sentry, Langfuse, Datadog, etc, OTel is still probably the best option. It's just a letdown that this is the best there is. I don't mean to disparage anyone working on OTel. I can appreciate that it has ambitious goals and it's not an easy problem to get alignment and interop here. Especially with all the stakeholders involved. But as a user, it feels simultaeneously over-engineered and under-engineered.
- nunez 25d ago
  - Paying Datadog $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$, or - Using and configuring a suite of tools (Jaeger for tracing, Vector or Fluentd for logs, Prometeheus for metrics)

by bilalq 26d ago

cyberax 26d ago

> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage. You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent. App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.

bilalq 26d ago
Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads. Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to. In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.
- clintonb 25d ago
  (I’m not the person you replied to, but have experience here.) I follow the [gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/ https://opentelemetry.io/docs/collector/deploy/gateway/). Everything sends telemetry to our gateway, which exports to ClickHouse (formerly Datadog). We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.
- cyberax 25d ago
  A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet. > In a runtime like Lambda, how would you avoid the need to run an edge collector? Here's my setup (in Go, very simplified): > // Instantiate a new slog logger > logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger)) > // Use the logger as needed My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.
  - bilalq 25d ago
    In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?
    - Game_Ender 25d ago
      This is what I have done with CLI apps the directly send to the OTEL vendor. It works great.
    - ojkelly 25d ago
      Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block. [0] https://github.com/open-telemetry/opentelemetry-lambda https://github.com/open-telemetry/opentelemetry-lambda

arcanemachiner 26d ago

So what's the alternative then? (Genuine question, not hypothetical snark.)

bilalq 26d ago
There isn't really a great alternative without vendor lock-in. If you go all-in on AWS Cloudwatch/X-Ray, it's a really easy setup with low effort. If you go all-in on Datadog, it's pretty easy. But if you want to mix Sentry, Langfuse, Datadog, etc, OTel is still probably the best option. It's just a letdown that this is the best there is. I don't mean to disparage anyone working on OTel. I can appreciate that it has ambitious goals and it's not an easy problem to get alignment and interop here. Especially with all the stakeholders involved. But as a user, it feels simultaeneously over-engineered and under-engineered.
nunez 25d ago
- Paying Datadog $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$, or - Using and configuring a suite of tools (Jaeger for tracing, Vector or Fluentd for logs, Prometeheus for metrics)