17 ms·

Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads. Gateway collectors are u

by bilalq 26d ago

Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads. Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to. In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.

clintonb 26d ago
(I’m not the person you replied to, but have experience here.) I follow the [gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/ https://opentelemetry.io/docs/collector/deploy/gateway/). Everything sends telemetry to our gateway, which exports to ClickHouse (formerly Datadog). We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.
cyberax 26d ago
A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet. > In a runtime like Lambda, how would you avoid the need to run an edge collector? Here's my setup (in Go, very simplified): > // Instantiate a new slog logger > logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger)) > // Use the logger as needed My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.
- bilalq 26d ago
  In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?
  - Game_Ender 25d ago
    This is what I have done with CLI apps the directly send to the OTEL vendor. It works great.
  - ojkelly 25d ago
    Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block. [0] https://github.com/open-telemetry/opentelemetry-lambda https://github.com/open-telemetry/opentelemetry-lambda
    - bilalq 25d ago
      That lambda layer comes with an incredibly heavy performance penalty. It doesn't block, but it does consume compute/memory resources and takes forever to startup[0][1]. To be fair, Rotel is promising in this regard[2]. [0]: https://github.com/open-telemetry/opentelemetry-lambda/issues/263 https://github.com/open-telemetry/opentelemetry-lambda/issue... [1]: https://github.com/aws-observability/aws-otel-lambda/issues/228 https://github.com/aws-observability/aws-otel-lambda/issues/... [2]: https://github.com/rotel-dev/rotel https://github.com/rotel-dev/rotel
  - cyberax 25d ago
    I don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.
    - bilalq 25d ago
      If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you now. The X-Ray daemon and SDKs are all deprecated now in favor of OTel. Things like enchrichment of resource level traces for things like the DynamoDB client in v3 of the AWS JS SDK don't work with the X-Ray SDK. And they never will now. You're now recommended to use the AWS Distro for OpenTelemetry setup and OTel SDKs. The performance overhead of this is heavy, with big cold-start penalties. Compare this with how the Datadog layer does adaptive flushing and performs relatively much better. Rotel is also promising in this space. But right now, OTel feels immature and things are being deprecated without the replacement being fully baked.

by bilalq 26d ago

clintonb 26d ago

(I’m not the person you replied to, but have experience here.) I follow the [gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/ https://opentelemetry.io/docs/collector/deploy/gateway/). Everything sends telemetry to our gateway, which exports to ClickHouse (formerly Datadog). We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.

cyberax 26d ago

A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet. > In a runtime like Lambda, how would you avoid the need to run an edge collector? Here's my setup (in Go, very simplified): > // Instantiate a new slog logger > logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger)) > // Use the logger as needed My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.

bilalq 26d ago
In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?
- Game_Ender 25d ago
  This is what I have done with CLI apps the directly send to the OTEL vendor. It works great.
- ojkelly 25d ago
  Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block. [0] https://github.com/open-telemetry/opentelemetry-lambda https://github.com/open-telemetry/opentelemetry-lambda
  - bilalq 25d ago
    That lambda layer comes with an incredibly heavy performance penalty. It doesn't block, but it does consume compute/memory resources and takes forever to startup[0][1]. To be fair, Rotel is promising in this regard[2]. [0]: https://github.com/open-telemetry/opentelemetry-lambda/issues/263 https://github.com/open-telemetry/opentelemetry-lambda/issue... [1]: https://github.com/aws-observability/aws-otel-lambda/issues/228 https://github.com/aws-observability/aws-otel-lambda/issues/... [2]: https://github.com/rotel-dev/rotel https://github.com/rotel-dev/rotel
- cyberax 25d ago
  I don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.
  - bilalq 25d ago
    If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you now. The X-Ray daemon and SDKs are all deprecated now in favor of OTel. Things like enchrichment of resource level traces for things like the DynamoDB client in v3 of the AWS JS SDK don't work with the X-Ray SDK. And they never will now. You're now recommended to use the AWS Distro for OpenTelemetry setup and OTel SDKs. The performance overhead of this is heavy, with big cold-start penalties. Compare this with how the Datadog layer does adaptive flushing and performs relatively much better. Rotel is also promising in this space. But right now, OTel feels immature and things are being deprecated without the replacement being fully baked.