6 ms·

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to select the instrumentations via configuration. Saves having to create your own tracer.ts/js file. Here's the PR that got merged earlier in the year: https://github.com/open-telemetry/opentelemetry-js-contrib/pull/1953 https://github.com/open-telemetry/opentelemetry-js-contrib/p... The env var config is `OTEL_NODE_ENABLED_INSTRUMENTATIONS` Anyways, love Opentelemetry success stories. Been working hard on it at my current company and yielding fruits already :)

roboben 2y ago
This is great. Last time I tried this I couldn’t even find a way in code to disable some.
tnolet 2y ago
That is awesome. Had no idea this was available as an env var. After diving into OTel for our backend, we also found some of this stuff is just too noisy. We switch it of using this code snippet, for anyone bumping into this thread: instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false, }, '@opentelemetry/instrumentation-net': { enabled: false, }, '@opentelemetry/instrumentation-dns': { enabled: false, },
- harisamin 2y ago
  yeah I totally turned all of those off...way too noisy :)
serverlessmom 2y ago
This is so cool! I’ve had this exact problem before.
Veserv 2y ago
Why would you disable instrumentation instead of just filtering the recorded log? That only makes sense if the instrumentation overhead itself is significant. But, for a efficient recording implementation that should only really start being a problem when your average span is ~1 us.
- tnolet 2y ago
  Oh, simple answer. The tools you use to inspect those traces just blow up with noise. Like a trace that shows 600+ of file reads that all take less than half a millisecond. This is all noise when you are trying to debug more common issues than your FS being too slow. + also storage cost. Most vendors charge by Mb stored or span recorded.
  - Veserv 2y ago
    That is why I mentioned post-filtering the recording as the alternative. Grab the full recording then filter to just the relevant results before inspection. For that matter, why are a few hundred spans a problem? Are the visualizers that poor? I usually use function tracing where hundreds of millions to billions of spans per second are the norm and there is no difficulty managing or understanding those.
    - tnolet 2y ago
      In most cases these traces are shipped over the wire to a vendor. Only that will cost $$. Then, not all vendors have tail sampling as a "free" feature. So, in many cases it's better to not record at all.
      - Veserv 2y ago
        That sounds positively dystopian. Is it really that hard to dump to private/non-vendor storage for local analysis using your own tools? I do not do cloud or web development, so this is just totally alien. I generate multi-gigabyte logs with billions of events for just seconds of execution and get to slice them however I want when doing performance analysis. The inability to even process your own logs seems crazy.
        cweld510 2y ago
        You can absolutely dump the traces somewhere and analyze them yourself. The problem is that this falls apart with scale. You are maybe serving thousands of requests per second. Your service has a ton of instances. Capturing all trace data for all requests from all services is just difficult. Where do you store all of it? How do you quickly find what you need? It gets very annoying very fast. When you pay a vendor, you pay them to deal with this.

roboben 2y ago

This is great. Last time I tried this I couldn’t even find a way in code to disable some.

tnolet 2y ago

That is awesome. Had no idea this was available as an env var. After diving into OTel for our backend, we also found some of this stuff is just too noisy. We switch it of using this code snippet, for anyone bumping into this thread: instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false, }, '@opentelemetry/instrumentation-net': { enabled: false, }, '@opentelemetry/instrumentation-dns': { enabled: false, },

harisamin 2y ago
yeah I totally turned all of those off...way too noisy :)

serverlessmom 2y ago

This is so cool! I’ve had this exact problem before.

Veserv 2y ago

Why would you disable instrumentation instead of just filtering the recorded log? That only makes sense if the instrumentation overhead itself is significant. But, for a efficient recording implementation that should only really start being a problem when your average span is ~1 us.

tnolet 2y ago
Oh, simple answer. The tools you use to inspect those traces just blow up with noise. Like a trace that shows 600+ of file reads that all take less than half a millisecond. This is all noise when you are trying to debug more common issues than your FS being too slow. + also storage cost. Most vendors charge by Mb stored or span recorded.
- Veserv 2y ago
  That is why I mentioned post-filtering the recording as the alternative. Grab the full recording then filter to just the relevant results before inspection. For that matter, why are a few hundred spans a problem? Are the visualizers that poor? I usually use function tracing where hundreds of millions to billions of spans per second are the norm and there is no difficulty managing or understanding those.
  - tnolet 2y ago
    In most cases these traces are shipped over the wire to a vendor. Only that will cost $$. Then, not all vendors have tail sampling as a "free" feature. So, in many cases it's better to not record at all.
    - Veserv 2y ago
      That sounds positively dystopian. Is it really that hard to dump to private/non-vendor storage for local analysis using your own tools? I do not do cloud or web development, so this is just totally alien. I generate multi-gigabyte logs with billions of events for just seconds of execution and get to slice them however I want when doing performance analysis. The inability to even process your own logs seems crazy.
      - cweld510 2y ago
        You can absolutely dump the traces somewhere and analyze them yourself. The problem is that this falls apart with scale. You are maybe serving thousands of requests per second. Your service has a ton of instances. Capturing all trace data for all requests from all services is just difficult. Where do you store all of it? How do you quickly find what you need? It gets very annoying very fast. When you pay a vendor, you pay them to deal with this.