6 ms·

I agree. I tried to get it to work recently with datadog, but there was so many hiccups. I ended up having to use datadogs solution mostly. The documentation ac

by cglan 2y ago

I agree. I tried to get it to work recently with datadog, but there was so many hiccups. I ended up having to use datadogs solution mostly. The documentation across everything is also kind of confusing

SomaticPirate 2y ago
imo Datadog is pretty hostile to OTel too. Ever since https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836 https://github.com/open-telemetry/opentelemetry-collector-co... was nearly killed by them I never felt like they fully supported the standard (perhaps for good reasons) OTel is a bear though. I think the biggest advantage it gives you is the ability to move across tracing providers
- bebop 2y ago
  I worry that vision is not going to become reality if the large observability vendors don't want to support the standard.
  - phillipcarter 2y ago
    FWIW the "datadog doesn't like otel" thing is kind of old hat, and the story was a little more complicated at the time too. Nowadays they're contributing more to the project directly and have built some support to embed the collector into their DD agent. Other vendors (splunk, dynatrace, new relic, grafana, honeycomb, sumo logic, etc.) contribute to the project a bunch and typically recommend using OTel to start instead of some custom stuff from before.
    - arccy 2y ago
      They support ingesting via otel (ie competing with other vendors for their customers) but won't support ingesting via their SDKs (they still try very hard to lock you in to their tooling).
- rikthevik 2y ago
  > the ability to move across tracing providers It's a nice dream. At Google Cloud Next last year, the vendors kinda of came in two buckets. Datadog, and everyone trying to replace Datadog's outrageous bills.
- hangonhn 2y ago
  Yeah their agent will accept traces from the standard Otel SDK but there is no way to change their SDK to send the traces to anyone other than Datadog when I last checked a couple(?) of years ago. I mean I understand why they did that but it really removes one of the most compelling parts about Otel. We ended doing the hard work of using the standard Otel libraries. I had to contribute a PR or two to get it all to work with our services but am glad that's the route we went because now we can switch vendors if needed (which is likely in the not too distant future in our case.
- jensensbutton 2y ago
  Pretty sure Datadog is literally one of the top contributors to OTel.
ljm 2y ago
The biggest barrier to setting up oTel for me is the development experience. Having a single open specification is fantastic, especially for portability, but the SDKs are almost overwhelmingly abstract and therefore difficult to intuit. I used to really like Datadog for being a one-stop observability shop and even though the experience of integrating with it is still quite simple, I think product and pricing wise they've jumped the shark. I'm much happier these days using a collection of small time services and self-hosting other things, and the only part of that which isn't joyful is the boilerplate and not really understanding when and why you should, say, use gRPC over HTTP, and stuff like that.
- mdaniel 2y ago
  And having to rebuild a golang binary based on this horseshit just to get a bugfixed collector is some horseshit: https://github.com/open-telemetry/opentelemetry-collector/tree/cmd/builder/v0.117.0/cmd/builder#tldr https://github.com/open-telemetry/opentelemetry-collector/tr... which is required (as best I can tell) because they text/template in the deps https://github.com/open-telemetry/opentelemetry-collector/blob/cmd/builder/v0.117.0/cmd/builder/internal/builder/templates/main.go.tmpl#L11-L16 https://github.com/open-telemetry/opentelemetry-collector/bl... Heaven help you if it's a contrib collector bugfix
- pdimitar 2y ago
  You are generally correct but I've used https://github.com/openobserve/openobserve https://github.com/openobserve/openobserve for several projects for dev-only complete OTel stack (dashboards included) and I liked it. There are better dashboards out there for sure, but for what I needed locally it did the job fantastically well. Zero complaints. It's extremely easy to self-host, either on a dev machine, a VPS, or in any Docker-based PaaS.
pranay01 2y ago
part of the reason for that experience is also because DataDog is not open telemetry native and all their docs and instructions encourage use of their own agents. Using DataDog with Otel is like trying to hold your nose round over your head You should try Otel native observability platforms like SigNoz, Honeycomb, etc. your life will be much simpler Disclaimer : i am one of the maintainers at SigNoz

by cglan 2y ago

SomaticPirate 2y ago

imo Datadog is pretty hostile to OTel too. Ever since https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5836 https://github.com/open-telemetry/opentelemetry-collector-co... was nearly killed by them I never felt like they fully supported the standard (perhaps for good reasons) OTel is a bear though. I think the biggest advantage it gives you is the ability to move across tracing providers

bebop 2y ago
I worry that vision is not going to become reality if the large observability vendors don't want to support the standard.
- phillipcarter 2y ago
  FWIW the "datadog doesn't like otel" thing is kind of old hat, and the story was a little more complicated at the time too. Nowadays they're contributing more to the project directly and have built some support to embed the collector into their DD agent. Other vendors (splunk, dynatrace, new relic, grafana, honeycomb, sumo logic, etc.) contribute to the project a bunch and typically recommend using OTel to start instead of some custom stuff from before.
  - arccy 2y ago
    They support ingesting via otel (ie competing with other vendors for their customers) but won't support ingesting via their SDKs (they still try very hard to lock you in to their tooling).
rikthevik 2y ago
> the ability to move across tracing providers It's a nice dream. At Google Cloud Next last year, the vendors kinda of came in two buckets. Datadog, and everyone trying to replace Datadog's outrageous bills.
hangonhn 2y ago
Yeah their agent will accept traces from the standard Otel SDK but there is no way to change their SDK to send the traces to anyone other than Datadog when I last checked a couple(?) of years ago. I mean I understand why they did that but it really removes one of the most compelling parts about Otel. We ended doing the hard work of using the standard Otel libraries. I had to contribute a PR or two to get it all to work with our services but am glad that's the route we went because now we can switch vendors if needed (which is likely in the not too distant future in our case.
jensensbutton 2y ago
Pretty sure Datadog is literally one of the top contributors to OTel.

ljm 2y ago

The biggest barrier to setting up oTel for me is the development experience. Having a single open specification is fantastic, especially for portability, but the SDKs are almost overwhelmingly abstract and therefore difficult to intuit. I used to really like Datadog for being a one-stop observability shop and even though the experience of integrating with it is still quite simple, I think product and pricing wise they've jumped the shark. I'm much happier these days using a collection of small time services and self-hosting other things, and the only part of that which isn't joyful is the boilerplate and not really understanding when and why you should, say, use gRPC over HTTP, and stuff like that.

mdaniel 2y ago
And having to rebuild a golang binary based on this horseshit just to get a bugfixed collector is some horseshit: https://github.com/open-telemetry/opentelemetry-collector/tree/cmd/builder/v0.117.0/cmd/builder#tldr https://github.com/open-telemetry/opentelemetry-collector/tr... which is required (as best I can tell) because they text/template in the deps https://github.com/open-telemetry/opentelemetry-collector/blob/cmd/builder/v0.117.0/cmd/builder/internal/builder/templates/main.go.tmpl#L11-L16 https://github.com/open-telemetry/opentelemetry-collector/bl... Heaven help you if it's a contrib collector bugfix
pdimitar 2y ago
You are generally correct but I've used https://github.com/openobserve/openobserve https://github.com/openobserve/openobserve for several projects for dev-only complete OTel stack (dashboards included) and I liked it. There are better dashboards out there for sure, but for what I needed locally it did the job fantastically well. Zero complaints. It's extremely easy to self-host, either on a dev machine, a VPS, or in any Docker-based PaaS.

pranay01 2y ago

part of the reason for that experience is also because DataDog is not open telemetry native and all their docs and instructions encourage use of their own agents. Using DataDog with Otel is like trying to hold your nose round over your head You should try Otel native observability platforms like SigNoz, Honeycomb, etc. your life will be much simpler Disclaimer : i am one of the maintainers at SigNoz