6 ms·

It is as complicated as you want or need it to be. You can avoid any magic and stick to a subset that is easy to reason about and brings the most value in your

It is as complicated as you want or need it to be. You can avoid any magic and stick to a subset that is easy to reason about and brings the most value in your context. For our team, it is very simple: * we use a library send traces and traces only[0]. They bring the most value for observing applications and can contain all the data the other types can contain. Basically hash-maps vs strings and floats. * we use manual instrumentation as opposed to automatic - we are deliberate in what we observe and have great understand of what emits the spans. We have naming conventions that match our code organization. * we use two different backends - an affordable 3rd party service and an all-on-one Jaeger install (just run 1 executable or docker container) that doesn't save the spans on disk for local development. The second is mostly for piece of mind of team members that they are not going to flood the third party service. [0] We have a previous setup to monitor infrastructure and in our case we don't see a lot of value of ingesting all the infrastructure logs and metrics. I think it is early days for OTEL metrics and logs, but the vendors don't tell you this.

buzzdenver 2y ago
Mind sharing that that affordable 3rd party service is?
- dimitar 2y ago
  honeycomb
hinkley 2y ago
I did not find that manual instrumentation made things simpler. You’re trading a learning curve that now starts way before you can demonstrate results for a clearer understanding of the performance penalties of using this Rube Goldberg machine. Otel may be okay for a green field project but turning this thing on in a production service that already had telemetry felt like replacing a tire on a moving vehicle.
- dmoy 2y ago
  I've not used otel for anything not greenfield, but I just wanted to say > felt like replacing a tire on a moving vehicle. Some people do this as a joke / dare. I mean literally replacing a car tire on a moving vehicle. You Saudi drift up onto one side, and have people climb out of the side in the air, and then swap the tire while the car is driving on two wheels. It's pretty insane stuff: https://youtu.be/Str7m8xV7W8?si=KkjBh6OvFoD0HGoh https://youtu.be/Str7m8xV7W8?si=KkjBh6OvFoD0HGoh
  - hinkley 2y ago
    That was the image I had in my head. My whole career I’ve been watching people on greenfield projects looking down on devs on already successful products for not using some tool they’ve discovered, missing the fact that their tool only functions if you build your whole product around the exact mental model of the tool (green field). Wisdom is learning to watch for people obviously working on brownfield projects espousing a tool. Like moving from VMs to Docker. Ansible to Kubernetes (maybe not the best example). They can have a faster adoption cycle and more staying power.
  - PeterCorless 2y ago
    SaS Institute used that exact same analogy & even this video in their talk about implementing ScyllaDB back in 2020 (check out 0:35 in the video): https://www.scylladb.com/2020/05/28/sas-institute-changing-all-four-tires-while-driving-an-adtech-engine-at-full-speed/ https://www.scylladb.com/2020/05/28/sas-institute-changing-a... Seems like moving to OTel might even be a bit more complex for some brownfield folks.
madeofpalk 2y ago
It's as complicated as you want, but it's not as easy as I want. The floor is pretty high. I'm still looking for an endpoint just to send simple one-off metrics to from parts of infrastructure that's not scrapable.
- pat2man 2y ago
  You can just send metrics via JSON to any otlphttp collector: https://github.com/open-telemetry/opentelemetry-proto/blob/v1.5.0/examples/metrics.json https://github.com/open-telemetry/opentelemetry-proto/blob/v...
  - madeofpalk 2y ago
    Shame none of this comes up whenever I search for it! Top google result, for me, for 'send metrics to otel' is https://opentelemetry.io/docs/specs/otel/metrics/ https://opentelemetry.io/docs/specs/otel/metrics/. If I go through the the Language APIs & SDK more whole bunch of useless junk https://opentelemetry.io/docs/languages/js/ https://opentelemetry.io/docs/languages/js/ Compare to the InfluxDB "send data" getting started https://docs.influxdata.com/influxdb/cloud/api-guide/client-libraries/nodejs/write/?t=Node.js https://docs.influxdata.com/influxdb/cloud/api-guide/client-... which gives you exactly it in a few lines.
    - phillipcarter 2y ago
      Maybe the confusion here is in comparing different things. The InfluxData docs you're linking to are similar to Observability vendor docs, which do indeed amount to "here's the endpoint, plug it in here, add this API key, tada". But OpenTelemetry isn't an observability vendor. You can send to an OpenTelemetry Collector (and the act of sending is simple), but you also need to stand that thing up and run it yourself. There's a lot of good reasons to do that, but if you don't need to run infrastructure right now then it's a lot simpler to just send directly to a backend. Would it be more helpful if the docs on OTel spelled this out more clearly?
      - 2y ago
Groxx 2y ago
>You can avoid any magic and stick to a subset... ... if (and only if) all the libraries you use also stick to that subset, yea. That is overwhelmingly not true in my experience. And the article shows a nice concrete example of why. For green-field projects which use nothing but otel and no non-otel frameworks, yea. I can believe it's nice. But I definitely do not live in that world yet.
mikestorrent 2y ago
Very sane advice. Most folks will already have something for metrics and logs and unless there's ROI on changing it out, why bother?

buzzdenver 2y ago

Mind sharing that that affordable 3rd party service is?

dimitar 2y ago
honeycomb

hinkley 2y ago

I did not find that manual instrumentation made things simpler. You’re trading a learning curve that now starts way before you can demonstrate results for a clearer understanding of the performance penalties of using this Rube Goldberg machine. Otel may be okay for a green field project but turning this thing on in a production service that already had telemetry felt like replacing a tire on a moving vehicle.

dmoy 2y ago
I've not used otel for anything not greenfield, but I just wanted to say > felt like replacing a tire on a moving vehicle. Some people do this as a joke / dare. I mean literally replacing a car tire on a moving vehicle. You Saudi drift up onto one side, and have people climb out of the side in the air, and then swap the tire while the car is driving on two wheels. It's pretty insane stuff: https://youtu.be/Str7m8xV7W8?si=KkjBh6OvFoD0HGoh https://youtu.be/Str7m8xV7W8?si=KkjBh6OvFoD0HGoh
- hinkley 2y ago
  That was the image I had in my head. My whole career I’ve been watching people on greenfield projects looking down on devs on already successful products for not using some tool they’ve discovered, missing the fact that their tool only functions if you build your whole product around the exact mental model of the tool (green field). Wisdom is learning to watch for people obviously working on brownfield projects espousing a tool. Like moving from VMs to Docker. Ansible to Kubernetes (maybe not the best example). They can have a faster adoption cycle and more staying power.
- PeterCorless 2y ago
  SaS Institute used that exact same analogy & even this video in their talk about implementing ScyllaDB back in 2020 (check out 0:35 in the video): https://www.scylladb.com/2020/05/28/sas-institute-changing-all-four-tires-while-driving-an-adtech-engine-at-full-speed/ https://www.scylladb.com/2020/05/28/sas-institute-changing-a... Seems like moving to OTel might even be a bit more complex for some brownfield folks.

madeofpalk 2y ago

It's as complicated as you want, but it's not as easy as I want. The floor is pretty high. I'm still looking for an endpoint just to send simple one-off metrics to from parts of infrastructure that's not scrapable.

pat2man 2y ago
You can just send metrics via JSON to any otlphttp collector: https://github.com/open-telemetry/opentelemetry-proto/blob/v1.5.0/examples/metrics.json https://github.com/open-telemetry/opentelemetry-proto/blob/v...
- madeofpalk 2y ago
  Shame none of this comes up whenever I search for it! Top google result, for me, for 'send metrics to otel' is https://opentelemetry.io/docs/specs/otel/metrics/ https://opentelemetry.io/docs/specs/otel/metrics/. If I go through the the Language APIs & SDK more whole bunch of useless junk https://opentelemetry.io/docs/languages/js/ https://opentelemetry.io/docs/languages/js/ Compare to the InfluxDB "send data" getting started https://docs.influxdata.com/influxdb/cloud/api-guide/client-libraries/nodejs/write/?t=Node.js https://docs.influxdata.com/influxdb/cloud/api-guide/client-... which gives you exactly it in a few lines.
  - phillipcarter 2y ago
    Maybe the confusion here is in comparing different things. The InfluxData docs you're linking to are similar to Observability vendor docs, which do indeed amount to "here's the endpoint, plug it in here, add this API key, tada". But OpenTelemetry isn't an observability vendor. You can send to an OpenTelemetry Collector (and the act of sending is simple), but you also need to stand that thing up and run it yourself. There's a lot of good reasons to do that, but if you don't need to run infrastructure right now then it's a lot simpler to just send directly to a backend. Would it be more helpful if the docs on OTel spelled this out more clearly?
    - 2y ago

Groxx 2y ago

>You can avoid any magic and stick to a subset... ... if (and only if) all the libraries you use also stick to that subset, yea. That is overwhelmingly not true in my experience. And the article shows a nice concrete example of why. For green-field projects which use nothing but otel and no non-otel frameworks, yea. I can believe it's nice. But I definitely do not live in that world yet.

mikestorrent 2y ago

Very sane advice. Most folks will already have something for metrics and logs and unless there's ROI on changing it out, why bother?