8 ms·

OpenTelemetry is a great concept, but in my experience not quite there yet. Docs especially fall into the common trap of handling the happy path hello world qui

by shipit1999 3y ago

OpenTelemetry is a great concept, but in my experience not quite there yet. Docs especially fall into the common trap of handling the happy path hello world quickstarts, then become increasingly useless as you want to get beyond that to real life use cases. Given the inherent tradeoff of complexity that comes from trying to unify different approaches around one standard, sometimes it seems like things that should be simple are more difficult than they should be. I'm sure it will keep improving.

mason55 3y ago
> Docs especially fall into the common trap of handling the happy path hello world quickstarts, then become increasingly useless as you want to get beyond that to real life use cases. Yeah, Java is what I'm most familiar with. The "Getting Started" shows how to do some basic manual instrumentation and collect the output with curl. Then the "Next Steps" are just random things with no guidance about why I would or wouldn't choose any of them for my next step. But, ok, I choose "Automatic Instrumentation", that sounds promising. And it actually is really easy to set up auto instrumentation. But then at the end it says > After you have automatic instrumentation configured for your app or service, you might want to annotate selected methods or add manual instrumentation to collect custom telemetry data. Uh... no... after I have automatic instrumentation enabled I want to do something with the output The two major flaws in the docs seem to be 1. The common failure of docs to explain to users why they might choose one thing or another. "If you want to do x.. If you want to do y.." what if I don't know? 2. Because otel is agnostic to the consumer of the output, there's very little in the way of explaining how to get value out of what otel produces. To connect the dots, you really need to use the docs of your observability tool. Which I understand, but then most of them have their own setup directions because they want some extra fields included in the data, or they have their own fork, so not everything in the otel docs is actually usable. I'm not sure what the answer is. It's not like I expect otel to document how to build a dashboard in Grafana. And a lot of frustration I've experienced has been with the observability tools themselves. But at the same time, I always feel like the otel docs just don't get you anywhere close to getting value out of the library. Which is a shame, because turning on auto-instrumentation and seeing all your traces with literally no extra work is a magical moment.
- Volundr 3y ago
  Hmmm... Yeah I setup open telemetry for a couple personal projects this year was pleased with the ease of setup, but by and large I knew what I was doing specifically I had my application, and I had Grafana and I wanted to get traces from A to B. Relooking at the docs from the eyes of a newcomers if you don't already have a destination in mind they don't really help you. It's a little tricky because my setup with Grafana will be somewhat different (but similar) from someone using honeycomb or signoz or what have you, but even just having a "want to visualize your data? Check out the list of compatible vendors", with a link that direction would probably go a long way.
  - structural 3y ago
    By comparison, I wanted to use opentelemetry for a series of projects, but could find absolutely no useful documentation on how to do anything else other than "send data from a webapp to a server / other cloud service that some vendor wants to sell you". All I wanted to do was instrument an application and write its telemetry data to a file in a standard way, and have some story regarding combining metrics, traces, and logs as necessary. Ideally this would use minimal system resources when idle. That's it.
    - bbkane 3y ago
      It doesn't read from files unfortunately, but https://openobserve.ai/ https://openobserve.ai/ is very easy to set up locally (single binary) and send otel logs/metrics/traces to. Here's how I run it locally for my little shovel project - https://github.com/bbkane/shovel#run-the-webapp-locally-with-openobserve https://github.com/bbkane/shovel#run-the-webapp-locally-with... . Also linked from that README is an Ansible playbook to start OpenObserve as a systems service on a Linux VM. Alternatively, see the shovel codebase I linked above for a "stdout" TracerProvider. You could do something like that to save to a file, and then use a tool to prettify the JSON. I have a small script to format json logs at https://github.com/bbkane/dotfiles/blob/2df9af5a9bbb40f2e1012dd5054fd4e9cdee0bc9/bin_common/bin_common/format_jsonl.py https://github.com/bbkane/dotfiles/blob/2df9af5a9bbb40f2e101...
hinkley 3y ago
Time and again I ran into two or three examples in different docs, and search engines sending me to the nonfunctional or ambiguous ones, complaining about it and having someone send me to a whole other doc I’ve never seen that is 3 clicks away from the overview doc while the broken ones are 0-2 clicks away. It’s all moving too fast and yet not fast enough.

mason55 3y ago

> Docs especially fall into the common trap of handling the happy path hello world quickstarts, then become increasingly useless as you want to get beyond that to real life use cases. Yeah, Java is what I'm most familiar with. The "Getting Started" shows how to do some basic manual instrumentation and collect the output with curl. Then the "Next Steps" are just random things with no guidance about why I would or wouldn't choose any of them for my next step. But, ok, I choose "Automatic Instrumentation", that sounds promising. And it actually is really easy to set up auto instrumentation. But then at the end it says > After you have automatic instrumentation configured for your app or service, you might want to annotate selected methods or add manual instrumentation to collect custom telemetry data. Uh... no... after I have automatic instrumentation enabled I want to do something with the output The two major flaws in the docs seem to be 1. The common failure of docs to explain to users why they might choose one thing or another. "If you want to do x.. If you want to do y.." what if I don't know? 2. Because otel is agnostic to the consumer of the output, there's very little in the way of explaining how to get value out of what otel produces. To connect the dots, you really need to use the docs of your observability tool. Which I understand, but then most of them have their own setup directions because they want some extra fields included in the data, or they have their own fork, so not everything in the otel docs is actually usable. I'm not sure what the answer is. It's not like I expect otel to document how to build a dashboard in Grafana. And a lot of frustration I've experienced has been with the observability tools themselves. But at the same time, I always feel like the otel docs just don't get you anywhere close to getting value out of the library. Which is a shame, because turning on auto-instrumentation and seeing all your traces with literally no extra work is a magical moment.

Volundr 3y ago
Hmmm... Yeah I setup open telemetry for a couple personal projects this year was pleased with the ease of setup, but by and large I knew what I was doing specifically I had my application, and I had Grafana and I wanted to get traces from A to B. Relooking at the docs from the eyes of a newcomers if you don't already have a destination in mind they don't really help you. It's a little tricky because my setup with Grafana will be somewhat different (but similar) from someone using honeycomb or signoz or what have you, but even just having a "want to visualize your data? Check out the list of compatible vendors", with a link that direction would probably go a long way.
- structural 3y ago
  By comparison, I wanted to use opentelemetry for a series of projects, but could find absolutely no useful documentation on how to do anything else other than "send data from a webapp to a server / other cloud service that some vendor wants to sell you". All I wanted to do was instrument an application and write its telemetry data to a file in a standard way, and have some story regarding combining metrics, traces, and logs as necessary. Ideally this would use minimal system resources when idle. That's it.
  - bbkane 3y ago
    It doesn't read from files unfortunately, but https://openobserve.ai/ https://openobserve.ai/ is very easy to set up locally (single binary) and send otel logs/metrics/traces to. Here's how I run it locally for my little shovel project - https://github.com/bbkane/shovel#run-the-webapp-locally-with-openobserve https://github.com/bbkane/shovel#run-the-webapp-locally-with... . Also linked from that README is an Ansible playbook to start OpenObserve as a systems service on a Linux VM. Alternatively, see the shovel codebase I linked above for a "stdout" TracerProvider. You could do something like that to save to a file, and then use a tool to prettify the JSON. I have a small script to format json logs at https://github.com/bbkane/dotfiles/blob/2df9af5a9bbb40f2e1012dd5054fd4e9cdee0bc9/bin_common/bin_common/format_jsonl.py https://github.com/bbkane/dotfiles/blob/2df9af5a9bbb40f2e101...

hinkley 3y ago

Time and again I ran into two or three examples in different docs, and search engines sending me to the nonfunctional or ambiguous ones, complaining about it and having someone send me to a whole other doc I’ve never seen that is 3 clicks away from the overview doc while the broken ones are 0-2 clicks away. It’s all moving too fast and yet not fast enough.