5 ms·
> just put it there, it might be useful later > Also note that we have never mentioned anything about cardinality. Because it doesn’t matter - any field can b
by rekwah 3y ago
> just put it there, it might be useful later
> Also note that we have never mentioned anything about cardinality. Because it doesn’t matter - any field can be of any cardinality. Scuba works with raw events and doesn’t pre-aggregate anything, and so cardinality is not an issue.
This is how we end up with very large, very expensive data swamps.
- _visgean 3y agothat depends on the sampling rate no? I would much rather have a rich log record sampled at 1% than more records that dont contain enough info to debug..
- growse 3y agoThe people feeling the pain of (and paying for) the expensive data swamp are often not the same people who are yolo'ing the sample rate to 100% in their apps, because why wouldn't you want to store every event? Put another way, you're in charge of a large telemetry event sink. How do you incentivise the correct sampling behaviour by your users?
- Spivak 3y agoDon't let the user pick the sampling rate. In Honeycomb land this is called the EMA Dynamic Sampler. https://docs.honeycomb.io/manage-data-volume/refinery/sampling-methods/#deterministic-sampler https://docs.honeycomb.io/manage-data-volume/refinery/sampli...
- kiitos 3y agoYou should never need to sample telemetry data.
- gtirloni 3y agoMetrics sample rate yes but logging sample? When an end-to-end transaction for a very important task breaks, do I get *some* breadcrumbs to debug it?
- isburmistrov 3y agoSampling can be smart, e.g. based on some field all events have (can be called traceId, haha).
- goosejuice 3y agoI agree. Sampling logs.. sounds dangerous. Obviously every system is different. At least in GCP you can apply a filter to prevent ingestion and set different expiries on log budgets. This can help control costs without missing important entries.
- _visgean 3y agoI have used that approach before with sentry. It was a non-issue. It depends on nature of the project of course, we had a system that was running every second so if it failed it generated a lot of data..
- kiitos 3y agoIt is a tragedy of the current generation of observability systems that they have inculcated the notion that telemetry data should be sampled. Absolute nonsense.
- deleted 3y ago[deleted]