Hi there,
To reduce cardinality due to histograms, i’m thinking of aggregating some percentiles in recording rules using the histogram buckets and then dropping those buckets. A recording rule is like this:
Rules:
Metric: rr_request_time_ms_50p
Query: histogram_quantile(.50, sum by (resource, method, le) (rate(request_time_ms_bucket{app="loadtest-service"}[5m])))
High number of pods in the system are vastly increasing the cardinality of timeseries. I’m trying to see how to fix that.
So, is it possible to drop metrics after aggregations?
Thanks!