Bug or feature in golang client: xxx already contains grouping label yyy

General Help/Support
1

I run fluent-bit with a plugin to export metrics to Prometheus. I found the following issue when running multiple instances of it: Instance pushes override each other · Issue #4 · neiman-marcus/fluent-bit-out-prometheus-metrics · GitHub

Tried to solve this problem by explicitly adding a Grouping instruction like

@@ -620,7 +621,8 @@ func FLBPluginInit(plugin unsafe.Pointer) int {
        }
 
        // Initialize new metric with push gateway
-       pCtx.Pusher = push.New(pCtx.URL, pCtx.Job).Gatherer(registry)
+       pCtx.Pusher = push.New(pCtx.URL, pCtx.Job).Grouping("instance", pCtx.ConstantLabels["instance"]).Gatherer(registry)

In this case I get errors from the library:

level=error plugin=prometheus_metrics caller=out.go:789 id=indexed_counter msg="Could not put push to pushgateway, resetting retry counter, declaring failure data will be lost" err="pushed metric fluentbit_indexed_record_total (label:<name:\"instance\" value:\"7de1c1fe4f33\" > label:<name:\"method\" value:\"GET\" > label:<name:\"status_code\" value:\"200\" > counter:<value:25 > ) already contains grouping label instance"

I have drilled down the problem to the following line in Prometheus golang client library: client_golang/push.go at main · prometheus/client_golang · GitHub

Is this a bug or a feature? I would think that the label should be allowed in both the grouping path as well as in the payload. The only test which makes sense to me is to check for equality of the value instead of the label name?

2

For some reason I can only use two links in my post, so here is the missing link wrt the diff: adding a Grouping instruction here

3

As you are the author of the mentioned lines (according to git blame), @beorn7, do you have an idea about it? Thx!

4

The behavior is deliberate. At least it is documented as such: push package - github.com/prometheus/client_golang/prometheus/push - pkg.go.dev

The basic idea is that grouping labels are akin to target labels and therefore should not collide with the metric’s own labels. I guess if the label value is the same, there is not really any harm done, it’s just a bit weird.

Can you perhaps just not set that label on the metric itself?