Hierarchical federated setup is not aggregating metrics

Prometheus server Scaling / Clustering / Long-Term Storage
1

Hi Team,

As per documentation Hierarchical federated setup does aggregation of metrics at the job level. I was trying out the same but it is not working for me.

Let’s assume the below setup.

Prometheus Server (P1) => Scarping from 3 nodes
eg: 3 nodes are exposing metrics

http_requests{instance: "n1", job:"j1"} = 10
http_requests{instance: "n2", job:"j1"} = 10
http_requests{instance: "n3", job:"j1"} = 10

Federated Prometheus(P2) => Scraping from P1
Expectation is P2 should have only one key(aggregated) http_requests{job:"j1"} = 30
Actually, In P2 I am getting 3 of the same instances not aggregated.

http_requests{instance: "n1", job:"j1"} = 10
http_requests{instance: "n2", job:"j1"} = 10
http_requests{instance: "n3", job:"j1"} = 10

I have a very large number of nodes and wanted aggregated metrics at the job level, not the instance level.

I tried relabeling/removing of instance label but it’s not helping here. Can someone please me with the correct configuration at P1 and P2?

Thanks in Advance,
Viral

1 Like