One cluster with Prometheus to scrape multiple clusters

Prometheus server
1

Hi all,

we are trying to monitor multiple clusters with one prometheus server but we could not figure out how can it be done.

We have find an article about this but it does not describe how this can be managed. Multiple Kubernetes cluster monitoring with Prometheus | Sysrant

Any help and suggestion would be appreciated.
With my best regards.
Seçil

2

As the linked document says there are multiple options. Which option are you hoping to try?

3

We have two clusters in different vm’s. One of them has Prometheus installed, one of them has no Prometheus. We are trying to pull metrics from the second one to Prometheus. How can we edit Prometheus configmap to get this data and visualize in Prometheus and Grafana. Second one vm, we add ServiceAccount to pull metrics from this cluster. Is there anything to do for getting metrics?

As the linked document, " One cluster with Prometheus to scrape multiple clusters" part we are looking at.
For this they say, “For this, I would just simply refer to the documentation of Prometheus itself. More specifically the kubernetes_sd_config: Configuration | Prometheus” we did not understand how to and where add this configuration.

Thank you so much.

4

If you are wanting to have Prometheus talk to another Kubernetes cluster you’d need to expose the API server & have the Prometheus access it. Using the kubernetes_sd_config with jobs that can then find all the metric endpoints you need. You’d also need to expose all those endpoints in a way that the Prometheus server can get to them (as if it isn’t within the cluster you’d need something like NodePorts or Ingresses as local pod/service access wouldn’t work).

Usually a simpler alternative is to run Prometheus (often in agent mode) within the other cluster, which makes access to the API server & ports to scrape much easier. Then you can use remote write/federation/etc to expose that data.

5

I have a slightly similar requirement (Federation as a way to "group" metrics and expose 1 interface instead of many) and was considering using federation for that, although I’ve also read that using federation to “read everything” is often a bad idea.

After seeing @stuart’s reply, searching about “prometheus agent mode” seems to show very interesting results and likely applicable to these scenarios. In case you’re interested: Introducing Prometheus Agent Mode, an Efficient and Cloud-Native Way for Metric Forwarding | Prometheus