We have two clusters in different vm’s. One of them has Prometheus installed, one of them has no Prometheus. We are trying to pull metrics from the second one to Prometheus. How can we edit Prometheus configmap to get this data and visualize in Prometheus and Grafana. Second one vm, we add ServiceAccount to pull metrics from this cluster. Is there anything to do for getting metrics?
As the linked document, " One cluster with Prometheus to scrape multiple clusters" part we are looking at.
For this they say, “For this, I would just simply refer to the documentation of Prometheus itself. More specifically the kubernetes_sd_config: Configuration | Prometheus” we did not understand how to and where add this configuration.
If you are wanting to have Prometheus talk to another Kubernetes cluster you’d need to expose the API server & have the Prometheus access it. Using the kubernetes_sd_config with jobs that can then find all the metric endpoints you need. You’d also need to expose all those endpoints in a way that the Prometheus server can get to them (as if it isn’t within the cluster you’d need something like NodePorts or Ingresses as local pod/service access wouldn’t work).
Usually a simpler alternative is to run Prometheus (often in agent mode) within the other cluster, which makes access to the API server & ports to scrape much easier. Then you can use remote write/federation/etc to expose that data.