Multi-Tenant prometheus.yml control

Hello all,

I’ve done some research and I’m not finding any solid information about my particular scenario. I thought I would try here to get some insight from the community, if possible.

Excuse my lack of understanding as I’m trying to learn everything I can, but my understanding thus far is limited and a huge work in progress.

My scenario is this, I’m looking to create an MSP service that will have multiple Prometheus installations and have those metrics federated and alerted upon. There are a couple of options there (PromScale, Thanos, Cortex etc.). Controlling the scraping config of Prometheus for all of these installations is a hurdle that I’m trying to find a solution to. The Prometheus installations will be at client’s locations and will be a major de-centralized component to what I’m trying to achieve. One requirement we have is to be able to adjust/alter the Prometheus.yml and scraping config to account for changes in the local environment (which would happen quite frequently). In an effort to always keep the monitoring up to date with these changes, my question is does a solution exist where the prometheus.yml file can be remotely controlled, edited, and perhaps even templatized? Is it possible to post a request to Prometheus API to make changes? Any other means that I’m not aware of?

I appreciate your time sharing your ideas. It is very much appreciated!

The expectation is that the prometheus.yml is read by Prometheus but that it is managed elsewhere. So there isn’t any way to edit the file from Prometheus itself. People use all sort of methods to manage the file, for example tools such as Ansible.

So it sounds like you need to use an existing tool such as this or to create your own to manage things the way you want to.