Hi all,
I’ve started my journey into understanding how the whole Prometheus system work.
The team I’m working with have setup a Prometheus Operator through kube-prometheus-stack and its doing a lot of magic behind the scenes that has gotten me slightly confused.
One of the thing I am interested and unable to find much is the affects of kuberentes annotations on Prometheus’s scraping logic.
In the configuration file, we have this service discovery in the scrape config.
- job_name: kubernetes-pods
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: ......
Now I understand that this is going to scrape everything at metrics path /metrics.
However, it is also scraping Envoy metrics at path /stats/prometheus which I cannot comprehend.
After searching the whole config file, I see nothing that says to look into /stats/prometheus for the metrics. However, I found in the annotation of the pod that there is the annotation prometheus.io/path: /stats/prometheus.
I’m assuming that this has something to do with how Prometheus decides where it scrapes.
Sadly, this is the only clue I have and I’m not sure where to find the docs that talks about this.
If anyone has any idea on how I can better clear myself of the whole system’s logic, I would very much appreciate it.
Thanks y’all