I am using the Bitnami version of Prometheus Node Exporter in my hybrid cloud kubernetes environment.
I’ve some regular cronjobs deployed and they are running every few minutes, the cronjobs will trigger a pod and deploying a job as a pod in the Kubernetes cluster. In the nodes that always running the cronjobs, I can see that the Prometheus scrape sample is going up and down when the cronjob pods are created and finished. However, the memory usage in those node exporters are unlike the scrape samples, it just keeps going up and leading to a memory leak issue in one day.
Does anyone has similar experience?
Is it an usual practice to restart the node exporter regularly?