Prometheus helm chart targets down

Exporters and Metrics
1

Hi! I am trying to monitor energy consumption along resource usage, I am using scaphandre installation with helm, and prometheus is required to be installed using helm too.

My problem is that when I install prometheus helm chart, some targets are down, and some metrics don’t show up (for example CONTAINER_X).

The commands I used for installation are the ones stated in scaphandre documentation:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update

helm install prometheus prometheus-community/prometheus \
--set alertmanager.persistentVolume.enabled=false \
--set server.persistentVolume.enabled=false

And if I get the pods, every pod is running but the alertmanager, which is in pending due to a failure in persistentvolumeclaims, even though it is supposed to be disabled.

The targets that are down are: kubernetes-nodes and kubernetes-nodes-cadvisor both errors are “server returned HTTP status 400 Bad Request”.

I’ve tried several things but haven’t being able to fix it. I only need those metrics to be accesible and it looks like one of the down targets may provide them, otherwise it shouldn’t be a problem since my app is relatively simple.

PD: I am using a kubernetes microk8s cluster and my OS is Ubuntu 20.04 LTS. When I install microk8s prometheus addon, everything works fine, but scaphandre does not work with other prometheus installations (doesn’t have plans to do so at the moments either) so I really need the prometheus helm chart installation.

2

I’m not sure if the instructions you mention are out of date or just
incorrect, but I don’t think that Helm --set option is correct. Having a
quick look suggests you should instead have --set
alertmanager.persistence.enabled=false

For the other error it suggests something is wrong with the request
being made, so presumably a different request is happening with the
microk8s prometheus addon. I’d suggest comparing the
prometheus.yaml/targets page when you set up the two different methods
of installing Prometheus and seeing what is different, and then applying
those differences to the standard Helm chart install.

Alternatively use the working Prometheus addon, as it looks like you are
just installing the latest Prometheus from the main chart repository
anyway, rather than using anything custom from the Scaphadre project.

3

Thanks for your answer! It is true that is persistence, I had tried it before writing the question but the problem didn’t solve. Now every pod is running properly.

The bad thing is that the problem might not be related to that, as I had thought. Because both targets are still down. I guess it has something to do with the metrics endpoint, since it queries the following endpoints:

https://kubernetes.default.svc/api/v1/nodes/myNodeName/proxy/metrics
https://kubernetes.default.svc/api/v1/nodes/myNodeName/proxy/metrics/cadvisor

(these endpoints are the ones queried in the down targets, and its error in prometheus GUI is “server returned HTTP status 400 Bad Request”).

I’ve double checked that the kubernetes service is properly running at my default namespace, so I don’t really know what could be happening.

About using the prometheus addon, I would love to. But if I do so, scaphandre metrics do not appear, I think it might be because helm chart prometheus and scaphandre are installed in default namespace, while microk8s prometheus is installed in monitoring namespace.