Server returned HTTP status 400 Bad Request on 1 Node

Exporters and Metrics node_exporter
1

Hey folks, I recently deployed the Prometheus helm chart to my default namespace.

All of my pods are up and working.

$ kubectl get pods -n default
NAME                                                 READY   STATUS    RESTARTS      AGE
prometheus-prometheus-node-exporter-9wkx8            1/1     Running   0             9h
prometheus-prometheus-node-exporter-cz98v            1/1     Running   0             9h
prometheus-prometheus-node-exporter-kxcl5            1/1     Running   0             9h
prometheus-prometheus-node-exporter-gjgst            1/1     Running   0             9h
prometheus-alertmanager-0                            1/1     Running   0             9h
prometheus-kube-state-metrics-7748c67656-5lgch       1/1     Running   0             9h
prometheus-prometheus-pushgateway-8457c6cff7-qrqgk   1/1     Running   0             9h
prometheus-server-799f594896-bms2z                   2/2     Running   0             40m

I seem to be facing an issue where my main my node (microk8s-controlplane) is receiving a server returned HTTP status 400 Bad Request under <prometheus-ip>/targets.

I’ve run through a lot of troubleshooting like:

  • Uninstalling the Microk8s metrics
  • Confirming my clusterole includes nodes/proxy and nodes/metrics
  • A few other things that I’m sure I’m forgetting.

Has anyone faced this issue? Is there anything else to look at and/or test?
Thanks!