Prometheus kuberentes-pods Get “https:// xx.xx.xx:443 /metrics”: dial tcp xx.xx.xx:443: connect: connection refused

Prometheus server
1

I have configured Prometheus on one of the kubernetes cluster node using this How To Install And Configure Prometheus Monitoring System On Linux after that I added following prometheus.yml file. I can list nodes and apiservers but for pods, all the pods shows down and error Get “https:// xx.xx.xx:443 /metrics”: dial tcp xx.xx.xx:443: connect: connection refused and for some pods the status is unknown. when i can list the nodes and api servers? why it shows connection refused? is it a bug or some network issue on my end?

Cat prometheus.yml



global:
  scrape_interval: 1m
 
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: \['localhost:9090'\]
 
# metrics for default/kubernetes api's from the kubernetes master
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
      bearer_token_file: /dfgdjk/token
      api_server: https://masterapi.com:3343
      tls_config:
        insecure_skip_verify: true
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /dfgdjk/token
    scheme: https
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: \[__meta_kubernetes_namespace\]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: \[__meta_kubernetes_pod_name\]
      action: replace
      target_label: kubernetes_pod_name
 
# metrics for default/kubernetes api's from the kubernetes master
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://masterapi.com:3343
      bearer_token_file: /dfgdjk/token
      tls_config:
        insecure_skip_verify: true
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /dfgdjk/token
    scheme: https
    relabel_configs:
    - source_labels: \[__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name\]
      action: keep
      regex: default;kubernetes;https][1]

image
image737×331 52.6 KB

2

If someone is having same issue, I used kube-state-metrics and cadvisor( it’s part of the nodes config ,but we need to explicitly give path as /metrics/cadvisor to get all the memory and n/w of the pods and containers). If someone wants to install prometheus outside kubernetes cluster, it can be done but not with service discovery option. We need to use kube-state-metrics and cadvisor and open firewall ports to the api server where Prometheus is installed.

3

You can still use Prometheus service discovery even if it is hosted outside the cluster, but you’d need to arrange for Prometheus to be able to access the correct APIs (so firewalls, authentication, etc.)

4

I tried that option but my application pods can’t scrape metrics and shows state as down and error as connection refused, that happens with only my application pods, I can see metrics for node and api servers but not for the pods in Prometheus, to avoid that I installed in kube-state-metrics.

image
image737×331 52.7 KB