Prometheus Dropping cadvisor metrics at relabel

General Help/Support
1

Hello! For some reason it seems that prometheus is dropping cadvisor metrics for about half of our ~250 nodes. It appears that prometheus is scraping nodes but I cannot see cadvisor metrics for many of our nodes. What could be causing this? I would expect prometheus to either keep metrics from all nodes or drop metrics from all nodes.

scrape_samples_scraped{job="kube-cadvisor", kubernetes_io_hostname="node-missing-metrics"} = 0

scrape_samples_post_metric_relabeling{job="kube-cadvisor", kubernetes_io_hostname="node-missing-metrics"}      =  ~1500

Environment

  • Kubernetes: 1.16.8
  • Prometheus: 2.26.0

** Kube-cadvisor job from Prometheus.yml**

  • Note - I checked, and our “samples_scraped” is under our sample_limit. I also don’t see any nodes labeled “prometheus_io_scrape=false” or anything similar.
 job_name: 'kube-cadvisor'
        sample_limit: 5000000
        scheme: https
        scrape_interval: 15s

        metrics_path: /metrics/cadvisor

        kubernetes_sd_configs:
          - role: node

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - source_labels: [__address__]
            action: replace
            regex: (.+):(?:\d+)
            replacement: ${1}:10250
            target_label: __address__
          - source_labels: [__address__]
            regex: (.+):(?:\d+)
            target_label: instance
            replacement: ${1}
          - source_labels: [__meta_kubernetes_node_label_prometheus_io_scrape]
            regex: 'false'
            action: drop

        metric_relabel_configs:
          # cAdvisor
        - source_labels: [pod_name]
          regex: .+
          action: keep
        - source_labels: [__name__]
          regex: container_tasks_state
          action: drop
        - source_labels: [__name__, interface]
          regex: container_network.+;(?:tun.+|dummy.+|kube-.+|docker.+)
          action: drop
        - source_labels: [__name__, scope]
          regex: container_memory_.+;hierarchy
          action: drop
        - source_labels: [__name__]
          regex: container_spec_memory_(limit_bytes|swap_limit_bytes)
          action: drop
        - source_labels: [__name__]
          regex: container_network_(tcp|udp)_usage_total
          action: drop
        - source_labels: [__name__]
          regex: container_cpu_(user_seconds_total|system_seconds_total|load_average_10s|cfs_.+)
          action: drop
        - source_labels: [__name__]
          regex: container_fs_(inodes_.+|limit_bytes|reads_merged_total|read_seconds_total|writes_merged_total|write_seconds_total|io_.+|sector_.+|usage_bytes|writes_total|reads_total)
          action: drop
        - source_labels: [__name__]
          regex: container_memory_(mapped_file)
          action: drop
        - regex: 'beta_kubernetes_io_arch'
          action: labeldrop
        - regex: 'beta_kubernetes_io_os'
          action: labeldrop
        - regex: 'beta_kubernetes_io_instance_type'
          action: labeldrop
        - regex: 'failure_domain_beta_kubernetes_io_region'
          action: labeldrop
        - regex: 'failure_domain_beta_kubernetes_io_zone'
          action: labeldrop
        - source_labels: [container_name]
          target_label: container
        - source_labels: [pod_name]
          target_label: pod
2

This is likely the problem. If the pod_name label doesn’t exist, all of the data will be dropped, as “keep” is exclusive. Anything non-matching is dropped, which is why you have no samples.

1 Like
3

So I changed

 - source_labels: [pod_name]
          regex: .+

to

 - source_labels: [pod_name, pod]
          regex: .+

And now it looks like those metrics are being kept. However, for some reason the “pod” label is being stripped off those metrics. They are definitely being originally exported with the “pod” label.

4

What’s happening now is the combination of pod_name and pod is ;. IE, the separator. So it will always match.

5

Ok, so I’ve just removed that source_labels block altogether

I’m still seeing similar behavior. I.e.
container_cpu_usage_seconds_total{namespace="service-namespace", kubernetes_io_hostname="broken-node-name"}

Outputs something like:

container_cpu_usage_seconds_total{cpu="total", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c33f05_b1db_4349_91d4_3381d6b04a52.slice/docker-08d7e4fe8247fa2330204432bcd6ea4a5e9ea42a4cfe2fdede33345128949e5a.scope", image="registry.example.com-service@sha256:48f0fa71faa023761cad5be41be39623b1d181369bd075acee2d2ae36d42ec43", instance="172.x.x.x", job="kube-cadvisor", kubernetes_io_arch="amd64", kubernetes_io_hostname="broken-node-name", kubernetes_io_os="linux", name="k8s_service-service-78f94d9d47-hz7nv_servicenamespace_10c33f05-b1db-4349-91d4-3381d6b04a52_0", namespace="service-namespace"} 12852.91696676 1623842126403

Edited for correctness

If I curl metrics from the server itself I get this:

container_cpu_usage_seconds_total{container="service",cpu="total",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c33f05_b1db_4349_91d4_3381d6b04a52.slice/docker-08d7e4fe8247fa2330204432bcd6ea4a5e9ea42a4cfe2fdede33345128949e5a.scope",image="registry.example.com/service@sha256:48f0fa71faa023761cad5be41be39623b1d181369bd075acee2d2ae36d42ec43",name="k8s_service_service-78f94d9d47-hz7nv_service-namespace_10c33f05-b1db-4349-91d4-3381d6b04a52_0",namespace="service-namespace",pod="service-78f94d9d47-hz7nv"} 12852.91696676 1623842126403

From what I can tell, then the label pod with it’s values should be being picked up right?

6

I’m not sure what you mean by the first datapoint. It’s impossible to have a metric result with no value in Prometheus. It must have an explicit value, even if it’s something like “NaN”.

Also, you’ve got one of the buggy versions of cAdvisor. They are incorrectly outputting timestamps on the data. This leads to strange behavior.

Set honor_timestamps: false in your Prometheus config. This is what the Prometheus operator configures.

7

Ah, my mistake. I’ve wasn’t referring to the value (I updated previous post with value).

Both prometheus and local curl show the same metric with the same value, however the local curl shows the additional and needed label “pod” with the pod-name. The prometheus query; however, only shows the metric without the “pod” label for some reason.

Also I’ve now added the “honor_timestamps: false” to this job as well.