# Prometheus Dropping cadvisor metrics at relabel

**URL:** <https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314>\
**Category:** General Help/Support\
**Created:** [June 16, 2021, 9:01am UTC](https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314 "2021-06-16T09:01:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![blockhead\_89](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/blockhead_89/32/167_2.png) [@blockhead\_89](https://discuss.prometheus.io/u/blockhead_89)\
**Post date:** [June 16, 2021, 9:01am UTC](https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314/1 "2021-06-16T09:01:05Z")

</div>

Hello! For some reason it seems that prometheus is dropping cadvisor metrics for about half of our ~250 nodes. It appears that prometheus is scraping nodes but I cannot see cadvisor metrics for many of our nodes. What could be causing this? I would expect prometheus to either keep metrics from all nodes or drop metrics from all nodes.

```auto
scrape_samples_scraped{job="kube-cadvisor", kubernetes_io_hostname="node-missing-metrics"} = 0

scrape_samples_post_metric_relabeling{job="kube-cadvisor", kubernetes_io_hostname="node-missing-metrics"} = ~1500

```

**Environment**

- Kubernetes: 1.16.8
- Prometheus: 2.26.0

\*\* Kube-cadvisor job from Prometheus.yml\*\*

- Note - I checked, and our “samples\_scraped” is under our sample\_limit. I also don’t see any nodes labeled “prometheus\_io\_scrape=false” or anything similar.

```auto
 job_name: 'kube-cadvisor'
        sample_limit: 5000000
        scheme: https
        scrape_interval: 15s

        metrics_path: /metrics/cadvisor

        kubernetes_sd_configs:
          - role: node

        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - source_labels: [__address__]
            action: replace
            regex: (.+):(?:\d+)
            replacement: ${1}:10250
            target_label: __address__
          - source_labels: [__address__]
            regex: (.+):(?:\d+)
            target_label: instance
            replacement: ${1}
          - source_labels: [__meta_kubernetes_node_label_prometheus_io_scrape]
            regex: 'false'
            action: drop

        metric_relabel_configs:
          # cAdvisor
        - source_labels: [pod_name]
          regex: .+
          action: keep
        - source_labels: [__name__]
          regex: container_tasks_state
          action: drop
        - source_labels: [__name__ , interface]
          regex: container_network.+;(?:tun.+|dummy.+|kube-.+|docker.+)
          action: drop
        - source_labels: [__name__ , scope]
          regex: container_memory_.+;hierarchy
          action: drop
        - source_labels: [__name__]
          regex: container_spec_memory_(limit_bytes|swap_limit_bytes)
          action: drop
        - source_labels: [__name__]
          regex: container_network_(tcp|udp)_usage_total
          action: drop
        - source_labels: [__name__]
          regex: container_cpu_(user_seconds_total|system_seconds_total|load_average_10s|cfs_.+)
          action: drop
        - source_labels: [__name__]
          regex: container_fs_(inodes_.+|limit_bytes|reads_merged_total|read_seconds_total|writes_merged_total|write_seconds_total|io_.+|sector_.+|usage_bytes|writes_total|reads_total)
          action: drop
        - source_labels: [__name__]
          regex: container_memory_(mapped_file)
          action: drop
        - regex: 'beta_kubernetes_io_arch'
          action: labeldrop
        - regex: 'beta_kubernetes_io_os'
          action: labeldrop
        - regex: 'beta_kubernetes_io_instance_type'
          action: labeldrop
        - regex: 'failure_domain_beta_kubernetes_io_region'
          action: labeldrop
        - regex: 'failure_domain_beta_kubernetes_io_zone'
          action: labeldrop
        - source_labels: [container_name]
          target_label: container
        - source_labels: [pod_name]
          target_label: pod

```

---

<div class="post-metadata">

**Author:** ![SuperQ](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/superq/32/4_2.png) [@SuperQ](https://discuss.prometheus.io/u/SuperQ)\
**Post date:** [June 16, 2021, 9:09am UTC](https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314/2 "2021-06-16T09:09:21Z")

</div>

> [@blockhead\_89](#):
>
> ```auto
> - source_labels: [pod_name]
> regex: .+
> action: keep
> 
> ```

This is likely the problem. If the `pod_name` label doesn’t exist, all of the data will be dropped, as “keep” is exclusive. Anything non-matching is dropped, which is why you have no samples.

---

<div class="post-metadata">

**Author:** ![blockhead\_89](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/blockhead_89/32/167_2.png) [@blockhead\_89](https://discuss.prometheus.io/u/blockhead_89)\
**Post date:** [June 16, 2021, 9:36am UTC](https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314/3 "2021-06-16T09:36:33Z")

</div>

So I changed

```auto
 - source_labels: [pod_name]
          regex: .+

```

to

```auto
 - source_labels: [pod_name, pod]
          regex: .+

```

And now it looks like those metrics are being kept. However, for some reason the “pod” label is being stripped off those metrics. They are definitely being originally exported with the “pod” label.

---

<div class="post-metadata">

**Author:** ![SuperQ](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/superq/32/4_2.png) [@SuperQ](https://discuss.prometheus.io/u/SuperQ)\
**Post date:** [June 16, 2021, 10:05am UTC](https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314/4 "2021-06-16T10:05:40Z")

</div>

> [@blockhead\_89](#):
>
> ```auto
> - source_labels: [pod_name, pod]
> regex: .+
> 
> ```

What’s happening now is the combination of `pod_name` and `pod` is `;`. IE, the separator. So it will always match.

---

<div class="post-metadata">

**Author:** ![blockhead\_89](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/blockhead_89/32/167_2.png) [@blockhead\_89](https://discuss.prometheus.io/u/blockhead_89)\
**Post date:** [June 16, 2021, 10:46am UTC](https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314/5 "2021-06-16T10:46:08Z")

</div>

Ok, so I’ve just removed that `source_labels` block altogether

I’m still seeing similar behavior. I.e.  
`container_cpu_usage_seconds_total{namespace="service-namespace", kubernetes_io_hostname="broken-node-name"}`

Outputs something like:

```auto
container_cpu_usage_seconds_total{cpu="total", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c33f05_b1db_4349_91d4_3381d6b04a52.slice/docker-08d7e4fe8247fa2330204432bcd6ea4a5e9ea42a4cfe2fdede33345128949e5a.scope", image="registry.example.com-service@sha256:48f0fa71faa023761cad5be41be39623b1d181369bd075acee2d2ae36d42ec43", instance="172.x.x.x", job="kube-cadvisor", kubernetes_io_arch="amd64", kubernetes_io_hostname="broken-node-name", kubernetes_io_os="linux", name="k8s_service-service-78f94d9d47-hz7nv_servicenamespace_10c33f05-b1db-4349-91d4-3381d6b04a52_0", namespace="service-namespace"} 12852.91696676 1623842126403

```

_Edited for correctness_

If I curl metrics from the server itself I get this:

```auto
container_cpu_usage_seconds_total{container="service",cpu="total",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c33f05_b1db_4349_91d4_3381d6b04a52.slice/docker-08d7e4fe8247fa2330204432bcd6ea4a5e9ea42a4cfe2fdede33345128949e5a.scope",image="registry.example.com/service@sha256:48f0fa71faa023761cad5be41be39623b1d181369bd075acee2d2ae36d42ec43",name="k8s_service_service-78f94d9d47-hz7nv_service-namespace_10c33f05-b1db-4349-91d4-3381d6b04a52_0",namespace="service-namespace",pod="service-78f94d9d47-hz7nv"} 12852.91696676 1623842126403

```

From what I can tell, then the label `pod` with it’s values should be being picked up right?

---

<div class="post-metadata">

**Author:** ![SuperQ](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/superq/32/4_2.png) [@SuperQ](https://discuss.prometheus.io/u/SuperQ)\
**Post date:** [June 16, 2021, 11:08am UTC](https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314/6 "2021-06-16T11:08:57Z")

</div>

I’m not sure what you mean by the first datapoint. It’s impossible to have a metric result with no value in Prometheus. It must have an explicit value, even if it’s something like “NaN”.

Also, you’ve got one of the buggy versions of cAdvisor. They are incorrectly outputting timestamps on the data. This leads to strange behavior.

Set `honor_timestamps: false` in your Prometheus config. This is what the Prometheus operator configures.

---

<div class="post-metadata">

**Author:** ![blockhead\_89](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/blockhead_89/32/167_2.png) [@blockhead\_89](https://discuss.prometheus.io/u/blockhead_89)\
**Post date:** [June 16, 2021, 11:17am UTC](https://discuss.prometheus.io/t/prometheus-dropping-cadvisor-metrics-at-relabel/314/7 "2021-06-16T11:17:27Z")

</div>

Ah, my mistake. I’ve wasn’t referring to the value (I updated previous post with value).

Both prometheus and local curl show the same metric with the same value, however the local curl shows the additional and needed label “pod” with the pod-name. The prometheus query; however, only shows the metric without the “pod” label for some reason.

Also I’ve now added the “honor\_timestamps: false” to this job as well.