How to add custom labels to alerting rules

Prometheus server
1 
groups: 
- name: cadvisor
  rules:
  - alert: ContainerCpuUsage
    expr: (sum(rate(container_cpu_usage_seconds_total[3m])) BY (instance, name) * 100) > 65
    for: 0m
    labels:
      severity: warning
      service : pod cpu near to full
      support_team: support-l1
    annotations:
      summary: Container CPU usage (instance {{ $labels.instance }})
      pod: POD name is (pod {{ $labels.pod }})
      namespace: Namespace is (namespace {{ $labels.namespace }})
      description: Container CPU usage is above 80%  VALUE = {{ $value }}  LABELS = {{ $labels }} external labels = {{ $externalLabels }}

Right now: $labels.pod is empty.

I need to get $labels.pod - which is custom label in the alert ( and $labels.container), is it possible? any help will be appreciated.

2

Welcome to our forums :slight_smile:

You can adapt your rule like this:

(rate(container_cpu_usage_seconds_total[3m]) * 100) > 65

It does not seem like the by is needed here.

3

Thanks… that helped