How to define and use Prometheus Labels variables?

General Help/Support
1

Hi experts,

I need your lights for defining and using properly labels variables.

Below you will find my prometheus config example.

I have defined

labels:
team: ‘team1’

and want to use “team” variable in alert.rules.yml

labels:
  severity: critical
  origin: {{ $labels.team }}

But Prometheus doesn’t recognize {{ $labels.team }} as a variable, if I replace it with {{ $labels.instance }} it works well.

Any idea on how to define new variables and use it properly?

Many thanks

prometheus.yml

  • job_name: ‘blackbox_probe’
    metrics_path: /probe
    params:
    module: [body_contains_ok]
    file_sd_configs:
    - files: [‘targets.yml’]
    scheme: https
    relabel_configs:
    - source_labels: [address]
    target_label: __param_target
    - source_labels: [__param_target]
    target_label: instance
    - target_label: address
    replacement: promtheus.blackbox.service.com:9115

======================
targets.yml

  • targets:
    • “myurl1/isAlive”
      labels:
      team: ‘team1’
  • targets:
    • “myurl2/isAlive”
      labels:
      team: ‘team2’

==========================
alert.rules.yml

groups:

  • name: service_down
    rules:
    • alert: IsAlive_down
      expr: (count by (instance) (probe_success{job=“blackbox_probe”}==0 ) )==1
      for: 10s
      labels:
      severity: critical
      origin: {{ $labels.team }}
      annotations:
      summary: “blabla”