I have some config Prometheus, Alert rules same format
Prometheus jobs (same format)
- job_name: "windows"
file_sd_configs:
- files:
- windows.yml
relabel_configs:
- source_labels: [__address__]
target_label: host
regex: ^(.*):\d+$
# target_label: address
replacement: $1
- job_name: "linux"
file_sd_configs:
- files:
- linux.yml
relabel_configs:
- source_labels: [__address__]
target_label: host
regex: ^(.*):\d+$
# target_label: address
replacement: $1
windows.yml
- labels:
job: "windows"
system: "windows01"
targets:
- "abc.def.ghi.xyz:9182"
linuxs.yml
- labels:
job: "linux"
system: "linux01"
targets:
- "abc.def.ghi.xyz:9100"
windows.rule
- alert: WindowsServerCpuUsage
expr: 100 - (avg by (instance) (rate(windows_cpu_time_total{mode="idle"}[2m])) * 100) > 95
for: 2m
labels:
severity: warning
annotations:
summary: high CPU load
description: (System {{ $labels.system }})(instance {{ $labels.instance }}) CPU load was > 95% for more than 2m\n VALUE = {{ $value }}\n
linux.rule
- alert: LinuxServerMemoryUsage
expr: 100 - ((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100) > 95
for: 2m
labels:
severity: warning
annotations:
summary: high Memory usage
description: (System {{ $labels.system }})(instance {{ $labels.instance }}) Memory usage was > 95% for more than 2m\n VALUE = {{ $value }}\n
But when I received email alert, content of Linux have more labels than Windows, I using same email template
Linux email content:
Labels
alertname = LinuxServerMemoryUsage
env = prod
host = abc.def.ghi.xyz
instance = abc.def.ghi.xyz:9100
job = linux
monitor = Myl-Infras
severity = warning
system = linux01
Annotations
description = (System linux01)(instance abc.def.ghi.xyz:9100) Memory usage was > 95% for more than 2m\n VALUE = 95.4328275548093\n
summary = high Memory usage
Source
Windows email content
Labels
alertname = WindowsServerMemoryUsage
instance = abc.def.ghi.xyz:9182
monitor = My-Infras
severity = warning
Annotations
description = (System )(instance abc.def.ghi.xyz:9182) Memory usage was > 95% for more than 2m\n VALUE = 95.4328275548093\n
summary = high Memory usage
Source
Why my email Windows missing some lables like: env; host, job; system when all my config as same format. I need include label “System” in my email beside host IP Address (my Infra have many server and I can not remember all IP of them).