A few weeks ago my alert manager started sending emails for InfoInhibitor alerts.
From my understanding of the configuration it should not do that and it did not do it before.
I do not remember exactly when it started so I cannot easily pinpoint the source of the issue.
Any assistance will be greatly appreciated
attached is the [redacted] config:
global:
resolve_timeout: 5m
http_config:
follow_redirects: true
enable_http2: true
smtp_hello: localhost
smtp_require_tls: true
smtp_tls_config:
insecure_skip_verify: false
slack_app_url: https://slack.com/api/chat.postMessage
pagerduty_url: https://events.pagerduty.com/v2/enqueue
opsgenie_api_url: https://api.opsgenie.com/
wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
telegram_api_url: https://api.telegram.org
webex_api_url: https://webexapis.com/v1/messages
rocketchat_api_url: https://open.rocket.chat/
route:
receiver: email receiver
group_by:
- alertname
continue: false
routes:
- receiver: "null"
matchers:
- alertname="Watchdog"
continue: false
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
inhibit_rules:
- source_matchers:
- severity="critical"
target_matchers:
- severity=~"warning|info"
equal:
- namespace
- alertname
- source_matchers:
- severity="warning"
target_matchers:
- severity="info"
equal:
- namespace
- alertname
- source_matchers:
- alertname="InfoInhibitor"
target_matchers:
- severity="info"
equal:
- namespace
- target_matchers:
- alertname="NodeDiskIOSaturation"
- severity="warning"
receivers:
- name: "null"
- name: email receiver
email_configs:
- send_resolved: true
to: [redacted]
from: [redacted]
hello: [redacted]
smarthost: [redacted]
auth_username: [redacted]
auth_password: [redacted]
headers:
From: Alertmanager <[redacted]>
Subject: '{{ template "email.default.subject" . }}'
To: [redacted]
html: '{{ template "email.default.html" . }}'
require_tls: true
tls_config:
insecure_skip_verify: false
templates:
- /etc/alertmanager/config/*.tmpl
And an example email:
View in Alertmanager
[1] Firing
Labels
alertname = InfoInhibitor
alertstate = pending
container = homepage
instance = [redacted]
namespace = homepage
pod = homepage-57bdb44f85-jshr5
prometheus = monitoring/prometheus-stack-kube-prom-prometheus
service = prometheus-stack-kube-prom-kubelet
severity = none
Annotations
description = This is an alert that is used to inhibit info alerts. By themselves, the info-level alerts are sometimes very noisy, but they are relevant when combined with other alerts. This alert fires whenever there's a severity="info" alert, and stops firing when another alert with a severity of 'warning' or 'critical' starts firing on the same namespace. This alert should be routed to a null receiver and configured to inhibit alerts with severity="info".
runbook_url = https://runbooks.prometheus-operator.dev/runbooks/general/infoinhibitor
summary = Info-level alert inhibition.