Hello, everyone
The issue regarding the frequency of sending alerts from Prometheus to Alertmanager and then to emails
The topological environment is as follows:
Two Prometheus instances (each running in a separate Pod and managed by a StatefulSet), each with a Thanos Sidecar (for remote reading/writing and uploading blocks to object storage)
Without using Thanos Ruler (i.e., alerting rules are evaluated locally by Prometheus, not uniformly by thanos Ruler)
Two Alertmanager instances form a high-availability cluster
Both Prometheuses send alerts directly to the same Alertmanager cluster
The relevant configuration for Prometheus is as follows：
The address “my-alertmanager-headless.monitor.svc.cluster.local” will perform DNS resolution for the IP addresses of all pods.
Part of Prometheus’s configuration
cat > /download/my-prometheus-config.yml << ‘EOF’
my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_timeout is set to the global default (10s).
external_labels:
replica: ${REPLICA_NAME}
Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- my-alertmanager-headless.monitor.svc.cluster.local:9093
alert_relabel_configs:
- action: labeldrop
regex: replica
up{job=“k8s_node_exporter”} == 1
This is a manual simulation of a fault.
cat <<‘EOF’ | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: my-prometheus-rules-config
namespace: monitor
data:
default.yml: |
groups:
- name: k8s-physical-node.rules
rules:
- alert: k8s-physical-node-down
expr: up{job=“k8s_node_exporter”} == 1
for: 1m
labels:
severity: critical
annotations:
summary: “k8s物理节点-{{ $labels.instance }}-停止工作。当前采集值为: {{ $value }}”
source_prometheus: “【replica: {{ $externalLabels.replica }}】”
source_expression: ‘up{job=“k8s_node_exporter”} == 1’
EOF
Altermanager Configuration
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: my-alertmanager-headless
namespace: monitor
spec:
clusterIP: None
ports:
- port: 9093
targetPort: 9093
name: web
- port: 9094
targetPort: 9094
name: mesh
selector:
app: my-alertmanager
publishNotReadyAddresses: true
type: ClusterIP
EOF
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: my-alertmanager-config
namespace: monitor
data:
alertmanager.yml: |
global:
resolve_timeout: 2m
smtp_from: ‘212353597@qq.com’
smtp_smarthost: ‘smtp.qq.com:465’
smtp_auth_username: ‘212353597@qq.com’
smtp_auth_password: ‘#############’
smtp_require_tls: false
templates:
- ‘/etc/alertmanager/template/*.template’
route:
group_by: [‘alertname’, ‘instance’]
group_wait: 90s
group_interval: 5m
repeat_interval: 2h
routes:
- match:
severity: emergency
repeat_interval: 15m
routes:
- receiver: ‘email’
continue: true
- receiver: ‘web.hook’
continue: true
- match:
severity: critical
repeat_interval: 10m
routes:
- receiver: ‘email’
continue: false
- receiver: ‘web.hook’
continue: true
- match:
severity: error
repeat_interval: 30m
routes:
- receiver: ‘email’
continue: true
- receiver: ‘web.hook’
continue: true
- match:
severity: warning
repeat_interval: 1h
routes:
- receiver: ‘email’
continue: true
- receiver: ‘web.hook’
continue: true
receiver: ‘null’
receivers:
- name: ‘email’
email_configs:
- to: ‘15051238103258@2331.com’
send_resolved: true
html: ‘{{ template “email.to.html” . }}’
- name: ‘web.hook’
webhook_configs:
- url: ‘http://127.0.0.1:5001/’
send_resolved: true
- name: ‘null’
inhibit_rules:
- source_match:
severity: ‘emergency’
target_match:
severity: ‘critical’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘emergency’
target_match:
severity: ‘error’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘emergency’
target_match:
severity: ‘warning’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘critical’
target_match:
severity: ‘error’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘critical’
target_match:
severity: ‘warning’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘error’
target_match:
severity: ‘warning’
equal: [‘alertname’, ‘instance’]
EOF
#The status of my altermanager cluster is all “ready”.
curl -s http://192.168.110.210:8080/api/v2/status | jq ‘.cluster’
{
"name": "01K96P2QRK4VGVH9E2JEW3WSPY",
"peers": [
{
"address": "10.244.218.252:9094",
"name": "01K96P2QRK4VGVH9E2JEW3WSPY"
},
{
"address": "10.244.157.190:9094",
"name": "01K96P559JYYX9P0DDGFVEF5Z0"
}
],
"status": "ready"
}
curl -s http://192.168.110.210:8081/api/v2/status | jq ‘.cluster’
{
"name": "01K96P559JYYX9P0DDGFVEF5Z0",
"peers": [
{
"address": "10.244.218.252:9094",
"name": "01K96P2QRK4VGVH9E2JEW3WSPY"
},
{
"address": "10.244.157.190:9094",
"name": "01K96P559JYYX9P0DDGFVEF5Z0"
}
],
"status": "ready"
}
【The first batch of received alerts】
#The time when the altermanager receives the alert
kubectl logs -n monitor pod/my-alertmanager-0 | grep “Received alert” -n
14117:{“time”:“2025-11-05T00:52:29.231953631Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.211:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.211:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.231843335+08:00”,"Timeout”:false}}
14118:{“time”:“2025-11-05T00:52:29.232151847Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.218:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.218:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.231843335+08:00”,"Timeout”:false}}
14119:{“time”:“2025-11-05T00:52:29.232204661Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.210:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.210:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.231843335+08:00”,"Timeout”:false}}
14120:{“time”:“2025-11-05T00:52:29.232228298Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.219:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.219:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.231843335+08:00”,"Timeout”:false}}
14121:{“time”:“2025-11-05T00:52:29.23945036Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.210:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.210:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.239281976+08:00”,"Timeout”:false}}
14122:{“time”:“2025-11-05T00:52:29.239601683Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.219:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.219:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.239281976+08:00”,"Timeout”:false}}
14123:{“time”:“2025-11-05T00:52:29.239681013Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.211:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.211:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.239281976+08:00”,"Timeout”:false}}
14124:{“time”:“2025-11-05T00:52:29.239715302Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.218:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.218:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.239281976+08:00”,"Timeout”:false}}
kubectl logs -n monitor pod/my-alertmanager-1 | grep “Received alert” -n
null
#The time when the alarm was successfully sent by the altermanager
kubectl logs -n monitor pod/my-alertmanager-0 | grep “Notify success” -n
14130:{“time”:“2025-11-05T00:52:30.457964557Z”,“level”:“DEBUG”,“source”:“notify.go:876”,“msg”:“Notify success”,“component”:“dispatcher”,“receiver”:“email”,“integration”:“email[0]”,“aggrGroup”:“{}/{severity="critical"}/{}:{alertname="k8s-physical-node-down", instance="192.168.110.211:9100"}”,“attempts”:1,“duration”:1030128705,“alerts”:“[k8s-physical-node-down[aa504b6][active]]”}
14132:{“time”:“2025-11-05T00:52:30.517906504Z”,“level”:“DEBUG”,“source”:“notify.go:876”,“msg”:“Notify success”,“component”:“dispatcher”,“receiver”:“email”,“integration”:“email[0]”,“aggrGroup”:“{}/{severity="critical"}/{}:{alertname="k8s-physical-node-down", instance="192.168.110.210:9100"}”,“attempts”:1,“duration”:1089951271,“alerts”:“[k8s-physical-node-down[61a5ca1][active]]”}
kubectl logs -n monitor pod/my-alertmanager-1 | grep “Notify success” -n
null
【The second batch of received alerts】
#The time when the altermanager receives the alert
kubectl logs -n monitor pod/my-alertmanager-0 | grep “Received alert” -n
14167:{“time”:“2025-11-05T00:57:29.229045333Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.219:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.219:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.228905739+08:00”,"Timeout”:false}}
14168:{“time”:“2025-11-05T00:57:29.22933875Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.211:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.211:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.228905739+08:00”,"Timeout”:false}}
14169:{“time”:“2025-11-05T00:57:29.229418053Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.218:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.218:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.228905739+08:00”,"Timeout”:false}}
14170:{“time”:“2025-11-05T00:57:29.229561876Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.210:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.210:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.228905739+08:00”,"Timeout”:false}}
14171:{“time”:“2025-11-05T00:57:29.231702787Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.211:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.211:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.231382656+08:00”,"Timeout”:false}}
14172:{“time”:“2025-11-05T00:57:29.231815435Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.218:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.218:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.231382656+08:00”,"Timeout”:false}}
14173:{“time”:“2025-11-05T00:57:29.231858085Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.210:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.210:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.231382656+08:00”,"Timeout”:false}}
14174:{“time”:“2025-11-05T00:57:29.231881344Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.219:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.219:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.231382656+08:00”,"Timeout”:false}}
kubectl logs -n monitor pod/my-alertmanager-1 | grep “Received alert” -n
null
#The time when the alarm was successfully sent by the altermanager
kubectl logs -n monitor pod/my-alertmanager-0 | grep “Notify success” -n
14180:{“time”:“2025-11-05T00:57:30.45846932Z”,“level”:“DEBUG”,“source”:“notify.go:876”,“msg”:“Notify success”,“component”:“dispatcher”,“receiver”:“email”,“integration”:“email[0]”,“aggrGroup”:“{}/{severity="critical"}/{}:{alertname="k8s-physical-node-down", instance="192.168.110.219:9100"}”,“attempts”:1,“duration”:1029726795,“alerts”:“[k8s-physical-node-down[eb4844f][active]]”}
14182:{“time”:“2025-11-05T00:57:30.53294889Z”,“level”:“DEBUG”,“source”:“notify.go:876”,“msg”:“Notify success”,“component”:“dispatcher”,“receiver”:“email”,“integration”:“email[0]”,“aggrGroup”:“{}/{severity="critical"}/{}:{alertname="k8s-physical-node-down", instance="192.168.110.218:9100"}”,“attempts”:1,“duration”:1104358612,“alerts”:“[k8s-physical-node-down[e95f07a][active]]”}
kubectl logs -n monitor pod/my-alertmanager-1 | grep “Notify success” -n
null
***
Question：
**1.**In the above section**【The first batch of received alerts】**, the configuration is as follows
The alertmanager received a total of 8 alerts.
Why were only two alerts successfully sent to the email?
{alertname=“k8s-physical-node-down”, instance=“192.168.110.211:9100”}
{alertname=“k8s-physical-node-down”, instance=“192.168.110.210:9100”}
I understand that after the deduplication process in the altermanager cluster, four alerts should be sent to the email.
{alertname=“k8s-physical-node-down”, instance=“192.168.110.211:9100”}
{alertname=“k8s-physical-node-down”, instance=“192.168.110.210:9100”}
{alertname=“k8s-physical-node-down”, instance=“192.168.110.218:9100”}
{alertname=“k8s-physical-node-down”, instance=“192.168.110.219:9100”}
**2.**In the above section**【The second batch of received alerts】**, the configuration is as follows
The alertmanager received a total of 8 alerts.
Why were only two alerts successfully sent to the email?
{alertname=“k8s-physical-node-down”, instance=“192.168.110.218:9100”}
{alertname=“k8s-physical-node-down”, instance=“192.168.110.219:9100”}
And it has been five minutes since the last successful transmission of two alerts.
{alertname=“k8s-physical-node-down”, instance=“192.168.110.211:9100”}
{alertname=“k8s-physical-node-down”, instance=“192.168.110.210:9100”}
Why is this? I don’t understand how they work.
After the received 8 alerts were deduplicated by the altermanager, why weren’t all 4 of them sent together? Instead, it sends only 2 messages each time, with an interval of 5 minutes between each transmission.
Why is the phenomenon like this? What kind of working logic is behind it? I don’t quite understand.
