# The issue regarding the frequency of sending alerts from Prometheus to Alertmanager and then to emails

**URL:** <https://discuss.prometheus.io/t/the-issue-regarding-the-frequency-of-sending-alerts-from-prometheus-to-alertmanager-and-then-to-emails/3477>\
**Category:** General Help/Support\
**Created:** [November 5, 2025, 3:06am UTC](https://discuss.prometheus.io/t/the-issue-regarding-the-frequency-of-sending-alerts-from-prometheus-to-alertmanager-and-then-to-emails/3477 "2025-11-05T03:06:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![cscowx](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/cscowx/32/1220_2.png) [@cscowx](https://discuss.prometheus.io/u/cscowx)\
**Post date:** [November 5, 2025, 3:06am UTC](https://discuss.prometheus.io/t/the-issue-regarding-the-frequency-of-sending-alerts-from-prometheus-to-alertmanager-and-then-to-emails/3477/1 "2025-11-05T03:06:33Z")

</div>

Hello, everyone

The issue regarding the frequency of sending alerts from Prometheus to Alertmanager and then to emails

**The topological environment is as follows:**

`Two Prometheus instances (each running in a separate Pod and managed by a StatefulSet), each with a Thanos Sidecar (for remote reading/writing and uploading blocks to object storage)`

`Without using Thanos Ruler (i.e., alerting rules are evaluated locally by Prometheus, not uniformly by thanos Ruler)`

`Two Alertmanager instances form a high-availability cluster`

`Both Prometheuses send alerts directly to the same Alertmanager cluster`

**The relevant configuration for Prometheus is as follows：**  
The address “my-alertmanager-headless.monitor.svc.cluster.local” will perform DNS resolution for the IP addresses of all pods.

**Part of Prometheus’s configuration**

```yaml
cat > /download/my-prometheus-config.yml << ‘EOF’

my global config

global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_timeout is set to the global default (10s).

external_labels:
replica: ${REPLICA_NAME}

Alertmanager configuration

alerting:
alertmanagers:
- static_configs:
- targets:
- my-alertmanager-headless.monitor.svc.cluster.local:9093
alert_relabel_configs:
- action: labeldrop
regex: replica

up{job=“k8s_node_exporter”} == 1
This is a manual simulation of a fault.

```

```yaml
cat <<‘EOF’ | kubectl apply -f -

apiVersion: v1
kind: ConfigMap
metadata:
name: my-prometheus-rules-config
namespace: monitor
data:
default.yml: |
groups:
- name: k8s-physical-node.rules
rules:
- alert: k8s-physical-node-down
expr: up{job=“k8s_node_exporter”} == 1
for: 1m
labels:
severity: critical
annotations:
summary: “k8s物理节点-{{ $labels.instance }}-停止工作。当前采集值为: {{ $value }}”
source_prometheus: “【replica: {{ $externalLabels.replica }}】”
source_expression: ‘up{job=“k8s_node_exporter”} == 1’
EOF

```

## **Altermanager Configuration**

```yaml
cat <<EOF | kubectl apply -f -

apiVersion: v1
kind: Service
metadata:
name: my-alertmanager-headless
namespace: monitor
spec:
clusterIP: None
ports:
- port: 9093
targetPort: 9093
name: web
- port: 9094
targetPort: 9094
name: mesh
selector:
app: my-alertmanager
publishNotReadyAddresses: true
type: ClusterIP
EOF

cat <<EOF | kubectl apply -f -

apiVersion: v1
kind: ConfigMap
metadata:
name: my-alertmanager-config
namespace: monitor
data:
alertmanager.yml: |
global:
resolve_timeout: 2m
smtp_from: ‘212353597@qq.com’
smtp_smarthost: ‘smtp.qq.com:465’
smtp_auth_username: ‘212353597@qq.com’
smtp_auth_password: ‘#############’
smtp_require_tls: false
templates:
- ‘/etc/alertmanager/template/*.template’
route:
group_by: [‘alertname’, ‘instance’]
group_wait: 90s
group_interval: 5m
repeat_interval: 2h
routes:
- match:
severity: emergency
repeat_interval: 15m
routes:
- receiver: ‘email’
continue: true
- receiver: ‘web.hook’
continue: true
- match:
severity: critical
repeat_interval: 10m
routes:
- receiver: ‘email’
continue: false
- receiver: ‘web.hook’
continue: true
- match:
severity: error
repeat_interval: 30m
routes:
- receiver: ‘email’
continue: true
- receiver: ‘web.hook’
continue: true
- match:
severity: warning
repeat_interval: 1h
routes:
- receiver: ‘email’
continue: true
- receiver: ‘web.hook’
continue: true
receiver: ‘null’
receivers:
- name: ‘email’
email_configs:
- to: ‘15051238103258@2331.com’
send_resolved: true
html: ‘{{ template “email.to.html” . }}’
- name: ‘web.hook’
webhook_configs:
- url: ‘http://127.0.0.1:5001/’
send_resolved: true
- name: ‘null’
inhibit_rules:
- source_match:
severity: ‘emergency’
target_match:
severity: ‘critical’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘emergency’
target_match:
severity: ‘error’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘emergency’
target_match:
severity: ‘warning’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘critical’
target_match:
severity: ‘error’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘critical’
target_match:
severity: ‘warning’
equal: [‘alertname’, ‘instance’]
- source_match:
severity: ‘error’
target_match:
severity: ‘warning’
equal: [‘alertname’, ‘instance’]
EOF

```

#The **status of my altermanager cluster is all “ready”.**

curl -s [http://192.168.110.210:8080/api/v2/status](http://192.168.110.210:8080/api/v2/status) | jq ‘.cluster’

```yaml
{

"name": "01K96P2QRK4VGVH9E2JEW3WSPY",

"peers": [

{

"address": "10.244.218.252:9094",

"name": "01K96P2QRK4VGVH9E2JEW3WSPY"

},

{

"address": "10.244.157.190:9094",

"name": "01K96P559JYYX9P0DDGFVEF5Z0"

}

],

"status": "ready"

}

```

curl -s [http://192.168.110.210:8081/api/v2/status](http://192.168.110.210:8081/api/v2/status) | jq ‘.cluster’

```yaml
{

"name": "01K96P559JYYX9P0DDGFVEF5Z0",

"peers": [

{

"address": "10.244.218.252:9094",

"name": "01K96P2QRK4VGVH9E2JEW3WSPY"

},

{

"address": "10.244.157.190:9094",

"name": "01K96P559JYYX9P0DDGFVEF5Z0"

}

],

"status": "ready"

}

```

**【The first batch of received alerts】**

#The **time when the altermanager receives the alert**  
kubectl logs -n monitor pod/my-alertmanager-0 | grep “Received alert” -n

```yaml
14117:{“time”:“2025-11-05T00:52:29.231953631Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.211:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.211:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.231843335+08:00”,"Timeout”:false}}

14118:{“time”:“2025-11-05T00:52:29.232151847Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.218:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.218:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.231843335+08:00”,"Timeout”:false}}

14119:{“time”:“2025-11-05T00:52:29.232204661Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.210:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.210:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.231843335+08:00”,"Timeout”:false}}

14120:{“time”:“2025-11-05T00:52:29.232228298Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.219:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.219:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.231843335+08:00”,"Timeout”:false}}

14121:{“time”:“2025-11-05T00:52:29.23945036Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.210:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.210:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.239281976+08:00”,"Timeout”:false}}

14122:{“time”:“2025-11-05T00:52:29.239601683Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.219:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.219:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.239281976+08:00”,"Timeout”:false}}

14123:{“time”:“2025-11-05T00:52:29.239681013Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.211:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.211:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.239281976+08:00”,"Timeout”:false}}

14124:{“time”:“2025-11-05T00:52:29.239715302Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.218:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.218:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T00:56:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:52:29.239281976+08:00”,"Timeout”:false}}

```

kubectl logs -n monitor pod/my-alertmanager-1 | grep “Received alert” -n

`null`

#The **time when the alarm was successfully sent by the altermanager**  
kubectl logs -n monitor pod/my-alertmanager-0 | grep “Notify success” -n

```yaml
14130:{“time”:“2025-11-05T00:52:30.457964557Z”,“level”:“DEBUG”,“source”:“notify.go:876”,“msg”:“Notify success”,“component”:“dispatcher”,“receiver”:“email”,“integration”:“email[0]”,“aggrGroup”:“{}/{severity="critical"}/{}:{alertname="k8s-physical-node-down", instance="192.168.110.211:9100"}”,“attempts”:1,“duration”:1030128705,“alerts”:“[k8s-physical-node-down[aa504b6][active]]”}

14132:{“time”:“2025-11-05T00:52:30.517906504Z”,“level”:“DEBUG”,“source”:“notify.go:876”,“msg”:“Notify success”,“component”:“dispatcher”,“receiver”:“email”,“integration”:“email[0]”,“aggrGroup”:“{}/{severity="critical"}/{}:{alertname="k8s-physical-node-down", instance="192.168.110.210:9100"}”,“attempts”:1,“duration”:1089951271,“alerts”:“[k8s-physical-node-down[61a5ca1][active]]”}

```

kubectl logs -n monitor pod/my-alertmanager-1 | grep “Notify success” -n

`null`

**【The second batch of received alerts】**

#The **time when the altermanager receives the alert**  
kubectl logs -n monitor pod/my-alertmanager-0 | grep “Received alert” -n

```yaml
14167:{“time”:“2025-11-05T00:57:29.229045333Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.219:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.219:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.228905739+08:00”,"Timeout”:false}}

14168:{“time”:“2025-11-05T00:57:29.22933875Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.211:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.211:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.228905739+08:00”,"Timeout”:false}}

14169:{“time”:“2025-11-05T00:57:29.229418053Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.218:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.218:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.228905739+08:00”,"Timeout”:false}}

14170:{“time”:“2025-11-05T00:57:29.229561876Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.210:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-1】”,“summary”:“k8s物理节点-192.168.110.210:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.228905739+08:00”,"Timeout”:false}}

14171:{“time”:“2025-11-05T00:57:29.231702787Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.211:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.211:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.231382656+08:00”,"Timeout”:false}}

14172:{“time”:“2025-11-05T00:57:29.231815435Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.218:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.218:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.231382656+08:00”,"Timeout”:false}}

14173:{“time”:“2025-11-05T00:57:29.231858085Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.210:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.210:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.231382656+08:00”,"Timeout”:false}}

14174:{“time”:“2025-11-05T00:57:29.231881344Z”,“level”:“DEBUG”,“source”:“dispatch.go:165”,“msg”:“Received alert”,“component”:“dispatcher”,“alert”:{“labels”:{“alertname”:“k8s-physical-node-down”,“instance”:“192.168.110.219:9100”,“job”:“k8s_node_exporter”,“severity”:“critical”},“annotations”:{“source_expression”:“up{job="k8s_node_exporter"} == 1”,“source_prometheus”:“【replica: my-prometheus-thanos-sidecar-0】”,“summary”:“k8s物理节点-192.168.110.219:9100-停止工作。当前采集值为: 1”},“startsAt”:“2025-11-04T02:52:29.224Z”,“endsAt”:“2025-11-05T01:01:29.224Z”,“generatorURL”:“http://192.168.110.210:39090/graph?g0.expr=up{job%3D"k8s_node_exporter"}+%3D%3D+1&g0.tab=1",“UpdatedAt”:“2025-11-05T08:57:29.231382656+08:00”,"Timeout”:false}}

```

kubectl logs -n monitor pod/my-alertmanager-1 | grep “Received alert” -n

`null`

#The **time when the alarm was successfully sent by the altermanager**  
kubectl logs -n monitor pod/my-alertmanager-0 | grep “Notify success” -n

```yaml
14180:{“time”:“2025-11-05T00:57:30.45846932Z”,“level”:“DEBUG”,“source”:“notify.go:876”,“msg”:“Notify success”,“component”:“dispatcher”,“receiver”:“email”,“integration”:“email[0]”,“aggrGroup”:“{}/{severity="critical"}/{}:{alertname="k8s-physical-node-down", instance="192.168.110.219:9100"}”,“attempts”:1,“duration”:1029726795,“alerts”:“[k8s-physical-node-down[eb4844f][active]]”}

14182:{“time”:“2025-11-05T00:57:30.53294889Z”,“level”:“DEBUG”,“source”:“notify.go:876”,“msg”:“Notify success”,“component”:“dispatcher”,“receiver”:“email”,“integration”:“email[0]”,“aggrGroup”:“{}/{severity="critical"}/{}:{alertname="k8s-physical-node-down", instance="192.168.110.218:9100"}”,“attempts”:1,“duration”:1104358612,“alerts”:“[k8s-physical-node-down[e95f07a][active]]”}

```

kubectl logs -n monitor pod/my-alertmanager-1 | grep “Notify success” -n

`null`

\*\*\*

**Question：**

\*\*1.\*\*In the above section\*\*【The first batch of received alerts】\*\*, the configuration is as follows

The alertmanager received a total of 8 alerts.  
Why were only two alerts successfully sent to the email?  
{alertname=“k8s-physical-node-down”, instance=“192.168.110.211:9100”}  
{alertname=“k8s-physical-node-down”, instance=“192.168.110.210:9100”}

I understand that after the deduplication process in the altermanager cluster, four alerts should be sent to the email.  
_{alertname=“k8s-physical-node-down”, instance=“192.168.110.211:9100”}  
{alertname=“k8s-physical-node-down”, instance=“192.168.110.210:9100”}  
{alertname=“k8s-physical-node-down”, instance=“192.168.110.218:9100”}  
{alertname=“k8s-physical-node-down”, instance=“192.168.110.219:9100”}_

 ![image](https://europe1.discourse-cdn.com/flex017/uploads/prometheus/original/2X/7/7860e4647f415109cc30a117b03d8da9e1b894bd.png)

\*\*2.\*\*In the above section\*\*【The second batch of received alerts】\*\*, the configuration is as follows

The alertmanager received a total of 8 alerts.  
Why were only two alerts successfully sent to the email?

{alertname=“k8s-physical-node-down”, instance=“192.168.110.218:9100”}  
{alertname=“k8s-physical-node-down”, instance=“192.168.110.219:9100”}

And it has been five minutes since the last successful transmission of two alerts.  
{alertname=“k8s-physical-node-down”, instance=“192.168.110.211:9100”}  
{alertname=“k8s-physical-node-down”, instance=“192.168.110.210:9100”}

Why is this? I don’t understand how they work.

After the received 8 alerts were deduplicated by the altermanager, why weren’t all 4 of them sent together? Instead, it sends only 2 messages each time, with an interval of 5 minutes between each transmission.

Why is the phenomenon like this? What kind of working logic is behind it? I don’t quite understand.

---

<div class="post-metadata">

**Author:** ![cscowx](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/cscowx/32/1220_2.png) [@cscowx](https://discuss.prometheus.io/u/cscowx)\
**Post date:** [November 5, 2025, 3:07am UTC](https://discuss.prometheus.io/t/the-issue-regarding-the-frequency-of-sending-alerts-from-prometheus-to-alertmanager-and-then-to-emails/3477/2 "2025-11-05T03:07:59Z")

</div>

Addition  
【The second batch of received alerts】  
The second batch of received alerts images

 ![image](https://europe1.discourse-cdn.com/flex017/uploads/prometheus/original/2X/b/bb2e4f9982417c5de1bc21d62140f23d666a95b6.png)