# Prometheus alert rule doesn't honor changed FOR clause

**URL:** <https://discuss.prometheus.io/t/prometheus-alert-rule-doesnt-honor-changed-for-clause/1268>\
**Category:** Prometheus server\
**Created:** [October 31, 2022, 8:39am UTC](https://discuss.prometheus.io/t/prometheus-alert-rule-doesnt-honor-changed-for-clause/1268 "2022-10-31T08:39:36Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![civa](https://avatars.discourse-cdn.com/v4/letter/c/b9e5f3/32.png) [@civa](https://discuss.prometheus.io/u/civa)\
**Post date:** [October 31, 2022, 8:39am UTC](https://discuss.prometheus.io/t/prometheus-alert-rule-doesnt-honor-changed-for-clause/1268/1 "2022-10-31T08:39:36Z")

</div>

Hi all,

I have a simple alert rule for my certificate expiration which goes like this (metrics come from `telegraf`):

```auto
groups:
  - name: ssl-alerts
    rules:
      - alert: cert_expiry_critical
        expr: x509_cert_expiry < 604800
        for: 60m
        annotations:
          summary: 'SSL/TLS certificate expires in less than a week'
          description: 'SSL/TLS certificate for {{ $labels.host }} expires in {{ .Value | humanizeDuration }}'
        labels:
          severity: "critical"
          env: test

```

Now, I’ve realized that alerts are too frequent thus I changed `FOR` clause to be `24h` to reduce alert notification spamming. Changed alert looks like this:

```auto
groups:
  - name: ssl-alerts
    rules:
      - alert: cert_expiry_critical
        expr: x509_cert_expiry < 604800
        for: 24h
        annotations:
          summary: 'SSL/TLS certificate expires in less than a week'
          description: 'SSL/TLS certificate for {{ $labels.host }} expires in {{ .Value | humanizeDuration }}'
        labels:
          severity: "critical"
          env: test

```

but I still receive alerts on `60m` or so. Already restarted `prometheus` and `alertmanager` daemons but no avail. Can anyone please explain why is that and how to properly increase alerts delaying in this case?

Thanks

---

<div class="post-metadata">

**Author:** ![stuart](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/stuart/32/18_2.png) [@stuart](https://discuss.prometheus.io/u/stuart)\
**Post date:** [October 31, 2022, 9:26am UTC](https://discuss.prometheus.io/t/prometheus-alert-rule-doesnt-honor-changed-for-clause/1268/2 "2022-10-31T09:26:24Z")

</div>

The `for:` in your alert rule just determines how long a condition has to be true before an alert starts firing. So it would delay the initial alert, but wouldn’t have anything to do with repeated notifications for alerts that keep firing.

Prometheus just generates alerts. It is up to Alertmanager to handle any notifications, including decisions around if or when to send additional notifications for alerts that have been firing for a while. So you will need to look at your Alertmanager config.

---

<div class="post-metadata">

**Author:** ![civa](https://avatars.discourse-cdn.com/v4/letter/c/b9e5f3/32.png) [@civa](https://discuss.prometheus.io/u/civa)\
**Post date:** [October 31, 2022, 11:00am UTC](https://discuss.prometheus.io/t/prometheus-alert-rule-doesnt-honor-changed-for-clause/1268/3 "2022-10-31T11:00:27Z")

</div>

> [@stuart](#):
>
> erates alerts. It is up to Alertmanager to handle any notifications, including decisions around if or when to send additional notifications for alerts that have been firing for a while. So you will need to look at your Alertmanager config.

Great, thanks for the explanation stuart - much clearer now.

It would be perfect if I could set `repeat_interval` as per individual alert, but seems like thats not supported.

So, on the `alertmanager` side, should I define `inhibit_rule` for the alert from the above to make it fire less frequently? Is that a best practice? I would like to only delay a single alert without affecting others. Please let me know if there is a better approach.

Thanks again!

---

<div class="post-metadata">

**Author:** ![stuart](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/stuart/32/18_2.png) [@stuart](https://discuss.prometheus.io/u/stuart)\
**Post date:** [October 31, 2022, 11:15am UTC](https://discuss.prometheus.io/t/prometheus-alert-rule-doesnt-honor-changed-for-clause/1268/4 "2022-10-31T11:15:36Z")

</div>

Inhibition rules are something different - they are to totally prevent an alert firing if another one is. The example use case would be a if there is say a central database failure having hundreds of alerts from each application would just obscure the actual issue and slow down response, so inhibit all the app alerts and just get the database alert.

For your use case you want an additional `route` with a custom `repeat_interval` value. You can then send things to that route via something like a label - set a “don’t repeat” label in the alert and set the route matching for that value.

---

<div class="post-metadata">

**Author:** ![civa](https://avatars.discourse-cdn.com/v4/letter/c/b9e5f3/32.png) [@civa](https://discuss.prometheus.io/u/civa)\
**Post date:** [October 31, 2022, 3:21pm UTC](https://discuss.prometheus.io/t/prometheus-alert-rule-doesnt-honor-changed-for-clause/1268/5 "2022-10-31T15:21:23Z")

</div>

Thanks @stuart for clarifying that out once again, but I couldn’t make it work using additional route, alerts are still being fired after `1h` + `group_interval` - here are my configs (decreased values from `24h` to `3h` for testing):

`alertmanager.yml `

```auto
route:
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  group_by: ['alertname']
  receiver: 'slack-alerts'

  routes:
    # additional route for delaying alerts
    - match:
        env: test
        severity: warning|critical
        postpone: true 
        delay: 3h
      receiver: 'slack-alerts'
      repeat_interval: 3h
      continue: false

    - match:
        env: test
        severity: warning|critical
      receiver: 'slack-alerts'

receivers:
...

```

And related `prometheus.yml`:

```auto
groups:
  - name: ssl-alerts
    rules:
      - alert: cert_expiry_critical
        expr: x509_cert_expiry < 604800
        for: 60m
        annotations:
          summary: 'SSL/TLS certificate expires in less than a week'
          description: 'SSL/TLS certificate for {{ $labels.san }} ({{ $labels.source }}) expires in {{ .Value | humanizeDuration }}'
        labels:
          severity: 'critical'
          postpone: true # <- added
          delay: 3h # <- added
          env: test 

```

Am I doing something wrong here?

Thanks!

---

<div class="post-metadata">

**Author:** ![civa](https://avatars.discourse-cdn.com/v4/letter/c/b9e5f3/32.png) [@civa](https://discuss.prometheus.io/u/civa)\
**Post date:** [November 8, 2022, 10:35am UTC](https://discuss.prometheus.io/t/prometheus-alert-rule-doesnt-honor-changed-for-clause/1268/6 "2022-11-08T10:35:18Z")

</div>

Can someone help with this? Pretty please?