Don't receive alert messages on discord

[TornaxO7]

Hello guys! I’m a bit frustrated at the moment to get alert-notifications on discord.

Context

I’m on NixOS and I’m configuring prometheus by using nix, however, I’ll post the generated config files here and not the config files written in nix.

I followed this tutorial and even got some messages on discord in the beginning but after the first 4 messages I didn’t retrieve any messages anymore…

Setup

So, here’s my config:

test_alert.yaml

Or rather /nix/store/3dpnn32gyxjknkng37lljx1354m43xrh-rules-checkrules-checked as in the second entry in rule_files from the prometheus config file:

groups:
- name: demo-service-alerts
  rules:
  - alert: DemoServiceHighErrorRate
    expr: |
      (
        sum without(status, instance) (
          rate(demo_api_request_duration_seconds_count{status=~"5..",job="demo"}[1m])
        )
      /
        sum without(status, instance) (
          rate(demo_api_request_duration_seconds_count{job="demo"}[1m])
        ) * 100 > 0.5
      )
    for: 1m
    labels:
      severity: critical
    annotations:
      title: 'High 5xx rate for {{ $labels.method }} on {{ $labels.path }}'
      description: 'The 5xx error rate for path {{$labels.path}} with method {{ $labels.method }} in {{ $labels.job }} is {{ printf "%.2f" $value }}%.'

Prometheus config file

From the prometheus → status → configuration page:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - follow_redirects: true
    enable_http2: true
    scheme: http
    timeout: 10s
    api_version: v2
    static_configs:
    - targets:
      - localhost:9093
rule_files:
- /nix/store/k7a1na4drygqwwxsd1nkb96q317r09ri-rules-checkrules-checked
- /nix/store/3dpnn32gyxjknkng37lljx1354m43xrh-rules-checkrules-checked
- /nix/store/v0wszamxlx5xq8qjx7lh9kpg9h52xd40-rules-checkrules-checked
scrape_configs:
- job_name: node
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 15s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  static_configs:
  - targets:
    - ' localhost:9100'
- job_name: zfs
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 15s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  static_configs:
  - targets:
    - localhost:9134
- job_name: demo
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 15s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  static_configs:
  - targets:
    - demo.promlabs.com:10000
    - demo.promlabs.com:10001
    - demo.promlabs.com:10002

alert-manager.yaml

The config file for the alert-manager. Its content is in json.

{
  "receivers": [
    {
      "discord_configs": [
        {
          "send_resolved": true,
          "webhook_url_file": "/run/agenix/discord-webhook"
        }
      ],
      "name": "discord"
    }
  ],
  "route": {
    "group_by": [
      "alertname",
      "job"
    ],
    "group_interval": "5m",
    " group_wait": "30s",
    "receiver": "discord",
    "repeat_interval": "3h"
  }
}

I also verified the URI in /run/agenix/discord-webhook and checked if the hook even worked, thanks to this stackoverflow answer.

Prometheus also seems to register the events:

but I’m still not getting any discord messages (I’ve created a new discord server with a webhook, assigned to a channel and as I said, the webhook works with the curl command as described in the stackoverflow answer which I linked).

Any help would be appreciated