Trying to set up alertmanager on a running prometheus instance. I can see alerts via Grafana, so I know alerts are happening. However, there are no log entries (as there are with other ideas) that say “alertmanager is not found” or "alertmanager not working because ". I have absolutely no indication that alertmanager is not working except for the lack of alerts.
I would like to open a github issue about this, but I need to know why alertmanager is failing to alert first, so I can ask for the inclusion of any log message that will give users a fighting chance to find out what is wrong.
I have the following in prometheus config:
alerting:
alertmanagers:
- static_configs:
- targets:
- 1.2.3.4:8765
I start alertmanager with
--cluster.listen-address='' --web.listen-address= 1.2.3.4:8765 --log.level=debug
Why isn’t this working?
Thanks in advance.
Do you see your alertmanager in the Runtime page of Prometheus?
Do you see alerts in the Alertmanager status page?
What are your prometheus and alertmanager versions?
Ok, yes I see alertmanager in the runtime page. I waited for an actual alert (I have grafana monitoring ALERTS_FOR_STATE atm) and -now- I see the same alert but not in the status page…on the alerts page.
I am running 0.20.0 of alertmanager and 2.26.0 of prometheus.
I had no indication (that I was familiar with) that alertmanager was even connected and working. Now that I see an alert, I think I can continue with trying to configure a few destinations.
To clarify, do you see the alerts in the alertmanager UI?
I only saw one alert briefly.
My issue is that when I restart the alertmanager, I don’t see a log message indicating that. In fact, the log file is purged apparently, so I am left with zero feedback to ensure that a configuration file change was accepted or works. Any previous log file is deleted, which is rather surprising. Apparently, the only way I can test alertmanager is to generate an alert myself with the alert manager UI open.