Hi all:
I am becoming increasingly frustrated with Prometheus in general and its Alertmanager in particular.
Even though I have modest monitoring and alerting needs, Prometheus’ Alertmanager lacks flexibility for common scenarios. Here is a good description of the main problem:
None of the solutions presented there is entirely satisfactory. The usual work-around of adding labels when scraping in order to check them later in the alert rules has the addional problem described here:
Non-cardinal labels
https://github.com/prometheus/prometheus/issues/17057
Some other people have realised of this too:
Those guys have gone to the extent of writing a custom pre-processor to generate alert rules.
Furthermore, I have an old “enterprise” Ethernet switch which I need to restart every week. I must not be the only one, because the switch has a built-in option to do schedule recurring restarts in its web interface. The switch takes around 3 minutes to restart, and then I get lots of alerts.
It turns out that working around this problem is difficult. There is no easy way to prevent creation of some alarms for some targets on Tuesday between 2:00 and 2:05. There are a number of tricks I have to learn, but nothing terse and direct.
I also lost time because of a stupid limitation and bad documentation which has been known for years:
Use the ‘mail’ tool instead of SMTP
https://github.com/prometheus/alertmanager/issues/4505
That is a minor problem in comparison, but it is a good example of simple things that are not deemed important and get silly excuses. The focus seems to be on docker-based enterprise needs. Even when I created a separate bug just to properly document the simple limitation, it still does not get great attention:
Document that smtp_smarthost does not support port 465
https://github.com/prometheus/alertmanager/issues/4507
It is apparently fine to let newcomers fall in the known trap for years.
Yet another limitation: Prometheus does not support constants or variables in its config syntax, nor in the Alertmanager config, so I have repeat names and values all over the place.
I short: obvious limitations and not a good perspective.
Some days ago I heard in a conference that Grafana can also generate alarms and send e-mail notifications. Before I waste more time in yet another rabbit whole, I wanted to ask here:
a) Is Grafana’s alerting implementation a worthy replacement to Prometheus’ Alertmanager? Or is it full with little annoying limitations too?
I heard that you must configure the alerts with the UI, that there is no config file, but I have seen that you can actually backup and restore the alarm configuration as a text file.
I also heard that the first versions where too simple, but that Grafana’s alerting has been improved in the mean time.
I am asking here in case someone coming from Grafana could share some of its drawbacks.
b) Are there any other alerting alternatives worth considering?
Thanks in advance,
rdiez