Hi,
I would like to create node/service parent-child dependencies, so we won’t receive unnecessary alerts from services running on node, when the node itself is down.
How to do that?
L.
You can do this using inhibition: Alertmanager | Prometheus Configuration | Prometheus
Thanks for reply.
I looked at inhibit rules already, but I am lacking some good examples.
Typically you don’t need this explicit logic in Prometheus.
Prometheus alerts on the presence of data. If the node is down, the service producing the data for alerting will have no data and therefore not alert.
For example, if you have an alert for application errors, and the service or node is down, it will not alert since there is no data to positively alert on.
1 Like
While that logic is good for host missing, there other reasons, like a database is down, all applications alerts will fail. For that inhibitions is the way to go. To better use that feature it is a good idea to first do a diagram of services dependencies so you better understand and control the inhibitions and do not miss some and not silence wrong alerts