Hello,
We use Prometheus to monitor a k8s cluster with several applications in it, and Karma to let monitoring team have a overview on alerts.
I’ve got a case I can’t solve, but I don’t know if the solution may be into Prometheus rules, Alert Manager or Karma …
Each pod related to a given application defines a label “application” with the application name as value.
Alerting rules are defined for situations like “pod crashes”, and in this case, the label is added to the alert thru a join with kube_pod_labels{} metric.
Monitoring team may use runbooks to find what to do in case of an alert, each runbook being on an internal wiki.
Application xyz is supposed to have its runbook into an URL like our.web.site/wiki/xyz
Thus, what I’d like to do is to display this link in Karma alerts, but I couldn’t find a clean way to do this.
- Prometheus allows relabelling, thus I can transform label
application: xyzintoapplication: web.site/wiki/xyzbut it displays the label “as is” into karma, without link interpretation - annotations containing urls are transformed into links, but to transform this label into an annotation, I have to define the annotation with an “if” explicitly for every alert rule.
- furthermore, if the alert falls in a case where the label is not defined (pod crash not related with an application for example), the annotation is generated empty, which results in a empty tag in Karma.
I didn’t find any way to relabel labels into annotations, or suppress empty annotations, or convert label into links.
Do you see any trick to achieve this ?