Alert rules of increasing severity

PromQL
1

I’m a newbie to Prometheus, so I’m sorry if I’ve missed this in the documentation …

In my “home-brew” network, Prometheus is collecting battery charge percentage, and I would like to create an alert for batteries below, say, 50% (i.e. “go and buy some more batteries”) and a more critical alert for batteries below, say, 25% (i.e. “change the batteries now!”).

I’ve started on my alert rules as follows:

groups:
  - name: Battery
    rules:
      - alert: Low
        expr: 25 > homeassistant_sensor_battery_percent and homeassistant_sensor_battery_percent <= 50
      - alert: VeryLow
        expr: homeassistant_sensor_battery_percent <= 25

I expect to flesh this out more fully, but even this feels clumsy:

  • Repetition of homeassistant_sensor_battery_percent;
  • Repetition of the 25 %threshold;
  • Easy to mess up to relational operators so the overlap or, worse, miss boundary values.

I wondered whether I could use the “limit” field set to 1 and order the rules from the most critical to least critical, such as:

groups:
  - name: Battery
    limit: 1
    rules:
      - alert: VeryLow
        expr: homeassistant_sensor_battery_percent <= 25
      - alert: Low
        expr: homeassistant_sensor_battery_percent <= 50

However, my reading of the “limit” field makes me feel that it is intended to manage a cascading failure case rather than “stop after the first matching rule”.

I couldn’t find any thing like python 50 <= homeassistant_sensor_battery_percent < 25 or some such. Nor could I find a way to define names constants, say, lower_limit: 25 that could be used inside the expr statements.

My use case is batteries, but I would imagine that similar rules might be wanted for many different resource usage gauges like disk utilisation, bandwidth utilisation, etc. Is there a pattern for this kind of thing in Prometheus? Any pointers would be welcome.

Many thanks,

Steve