Trigger an alarm when no data received or no more data received


I am building a alerting mechanism an also a graph when I need to inform when there is no data coming since 6 hours (missing or no increase). I need to count only office opening hours (from 8am to 7pm, no week-ends).
So if I had no data yesterday since 4pm. I will be notified only at 11am today.

I have something that is pretty good for missing data (only for weekday opening hours for the moment)

absent_over_time(data_flux{country="Country"}[6h]) if (hour() > 8 < 19) 

But I am trying to do the same when there is no increase.

I tried to look at other _over_time but didn’t found the right function. I also tried to offset, but with the “only opening hours” it is not easy to find a path.

If I try something like


It never goes to 0.

I think I might have found the solution but it is not yet ok with the “only working hours” interval.

sum(max_over_time(increase(data_flux{country="country"}[$__rate_interval])[6h]) if (hour() > 8 < 18))

The fact that the condition does not apply during the closed hours but should continue where it left off I think is not going to be really easy to overcome. What I mean by this is the fact that if you start missing data 2 hours before closing time Friday, when opening time Mon arrives the “remaining time” before trigger is 4 hours rather than six. I don’t see an easy way to overcome this, at least at first glance.

For alerting, you can take advantage of the for: <time-range> on the rule definition while leveraging [mute|active]_time_intervals in the alert manager configuration to achieve what you want.
By doing this you wouldn’t need to use *_over_time() functions nor hour() conditionals.


[1] Alerting Docs
[2] Alert Rules Docs

1 Like

Thanks for your answer.

absent_over_time works well but I need to have missing data instead of 0. Do you know how to trick data an replace 0 to NoData ?

you can use a condition (like promql-query > 0). That will basically remove all datapoints from that time-series that not match the condition.

1 Like