Hello
I am building a alerting mechanism an also a graph when I need to inform when there is no data coming since 6 hours (missing or no increase). I need to count only office opening hours (from 8am to 7pm, no week-ends).
So if I had no data yesterday since 4pm. I will be notified only at 11am today.
I have something that is pretty good for missing data (only for weekday opening hours for the moment)
absent_over_time(data_flux{country="Country"}[6h]) if (hour() > 8 < 19)
But I am trying to do the same when there is no increase.
I tried to look at other _over_time but didn’t found the right function. I also tried to offset, but with the “only opening hours” it is not easy to find a path.
If I try something like
sum(max_over_time(data_flux{country="Country"}[2h]))
It never goes to 0.
I think I might have found the solution but it is not yet ok with the “only working hours” interval.
sum(max_over_time(increase(data_flux{country="country"}[$__rate_interval])[6h]) if (hour() > 8 < 18))
The fact that the condition does not apply during the closed hours but should continue where it left off I think is not going to be really easy to overcome. What I mean by this is the fact that if you start missing data 2 hours before closing time Friday, when opening time Mon arrives the “remaining time” before trigger is 4 hours rather than six. I don’t see an easy way to overcome this, at least at first glance.
For alerting, you can take advantage of the for: <time-range> on the rule definition while leveraging [mute|active]_time_intervals in the alert manager configuration to achieve what you want.
By doing this you wouldn’t need to use *_over_time() functions nor hour() conditionals.
Source:
[1] Alerting Docs
[2] Alert Rules Docs
1 Like
Thanks for your answer.
absent_over_time works well but I need to have missing data instead of 0. Do you know how to trick data an replace 0 to NoData ?
you can use a condition (like promql-query > 0). That will basically remove all datapoints from that time-series that not match the condition.
1 Like
