I extract statistics from Jenkins with the Prometheus Metrics plugin.
I have created a query in PromQL to check if a Jenkins job builds time has increased by 50% from the average successful build time:
default_jenkins_builds_last_build_duration_milliseconds > 1.5 * (avg_over_time(default_jenkins_builds_last_build_duration_milliseconds[180d]) and default_jenkins_builds_last_build_result_ordinal == 0)
However there is a problem with this query. The result is getting diluted over time because the query keeps adding each value from the time-series to the total average result. There may be values that haven’t changed but they keep adding.
I expected to create a query that calculates the ‘delta’ from the current successful build time against the previous one, but there doesn’t seem to be a metric that represents the previous build (or I can’t find it), so I ended up using the average_over_time.
I have also tried to calculate the delta with the offset modifier by one minute (because the Prometheus scrapes the Jenkins exporter every 1 minute), but the problem there is that some times the time-series returns Nan results and it cant calculate the time delta from each build. I was expecting that in a graph i would see a line with some ups and downs if the build time increased or decreased but NaN values break this graph.
How can this query be refactored in order to yield the expected result ???