Hello,
I have a Prometheus client that counts the number of events processed by my application. I want to be able to recover that counter after my application crash and restarts.
For example: say that 500 events were counted before the crash. When my application restarts, I want to set my counter to start counting from 500 (not 0).
To do so, I want to query Prometheus server for the last known value (500 in my example).
Is it possible to do it?
Attached is a screenshot that shows the counter’s query in the server before and after the crash. You can see that the counter’s value before the crash is 500 and after the crash is 0 (while I want it to be 500).
Hi @stuart,
Thank you for your response.
I understand that it’s not a common practice with counters. I may replace the counter with a gauge.
Actually I found a way to get the result I want with the following query: max(max_over_time(dpu_src_event_count[1h]))
My only problem now is to figure out how to send this query from the client (I am using prometheus-cpp). It seems that my best option is using HTTP API.
What do you think?
All alerts, dashboards, etc. should be using the rate so the actual value of the counter doesn’t matter. rate() handles counter resets automatically so everything should “just work”.
I want to count the number of events processed by my system.
I don’t think that the rate function will help me, but perhaps if there’s a way to sum all the previous counter results when my application comes up after a crash, that can be useful for me instead of setting the counter to a non-zero value.
To be more specific, if you look at the screenshot in my first post, is it possible to sum the 2 queries there?
