# Recovering counter value after Prometheus client crash

**URL:** <https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694>\
**Category:** General Help/Support\
**Created:** [January 6, 2022, 8:57am UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694 "2022-01-06T08:57:52Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![noamz](https://avatars.discourse-cdn.com/v4/letter/n/278dde/32.png) [@noamz](https://discuss.prometheus.io/u/noamz)\
**Post date:** [January 6, 2022, 8:57am UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694/1 "2022-01-06T08:57:52Z")

</div>

Hello,  
I have a Prometheus client that counts the number of events processed by my application. I want to be able to recover that counter after my application crash and restarts.  
For example: say that 500 events were counted before the crash. When my application restarts, I want to set my counter to start counting from 500 (not 0).  
To do so, I want to query Prometheus server for the last known value (500 in my example).  
Is it possible to do it?

Attached is a screenshot that shows the counter’s query in the server before and after the crash. You can see that the counter’s value before the crash is 500 and after the crash is 0 (while I want it to be 500).

Thanks.

 ![Screenshot from 2022-01-06 10-24-46](https://europe1.discourse-cdn.com/flex017/uploads/prometheus/original/1X/abb3636d263f1ed33233eeca5424d10682ffd7d8.png)

---

<div class="post-metadata">

**Author:** ![stuart](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/stuart/32/18_2.png) [@stuart](https://discuss.prometheus.io/u/stuart)\
**Post date:** [January 6, 2022, 10:45am UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694/2 "2022-01-06T10:45:21Z")

</div>

This isn’t something that is generally done or recommended. Counters are expected to reset to 0 when the application restarts.

---

<div class="post-metadata">

**Author:** ![noamz](https://avatars.discourse-cdn.com/v4/letter/n/278dde/32.png) [@noamz](https://discuss.prometheus.io/u/noamz)\
**Post date:** [January 6, 2022, 10:51am UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694/3 "2022-01-06T10:51:03Z")

</div>

Hi @stuart,  
Thank you for your response.  
I understand that it’s not a common practice with counters. I may replace the counter with a gauge.  
Actually I found a way to get the result I want with the following query: `max(max_over_time(dpu_src_event_count[1h]))`  
My only problem now is to figure out how to send this query from the client (I am using prometheus-cpp). It seems that my best option is using HTTP API.  
What do you think?

---

<div class="post-metadata">

**Author:** ![stuart](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/stuart/32/18_2.png) [@stuart](https://discuss.prometheus.io/u/stuart)\
**Post date:** [January 6, 2022, 11:02am UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694/4 "2022-01-06T11:02:50Z")

</div>

As I said this wouldn’t be recommended. Why are you trying to do this?

---

<div class="post-metadata">

**Author:** ![noamz](https://avatars.discourse-cdn.com/v4/letter/n/278dde/32.png) [@noamz](https://discuss.prometheus.io/u/noamz)\
**Post date:** [January 6, 2022, 11:04am UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694/5 "2022-01-06T11:04:51Z")

</div>

I have a client application that may fail and restart.  
After the application restarts I want to continue counting from the last known value.

---

<div class="post-metadata">

**Author:** ![stuart](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/stuart/32/18_2.png) [@stuart](https://discuss.prometheus.io/u/stuart)\
**Post date:** [January 6, 2022, 2:41pm UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694/6 "2022-01-06T14:41:40Z")

</div>

I mean why are you wanting to do that?

All alerts, dashboards, etc. should be using the rate so the actual value of the counter doesn’t matter. rate() handles counter resets automatically so everything should “just work”.

---

<div class="post-metadata">

**Author:** ![noamz](https://avatars.discourse-cdn.com/v4/letter/n/278dde/32.png) [@noamz](https://discuss.prometheus.io/u/noamz)\
**Post date:** [January 9, 2022, 7:47am UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694/7 "2022-01-09T07:47:07Z")

</div>

I want to count the number of events processed by my system.  
I don’t think that the rate function will help me, but perhaps if there’s a way to sum all the previous counter results when my application comes up after a crash, that can be useful for me instead of setting the counter to a non-zero value.  
To be more specific, if you look at the screenshot in my first post, is it possible to sum the 2 queries there?

---

<div class="post-metadata">

**Author:** ![drthornt](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/drthornt/32/1050_2.png) [@drthornt](https://discuss.prometheus.io/u/drthornt)\
**Post date:** [September 17, 2024, 7:07pm UTC](https://discuss.prometheus.io/t/recovering-counter-value-after-prometheus-client-crash/694/8 "2024-09-17T19:07:54Z")

</div>

So you want a graph that just goes up and up and up? If you want to graph a count over time, then it will just go up and up and up yeah?

Typically when you emit a count of activities metric you are concern with the _rate_ at which that activity occurs, which is to say you measure the change in count over time. the rate() function handles a lot of the details of that sort of metric for you. For example counter resets, and new pods doing the same work.

In you exmaple you might try simply doing “rate(dpu\_src\_event\_count[5m]) by (pod)” and see how it _automatically_ handles when a pod gets restarted .

or maybe you want a songle line : "sum(rate(dpu\_src\_event\_count[5m]) )

try those and let us know if they are closer to what you want.
