# Subtraction, same label, different value

**URL:** <https://discuss.prometheus.io/t/subtraction-same-label-different-value/201>\
**Category:** PromQL\
**Created:** [May 6, 2021, 7:46am UTC](https://discuss.prometheus.io/t/subtraction-same-label-different-value/201 "2021-05-06T07:46:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![tonymurphy](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/tonymurphy/32/113_2.png) [@tonymurphy](https://discuss.prometheus.io/u/tonymurphy)\
**Post date:** [May 6, 2021, 7:46am UTC](https://discuss.prometheus.io/t/subtraction-same-label-different-value/201/1 "2021-05-06T07:46:28Z")

</div>

I’m using Feign/Hystrix, micrometer, prometheus.

The following query works, but I need to modify it and I don’t know how:

```auto
sum by(group, key) (increase(hystrix_execution_total{event="exception_thrown"}[1m])) / sum by(group, key) (increase(hystrix_execution_terminal_total[1m])) * 100 >= 5

```

Basically `exception_thrown` is one event emitted by Hystrix, but Hystrix also emits another metric `bad_request` which signifies a client error (4…). To get an accurate measure of upstream server errors, I need to subtract two metrics, which have the same labels, but different values, and then get a rate per minute to alert on

The following doesn’t work, No datapoints found.

```auto
hystrix_execution_total{job="auth",key="authenticate",event="exception_thrown"} - hystrix_execution_total{job="auth",key="authenticate",event="bad_request"}

```

any ideas if it is possible to achieve this?

Here is a graph to help explain the problem, it’s hystrix calls related to authentication, most errors are users submitting wrong credentials (bad request), I need to distinguish those from our authentication server 5… errors

 ![Screenshot 2021-05-05 at 14.27.00](https://europe1.discourse-cdn.com/flex017/uploads/prometheus/original/1X/6fc25b81cb2365f667c65ed3cf16d8f6f6f44ce1.png)

---

<div class="post-metadata">

**Author:** ![roidelapluie](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/roidelapluie/32/6_2.png) [@roidelapluie](https://discuss.prometheus.io/u/roidelapluie)\
**Post date:** [May 11, 2021, 9:33am UTC](https://discuss.prometheus.io/t/subtraction-same-label-different-value/201/2 "2021-05-11T09:33:44Z")

</div>

Can you try:

```auto
hystrix_execution_total{job="auth",key="authenticate",event="exception_thrown"} - ignoring(event) hystrix_execution_total{job="auth",key="authenticate",event="bad_request"}

```

if event is different on both sides, you will need to ask Prometheus not to use it to compare.

---

<div class="post-metadata">

**Author:** ![tonymurphy](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/tonymurphy/32/113_2.png) [@tonymurphy](https://discuss.prometheus.io/u/tonymurphy)\
**Post date:** [May 11, 2021, 3:07pm UTC](https://discuss.prometheus.io/t/subtraction-same-label-different-value/201/3 "2021-05-11T15:07:54Z")

</div>

Unfortunately I still get no data points.

I think I need event, as I need to subtract based on that labels values…

Please find attached 3 screenshots - one with ignoring clause, and one for exceptions\_thrown and bad\_request (had to combine into one to workaround new user restriction of 1 image )

 ![Screenshot 2021-05-11 at 16.02.11](https://europe1.discourse-cdn.com/flex017/uploads/prometheus/original/1X/b7875cd2fb4edf0fdde78cbf893ff1d9af7bddb6.jpeg)

---

<div class="post-metadata">

**Author:** ![roidelapluie](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/roidelapluie/32/6_2.png) [@roidelapluie](https://discuss.prometheus.io/u/roidelapluie)\
**Post date:** [May 11, 2021, 3:10pm UTC](https://discuss.prometheus.io/t/subtraction-same-label-different-value/201/4 "2021-05-11T15:10:32Z")

</div>

It seems that terminal is also different. Can you try ignoring (terminal, event) ?

---

<div class="post-metadata">

**Author:** ![tonymurphy](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/tonymurphy/32/113_2.png) [@tonymurphy](https://discuss.prometheus.io/u/tonymurphy)\
**Post date:** [May 11, 2021, 6:34pm UTC](https://discuss.prometheus.io/t/subtraction-same-label-different-value/201/6 "2021-05-11T18:34:20Z")

</div>

WOW, thank you very much, that has fixed it 🙏

---

<div class="post-metadata">

**Author:** ![roidelapluie](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/roidelapluie/32/6_2.png) [@roidelapluie](https://discuss.prometheus.io/u/roidelapluie)\
**Post date:** [May 11, 2021, 6:48pm UTC](https://discuss.prometheus.io/t/subtraction-same-label-different-value/201/7 "2021-05-11T18:48:17Z")

</div>

```auto
sum(hystrix_execution_total{job="auth",key="authenticate",event="exception_thrown"} ) without(event,terminal) - sum(hystrix_execution_total{job="auth",key="authenticate",event="bad_request"}) without(event,terminal) 

```

Is better then.

---

<div class="post-metadata">

**Author:** ![tonymurphy](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/tonymurphy/32/113_2.png) [@tonymurphy](https://discuss.prometheus.io/u/tonymurphy)\
**Post date:** [May 12, 2021, 7:45am UTC](https://discuss.prometheus.io/t/subtraction-same-label-different-value/201/8 "2021-05-12T07:45:40Z")

</div>

Thanks, really appreciate the support