Is it a right practice to use empty label value in instrumentation?

General Help/Support
1

I want to add a label for the request to hint that the system is congested. I think I can instrument it in 2 ways.

I don’t understand as of now was the impact on TSDB (compaction and disk usage) and query support on the below patterns:

Pattern-1: Empty label value
my_requests_1_total{cause="",code=“200”} 100.0
my_requests_1_total{cause="",code=“201”} 100.0
my_requests_1_total{cause=“congestion”,code=“503”} 41.0
my_requests_1_total{cause="",code=“503”} 330.0
my_requests_1_total{cause="",code=“404”} 1.0
my_requests_1_total{cause="",code=“400”} 1.0

Pattern-2: True/False based value
my_requests_2_total{code=“503”,is_congested=“1”} 41.0
my_requests_2_total{code=“503”,is_congested=“0”} 330.0
my_requests_2_total{code=“404”,is_congested=“0”} 1.0
my_requests_2_total{code=“400”,is_congested=“0”} 1.0
my_requests_2_total{code=“200”,is_congested=“0”} 100.0
my_requests_2_total{code=“201”,is_congested=“0”} 100.0

I have asked this question in Prometheus slack channel, slack maintainer is also asking the community to help. See this conversation

Could you help me understand the right instrumentation practice to this case.

Thanks!
Teja

2

From a technical perspective, an empty label is the same as no label at all. So a boolean 0/1 is probably better.

That said, it looks like what you have should be two separate metrics, not something on the status code. I don’t know what “congested” means in your context, but typically that seems like a separate signal, so a new metric name.

3

Thanks Ben for your sharing your views!

The term congested is equivalent to overloaded, I want to know if requested is discarded as the microservice is overloaded. Overloaded service discards a request with “503” and labeled reason:“congested”.

What kind of problems do we see with empty labels? Is there a problem with querying or joining metrics or with managing the disk (more space or compaction issues)?

  1. This is in accordance with OpenMetrics standard which says “Empty label values SHOULD be treated as if the label was not present.” Prometheus has implemented this in 2015, Match empty labels. by fabxc · Pull Request #810 · prometheus/prometheus · GitHub. Earlier people can’t query empty label values using Prometheus.

I would like to know the right instrumentation practice, there is no such handy material available on this topic. Do you guys have any idea on boolean preference?

Thanks!
Teja