Hello guys!
I have the following issue:
I am trying to count my requests for some label combinations (client_id - ~100 distincts, endpoint - ~5 distincts). The app that produces the logs is deployed on Azure. Performing requests manually, makes the counter increase and behave normal, but the issue is there are those gaps which I am not sure why they appear. For example if i’ve had 6 requests, even if it gapped to 3, when i’ll do 3 other more requests, it would jump straightforward to 9, but the gap would still be created, as seen below:
I understand that rate is supposed to solve these ‘gaps’ and should be fine, but the issue is when I am trying to find the count of requests within a certain timeframe. I understood for that I have to use ‘increase’. From how it look, the increase gets affected by those gaps as it increases when this gaps occur.
Could someone help me understand why those ‘gaps’ occur? I am not using kubernetes and there aren’t restarts occurring on the service, so not sure what might cause those drops. If i’ve host the service locally, and set that as target, the gaps don’t seem to appear. If somebody encountered it or might know might cause it, it would be really helpful.
To mention that if i would start my server locally and use that as target, the fluctuations doesn’t appear, but somehow on the target from azure they do appear, even if only 1 instance and no traffic
Thanks!