Hello Team,
when target server goes down prometheus doesn’t send alert and its always in pending state
below is the error we are getting from logs as well as target in gui
Get “http://xx.xx.xx.xx:11300/metrics”: context deadline exceeded
Though we tried to increase below but doesn’t help
scrape_interval: 60s
scrape_timeout: 59s
stuart
2
That suggests the target is unavailable. Changing the timeout won’t help if the target isn’t responding.
Are you using the “up” metric for your alert?
stuart
4
And what is that returning if you graph it?
stuart
6
What is the full alert code?
stuart
8
What is the full alerting rule, not just the query?