Hi,
I just wanted to know if there is a way to force scrape a node again or do I always have to wait until the interval is over? I didn’t find anything for this issue.
Yes you just need to wait for the next scrape, which should never be more than a few minutes away.
I was afraid of that. Because we try to migrate from a simple scheduled monitoring. If our oncall gets a call at 3 AM, they’d like to have the option to immediately check if our monitoring is alright again.
Would waiting 30s or a minute for the next scrape really be a problem?
Also, I’d suggest being very careful about how you create your alerting rules to ensure you don’t alert too quickly - the “for” option is very useful here. There’s no point contacting an on-call person if a single scrape fails. I usually suggest 5, 10 or 15 minutes of continuous failure before alerting someone (given that they might take 30 minutes to wake up & come online anyway, so it doesn’t delay things all that much).
2 Likes
30s to 1 minute is fine tbh. We are currently in the evaluation phase of a new monitoring system and Prometheus is the hottest candidate for us right now. It gives developers the opportunity to monitor the application in new ways. Also your suggestions for 5, 10 or 15 minutes of continues failures before we alert anyone, sounds very good.