Hello,
Got strange Prometheus behavior after some time, they just stop responding. No entries in logs, nothing. Log level is debug. Restart not help, only thing what fix that - empty database. Then couple days all working fine.
Is it possible to know what is going on?
Thanks.
prometheus v2.27.1 installed from docker hub
Is the /debug/pprof endpoint still responding?
While filesystem are you using?
Are you monitoring prometheus with another prometheus as well?
/debug/pprof was not checked, will try check on next time.
Problem was start on btrfs on 2 NVMe disk in md raid, but then i reformat array to ext4. That was not help.
No, just one prometheus. Need install one more for get metrics from first?
Installed second bundle prometheus+grafana with Prometheus2.0 (by FUSAKLA) dashboard for Grafana | Grafana Labs dashboard and node exporter stats for monitor fist prometheus. Will collect data.