Prometheus stop respond after some time


Got strange Prometheus behavior after some time, they just stop responding. No entries in logs, nothing. Log level is debug. Restart not help, only thing what fix that - empty database. Then couple days all working fine.

Is it possible to know what is going on?


prometheus v2.27.1 installed from docker hub

Is the /debug/pprof endpoint still responding?

While filesystem are you using?
Are you monitoring prometheus with another prometheus as well?

/debug/pprof was not checked, will try check on next time.

Problem was start on btrfs on 2 NVMe disk in md raid, but then i reformat array to ext4. That was not help.

No, just one prometheus. Need install one more for get metrics from first?

Installed second bundle prometheus+grafana with Prometheus2.0 (by FUSAKLA) dashboard for Grafana | Grafana Labs dashboard and node exporter stats for monitor fist prometheus. Will collect data.