Periodic dropouts

Like clockwork, 1 of my Prometheus servers sees rate calculations drop every 30 minutes. I noticed that prometheus_tsdb_out_of_order_samples_total was rising at the exact same time, so I tried setting storage.tsdb.out_of_order_time_window to 1s. That fixed the periodic rise in prometheus_tsdb_out_of_order_samples_total, but not in any of my calculated data.
I also noticed that there is a gap in timestamp(up{job="$JOBNAME"}) at the same times. It looks like the data from scrapes at that those times is arriving late. (We scrape every 15s).
Other data we have drops out at 2 hour intervals.

I’ve checked for cron jobs or rises in resource utilization, but I haven’t been able to find anything. We’ve got other servers with periodic issues, but not the same period, and others with no issues. They’re all using the same basic configs, with different scrape targets. There doesn’t seem to be any correlation to the number of targets and the dropout periods.

What else could it be?

I think this is what we’re experiencing

It was fixed in 2.47, we’re on 2.45. I’ll update when I’ve had a chance to upgrade and see if that fixes our issues.

Upgrading to 2.52.0 fixed our problem.