Hello,
I have some doubts about the following situation:
I am running Prometheus server on localhost and sending data every minute to Grafana Cloud via remote write. In case the internet connection is lost, and let’s say the connection recovers in 1 day… is it possible to update the gaps that are created due to missing data points in the Cloud service (Grafana Cloud )? Do you guys know if this is possible to achieve? Is remote write query capacity important in this case?
Thanks in advance!!
Remote write will retry sending the data the data for a while, so assuming it hasn’t reached the end of the reties the data should be sent over. However that also relies on the data being accepted - the remote system may reject data that is “too old”. It might also take a little while to catchup depending on the volume of data it is sending (if it has millions of samples for example).
It is much more likely that an outage of, say, half an hour recovers seamlessly, compared to one that lasts several days.
1 Like
Thanks for the reply. Your answer touches on several interesting points to keep in mind when handling prometheus and a cloud service.
These days, Grafana’s technical service has told me that data recovery is not possible. And I have also been able to verify it myself, having had a loss of information and it could not be recovered despite recovering the connection.
However, this answer is regarding the use of Grafana Cloud, so I think it will depend on the service to use.