Remote read only


I have a Prometheus instance using remote_read/remote_write to InfluxDB 1.8 for TSDB and Grafana for visualisation. I have a set of requirements where data needs to be down sampled for long term storage. I can view the data in full from the default autogen in Grafana however, I am really struggling with the visualisation of data from other retention policies.

I can’t determine from the documentation whether I can tag the retention policy directly onto the Prometheus data source as a parameter so I have created several Prometheus instances using remote_read and specifying the rp in the URL. I have found this approach looks to be the most straightforward and not require lots of edits to dashboards.

The issue I have is Prometheus doesn’t appear to read the remote data so no metrics are available despite them being present in InfluxDB and can be retrieved through InfluxDB queries or using Chronograph. I have seen metric labels become available when I add a scrape job in my Prometheus config. Is there a dependency where Prometheus stores write data locally to build up a list of available metrics and isn’t actually able to exclusively read from the remote and construct a list of metrics from InfluxDB?