I am planning to setup Prometheus monitoring for monitor databases Service. As i want to store the historic metrics also.
So can anyone advice me How to make the decisions to choose the popular remote storage?
I will use Grafana for visualisation? As Grafana also has data source option i do not understand what is the use of that?
As some of the remote storage only support write api only, Does it from remote storage we can read the metrics?
Grafana is a tool for visualising & querting data. It isn’t limited to only working with Prometheus, but via different datasource plugins it can also work with other metrics systems (such as Datadog) as well as logs and traces (for example Loki or Cloudwatch Logs). It can even talk to databases to fetch information. So if you are only using it for Prometheus you can ignore all the other datasource options.
Remote storage with Prometheus can be a useful addition, but is by no means required. Prometheus is more than capable of storing years of metrics locally, however some people want to store data in other locations (e.g. a different geographic region) or interoperate with other systems (e.g. for reporting & analysis). Many tools now accept Prometheus metrics via the remote write feature, such as Timescale and InfluxDB.
In addition to remote write, which sends data to an external system, there is also remote read which will query an external system for metric data. This can be used in conjunction with remote write (so you can query the data you sent externally) but can also be used separately to allow Prometheus queries (PromQL) to access things created elsewhere (for example projections or data from an event store).
It is up to the third party tooling (which might be native or via plugins/protocol convertors) if they support remote write, remote read or both.
Thanks for sharing some knowledge on it. I have added a image which is describing my use case. My mainly requirement is to store the Prometheus metric in other database/storage. I do not want to use the Default Prometheus option to store the metrics.
Can you share some opinion on popular storage and How to make decisions on it to store the Prometheus metrics and so that it can full-fill the use case requirements?