Hi
Sorry for long question, but I try to describe the situation.
I’m not an expert in Prometheus but I’ve got a task to decrease amount of metrics that prometheus are sending to Nerwrelic by remote_write.
I use Quay in my infrastructure.
I have special config for remote-writes where metrics divided into groups. Each group has specific regexp like:
- url: someurl
name: groupname
bearerToken: mybearertoekn
writeRelabelConfigs:
- sourceLabels: ['__name__']
regex: ^metric_.*
action: keep
I removed some groups from config, applied it to prometheus but still see that metrics are going to Newrelic.
Of course I looked into the prometheus config - there is no section that I’ve deleted.
Also after applying in the prometheus logs I see that remote storage was stopped:
2022-05-24 19:27:17,"ts=2022-05-24T19:27:17.530Z caller=dedupe.go:112 component=remote level=info remote_name=mymetric url=""https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=mymetric"" msg=""Remote storage stopped."""
2022-05-24 19:27:17,"ts=2022-05-24T19:27:17.529Z caller=dedupe.go:112 component=remote level=info remote_name=mymetric url=""https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=mymetric"" msg=""Stopping remote storage..."""
2022-05-24 19:27:17,"ts=2022-05-24T19:27:17.529Z caller=dedupe.go:112 component=remote level=info remote_name=mymetric url=""https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=mymetric"" msg=""WAL watcher stopped"" queue=mymetric"
2022-05-24 19:27:17,"ts=2022-05-24T19:27:17.529Z caller=dedupe.go:112 component=remote level=info remote_name=mymetric url=""https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=mymetric"" msg=""Stopping metadata watcher..."""
2022-05-24 19:27:17,"ts=2022-05-24T19:27:17.529Z caller=dedupe.go:112 component=remote level=info remote_name=mymetric url=""https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=mymetric"" msg=""Scraped metadata watcher stopped"""
I tried to delete/recreate pod with prometheus to apply new config to the new pod - but situation is the same.
Tried googling such problems but met only reverse task, like “my remote-write does not pass data to remote storage”.
How is it possible?
What I need to analyze to investigate this issue further?
Thank You