Hello everyone,
We are currently facing an issue where scrape targets are being duplicated across multiple replicas.
Prometheus - Run as operator in server mode
Shards - 4
Replica - 1
Modulus - 4
Includes recording rules as well
Below config:
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
regex: “true”
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
regex: (.+)
target_label: metrics_path
action: replace
- source_labels: [address, __meta_kubernetes_pod_annotation_prometheus_io_port]
regex: ([^:]+)(?::\d+)?;(\d+)
target_label: address
replacement: $1:$2
action: replace
- source_labels: [__meta_kubernetes_pod_label_nfId]**
modulus: 4
target_label: __tmp_hash**
action: hashmod**
- source_labels: [__tmp_hash]**
action: keep**
- action: labelmap
regex: _meta_kubernetes_pod_label(.+)
- source_labels: [__meta_kubernetes_namespace]
regex: (.)
target_label: kubernetes_namespace
action: replace
- source_labels: [__meta_kubernetes_pod_name]
regex: (.)
target_label: kubernetes_pod_name
action: replace
Concerns:
- Scrape targets are duplicated across multiple replica
- Duplicates have been observed in the metrics scraped
Kindly share your inputs or if anyone faced a similar issue