Horizontally scalable prometheus - support for duplicate data

Hello everyone,

We are currently facing an issue where scrape targets are being duplicated across multiple replicas.

Prometheus - Run as operator in server mode
Shards - 4
Replica - 1
Modulus - 4
Includes recording rules as well

Below config:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
regex: “true”
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
regex: (.+)
target_label: metrics_path
action: replace
- source_labels: [address, __meta_kubernetes_pod_annotation_prometheus_io_port]
regex: ([^:]+)(?::\d+)?;(\d+)
target_label: address
replacement: $1:$2
action: replace
- source_labels: [__meta_kubernetes_pod_label_nfId]**
modulus: 4
target_label: __tmp_hash**
action: hashmod**
- source_labels: [__tmp_hash]**
action: keep**
- action: labelmap
regex: _meta_kubernetes_pod_label(.+)
- source_labels: [__meta_kubernetes_namespace]
regex: (.)
target_label: kubernetes_namespace
action: replace
- source_labels: [__meta_kubernetes_pod_name]
regex: (.
target_label: kubernetes_pod_name
action: replace


  1. Scrape targets are duplicated across multiple replica
  2. Duplicates have been observed in the metrics scraped

Kindly share your inputs or if anyone faced a similar issue