Prometheus in Agentmode not scrapping default openshift prometheus

Hi guys,
I’ve currently deployed prometheus(prometheus1) in agent mode and have configured it to scrape the prometheus in openshift-monitoring namespace along with agent itself and federated prometheus in the same cluster. I’m also trying to send this scrapped metrics to another prometheus instance(prometheus2) through remote_write. Below is my prometheus ConfigMap.

  scrape_interval: 300s
  #evaluation_interval: 50s
  scrape_timeout: 100s
  - job_name: 'prometheus'
    honor_labels: true
    metrics_path: '/metrics'
    - targets: ['localhost:9090']
  - job_name: openshift-prometheus
    honor_labels: true
    metrics_path: '/federate'
      type: Bearer
      credentials:  <credential>
    scheme: https
      insecure_skip_verify: true
    - targets: ['<target-url>']
  - job_name: prometheus-ap
    honor_labels: true
    metrics_path: '/metrics'
      username: <username>
      password: <password>
    scheme: https
      insecure_skip_verify: true
    - targets: ['<target-url>']
  - url: 'http://<svc-name>:9090/api/v1/write'
    # tls_config:
      # insecure_skip_verify: true
      capacity: 10 # default = 500
      max_shards: 4 # default = 1000
      min_shards: 1 # default = 1
      max_samples_per_send: 1 # default = 100
      # batch_send_deadline: 5s # default = 5s
      # min_backoff: 30ms # default = 30ms
      # max_backoff: 100ms # default = 100ms

But in my prometheus2 instance I’m not seeing metrics scrapped by openshift-monitoring prometheus. I can see the metric value up=1 for all the targets and also no error logs in prometheus pod. Below is the result for queries scrape_samples_scraped

scrape_samples_scraped{instance="<url>:443", job="prometheus-ap"}   731
scrape_samples_scraped{instance="<openshift-prometheus-url>:443", job="openshift-prometheus"} 0
scrape_samples_scraped{instance="localhost:9090", job="prometheus"}  381

up{instance="<url>:443", job="prometheus-ap"} 1
up{instance="<openshift-prometheus-url>:443", job="openshift-prometheus"} 1
up{instance="localhost:9090", job="prometheus"} 1

Hey. Does the endpoint belonging to the target openshift-prometheus expose metrics correctly? Have you tried hitting the endpoint: https://<basic auth credentials>@<target-url>/federate and validate its output?

It looks like it is actually able to scrape it but no metrics are returned.

Yes. After adding the basic_auth credentials I’m able to see the scraped metrics. Thanks

Also I’ve seen that remote-write-receiver capability of prometheus should be used only for low-volume use cases with caution. This is confusing. Will it be able to handle the load from multiple agents deployed in different clusters that is scraping the openshift-prometheus? Is there any extra cases or precautions to be taken while enabling remote receive capability?