Prometheus remote_write server returned HTTP status 405 method not allowed

Prometheus server
1

Hi All,

I am working on sending data from prometheus to elasticsearch using remote_write in k8s and I am using Prometheus 2.28 image.

Below is the remote write configuration defined in configMap (k8s).

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: monitor-config
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    scrape_configs:
      - job_name: opensearch
        scrape_interval: 10s
        metrics_path: "/_prometheus/metrics"
        scheme: https
        static_configs:
        - targets:
          - opensearch-cluster-master:9200
          - opensearch-cluster-data:9200
        tls_config:
          insecure_skip_verify: true
        basic_auth:
          username: 'admin'
          password: 'admin'
    remote_write:
      - url: "https://opensearch-cluster-master:9200/write"
        remote_timeout: 30s
        queue_config:
          capacity: 10000
          max_shards: 10
          min_shards: 2
          max_samples_per_send: 5000
          batch_send_deadline: 30m
          min_backoff: 50ms
          max_backoff: 1s
        basic_auth:
          username: 'admin'
          password: 'admin'
        tls_config:
          insecure_skip_verify: true
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://127.0.0.1:9090

Once the prometheus pod starts running below is the error I am getting in pod logs,

ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg=“non-recoverable error” count=495 exemplarCount=0 err=“server returned HTTP status 405 Method Not Allowed: {“error”:“Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]”,“status”:405}”**

[root@k8master1 prometheus-grafana]# kubectl logs monitoring-6c7546db6c-h9djx -n es -c prometheus -f
level=info ts=2022-04-18T09:34:16.086Z caller=main.go:148 msg="Experimental remote-write-receiver enabled"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:388 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:426 msg="Starting Prometheus" version="(version=2.28.0, branch=HEAD, revision=ff58416a0b0224bab1f38f949f7d7c2a0f658940)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:431 build_context="(go=go1.16.5, user=root@32b9079a2740, date=20210621-15:45:36)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:432 host_details="(Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 monitoring-6c7546db6c-h9djx (none))"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:433 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2022-04-18T09:34:16.089Z caller=main.go:434 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2022-04-18T09:34:16.094Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-04-18T09:34:16.095Z caller=main.go:807 msg="Starting TSDB ..."
ts=2022-04-18T09:34:16.098Z caller=log.go:124 component=web level=info msg="TLS is disabled." http2=false
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=6.709µs
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2022-04-18T09:34:16.102Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.859µs wal_replay_duration=322.23µs total_replay_duration=364.154µs
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:834 fs_type=EXT4_SUPER_MAGIC
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:837 msg="TSDB started"
level=info ts=2022-04-18T09:34:16.104Z caller=main.go:964 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg="Starting WAL watcher" queue=e3afd4
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg="Starting scraped metadata watcher"
ts=2022-04-18T09:34:16.111Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg="Replaying WAL" queue=e3afd4
level=info ts=2022-04-18T09:34:16.112Z caller=main.go:995 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=8.219377ms remote_storage=5.66082ms web_handler=637ns query_engine=877ns scrape=870.296µs scrape_sd=39.662µs notify=1.047µs notify_sd=4.678µs rules=4.199µs
level=info ts=2022-04-18T09:34:16.112Z caller=main.go:779 msg="Server is ready to receive web requests."
ts=2022-04-18T09:34:23.455Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg="Done replaying WAL" duration=7.344249957s
ts=2022-04-18T09:35:03.459Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg="non-recoverable error" count=5000 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"
ts=2022-04-18T09:35:06.119Z caller=dedupe.go:112 component=remote level=info remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg="Remote storage resharding" from=2 to=10
**ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg="non-recoverable error" count=494 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"
ts=2022-04-18T09:35:06.123Z caller=dedupe.go:112 component=remote level=error remote_name=e3afd4 url=https://opensearch-cluster-master:9200/write msg="non-recoverable error" count=495 exemplarCount=0 err="server returned HTTP status 405 Method Not Allowed: {\"error\":\"Incorrect HTTP method for uri [/write] and method [POST], allowed: [DELETE, GET, HEAD, PUT]\",\"status\":405}"**

Please let me know how we can resolve this issue and correct me If I am doing anything wrong in the setup.

Thanks,
Ganeshbabu R