Unable to remote write to AWS managed Prometheus workspace

General Help/Support
1

I’m unable to write to AWS managed prometheus workspace. I’m using vanilla prometheus running in a pod, and attempting to write to the managed aws prometheus workspace. I’m using sig4 for authentication to the remote write. It looks like i am authenticated but the request is too large. Im just scraping prometheus itself and sending the metrics via the remote write. What am i doing wrong? is there an optional setting i need to add?

    remote_write:
        - url: https://aps-workspaces.<region>.amazonaws.com/workspaces/<workspaceid>/api/v1/remote_write
          sigv4:
            region: us-west-2
            access_key: 
            secret_key: 
          queue_config:
            max_samples_per_send: 10000
            max_shards: 2000
            capacity: 25000

msg="non-recoverable error" count=10000 err="server returned HTTP status 400 Bad Request: series has too many labels (actual: 138, limit: 70) series: 

"server returned HTTP status 413 Request Entity Too Large: <RequestEntityTooLargeException>"
2

I’ve not had any issues sending data to AMP using the defaults (i.e. not setting things within the queue_config block). The error suggests that 10,000 samples per send is too high, so I’d suggest looking to reduce that.

3

I tried to reduce that to below, and im still seeing issues. How can i get past this?

server returned HTTP status 400 Bad Request: series has too many labels (actual: 121, limit: 70) series:

         queue_config:
            max_samples_per_send: 100
            max_shards: 200
            capacity: 300
4

You would need to exclude the series with over 70 labels as the error suggests. Take a look at relabling rules. However even 70 labels is a lot, so I’d suggest looking into why there are so many labels attached to some of your metrics.

5

i commented out some of the labels, im deploying via helm manifest. I only enabled 1 job: prometheus to scrape itself, and then the remote write. GOing to test to see if data got sent to the remote write. How can we confirm from logs if the remote write is being processed. Right now in logs i see “remote_write msg=“Done replaying WAL” duration=34.762575077s”

6

do you happen to have a sample prometheus.yml file that i can refer to. I dont have kube-state-exporter, or node exporter running, just the prometheus server.

7

How do you mean you have commented out some of the labels?

If you go to the Prometheus UI what is the configuration listed?