Prometheus remote write to AMP

Prometheus server
1

Hello
I am using vanilla prometheus which is running on a server, I have a workspace created as an amazon managed service for prometheus. I want to remote _ write from prometheus to AMP. If I use access key and secret key, error logs say: security token is invalid, but there is no option to specify security token here. Please help!

Thanks

2

You have to use Prometheus 2.26 and the new “sigv4” settings.

3

Hey, I am using prometheus 2.26 and sigv4 settings. In here when specify my access key and secret key, it says security token invalid.

4

Can you please provide us your configuration (after removing the passwords)

5

Sure,
This is the remote_write part in prometheus.yml

remote_write:
  - url: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/wsID/api/v1/remote_write"
    sigv4:
       region: us-east-1
       access_key: ACCESSKEY
       secret_key: SECRETKEY
6

This looks like. Is is possible that your access_key / secret_key lack sufficient permissions to be used ?

7

It seems that your access key must have the aps:RemoteWrite permission. is it the case?

https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-and-IAM.html

8

I’ll check the RemoteWrite permission once

9

This is not the case. It has all required permissions

10

Can you try something please? Use awscurl to check what you get from this:

$ awscurl --region us-east-1  --service aps 
         "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/$YOUR_AMP_WORKSPACE_ID/api/v1/query?query=prometheus_api_remote_read_queries"

Also, after the merge of the sigv4 proxy side car in upstream, we updated the launch blog post Getting Started with Amazon Managed Service for Prometheus.

I’m not sure where you have your remote_write config from but you shouldn’t have the AWS static credentials in there but an IAM role with an IAM policy that has permissions to remote-write into the AMP workspace (details see above blog post). Do you have said IAM role created and if so, did you check if the service account used by Prom is annotated accordingly, something like the following:

annotations:
  eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/amp-iamproxy-ingest-role"
11

I’ve tried awscurl command, this is what I got:
Error 405 Client Error: Method Not Allowed for url
I’ve also checked the blog that you’ve linked - I see it uses helm and kubernetes clusters. My scenario: I have vanilla prometheus installed, configured and running on a sever and I want to write to AMP using this. The remote_write config part is included in my prometheus.yml file (Referred from : Configuration | Prometheus). Please check if you can help.

12

Maybe you did invert the access key and the secret key? Did you double check the credentials?

1 Like
13

Great point and also, please check the credential provider chain, that is, the order in which creds are picked up.

14

Log diving for this. It looks like the reason was because of a GET on remote_write (which seems to correlate with the use of awscurl)

I am inline with roidelapluie recommendation, about original error

This usually happens when credentials are not set properly.

1 Like
15

Thank you everyone for the suggestions. I realized I was using credentials that were refreshed every 5 min, hence the security token invalid error. Created and used static credentials for remote_write and it worked.

1 Like
16

@Rakshitha Could you please help me with your prome configuration? I am trying to achieve something similar to your case in this post.