# Prometheus remote write to AMP **URL:** \ **Category:** Prometheus server\ **Created:** [May 10, 2021, 10:32am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215 "2021-05-10T10:32:06Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Rakshitha](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Rakshitha](https://discuss.prometheus.io/u/Rakshitha)\ **Post date:** [May 10, 2021, 10:32am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/1 "2021-05-10T10:32:06Z")
Hello I am using vanilla prometheus which is running on a server, I have a workspace created as an amazon managed service for prometheus. I want to remote \_ write from prometheus to AMP. If I use access key and secret key, error logs say: security token is invalid, but there is no option to specify security token here. Please help! Thanks ---
**Author:** ![roidelapluie](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/roidelapluie/32/6_2.png) [@roidelapluie](https://discuss.prometheus.io/u/roidelapluie)\ **Post date:** [May 10, 2021, 11:06am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/2 "2021-05-10T11:06:50Z")
You have to use Prometheus 2.26 and the new “sigv4” settings. ---
**Author:** ![Rakshitha](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Rakshitha](https://discuss.prometheus.io/u/Rakshitha)\ **Post date:** [May 11, 2021, 8:58am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/3 "2021-05-11T08:58:30Z")
Hey, I am using prometheus 2.26 and sigv4 settings. In here when specify my access key and secret key, it says security token invalid. ---
**Author:** ![roidelapluie](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/roidelapluie/32/6_2.png) [@roidelapluie](https://discuss.prometheus.io/u/roidelapluie)\ **Post date:** [May 11, 2021, 9:23am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/4 "2021-05-11T09:23:10Z")
Can you please provide us your configuration (after removing the passwords) ---
**Author:** ![Rakshitha](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Rakshitha](https://discuss.prometheus.io/u/Rakshitha)\ **Post date:** [May 11, 2021, 9:43am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/5 "2021-05-11T09:43:22Z")
Sure, This is the remote\_write part in prometheus.yml ```auto remote_write: - url: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/wsID/api/v1/remote_write" sigv4: region: us-east-1 access_key: ACCESSKEY secret_key: SECRETKEY ``` ---
**Author:** ![roidelapluie](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/roidelapluie/32/6_2.png) [@roidelapluie](https://discuss.prometheus.io/u/roidelapluie)\ **Post date:** [May 11, 2021, 9:45am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/6 "2021-05-11T09:45:14Z")
This looks like. Is is possible that your access\_key / secret\_key lack sufficient permissions to be used ? ---
**Author:** ![roidelapluie](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/roidelapluie/32/6_2.png) [@roidelapluie](https://discuss.prometheus.io/u/roidelapluie)\ **Post date:** [May 11, 2021, 9:48am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/7 "2021-05-11T09:48:29Z")
It seems that your access key must have the `aps:RemoteWrite` permission. is it the case? [https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-and-IAM.html](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-and-IAM.html) ---
**Author:** ![Rakshitha](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Rakshitha](https://discuss.prometheus.io/u/Rakshitha)\ **Post date:** [May 11, 2021, 9:55am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/8 "2021-05-11T09:55:20Z")
I’ll check the RemoteWrite permission once ---
**Author:** ![Rakshitha](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Rakshitha](https://discuss.prometheus.io/u/Rakshitha)\ **Post date:** [May 11, 2021, 10:06am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/9 "2021-05-11T10:06:50Z")
This is not the case. It has all required permissions ---
**Author:** ![mh9](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/mh9/32/19_2.png) [@mh9](https://discuss.prometheus.io/u/mh9)\ **Post date:** [May 11, 2021, 11:15am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/10 "2021-05-11T11:15:09Z")
Can you try something please? Use [awscurl](https://github.com/okigan/awscurl) to check what you get from this: ```auto $ awscurl --region us-east-1 --service aps "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/$YOUR_AMP_WORKSPACE_ID/api/v1/query?query=prometheus_api_remote_read_queries" ``` Also, after the merge of the sigv4 proxy side car in upstream, we updated the launch blog post [Getting Started with Amazon Managed Service for Prometheus](https://aws.amazon.com/blogs/mt/getting-started-amazon-managed-service-for-prometheus/). I’m not sure where you have your `remote_write` config from but you shouldn’t have the AWS static credentials in there but an IAM role with an IAM policy that has permissions to remote-write into the AMP workspace (details see above blog post). Do you have said IAM role created and if so, did you check if the service account used by Prom is annotated accordingly, something like the following: ```auto annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/amp-iamproxy-ingest-role" ``` ---
**Author:** ![Rakshitha](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Rakshitha](https://discuss.prometheus.io/u/Rakshitha)\ **Post date:** [May 12, 2021, 6:33am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/11 "2021-05-12T06:33:46Z")
I’ve tried awscurl command, this is what I got: ` Error 405 Client Error: Method Not Allowed for url` I’ve also checked the blog that you’ve linked - I see it uses helm and kubernetes clusters. My scenario: I have vanilla prometheus installed, configured and running on a sever and I want to write to AMP using this. The remote\_write config part is included in my prometheus.yml file (Referred from : [Configuration | Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write)). Please check if you can help. ---
**Author:** ![roidelapluie](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/roidelapluie/32/6_2.png) [@roidelapluie](https://discuss.prometheus.io/u/roidelapluie)\ **Post date:** [May 12, 2021, 7:12am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/12 "2021-05-12T07:12:55Z")
Maybe you did invert the access key and the secret key? Did you double check the credentials? ---
**Author:** ![mh9](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/mh9/32/19_2.png) [@mh9](https://discuss.prometheus.io/u/mh9)\ **Post date:** [May 12, 2021, 3:43pm UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/13 "2021-05-12T15:43:26Z")
Great point and also, please check the [credential provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html), that is, the order in which creds are picked up. ---
**Author:** ![jeromeinbc](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/jeromeinbc/32/130_2.png) [@jeromeinbc](https://discuss.prometheus.io/u/jeromeinbc)\ **Post date:** [May 12, 2021, 4:12pm UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/14 "2021-05-12T16:12:58Z")
> [@Rakshitha](#): > > `Error 405 Client Error: Method Not Allowed for url` Log diving for this. It looks like the reason was because of a GET on remote\_write (which seems to correlate with the use of awscurl) I am inline with roidelapluie recommendation, about original error > [@Rakshitha](#): > > security token is invalid This usually happens when credentials are not set properly. ---
**Author:** ![Rakshitha](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@Rakshitha](https://discuss.prometheus.io/u/Rakshitha)\ **Post date:** [May 17, 2021, 9:54am UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/15 "2021-05-17T09:54:40Z")
Thank you everyone for the suggestions. I realized I was using credentials that were refreshed every 5 min, hence the security token invalid error. Created and used static credentials for remote\_write and it worked. ---
**Author:** ![KavyaShree25](https://dub1.discourse-cdn.com/flex017/user_avatar/discuss.prometheus.io/kavyashree25/32/504_2.png) [@KavyaShree25](https://discuss.prometheus.io/u/KavyaShree25)\ **Post date:** [August 29, 2022, 6:30pm UTC](https://discuss.prometheus.io/t/prometheus-remote-write-to-amp/215/16 "2022-08-29T18:30:51Z")
@Rakshitha Could you please help me with your prome configuration? I am trying to achieve something similar to your case in this post.