I have a Prometheus docker container (v2.35.0) running in AWS ECS, and I’m struggling with configuring it correctly so that targets from EC2 instances in multiple regions are scraped.
My current ‘scrape_configs’ section in ‘prometheus.yml’ looks like this:
scrape_configs:
- job_name: ec2_instances
metrics_path: /metrics
ec2_sd_configs:
- region: eu-central-1
profile: arn:aws:iam::67.....
port: 18080
filters:
- name: "tag:erlendo:prometheus-filter"
values:
- erlendo_1
- region: eu-west-1
profile: arn:aws:iam::67.....
port: 18080
filters:
- name: "tag:erlendo:prometheus-filter"
values:
- erlendo_2
With this configuration, only EC2 instances running in ‘eu-central-1’ gets scraped, and I see the following error message in Prometheus log:
ts=2022-05-13T09:36:47.949Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=ec2 msg="Unable to refresh target groups" err="could not describe instances: MissingEndpoint: 'Endpoint' configuration is required for this service"
If I switch the order of the 2 ‘ec2_sd_config’ entries, only the EC2 instances in ‘eu-west-1’ gets scraped, and the same error message appears in Prometheus log.
I also tried adding ‘endpoint: “https://ec2.eu-west-1.amazonaws.com”’ to the second ‘ec2_sd_config’ entry, but then I got the following error in Prometheus log:
ts=2022-05-12T14:09:15.831Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=ec2 msg="Unable to refresh target groups" err="could not describe instances: AuthFailure: AWS was not able to validate the provided access credentials\n\tstatus code: 401, request id: 5a70393a-790d-4114-9925-2ac38d3ff08e"
I don’t have much experience in using Prometheus, so I would appreciate if anyone could help making this scenario work.