Hi. I’m trying to set up my prometheus for scraping metrics from Spring Boot app, however so far I was able to scrape only from the URL of the Load Balancer, so I never know which instance is scraped.
How do I set up the scraping to get each instance reliably?
Service is running in AWS ECS.
Looks like the solution is to use GitHub - teralytics/prometheus-ecs-discovery: A Prometheus discoverer that scrapes Amazon ECS and a generates file SD configuration file.
dockerLabels for all 4 listed labels need to be added to each container in order for the tool to detect it. The tools updates a config file, which should be reloaded by Prometheus server
If anyone is looking for dynamic ECS service discovery of ECS Clusters - check out this blog post:
It provides dynamic service discovery of your ECS Tasks running in AWS ECS Cluster and exposes HTTP endpoint that is compatible with http_sd_target of Prometheus.
Also available on GitHub:
I internally implemented an alternative approach, without the use of CloudMap.
Instead of relying on CloudMap, I am instead adding dockerLabel METRICS_PATH to my containers that need to be scraped. Then I am using python script to look for services deployed in my ECS cluster, look in Task Definitions and if the definition contains the dockerLabel, I get the task IP and PORT and then using the value of the label I form the endpoint that prometheus can query. Because our apps have different endpoints where the metrics is exposed, this approach allows more flexibility and does not require registering with CloudMap.