My Setup
I have setup prometheus and grafana on a machine - M1 I have a AWS Load balancer (LB1) which is forwarding the requests to app instances and handling the ssl part. Load Balancer has one listener listening on https:443 and is forwarding requests to Target group(TG1) which has 2 EC2 instances (I1, I2). Each EC2 instance is also running nginx.
Request lifecycle:
Browser requests → https://api.example.com
- Request goes to load balancer.
- Load balancer redirects it to one of the instances in TG1
- Nginx running on instance further forwards the request to one of the app workers.
The problem is coming when i am trying to fetch metrics from prometheus using node exporter from each of the instances(I1 & I2). I have installed the node exporter in each of the instances which is running. How do I register Instance I1 & Instance I2 as targets in prometheus.yml file?
I have tried 2 approaches:
Approach 1:
Using load balancer -
I simply installed node exporter in each of the instances and running node exporter on port 9100. Incoming requests to the machine from load balancer will be forwarded to localhost:9100/metrics by nginx.
Example prometheus.yml file:
- job_name: "node_exporter"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
scheme: "https"
metrics_path: "/node-exporter/metrics"
static_configs:
- targets: ["api.example.com:443"]
basic_auth:
username: 'username'
password: 'password'
The problem here is it registers just one host as the target. I was expecting to see two hosts on prometheus with the same name.
Question:
- Is there a way I can forward requests to a unique instance in loadbalancer target group based on port? This way, i can run node exporter on different ports on different instances and specify which instance to forward the request to based on port.
- Is there a way I can forward requests to a unique instance in loadbalancer target group based on ip address in url? This way, i can run node exporter on same ports on different instances and just forward the request to individual machine based on the ip address in the url.
Approach 2:
Using IP address -
I registered the IP address as a prometheus target and opened the ports on machines.
Example prometheus.yml file:
- job_name: "node_exporter"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
scheme: "http"
metrics_path: "/node-exporter/metrics"
static_configs:
- targets: ["162.x.y.1:9100", "162.x.y.2:9100"]
basic_auth:
username: 'username'
password: 'password'
The problem here how do i make it work with https scheme?
Question:
- How do i enable ssl for an ip address and for sending requests to port other than 443? Is it possible?
Summary:
What is the best way to set up prometheus to start gathering metrics from machines running behind AWS Load balancer in a single target group and accessible via common domain name? How do i register the prometheus targets in this case?