How to register ec2 instances running behind a load balancer as targets in prometheus

Exporters and Metrics
1

My Setup

I have setup prometheus and grafana on a machine - M1 I have a AWS Load balancer (LB1) which is forwarding the requests to app instances and handling the ssl part. Load Balancer has one listener listening on https:443 and is forwarding requests to Target group(TG1) which has 2 EC2 instances (I1, I2). Each EC2 instance is also running nginx.

Request lifecycle:
Browser requests → https://api.example.com

  1. Request goes to load balancer.
  2. Load balancer redirects it to one of the instances in TG1
  3. Nginx running on instance further forwards the request to one of the app workers.

The problem is coming when i am trying to fetch metrics from prometheus using node exporter from each of the instances(I1 & I2). I have installed the node exporter in each of the instances which is running. How do I register Instance I1 & Instance I2 as targets in prometheus.yml file?

I have tried 2 approaches:

Approach 1:

Using load balancer -

I simply installed node exporter in each of the instances and running node exporter on port 9100. Incoming requests to the machine from load balancer will be forwarded to localhost:9100/metrics by nginx.

Example prometheus.yml file:

  - job_name: "node_exporter"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    scheme: "https"
    metrics_path: "/node-exporter/metrics"
    static_configs:
      - targets: ["api.example.com:443"]
    basic_auth:
      username: 'username'
      password: 'password'

The problem here is it registers just one host as the target. I was expecting to see two hosts on prometheus with the same name.

Question:

  1. Is there a way I can forward requests to a unique instance in loadbalancer target group based on port? This way, i can run node exporter on different ports on different instances and specify which instance to forward the request to based on port.
  2. Is there a way I can forward requests to a unique instance in loadbalancer target group based on ip address in url? This way, i can run node exporter on same ports on different instances and just forward the request to individual machine based on the ip address in the url.

Approach 2:

Using IP address -

I registered the IP address as a prometheus target and opened the ports on machines.

Example prometheus.yml file:

  - job_name: "node_exporter"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    scheme: "http"
    metrics_path: "/node-exporter/metrics"
    static_configs:
      - targets: ["162.x.y.1:9100", "162.x.y.2:9100"]
    basic_auth:
      username: 'username'
      password: 'password'

The problem here how do i make it work with https scheme?

Question:

  1. How do i enable ssl for an ip address and for sending requests to port other than 443? Is it possible?

Summary:

What is the best way to set up prometheus to start gathering metrics from machines running behind AWS Load balancer in a single target group and accessible via common domain name? How do i register the prometheus targets in this case?

2

You need to be configuring Prometheus to scrape each machine separately, not via a load balancer. If you use a load balancer you just get a single instance inside Prometheus with a fairly useless mix of data from different backend servers.

For your second approach, which is talking to each server individually (you can use IP addresses or DNS names if they exists, as well as other more dynamic service discovery methods), you can enable HTTPS if you wish. On the Prometheus end you just need to set the scheme to https. On the exporter side you have various options.

You could use some form of TLS offloading proxy (such as Nginx) that talks to the exporter locally over HTTP and exposes an encrypted endpoint to be scraped. Some exporters also have native HTTPS support, such as the TLS options for the node exporter: GitHub - prometheus/node_exporter: Exporter for machine metrics

3

Hey Stuart, thanks for the reply.

As long as things were on the HTTP scheme, everything worked great. I was facing challenges when I tried to use HTTPS scheme for multiple targets under a job.

Since I have 2 EC2 instances running behind a load balancer, so I don’t have DNS names for them. I had to work with the IP addresses.

Finally, I was able to make it work using self signed ssl certificates which I used for for nginx and Configure Prometheus to ignore ssl verification.

Now, the only thing is that I have to do this on each of the instances that I want to monitor.

My prometheus.yml file looks something like this:

- job_name: "node_exporter"
    scheme: "https"
    static_configs:
      - targets: ["192.x.y.z:443"]
    tls_config:
      insecure_skip_verify: true
    basic_auth:
      username: 'username'
      password: 'password'