Prometheus.yml targets in separate file?

Prometheus server
1

Hello, he have a high performance compute environment where I have thousands of node exporter nodes in multiple clusters to add to to my prometheus. Rather than list all of the target nodes directly in prometheus.yml, I would like to list them in a separate file, which is then read in by Prometheus.

Is this possible? If so, what is the syntax for including the outside file of targets in prometheus.yml?

To clarify I’m envisioning adding nodes to Prometheus something like this:

  • job_name: clusterA
    static_configs:
    • targets:
      clusterA_nodes_list.txt
  • job_name: clusterB
    static_configs:
    • targets:
      clusterB_nodes_list.txt
  • job_name: clusterC
    static_configs:
    • targets:
      clusterC_nodes_list.txt

Thanks.

2

You can do that with file_sd: Use file-based service discovery to discover scrape targets | Prometheus

3

Thanks, I’ll check that out.

4

I found another service discovery that looks even better – dns_sd_config. As I understand it will find all node_exporter nodes automatically.

but when I added the three items needed to configure it to my prometheus.yml , I get this error when cycling prometheus:

“Error reloading config” err=“couldn’t load configuration (–config.file=“prometheus.yml”): parsing YAML file prometheus.yml: yaml: line 8: could not find expected ‘:’”

Here’s my prometheus.yml Line 8 is the “names”

global:
  scrape_interval: 10s
  evaluation_interval: 10s

#BEGIN DNS Node Scan
dns_sd_config

 - names: pace.gatech.edu

# The type of DNS query to perform. One of SRV, A, or AAAA.
#[ type: <string> | default = 'SRV' ]

# The port number used if the query type is not SRV.
 - port: 9100

 - refresh_interval: 1800

#END DNS Node Scan


I get the same error if I format names: like so (on two lines)

 - names:
     pace.gatech.edu
5

Can you try:

scrape_configs:
   - dns_sd_configs:
      - names: [pace.gatech.edu]
        port: 9100
        refresh_interval: 1800
6

For DNS SD, you need to have SRV records at this address.

7

Thanks Julien,

results, unfortunately: parsing YAML file prometheus.yml: not a valid duration string: “1800"”

I’ve reviewed the documentation for configuration, and am puzzled by syntax. You put the names: value within square brackets, which follows the below example. But you did not put port: or refresh_value: into square brackets, which is shown below. This seems inconsistent.

Further, you prepended “names:” as " - names", but that item is not prepended with " - " in the documented syntax. Rather, it is the value which is prepended. The other three items (type, port, refresh_interval) are not prepended. It seems there’s a lot of inconsistency within the config doc, and also between the doc, and the syntax you have recommended. Can you help me to understand the seeming discrepancies, so that I can follow docs for the .yml files successfully?

names:
  [ - <string> ]

# The type of DNS query to perform. One of SRV, A, or AAAA.
[ type: <string> | default = 'SRV' ]

# The port number used if the query type is not SRV.
[ port: <int>]

# The time after which the provided names are refreshed.
[ refresh_interval: <duration> | default = 30s ]

@SuperQ , I’m going to explicitly try out SRV, A, and AAAA, see which one works.

8

Sorry, yes you need 1800s

9

brackets means that it is not mandatory, it might be a bug in the documentation.

- means it is a list, which is also written [] in yaml if you want to write it in a short form.

10

OK, the correct syntax (or at least which eliminated the last error) is:

refresh_interval: 1800s < "s"

Now that that is solved, I have new error which did not manifest until I added dns_sd_configs to my .yml:

“Error reloading config” err=“couldn’t load configuration (–config.file=“prometheus.yml”): parsing YAML file prometheus.yml: job_name is empty”

Of course I have job_name further down in my .yml, but each entry has a value:

scrape_configs:

  • job_name: prometheus
    static_configs:
    • targets:
      • localhost:9100
        #BEGIN node_exporter config of my Prom node
  • job_name: greg-test
    static_configs:
    • targets:
      • localhost:9100
        #BEGIN node_exporter config of PROD coda nodes
  • job_name: coda
    static_configs:
    • targets:
      < — snip —>

#END node_exporter config

11

bump, any thoughts on the prometheus.yml: job_name is empty error, when all of my job_names have a value?

Thanks

12

It seems malformed yaml, hard to say without the complete file. Can you check the indentation?