I need to change the port that Prometheus scrapes from. In the YML file, no matter what I change the targets parameter to, Prometheus continues to scrape from the default 9090 port. I’ve even changed the target port to targets: ['foo'] and I’m still able to run Prometheus without error. This tells me the YML file isn’t getting loaded or updating. Below is the troubleshooting I’ve gone through.
After downloading Prometheus, I change directories into the directory containing the YML file. There is only one YML file in the entire directory, so I know I am editing the correct YML file.
(py3_venv) jesse@ldp-jesse:~/Downloads/prometheus-2.28.0.linux-amd64$ ls
console_libraries consoles data LICENSE NOTICE prometheus prometheus.yml promtool
(py3_venv) jesse@ldp-jesse:~/Downloads/prometheus-2.28.0.linux-amd64$ find . -name “*.yml”
./prometheus.yml
I’m running the Prometheus server with the following command: ./prometheus --config.file=prometheus.yml
Here is the scrape target in prometheus.yml:
scrape_configs:
-job_name: ‘prometheus’
static_configs:
-targets: [‘localhost:9190’]
The target was originally ['localhost:9090'] and note I changed it to ['localhost:9190']. I’ve also tried changing it to ['0.0.0.0:9190'], ['127.0.0.1:9190'] and ['foo']. After making changes, I’ve tried re-starting the Prometheus server, restarting my computer, and verifying my changes using VIM. No matter what, when I use the command netstat -tanp | grep LISTEN, the below line keeps showing that Prometheus continues to scrape from 9090:
tcp6 0 0 :::9090 :::* LISTEN 17383/./prometheus
I can’t understand it because, when I run the Prometheus server with the command ./prometheus --config.file=prometheus.yml, I get the below printout, which states towards the end that the file prometheus.yml was loaded.
(py3_venv) jesse@ldp-jesse:~/Downloads/prometheus-2.28.0.linux-amd64$ ./prometheus --config.file=prometheus.yml
level=info ts=2021-06-29T18:29:25.426Z caller=main.go:388 msg=“No time or size retention was set so using the default time retention” duration=15d
level=info ts=2021-06-29T18:29:25.426Z caller=main.go:426 msg=“Starting Prometheus” version=“(version=2.28.0, branch=HEAD, revision=ff58416a0b0224bab1f38f949f7d7c2a0f658940)”
level=info ts=2021-06-29T18:29:25.426Z caller=main.go:431 build_context=“(go=go1.16.5, user=root@32b9079a2740, date=20210621-15:45:36)”
level=info ts=2021-06-29T18:29:25.426Z caller=main.go:432 host_details=“(Linux 5.4.0-47-generic #51~18.04.1-Ubuntu SMP Sat Sep 5 14:35:50 UTC 2020 x86_64 ldp-jesse.corp.vicarious.com (none))”
level=info ts=2021-06-29T18:29:25.426Z caller=main.go:433 fd_limits=“(soft=1024, hard=1048576)”
level=info ts=2021-06-29T18:29:25.426Z caller=main.go:434 vm_limits=“(soft=unlimited, hard=unlimited)”
level=info ts=2021-06-29T18:29:25.428Z caller=web.go:541 component=web msg=“Start listening for connections” address=0.0.0.0:9090
level=info ts=2021-06-29T18:29:25.428Z caller=main.go:807 msg=“Starting TSDB …”
level=info ts=2021-06-29T18:29:25.429Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1624562004657 maxt=1624572000000 ulid=01F99NNNC0RNW87SGK4P6XG8T3
level=info ts=2021-06-29T18:29:25.429Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1624635909656 maxt=1624644000000 ulid=01F9A6RP7GPN84YPQ2X4WNSVH7
level=info ts=2021-06-29T18:29:25.429Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1624896099655 maxt=1624924800000 ulid=01F9B7EWJEX6XMRBPG8PNDHS76
level=info ts=2021-06-29T18:29:25.429Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1624948306437 maxt=1624953600000 ulid=01F9C5XKDRCAPEEMJQV2M6MSVA
level=info ts=2021-06-29T18:29:25.429Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1624924800000 maxt=1624946400000 ulid=01F9C5XKED2NVAMQF3EA4DQ8RR
ts=2021-06-29T18:29:25.430Z caller=log.go:124 component=web level=info msg=“TLS is disabled.” http2=false
level=info ts=2021-06-29T18:29:25.432Z caller=head.go:780 component=tsdb msg=“Replaying on-disk memory mappable chunks if any”
level=info ts=2021-06-29T18:29:25.433Z caller=head.go:794 component=tsdb msg=“On-disk memory mappable chunks replay completed” duration=364.936µs
level=info ts=2021-06-29T18:29:25.433Z caller=head.go:800 component=tsdb msg=“Replaying WAL, this may take a while”
level=warn ts=2021-06-29T18:29:25.441Z caller=head.go:767 component=tsdb msg=“Unknown series references” samples=10224 exemplars=0
level=info ts=2021-06-29T18:29:25.441Z caller=head.go:826 component=tsdb msg=“WAL checkpoint loaded”
level=info ts=2021-06-29T18:29:25.441Z caller=head.go:854 component=tsdb msg=“WAL segment loaded” segment=27 maxSegment=31
level=info ts=2021-06-29T18:29:25.442Z caller=head.go:854 component=tsdb msg=“WAL segment loaded” segment=28 maxSegment=31
level=info ts=2021-06-29T18:29:25.442Z caller=head.go:854 component=tsdb msg=“WAL segment loaded” segment=29 maxSegment=31
level=info ts=2021-06-29T18:29:25.442Z caller=head.go:854 component=tsdb msg=“WAL segment loaded” segment=30 maxSegment=31
level=info ts=2021-06-29T18:29:25.442Z caller=head.go:854 component=tsdb msg=“WAL segment loaded” segment=31 maxSegment=31
level=info ts=2021-06-29T18:29:25.442Z caller=head.go:860 component=tsdb msg=“WAL replay completed” checkpoint_replay_duration=8.494289ms wal_replay_duration=931.63µs total_replay_duration=9.807008ms
level=info ts=2021-06-29T18:29:25.443Z caller=main.go:834 fs_type=EXT4_SUPER_MAGIC
level=info ts=2021-06-29T18:29:25.443Z caller=main.go:837 msg=“TSDB started”
level=info ts=2021-06-29T18:29:25.443Z caller=main.go:964 msg=“Loading configuration file” filename=prometheus.yml
level=info ts=2021-06-29T18:29:25.444Z caller=main.go:995 msg=“Completed loading of configuration file” filename=prometheus.yml totalDuration=362.341µs remote_storage=1.17µs web_handler=230ns query_engine=498ns scrape=150.652µs scrape_sd=17.733µs notify=15.123µs notify_sd=6.966µs rules=712ns
level=info ts=2021-06-29T18:29:25.444Z caller=main.go:779 msg=“Server is ready to receive web requests.”