Prometheus start up stuck at "Starting rule manager"

Prometheus server
1

Hi everyone,
I’m having a strange problem. I had Prometheus setup on a host machine (Windows), which I ran with docker run -p 9090:9090 prom/prometheus and it was working fine, collecting metrics as it should and presenting them at [ip]:[port]/metrics.

The goal was to have multiple servers being monitored on the host. I put a node exporter into my test server, and it was collecting metrics but they weren’t showing up on the host. However, I decided to restart Prometheus on the host, and from that point onwards, running the docker run command written above always gets the setup stuck at “Starting rule manager”. I’ve already tried to reset everything, and it always hangs there.
The test server is still running fine, I can access its metrics page from the host without issue.

prometheus.yml:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

A part that confused me in the log was this:

ts=2023-08-23T16:55:05.304Z caller=main.go:1231 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml

since the host is Windows and not Linux…?

Strangely enough, this seems to function somehow, so I think it might be just looking for the .yml file in the root directory from which docker run is called?

ts=2023-08-23T16:55:05.305Z caller=main.go:1268 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=527.906µs

This is the part where it gets stuck on:

ts=2023-08-23T16:55:05.305Z caller=main.go:1011 level=info msg="Server is ready to receive web requests."
ts=2023-08-23T16:55:05.305Z caller=manager.go:1009 level=info component="rule manager" msg="Starting rule manager..."

It doesn’t abort the setup, it just hangs on this line.
No clue what to do, any help?

1 Like
2

I met absolutely the same situation!
My prometheus server stuck at “Starting rule manager” at the first time although I’m using ubuntu20.04:(

If you solve this strange problem please contact me!

3

same problem.
my machine is:
Linux dtmaster 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:29:27 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

4

Hi!

Any update here?
Maybe someone solved the problem, i 've got now the same issue too

5

Hi!

Any updates?
Same issue

6

Having the same issue here. Debian 11

7

Had the same issue after removing the last target of a static scrape config.

With this config I had problems, but no error message in the logs:

- job_name: https_blackbox_withproxy
    metrics_path: /probe
    scrape_interval: 2m
    scrape_timeout: 1m
    params:
      module: [https_2xx_withproxy]
    static_configs:
      - targets:
#      - https://anyurl.com

I have solved this by removing the complete scrape config job.

Maybe prometheus does not like empty yaml lists / elements?

8

Unlikely to be related to the original posters issue, but when I was first launching prometheus my browser would also not load anything, and the CLT’s last line was the “Starting rule manager” line.

I was fishing around trying to debug searching through posts like this, and ChatGPT, when it suggested to load localhost:9090 through incognito mode and it worked. When asking for an explanation it mentioned caching, privacy settings and extensions can all conflict with the ability to load the web server. Reposting here in case someone runs into the same thing and this saves them some time.