Periodically getting SIGTERM. See you next time!

General Help/Support
1

Hi,

I keep getting caller=main.go:807 level=warn msg="Received SIGTERM, exiting gracefully..." about once a day.

And the server shuts down. Anyone know what might cause that?

I’m using prometheus-2.37.5.linux-amd64.

I don’t see anything else unusual going on.

Here’s what some of the other logs lines look like

ts=2023-01-22T15:00:29.437Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1674388814368 maxt=1674396000000 ulid=01GQCZXG2TPPNXZX0PTHNFQJCX duration=35.743481ms
ts=2023-01-22T15:00:29.439Z caller=head.go:842 level=info component=tsdb msg="Head GC completed" duration=642.35µs
ts=2023-01-22T15:00:29.439Z caller=checkpoint.go:97 level=info component=tsdb msg="Creating checkpoint" from_segment=191 to_segment=192 mint=1674396000000
ts=2023-01-22T15:00:29.448Z caller=head.go:1018 level=info component=tsdb msg="WAL checkpoint complete" first=191 last=192 duration=9.577716ms
ts=2023-01-22T16:03:45.981Z caller=main.go:807 level=warn msg="Received SIGTERM, exiting gracefully..."
ts=2023-01-22T16:03:45.981Z caller=main.go:831 level=info msg="Stopping scrape discovery manager..."
ts=2023-01-22T16:03:45.981Z caller=main.go:845 level=info msg="Stopping notify discovery manager..."
ts=2023-01-22T16:03:45.981Z caller=manager.go:955 level=info component="rule manager" msg="Stopping rule manager..."
ts=2023-01-22T16:03:45.981Z caller=manager.go:965 level=info component="rule manager" msg="Rule manager stopped"
ts=2023-01-22T16:03:45.981Z caller=main.go:827 level=info msg="Scrape discovery manager stopped"
ts=2023-01-22T16:03:45.981Z caller=main.go:841 level=info msg="Notify discovery manager stopped"
ts=2023-01-22T16:03:45.981Z caller=main.go:882 level=info msg="Stopping scrape manager..."
ts=2023-01-22T16:03:45.985Z caller=main.go:874 level=info msg="Scrape manager stopped"
ts=2023-01-22T16:03:45.994Z caller=notifier.go:608 level=info component=notifier msg="Stopping notification manager..."
ts=2023-01-22T16:03:45.996Z caller=main.go:1103 level=info msg="Notifier manager stopped"
ts=2023-01-22T16:03:45.996Z caller=main.go:1115 level=info msg="See you next time!"

See you next time,
Jonathan

2

It is saying something is sending Prometheus a SIGTERM which is a request for it to quit.

How are you running Prometheus? Within Kubernetes, using systemd, via some other mechanism?

3

I am just running it on EC2 with nohup ./prometheus --web.enable-admin-api &

Any idea who might be sending the SIGTERM or why?

Or is there a better way I should run it that protects it or restarts it automatically?

4

Generally you’d use something like systemd to manage a daemon

5

Thanks! I’ll try that.

6

Hello, have you fixed the problem? I have the same problem…