Daemonize prometheus on Amazon EC2 with init.d

Prometheus server
1

Greetings. I am attempting to install and run Prometheus on an AWS EC2 instance that is running “Amazon Linux.” It’s important to note that it is not Amazon Linux 2. We’d like to have Prometheus start with the instance and be manageable as a service, but this flavor of Linux does not have systemd, so requires an init.d script. The system also does not have “start-stop-daemon” installed, and I’d rather not add it. I’ve been trying to write a script (sorry, I’m new to this stuff), but have not had much success. Here’s what I have so far:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          Prometheus
# Required-Start:    $local_fs $network $named $time $syslog
# Required-Stop:     $local_fs $network $named $time $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Description:       Prometheus - Monitoring system & time series database
### END INIT INFO

EXEC="/opt/prometheus/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --web.listen-address=0.0.0.0:9090 --web.external-url="
NAME="prometheus"
USER="prometheus"
PIDFILE=/var/run/$name.pid
LOGFILE=/var/log/prometheus/$name.log
LOCKFILE=/var/lock/subsys/$name

# execute init.d functions
. /etc/rc.d/init.d/functions

do_start() {
    echo -n "Starting prometheus"
    if [ -f $PIDFILE ]; then
        pid = `cat $PIDFILE`
        echo prometheus already running: $pid
        exit 2;
    else
        daemon --user $USER --pidfile=$PIDFILE $EXEC >> $LOGFILE 2>&1
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $LOCKFILE
        return $RETVAL
    fi
}

do_stop() {
    echo -n "Shutting down prometheus"
    echo
    killproc -p $PIDFILE $NAME
    echo
    rm -f $LOCKFILE
}

case "$1" in
    start)
        do_start
        ;;
    stop)
        do_stop
        ;;
    status)
        status $NAME
        ;;
    restart)
        do_stop
        do_start
        ;;
    *)
        echo "Usage: {start|stop|status|restart}"
        exit 1
        ;;
esac

exit 0

If I run the script…

sudo /etc/init.d/prometheus start

…it echoes the “Starting prometheus” message and then hangs. When I break the hanging init.d script, I can find that Prometheus is indeed running (using ps ax), but the PID and log files do not exist.

How do I write this script so it behaves like an init.d script, e.g., firing Prometheus in a separate daemon process? And why aren’t my log and PID files working? Again, this is my first time writing an init.d script, thanks for your patience.

2

I have no fresh experience with init.d. Maybe that prometheus was started once outside of the init.d script?

On RHEL6, here is what I am using (based on upstart):

description "Prometheus Service"

start on startup
stop on shutdown

respawn
respawn limit 10 300

script
    . /etc/default/prometheus
    exec chroot --userspec prometheus:prometheus / /usr/bin/prometheus \
          --config.file=/etc/prometheus/prometheus.yml $PROMETHEUS_ARGS

Write that to /etc/init/prometheus.conf
and run it with initcl start prometheus.

3

Thanks. I ended up going a different direction and using PM2 (https://pm2.keymetrics.io/) to manage Prometheus and Alert Manager. We were already using it for some Node processes so it worked out nicely.