Running node-exporter on Kubernetes without Root Privileges

Exporters and Metrics
1

We use node-exporter on Kubernetes (k8s) to monitor /dev/mapper devices. Unfortunately, node-exporter can only retrieve information about a device when it is running with root privileges, as indicated by the following securityContext configuration:

securityContext:
    runAsUser: 0

Without this configuration, we encounter the ‘node_filesystem_device_error’ with details like the device path, file system type (fstype), and mount point:

node_filesystem_device_error{device="/dev/mapper/3624a937010ed65ee4214591800042af7", fstype="ext4", mountpoint="/var/lib/kubelet/pods/7adbc046-d10e-41e7-b64a-774047217cd2/volumes/kubernetes.io~portworx-volume/pvc-3322b508-d83d-4f11-a570-30f606bdb53a"} 1

However, for security reasons, we want to avoid running node-exporter as the root user. Is there a possible solution to address this security concern?

2

You can take a look at adding just the capabilities you need instead of running as root, but be aware that the node exporter does need a fair amount of privilege to do its work.

3

Technically the node_exporter should not need root for anything. It’s explicitly in the code policy that root should not be required for any feature.

However, some permissions/configuration may be required for some features.

In addition, monitoring PVCs is actually provided by kube-state-metrics. Monitoring them with the node_exporter is not correct.

4

Thanks for the replies. This Move RO status before error return by metbog · Pull Request #2807 · prometheus/node_exporter · GitHub should help with our problem. Does anyone know when the new release of node_exporter will be out? Is there a schedule for it?

5

I just discovered that node_dmi_info for my fleet is missing product_serial, most likely because

# ls -l /sys/class/dmi/id/
total 0
-r--r--r-- 1 root root 4096 Sep 13 09:41 bios_date
-r--r--r-- 1 root root 4096 Sep 13 09:41 bios_vendor
-r--r--r-- 1 root root 4096 Sep 13 09:41 bios_version
-r--r--r-- 1 root root 4096 Sep 13 09:41 board_asset_tag
-r--r--r-- 1 root root 4096 Sep 13 09:41 board_name
-r-------- 1 root root 4096 Sep 13 09:41 board_serial
-r--r--r-- 1 root root 4096 Sep 13 09:41 board_vendor
-r--r--r-- 1 root root 4096 Sep 13 09:41 board_version
-r--r--r-- 1 root root 4096 Sep 13 09:41 chassis_asset_tag
-r-------- 1 root root 4096 Sep 13 09:41 chassis_serial
-r--r--r-- 1 root root 4096 Sep 13 09:41 chassis_type
-r--r--r-- 1 root root 4096 Sep 13 09:41 chassis_vendor
-r--r--r-- 1 root root 4096 Sep 13 09:41 chassis_version
-r--r--r-- 1 root root 4096 Mar 13 16:46 modalias
drwxr-xr-x 2 root root    0 Sep 13 09:42 power
-r--r--r-- 1 root root 4096 Sep 13 09:41 product_name
-r-------- 1 root root 4096 Sep 13 09:41 product_serial
-r-------- 1 root root 4096 Sep 13 09:41 product_uuid
-r--r--r-- 1 root root 4096 Sep 13 09:41 product_version
lrwxrwxrwx 1 root root    0 Mar 13 16:46 subsystem -> ../../../../class/dmi
-r--r--r-- 1 root root 4096 Sep 13 09:41 sys_vendor
-rw-r--r-- 1 root root 4096 Sep 13 09:42 uevent
#

So to get the chassis/product serial numbers it would seem that I either need to chmod o+r /sys/class/dmi/id/* with a boot-time cron job (ugh) or run node_exporter as root. If anyone has a better solution, I’d love to know. I’d also love to know why this is not u+r by default.