I am a great fan of using munin for monitoring my servers and I have quite an extensive setup that even sends me Telegram messages when something is wrong.
This weekend I ran into an interesting problem with a backupserver where the ‘df’ plugin on the node did not want to list the backup disk, and I’ve spent quite some time trying to figure out what the problem was.
Here’s the fstab setup for my machine:
proc /proc proc defaults 0 0 PARTUUID=698b7a16-01 /boot vfat defaults 0 2 PARTUUID=698b7a16-02 / ext4 defaults,noatime 0 1 UUID=6ef2cb75 /home/pi/usbdisk ext4 defaults 0 2
When I run ‘df’ on my machine it returns the expected results:
# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 29522500 1377884 26912408 5% / devtmpfs 465280 0 465280 0% /dev tmpfs 498560 0 498560 0% /dev/shm tmpfs 498560 19128 479432 4% /run tmpfs 5120 4 5116 1% /run/lock tmpfs 498560 0 498560 0% /sys/fs/cgroup /dev/mmcblk0p1 258095 49767 208329 20% /boot /dev/sda1 3845568008 1698470112 1951731164 47% /home/pi/usbdisk tmpfs 99712 0 99712 0% /run/user/1000
When I use munin-run to check the output of the ‘df’ plugin this also works:
# munin-run df fetch _dev_root.value 4.87051883381055 _dev_shm.value 0 _run.value 3.83664955070603 _run_lock.value 0.078125 _sys_fs_cgroup.value 0 _dev_mmcblk0p1.value 19.2823600520736 _dev_sda1.value 46.5308618230838
But for some mysterious reason, /dev/sda1 did not show up when connecting through the network:
# nc localhost 4949 # munin node at backuppi fetch df _dev_root.value 4.87051883381055 _dev_shm.value 0 _sys_fs_cgroup.value 0 _run.value 3.83664955070603 _run_lock.value 0.078125 _dev_mmcblk0p1.value 19.2823600520736 .
After hours of debugging the perl script (I can not recommend that to anyone) and trying restarting, rebooting and other voodoo, I found a non-related redit post about systemd and protecting home directories. The keen-eyed reader has spotted that my disk is mounted on /home/pi/usbdisk (let’s say that’s for artistic reasons).
A quick peek in /lib/systemd/system/munin-node.service shows that this is property indeed set to ProtectHome=true (which makes sense). Of course I have the option to mount the disk somewhere else, but since my raspberry is behind 2 (!) firewalls and only operated by myself I solved the problem as follows:
Edit /lib/systemd/system/munin-node.service and set ProtectHome=false.
# systemctl daemon-reload
# systemctl restart munin-node
A quick check shows that the disk is now indeed listed:
# nc localhost 4949 # munin node at backuppi fetch df _dev_root.value 4.87051883381055 _dev_shm.value 0 _sys_fs_cgroup.value 0 _run.value 3.83664955070603 _run_lock.value 0.078125 _dev_mmcblk0p1.value 19.2823600520736 _dev_sda1.value 46.5308618230838
Success!