JSON alerts for Munin

Munin is a great and easy way to monitor your servers. The documentation for Munin is kind of all over the place, and sometimes not all that clear.

• There is the recent “Munin Guide” which is quite informative and well laid out.
• Sometimes additional info and background can be found on the old munin wiki.

Getting the standard mail alerts working on munin is simplest, but as soon as you want custom layouts or send alerts as a push message to a phone, it can take quite some work. Here’s what I figured out to get the Munin alert available to you as JSON inside your own Python script so you can do anything you want with it.

On the munin master, create the file /etc/munin/munin-conf.d/alerts.conf with the following contents:

contact.python.command /etc/munin/scripts/alert.py
contact.python.text  { "group":"${var:group}", "host":"${var:host}",\
                       "graph_category":"${var:graph_category}",\
                       "graph_title":"${var:graph_title}",\
                       "warning":[ \
                           ${loop<,>:wfields {"label":"${var:label}",\
                                              "value":"${var:value}",\
                                              "w":"${var:wrange}",\
                                              "c":"${var:crange}",\
                                              "extra":"${var:extinfo}"} }\
                       ],\
                       "critical":[ \
                           ${loop<,>:cfields {"label":"${var:label}",\
                                              "value":"${var:value}",\
                                              "w":"${var:wrange}",\
                                              "c":"${var:crange}",\
                                              "extra":"${var:extinfo}"} }\
                       ],\
                       "unknown":[ \
                           ${loop<,>:ufields {"label":"${var:label}",\
                                              "value":"${var:value}",\
                                              "w":"${var:wrange}",\
                                              "c":"${var:crange}",\
                                              "extra":"${var:extinfo}"} }\
                       ]\
                     }

This will make Munin output JSON to the “python” user, you can chose another name if needed. After this, you can create a Python script looking somewhat like this:

#!/usr/bin/python
#
# Save as /etc/munin/scripts/alert.py
# chown munin:munin /etc/munin/scripts/alert.py
# chmod +x /etc/munin/scripts/alert.py
#

import sys
import os
import json

def humanreadable( jsonalert ):
  hascritical = (len(jsonalert['critical']) > 0)
  haswarning  = (len(jsonalert['warning'])  > 0)
  hasunknown  = (len(jsonalert['unknown'])  > 0)

  report = ""

  if hascritical:
    for i in jsonalert['critical']:
      report += "CRITICAL: {0} {1}: {2}={3}".format(jsonalert['host'],
                       jsonalert['graph_title'], i['label'],i['value'])

  if haswarning:
    for i in jsonalert['warning']:
      report += "WARNING: {0} {1}: {2}={3}".format(jsonalert['host'],
                       jsonalert['graph_title'], i['label'],i['value'])

  if hasunknown:
    for i in jsonalert['unknown']:
      report += "UNKNOWN: {0} {1}: {2}={3}".format(jsonalert['host'],
                       jsonalert['graph_title'], i['label'],i['value'])

  if hascritical + haswarning + hasunknown == 0:
    report += "OK: {0} {1}".format(jsonalert['host'],jsonalert['graph_title'])

  return report

# Get the output from Munin stdout and print it.
for line in sys.stdin:
  j = json.loads(line)
  readable = humanreadable(j)
  print readable

There is no debugging without testruns. To see if this works, you can run this command:

sudo -u munin /usr/share/munin/munin-limits --force

This should give you output which looks somewhat like this:

WARNING: monitor.local Logged in users: pts=4.00
OK: monitor.local Inode usage in percent
OK: monitor.local eth0 errors
OK: monitor.local File table usage
OK: monitor.local Disk usage in percent
OK: monitor.local routerboard CPU usage
OK: monitor.local Disk latency per device :: Average latency for /dev/mmcblk0

Bingo! Munin sends JSON to your Python script! Now you can use your awesome Pyhton skills to change this into whatever you want or send. To get you inspired, here’s a piece of code I use to send alerts to a Mattermost server:

import requests

mattermost_url = "https://<your-server-here>/hooks/<your-hook-here>"

def sendmattermost( alerts ):
  payload = '{"username": "Monitor", "text": "'

  if "WARNING:" in alerts or "CRITICAL:" in alerts:
    payload += ':rotating_light:\n'
  else:
    payload += ":pager:\n"

  payload += alerts
  payload += '"}'

  print "Sending payload: '" + payload +"'."
  r = requests.post(mattermost_url, data=payload)
  if r.status_code != 200:
    print "PANIC " + r.reason +" : " + r.text

You can find this trick and other munin plugin stuff in my munin-plugins repository.

Happy alerting!