Plugins

Every DRSM check is a plugin: a folder with a small manifest and one BASH script. bin/drsm_run.sh finds the plugins each host needs, runs them, and writes the reports the dashboard and alerts use. Adding a check means adding a folder, not editing a core script.

Plugins included

These are the checks from bin/*_checks.sh and system_check.sh with the same messages and default levels. Their settings are listed in Thresholds and alerts.

Running checks with drsm_run.sh

$ ~/drsm/bin/drsm_run.sh -l             # list hosts and the plugins that would run
$ ~/drsm/bin/drsm_run.sh -n             # all hosts, no alerts
$ ~/drsm/bin/drsm_run.sh -n -H cloud1   # one host
$ ~/drsm/bin/drsm_run.sh -n -g dev      # the dev group
$ ~/drsm/bin/drsm_run.sh -m probes      # connectivity probes only

Exit code: 0 all good, 1 warnings, 2 errors.

Host files

One file per host in etc/hosts, named <host>.conf. The files are plain KEY=value lines, read but never run, so a typo cannot execute anything. Values with spaces go in double quotes.

HOST=cloud1
DESCRIPTION="AWS web server"
IMPACT="Public website is down"
GROUP=production
PLATFORM=linux
COLLECTOR=ssh
PROBES="network/ping network/https"
PLUGINS="storage/disk storage/raid core/cpu core/load core/memory network/nics core/users"

# Plugin settings: <category>_<name>_<SETTING>=value
storage_disk_WARN=85
storage_disk_SKIP="/mnt/scratch"
network_https_URL=https://www.example.com/
core_cpu_TIMEOUT=60

_defaults.conf is read before every host file, so put the common PLUGINS list and site wide settings there. Files starting with _ are not hosts. If network/ping fails the host is reported down and its remote plugins are skipped.

To convert systems.dat, dev_systems.dat and the custom levels in <host>_profile.sh scripts, run utils/migrate_systems_dat.sh.

Ansible inventory

If your Linux hosts are already in an Ansible inventory, DRSM can use it as its host list, so there is one list to keep. Add to ~/.drsm.sh:

export ANSIBLE_INVENTORY=/etc/ansible/hosts
export ANSIBLE_TIERS="production=prod qa=qa dev=dev"   # tier=group[,group]
export ANSIBLE_DEFAULT_TIER=""                          # tier for other hosts, empty leaves them out
export ANSIBLE_SHORT_NAMES=yes                          # web1.example.com becomes web1
export ANSIBLE_EXCLUSIVE=no                             # yes: Linux hosts only from Ansible

With ANSIBLE_EXCLUSIVE=yes, Linux host files of hosts the inventory does not list are skipped, so removing a host from Ansible stops its checks from the next run; Windows and other host files still run, and so do probe only hosts (PLUGINS="" or COLLECTOR=none, such as cluster addresses). Its last report stays on the dashboard, shown as not reporting, until utils/drsm_remove_host.sh <host> --apply removes it.

Hosts in a group named in ANSIBLE_TIERS, or in any group under it through [group:children] at any depth, are checked as Linux hosts over SSH in that tier, with the _defaults.conf plugins. A host in two tiers goes to the first one listed. A host file with the same short or full name, or the same HOST=, adds settings such as DESCRIPTION, PLUGINS or plugin levels, but the tier always comes from Ansible. Host files for hosts Ansible does not list, such as Windows hosts and network devices, work as before. Check the result with drsm_run.sh -l.

Ansible Automation Platform or AWX

To read the inventory from Ansible Automation Platform (AAP, formerly Ansible Tower) or AWX instead of a file:

export ANSIBLE_AAP_URL=https://aap.example.com
export ANSIBLE_AAP_INVENTORY=12                     # ID or name
export ANSIBLE_AAP_TOKEN_FILE=~/.drsm_aap_token     # OAuth2 token, mode 600
export ANSIBLE_AAP_CA_FILE=""                       # CA bundle for a private CA
export ANSIBLE_AAP_CACHE_MINUTES=15
export ANSIBLE_AAP_API=""                           # empty tries /api/controller/v2, then /api/v2

DRSM fetches the inventory script (<api>/inventories/<id>/script/) with curl and jq, keeps an INI copy in $VARdir for the cache time and reads it like a local inventory with ANSIBLE_TIERS. AAP 2.5 and later serve the controller API at /api/controller/v2, older AAP, Tower and AWX at /api/v2; both are tried. The older ANSIBLE_TOWER_* names of these settings still work. If the platform cannot be reached, the last good copy is used and a warning is printed. A read-only token is enough; it is sent from a header file, not on the command line. utils/drsm_ansible_hosts.sh (-f to fetch now) lists the hosts and tiers DRSM takes from either source.

INI inventories are read: [group], [group:children], numeric ranges such as web[01:03].example.com; [group:vars] and host variables are skipped. YAML inventories are not read; ansible-inventory -i inv.yml --list can show their groups for an INI copy.

Organizations and sites

To monitor several customers or offices from one dashboard, put hosts in organizations and sites: organization > site > group > host. Each site has its own reports, alerts and card on the Organizations page, and two sites may have a host with the same name. Hosts in no site are in the default/default site, which keeps the reports where they have always been, so nothing changes until you add a site.

etc/sites/acme.org            # NAME="Acme Corp", CONTACT="..."
etc/sites/acme/hq.site        # the hq site of acme

Any host key or plugin setting in a site file is a default for the site's hosts, read after _defaults.conf and before the host file. GROUP is one too, so a whole site can be a QA or DEV site. A host joins a site with SITE=<org>/<site>, or every host of a monitor host with DRSM_SITE in ~/.drsm.sh. IDs use a-z, 0-9, _ and -.

Site reports go to reports/orgs/<org>/<site> with the usual layout inside, plus site.json. Each site has its own lock, log and alert throttle, and the alert subject names the site. Plugins get DRSM_ORG and DRSM_SITE.

Tiers

GROUP sorts hosts into tiers: dev, qa and ops. The dashboard shows them as DEV, QA and OPS, with a tier filter on the Organizations, Heat map and Hosts pages. ops is another name for production, the default group: ops hosts run with the production hosts, write the production report names with no _<group> suffix and get texts, so existing production files keep their names. Run each tier from cron:

15,45 * * * * ~/drsm/bin/drsm_run.sh &> /dev/null                # OPS
55 09-16 * * mon-fri ~/drsm/bin/drsm_run.sh -g dev &> /dev/null
55 09-16 * * mon-fri ~/drsm/bin/drsm_run.sh -g qa &> /dev/null

DEV and QA alerts never send texts. A site file can send each tier's alerts to its own addresses with ALERT_EMAIL_DEV, ALERT_EMAIL_QA and ALERT_EMAIL_OPS. A tier without one uses ALERT_EMAIL, then alert.email.list. Other group names still work and show as their own tier, in upper case.

Site links

Site and org files can list links for the dashboard's Site links menu, one LINK line each:

LINK="Wiki|https://wiki.example.com/"
LINK="Grafana/Database|https://grafana.example.com/d/db"
LINK="Apps/OSA/Admin|https://osa.example.com/admin|ops"

The first field is the name, and each / in it makes a submenu, up to six levels. The second is the URL: only http and https links are kept, others are reported by drsm_run.sh and skipped, and the dashboard checks them again. The optional third field is a tier: the link shows only when the tier filter is All or that tier. Links in <org>.org show for every site of the organization, in an Org links section of the same menu.

drsm_run.sh puts the links in site.json, so drsm_web_sync.sh takes them to a hosted dashboard, and users only see the links of sites they may see.

Windows hosts

Windows hosts run the Windows Reporting Agent, a small Windows service from the agent distribution. Every 45 minutes it sends a text report (installed programs, Windows Defender events, Windows Update history, hotfixes, users, IP addresses) to the monitor side by signed HTTPS upload, a CIFS share, or SCP. DRSM never connects to the Windows host: the wra collector runs the windows/* plugins on the monitor host against each host's last report, $WRA_REPORTS_DIR/<host>_windows_report_agent.txt.

A Windows host file, also in etc/hosts/windows.conf_example:

HOST=laptop1
PLATFORM=windows
COLLECTOR=wra
PROBES=""
PLUGINS="windows/checkin windows/defender windows/patches windows/apps windows/resources"

PROBES="" turns off the ping probe, laptops move between networks. Set WRA_REPORTS_DIR once in _defaults.conf: it is reports_dir from the agent's server.ini, or the folder the CIFS or SCP delivery writes to. The default is /var/www/html/windows/reports. Set WRA_NAME when the agent reports under a name other than HOST. Linux plugins in _defaults.conf are skipped on Windows hosts.

For many hosts, utils/wra_hosts.sh makes one host file per report. It only lists what it would write until run with --apply, and never changes a host file that exists:

$ ~/drsm/utils/wra_hosts.sh -d /var/www/html/windows/reports
$ ~/drsm/utils/wra_hosts.sh -d /var/www/html/windows/reports --apply

windows/resources uses the metric names of the Linux plugins (cpu_used_pct, mem_available_mb, disk_used_pct{mount="C:"}, nic_rx_bytes{nic="Wi-Fi"} and so on), so Windows hosts get the same gauges and trends. Its sample_etime metric is the time the agent read the counters: the trends page times network rates by the reports, and counts a report that two runs saw only once.

The windows/apps lists are case insensitive regular expressions separated by semicolons, because program names have spaces: windows_apps_REQUIRED="Microsoft Defender;Mozilla Firefox".

A host that has never sent a report, or whose report file is empty, is an ERROR from windows/checkin; the other Windows plugins are UNKNOWN until a report arrives. To run a Windows plugin by hand:

$ DRSM_TARGET=laptop1 DRSM_WRA_REPORT=/var/www/html/windows/reports/laptop1_windows_report_agent.txt \
    bash ~/drsm/plugins/windows/checkin/check.sh

Writing a plugin

$ ~/drsm/utils/new_plugin.sh local/myapp            # runs on the host
$ ~/drsm/utils/new_plugin.sh local/myapp_url probe  # runs from the monitor against the host

This creates the folder, a commented plugin.conf, and a script you can run by hand right away:

$ DRSM_WARN=50 bash ~/drsm/plugins/local/myapp/check.sh
$ DRSM_TARGET=cloud1 bash ~/drsm/plugins/local/myapp_url/probe.sh

Then add local/myapp to PLUGINS (or PROBES) in a host file. Site plugins live in plugins/local, which updates never overwrite.

Manifest

NAME=myapp
CATEGORY=local
DESCRIPTION="Order queue is draining"
PLATFORMS="linux"      # linux, windows, or any
MODE=remote            # remote runs check.sh on the host, probe runs probe.sh on the monitor
TIMEOUT=30             # seconds, enforced by the runner
REQUIRES="curl jq"     # commands that must exist, otherwise the plugin is UNKNOWN
NEEDS_SUDO=no          # informational: no, yes, or optional

# Every other key is a setting with its default
WARN=100
CRIT=500
QUEUE_DIR=/var/spool/myapp

Each setting reaches the script as DRSM_<KEY>. A host overrides it with <category>_<name>_<KEY>, for example local_myapp_WARN=200, and can set <category>_<name>_TIMEOUT.

Script

#!/bin/bash
source "${DRSM_LIB:-$(dirname "$0")/../../../lib}/drsm_plugin.sh"

WARN=$(drsm_setting WARN 100)
CRIT=$(drsm_setting CRIT 500)
QUEUE_DIR=$(drsm_setting QUEUE_DIR /var/spool/myapp)

count=$(find "${QUEUE_DIR}" -type f | wc -l)
if [ "${count}" -ge "${CRIT}" ]; then
    drsm_crit "${HOST} myapp queue has ${count} files"
elif [ "${count}" -ge "${WARN}" ]; then
    drsm_warn "${HOST} myapp queue has ${count} files"
else
    drsm_ok "${HOST} myapp queue checks good at ${count} files"
fi
drsm_metric myapp_queue_files "${count}"

drsm_exit

States and exit codes

Exit codes follow the Nagios plugin convention:

0 OK 1 WARNING 2 ERROR 3 UNKNOWN

• An ERROR - or WARNING - line counts even if the script exits 0.
• A plugin that runs past its TIMEOUT is stopped and reported as a WARNING.
• A missing REQUIRES command makes the plugin UNKNOWN, which shows in the report but does not alert.
• A plugin that prints nothing is reported as an ERROR.

How a run works

For each host the runner builds one job script with a line per plugin, its settings and its timeout. Probe jobs run on the monitor host. Remote jobs go through the host's collector:

• local runs the job in place from $DRSMHOME.
• wra runs the windows/* plugins in place from $DRSMHOME against the host's last Windows Reporting Agent report, see Windows hosts.
• ssh-linux copies lib/drsm_plugin.sh and the host's plugins to ~/drsm_agent as one tar stream, skipped when the checksum has not changed, then runs the whole job in one SSH session. Set DRSM_REMOTE_BASE for another directory.

Each plugin's output is a section of the host report:

=== storage/disk [WARNING] 0s ===
Disk usage checks for cloud1
WARNING - cloud1:/var is at 92% capacity.
METRIC disk_used_pct{mount="/var"}=92

Groups other than production (and ops, its other name) add _<group> to the group report names. Alerts use bin/text_email_alert.sh with the same subjects and 4 hour throttle as before.

Files

bin/drsm_run.sh               plugin runner
lib/drsm_plugin.sh            helpers every plugin sources
lib/drsm_conf.sh              KEY=value reader for manifests and host files
lib/drsm_job.sh               runs plugins on the target, with timeouts
collectors/local.sh           runs plugins on the monitor host itself
collectors/ssh-linux.sh       copies and runs plugins on Linux hosts over SSH
collectors/wra.sh             reads Windows Reporting Agent reports for Windows hosts
plugins/<category>/<name>/    plugin.conf plus check.sh or probe.sh
plugins/local/                your own plugins, never overwritten by updates
etc/hosts/_defaults.conf      settings shared by all hosts
etc/hosts/<host>.conf         one file per host
utils/new_plugin.sh           creates a plugin from a template
utils/migrate_systems_dat.sh  converts systems.dat and profiles to host files
utils/wra_hosts.sh            creates host files for Windows agent hosts

Roadmap

Later phases of the plugin design add more checks (SMART, mounts, backup age, TCP ports, TLS expiry, DNS, time sync, failed services), generic database, container, service and process plugins with a custom/command catch all, live Windows checks over OpenSSH and PowerShell alongside the Windows Reporting Agent, and configurable notifiers (email, SMS, Slack, ntfy, webhook) with alert on change.