sys_monitor(8)

NAME

sys_monitor - monitor zpool capacity, stale pool locks and host time sync

SYNOPSIS

/opt/omnicube/sbin/sys_monitor

sys_monitor

DESCRIPTION

sys_monitor is the host-level health probe of an OmniCube node. In one pass it grades the capacity of every ZFS pool, detects OmniCube pool locks that have been left behind by a killed job, re-synchronises the clock and makes sure the system logger is still writing. It takes no options and no operands and is meant for unattended periodic execution from root's crontab, alongside the other periodic jobs whose sample entries are shipped in /opt/omnicube/share/README. It may also be run by hand; a hand run that collides with the cron run exits 0 without output.

Unlike the other monitors in /opt/omnicube/sbin this script does not set its own PATH, so the invoking environment must already resolve zpool, zfs, svcprop, svcs, netstat and pfexec. ntpdate is called by absolute path as /usr/sbin/ntpdate. It sources /opt/omnicube/lib/common/utils.sh for logging, locking, the run-level guard and the site property prefix.

Locking

A single-instance lock directory /var/run/omnicube/<site>sys_monitor.sys_monitor.lock is created atomically with acquire_lock_or_exit(). If it already exists the script exits 0 without output, because overlapping runs would double-increment the per-pool lock-streak counters described below and produce a false alert. The lock is released by a trap on EXIT, SIGHUP, SIGINT and SIGTERM, so it is also released when the run-level guard aborts the run.

Run-level guard

Immediately after taking the lock the script calls abort_if_shutting_down and exits 0 silently when the host is at run level 0, 1, 5, 6, S or s, or when /etc/nologin exists. The guard is re-evaluated at the top of each pool iteration, so a run-level change part way through the capacity sweep stops the pass before the next pool. OC_IGNORE_RUNLEVEL set to a non-empty value overrides it.

Pool capacity grading

Every pool from zpool list is considered, except those listed exactly in config/monitor_exclude_pools. A pool whose capacity is below config/monitor_pool_min_level, default 70 percent, is skipped silently. For the rest, each dataset in the pool that actually consumes space in snapshots, that is, whose usedbysnapshots is not 0B, contributes a block containing the pool capacity and the output of

zfs get usedbysnapshots,referenced,used,available <dataset>

If no dataset in the pool consumes snapshot space, one such block is emitted for the pool itself. Blocks are graded by pool capacity and accumulated into three messages, each emitted once at the end of the sweep:

90 percent and above: Prefixed CAPACITY_ERROR> and reported with error(), which mails oc-alert.
80 to 89 percent: Prefixed CAPACITY_WARN> and reported with warning(), which mails oc-warn.
below 80 percent: Also prefixed CAPACITY_WARN> but reported with info(), which only logs. Reaching this branch requires config/monitor_pool_min_level to be set below 80.

Stale pool lock detection

OmniCube marks a pool as busy with the ZFS user property site:is_locked while autosync(8), autocleansnap(8) or a manage_zone.sh(8) relocation is in flight. A job killed between setting and clearing that property leaves the pool locked forever, which silently blocks later jobs. sys_monitor therefore keeps a persistent streak counter per pool in /var/run/omnicube/sys_monitor/lock_streak.<pool>, holding a decimal count of consecutive observations of is_locked=true. The first observation of an unlocked pool deletes the counter file. From the third consecutive locked observation onward, and on every run after that until the property is cleared, zpool <pool> is locked (streak=<n>) is reported with warning(). Below the threshold the lock is assumed to belong to a legitimate in-flight job and nothing is said.

The counter directory is created with mode 1775 and group config/lock_group, so root (cron, SMF) and RBAC operators can both update the counters whoever ran first. If it cannot be prepared, the script logs cannot prepare the directory and lock-streak detection disabled this run with error() and continues with the capacity sweep only. Counter files for pools that no longer exist are reaped at the start of every run. The counters live on /var/run (tmpfs) and therefore reset at boot, consistently with the property being runtime-only.

Generic host checks

The clock is re-synchronised with /usr/sbin/ntpdate -bsq against the servers listed in config/ntp_servers, or, when that property is empty, against the default route's gateway as reported by netstat -rn. Finally, if /var/log/misc.log is empty or missing, the logger is restarted with pfexec svcadm restart rsyslog, on the assumption that an empty log means nothing is being written.

Alerting

info(), warning() and error() from utils.sh write to stdout or stderr and to syslog as user.info, user.warn and user.error under the tag <site>sys_monitor, which /etc/rsyslog.d/omnicube.conf routes to /var/log/misc.log. warning() mails oc-warn and error() mails oc-alert, from $HOSTNAME@<config/mail_domain>, throttled to one mail per (severity, message) pair per config/mail_throttle_window seconds, default 3600.

OPERANDS

This command takes no options and no operands. Any argument given is ignored.

SMF PROPERTIES

Read from svc:/omnicube/sysadm:default unless OC_SMF_INSTANCE selects another instance.

config/site: Mandatory. Prefix of the site:is_locked ZFS property, of the syslog tag and of the lock directory name. If unset, sourcing utils.sh fails and the script exits 78.
config/monitor_pool_min_level: Capacity percentage below which a pool is not reported at all. Default 70 when the property is unset or not numeric.
config/monitor_exclude_pools: String list of pool names to skip entirely. Matched literally, character for character, so a name containing a dot is not treated as a wildcard. Empty or unset means no exclusions.
config/ntp_servers: String list of NTP servers for the ntpdate sync. Empty or unset falls back to the default route's gateway.
config/lock_group: Group applied to /var/run/omnicube and to the streak counter directory (mode 1775). Default sysadmin.
config/mail_domain: Domain used in the envelope sender of alert mail.
config/mail_throttle_window: Alert-mail throttle window in seconds. Default 3600.

EXAMPLES

Example 1: hourly cron entry

16 * * * * /opt/omnicube/sbin/sys_monitor >/dev/null

Example 2: raise the reporting floor and exclude a scratch pool

svccfg -s svc:/omnicube/sysadm:default \\
    setprop config/monitor_pool_min_level = astring: 85
svccfg -s svc:/omnicube/sysadm:default \\
    setprop config/monitor_exclude_pools = astring: ( "scratch" "tmppool" )
svcadm refresh svc:/omnicube/sysadm:default

Example 3: investigate a stale pool lock warning

Read the streak counter, then clear the property by hand once the owning job is known to be dead.

cat /var/run/omnicube/sys_monitor/lock_streak.zn01pool
zfs get -H -o value site:is_locked zn01pool
pfexec zfs inherit site:is_locked zn01pool

Example 4: run one pass by hand during single-user maintenance

OC_IGNORE_RUNLEVEL=1 pfexec /opt/omnicube/sbin/sys_monitor

EXIT STATUS

sys_monitor does not set an explicit exit status on its normal path; it returns the status of the last check in generic_mon, which makes the status useless as a health verdict. Do not gate anything on it.

0: Returned silently when the run-level guard trips and when another instance already holds the lock. Also returned on a completed pass in which /var/log/misc.log was empty and the logger restart succeeded.
1: Returned by a completed pass in which /var/log/misc.log was not empty, that is, on the ordinary healthy path: the final test on the log file is the last statement executed and its false result becomes the script's status. The same value is returned when the lock directory could not be created for a reason other than "already exists"; that case is distinguishable only by the accompanying acquire_lock_or_exit: cannot create lock message.
78: config/site is not set on the SMF instance, so utils.sh refused to initialise.

FILES

/var/run/omnicube/<site>sys_monitor.sys_monitor.lock: Single-instance lock directory, released by the EXIT trap.
/var/run/omnicube/sys_monitor/: Persistent lock-streak state, mode 1775, group config/lock_group.
/var/run/omnicube/sys_monitor/lock_streak.<pool>: Consecutive count of site:is_locked observations for one pool. Removed when the pool unlocks or disappears.
/var/run/omnicube/mail_throttle/<user>/: Per-user alert-mail throttle markers, mode 0700.
/var/log/misc.log: Destination of the syslog records, and the file whose emptiness triggers the logger restart. Rotated by the omnicube entry in /etc/logadm.d/omnicube.conf.
/opt/omnicube/lib/common/utils.sh: Shared shell library, sourced at start-up.
/etc/nologin: Presence means a shutdown is in progress; the script exits 0.

ENVIRONMENT

OC_SMF_INSTANCE: Selects the svc:/omnicube/sysadm instance whose config properties are read. Unset means default.
OC_IGNORE_RUNLEVEL: When non-empty, bypasses the run-level guard.
PATH: Inherited, not set by the script. It must resolve the ZFS, SMF and networking utilities listed under DESCRIPTION.
HOSTNAME: Used in the envelope sender of alert mail.

SECURITY

The only privileged action is pfexec svcadm restart rsyslog, which requires the OmniCube Administration profile from /etc/security/prof_attr.d/omnicube and the /usr/sbin/svcadm entry in /etc/security/exec_attr.d/omnicube. The ZFS and pool queries are read-only and need no privilege. The streak counter directory is mode 1775 with the sticky bit set, so an operator cannot remove another user's counter files.

NOTES

A stale-lock warning is a report, not a repair. sys_monitor never clears site:is_locked itself, because it cannot distinguish a very long legitimate job from a dead one; clearing the property is an operator decision.

sys_monitor does not consult the isolation marker /var/run/omnicube/isolate_node.lock, so capacity and lock-streak alerts continue to be raised, and mailed, while the node is isolated. See isolate_node.sh(8).

Because the streak state lives on tmpfs, a reboot resets every counter and the threshold of three consecutive observations must be reached again before a genuinely orphaned lock is reported.

The capacity messages embed live zfs get output, which changes between runs, so consecutive reports about the same pool hash to different alert-mail throttle keys and may mail more than once per window.