NAME
pool_monitor.sh - monitor and repair iSCSI sessions, bridges and ZFS pools
SYNOPSIS
/opt/omnicube/sbin/pool_monitor.sh
pool_monitor.sh
DESCRIPTION
pool_monitor.sh is the storage-path watchdog of an OmniCube node. It is unattended: it takes no options and no operands, and it is normally driven periodically from root's crontab, in the same manner as the other periodic jobs whose sample entries are shipped in /opt/omnicube/share/README. It can also be run by hand at any time; a hand run that collides with the cron run simply exits 0, see Locking below.
The script sets an explicit, exported PATH of /usr/sbin:/sbin:/opt/ooce/sbin:/usr/bin:/usr/gnu/bin:/opt/ooce/bin and does not inherit the caller's. This is required for correctness rather than convenience: pfexec matches the entries of /etc/security/exec_attr.d/omnicube on the resolved path of the binary, so sqlite3 must resolve to /opt/ooce/bin/sqlite3 to pick up its privilege grant, and /usr/gnu/bin is needed for tac.
It then sources /opt/omnicube/lib/common/utils.sh for logging, locking, the run-level guard and the node list, and reads its configuration from the SMF instance svc:/omnicube/sysadm:default as described under SMF PROPERTIES below.
Run-level guard
Before any lock is taken the script calls abort_if_shutting_down and exits 0 silently when the host is at run level 0, 1, 5, 6, S or s, or when /etc/nologin exists. Pool repair must never race a shutdown, a reboot or a drop to single-user. The guard is re-evaluated inside the per-device repair loop with is_shutting_down, so a run-level change part way through a long pass aborts before the next zpool clear or zpool scrub. Setting OC_IGNORE_RUNLEVEL to a non-empty value bypasses the guard for deliberate interactive maintenance.
Locking
Single-instance behaviour comes from the atomic creation of the directory /var/run/omnicube/pool_monitor.lock. If it already exists a concurrent run is assumed and the script exits 0 without output, so overlapping cron ticks generate no mail. The lock is removed on the normal exit path and by the signal handler, which traps SIGHUP, SIGINT, SIGQUIT, SIGABRT, SIGALRM and SIGTERM, logs sig_cleanup called through error() and exits 96.
Isolation check
At four points in the pass (before the bridge check, before the scrub census, after each repaired device, and before the degraded-pool sweep) the script tests for the shared isolation marker directory /var/run/omnicube/isolate_node.lock. When it is present the script first runs
/opt/omnicube/sbin/isolate_node.sh reconcile
to give a temporary (auto) isolation the chance to heal: if the remote node answers again, reconcile removes the marker and this pass continues normally, which is what brings the offlined pools back. A manual isolation is deliberately left untouched by reconcile; the marker therefore survives, and pool_monitor.sh logs Isolation mode detected, aborting at warning level, releases its lock and exits 0. See isolate_node.sh(8).
Bridge repair
dladm show-bridge is enumerated and, for every bridge, each link reported disabled is removed from and re-added to the bridge with pfexec dladm remove-bridge -l and pfexec dladm add-bridge -l, with a two-second pause after each link.
iSCSI session census
The expected session count is the number of storage nodes multiplied by config/iscsi_num_sessions. The node list is taken from config/storage_nodes and falls back to config/nodes; an empty list counts as one node. The observed count is the sum of the Connections fields of pfexec iscsiadm list target. An exact match ends the check. A count of at least half the expected value is treated as degraded and triggers a reconnect attempt at level 0 followed by level 1. Anything lower is reported through error(), which sends mail to oc-alert, together with the full iscsiadm list target -v output and the text manual intervention needed; a level-2 reconnect attempt is made and the script exits 2.
iSCSI reconnect levels
Per-target connection counts are accumulated from iscsiadm list target -v and compared against config/iscsi_num_sessions. For each target below that value the discovery address is looked up in the iscsi_sessions table (columns target and discover_addr) of /opt/efitsrv/sysinfo.db using pfexec sqlite3, and probed with nc -vz addr 3260. Both the target name and the address are matched against ^[a-zA-Z0-9.:_-]+$ before use; a value containing anything else is skipped with a Suspicious warning, because both are external input that would otherwise flow into an SQL string literal and a privileged command line. Then:
Device error repair
zpool status is scanned for devices matching the c0tXXXd0 naming convention, with an optional slice or partition suffix, that are ONLINE yet carry a non-zero READ, WRITE or CKSUM counter. For each such device the LUN id is derived from the device name and looked up with pfexec /usr/sbin/stmfadm list-lu -v locally, and over ssh on the other hosts in config/nodes. Remote nodes are validated with validate_name() and pre-probed with ssh_reachable(), so a peer that is down costs one fast probe instead of a long ssh timeout per LUN per tick, and an ssh failure is not mistaken for "LU not defined here". The zone name is recovered from the logical unit's Data File path; because a zone's pool is not guaranteed to carry the zone's name, the owning pool is then resolved from the device itself out of zpool status. A pool whose site:is_locked ZFS property is true is left alone. Otherwise the failure is reported through warning(), zpool status for the pool is printed, and repair proceeds with
pfexec zpool clear <pool> c0t<lun>d0 pfexec zpool scrub <pool>
A device that cannot be mapped to a pool produces a Could not map device warning and is skipped.
Scrub throttle
The number of pools reporting scrub in progress or resilver in progress is counted before the repair loop, and each repair started by this pass increments it. At three or more the script stops and exits 0, so a node recovering from multiple faults is not flooded with concurrent scrubs.
Degraded pool sweep
Finally, every pool other than rpool and other than those whose name contains this host's short name (the hostname truncated at the first hyphen) is examined for a DEGRADED, UNAVAIL, FAULTED, OFFLINE or REMOVED state. The candidate list is built with shell pattern matching rather than a regular expression so that a metacharacter in the hostname cannot act as a wildcard. If any pool is degraded a warning is issued, pfexec format is run with stdin from /dev/null to force a device rescan, and the zone service is recovered from svcs -xv omnicube/zone:default by pfexec svcadm enable svc:/service/omnicube/zone:default when it is disabled, or pfexec svcadm clear on the same FMRI when it is in maintenance.
Alerting
info(), warning() and error() from utils.sh write to stdout or stderr and to syslog as user.info, user.warn and user.error under the tag <site>pool_monitor, which /etc/rsyslog.d/omnicube.conf routes to /var/log/misc.log. In addition, warning() mails oc-warn and error() mails oc-alert, from $HOSTNAME@<config/mail_domain>. Mail is throttled per (severity, message) pair: an identical message is mailed at most once per config/mail_throttle_window seconds, default 3600.
OPERANDS
This command takes no options and no operands. Any argument given is ignored.
SMF PROPERTIES
Read from svc:/omnicube/sysadm:default unless OC_SMF_INSTANCE selects another instance.
EXAMPLES
Example 1: hourly cron entry
Run the monitor every hour, discarding routine stdout so that only genuine diagnostics generate cron mail.
56 * * * * /opt/omnicube/sbin/pool_monitor.sh >/dev/null
Example 2: run one pass by hand and read the log
pfexec /opt/omnicube/sbin/pool_monitor.sh tail /var/log/misc.log
Example 3: run a pass while the host is in single-user
The run-level guard normally aborts. Override it deliberately, for one invocation only.
OC_IGNORE_RUNLEVEL=1 pfexec /opt/omnicube/sbin/pool_monitor.sh
Example 4: check why the monitor keeps aborting
/opt/omnicube/sbin/isolate_node.sh status /opt/omnicube/sbin/isolate_node.sh release
EXIT STATUS
FILES
ENVIRONMENT
SECURITY
Every privileged operation is performed through pfexec, using the OmniCube Administration profile in /etc/security/prof_attr.d/omnicube and the command list in /etc/security/exec_attr.d/omnicube; ${PFEXEC} expands to nothing when the caller is already root. The profile grants zpool, dladm, format, iscsiadm, stmfadm, svcadm and sqlite3 as uid 0 and is root-equivalent as a whole, so it must only be assigned to accounts trusted with root. iSCSI target names and discovery addresses read from /opt/efitsrv/sysinfo.db are validated before they are interpolated into SQL literals or privileged command lines.
SEE ALSO
isolate_node.sh(8), monitor_nvme.sh(8), sync_pool.sh(8), sys_monitor(8), zone_monitor.sh(8), zones_srv_monitor.sh(8), oc_lock(3), oc_log(3), oc_runlevel(3), oc_ssh(3), oc_validate(3), omnicube_utils(3), omnicube(7).
NOTES
The script repairs cluster-wide state: it clears and scrubs pools, rewrites iSCSI discovery addresses, re-attaches bridge links and can put the whole node into isolation. All of these are visible to the peer node and to every hosted zone.
Exit status 2 is not a transient condition. The session count has already been found to be less than half of what the configuration expects, and no reconnect was attempted at that level; the accompanying oc-alert mail is the actionable artefact.
Because the alert-mail throttle keys on the exact message text, warnings that embed changing values (counters, pool state dumps) hash differently from run to run and may mail more than once per window.