oc_runlevel(3)

NAME

oc_runlevel, _current_runlevel, is_shutting_down, abort_if_shutting_down - stop OmniCube jobs when the host is not in a normal multi-user state

SYNOPSIS

amp;. /opt/omnicube/lib/common/utils.sh

_current_runlevel
is_shutting_down
abort_if_shutting_down

DESCRIPTION

A snapshot, a zfs send or a zone relocation that races a shutdown is the worst class of failure this suite can produce: services are half torn down, pools are being exported, filesystems are unmounting, and the job fails somewhere in the middle of a sequence whose recovery is manual. The run-level guard exists so that such a job never starts, and stops at the next iteration if the state changes while it runs.

Consumers call abort_if_shutting_down() immediately after sourcing the library and again at the top of every long loop body; see autosnap(8), autosync(8), sys_monitor(8), zones_srv_monitor.sh(8) and monitor_nvme.sh(8).

_current_runlevel

Prints the current numeric or letter run level on standard output, or nothing if it cannot be determined. Implemented as who -r piped through awk(1) which scans the fields for the literal token run-level and prints the field after it.

The token scan, rather than a fixed column, is what makes this portable. On illumos who -r prints

amp;.       run-level 3  Jul 21 17:01     3      0  S

so the level is the third field, while on Linux the token is the first field and the digit is the second. Hard-coding awk '{print $2}' would read the string run-level on illumos, match none of the shutdown levels, and silently disable the entire guard.

is_shutting_down

A pure predicate: no output, no exit, usable in a conditional, in && chains and inside command substitution. Returns true (0) when either

/etc/nologin: exists, or
_current_runlevel: prints one of 0, 1, 5, 6, S or s.

Those levels are, per illumos init(8): 0 going to firmware or halt, 1 administrative single-user, 5 shut down and power off, 6 reboot, and S s single-user with the root filesystem only. Levels 2 and 3 are the normal multi-user states and are the only ones treated as “running normally”; any level not in the list, and an empty result from _current_runlevel(), is treated as normal, so a host where who -r gives no answer keeps working.

/etc/nologin is checked first because the shutdown sequence creates it a few minutes before the run level actually changes. Treating its presence as “going down” trips the guard earlier and covers a shutdown -g or an init that is in progress but has not yet changed level.

abort_if_shutting_down

The action wrapper. If OC_IGNORE_RUNLEVEL is set to a non-empty value it returns 0 at once. Otherwise it calls is_shutting_down() and returns 0 if the host is normal. If the host is going down it composes a reason,

/etc/nologin present (shutdown in progress)

run-level <level>

logs one line through info() in the form

<script>: system not in multi-user state (<why>); exiting

and terminates the whole script with exit 0.

Two design points follow from that:

exit, not return: A long-running loop must stop for good, not merely skip one iteration. The caller's EXIT, HUP, INT and TERM traps still run, so a lock taken with acquire_lock_or_exit() is released cleanly — see oc_lock(3).
status 0: A shutdown abort is not a job failure. Exiting 0 keeps cron from mailing the operator every time the machine is rebooted, while the info() line preserves the record in syslog.

Because it calls exit, abort_if_shutting_down() must be called from the script body. Inside $(), a pipeline or any other subshell the exit only leaves the subshell and the script continues; use the bare is_shutting_down() predicate in those positions.

RETURN VALUES

_current_runlevel(): The status of the who -r/awk pipeline. The result is the text on standard output, not the status.
is_shutting_down(): 0 if /etc/nologin exists or the run level is 0, 1, 5, 6, S or s; 1 otherwise.
abort_if_shutting_down(): 0 when OC_IGNORE_RUNLEVEL is set or the host is in a normal multi-user state. When the host is going down it does not return at all: it calls exit 0.

FILES

/etc/nologin: Created by the shutdown sequence before the run level changes; its mere presence trips the guard.

ENVIRONMENT

OC_IGNORE_RUNLEVEL: Escape hatch for interactive maintenance. When non-empty, abort_if_shutting_down() returns immediately and the job runs regardless of run level. Intended for an administrator who has deliberately dropped the host to single-user in order to run one of these tools by hand. Cron and SMF invocations never set it, so automated jobs still stop. It has no effect on is_shutting_down(), which remains a truthful predicate.

EXAMPLES

Example 1: guard the job, then guard each iteration

amp;. /opt/omnicube/lib/common/utils.sh
abort_if_shutting_down
for DS in ${DSLIST}; do
    abort_if_shutting_down
    ${PFEXEC} zfs snapshot "${DS}@${SNAPNAME}"
done

Example 2: the predicate where exit would not work

if is_shutting_down; then
    info "skipping remote work, host is going down"
else
    ${_SSH_CMD} "${node}" "pfexec zpool import ${pool}"
fi

Example 3: single-user maintenance by hand

OC_IGNORE_RUNLEVEL=1 /opt/omnicube/sbin/manage_zone.sh -z web01 -b

NOTES

The guard is advisory and coarse. It cannot detect a shutdown that has been requested but has neither created /etc/nologin nor changed the run level yet, so a job started in that window still races the shutdown; the per-iteration call is what limits the damage.

Setting OC_IGNORE_RUNLEVEL in a cron environment or in an SMF method disables the protection for every job that inherits it. It is an interactive tool only.