isolate_node.sh(8)

NAME

isolate_node.sh - place an OmniCube node in isolation mode, or take it back out

SYNOPSIS

/opt/omnicube/sbin/isolate_node.sh [ auto | manual | reconcile | release | status ]

isolate_node.sh [verb]

DESCRIPTION

isolate_node.sh puts the local node into isolation mode by taking every remote pool and LUN offline, and takes it back out again. A node is isolated when it can no longer reach its peer's iSCSI targets: continuing to drive half-visible remote devices is worse than standing down until the peer returns.

Exactly one verb is accepted, and it defaults to auto when no argument is given, which preserves the bare invocation used by pool_monitor.sh(8). Both isolation modes share the same on-disk marker, the directory /var/run/omnicube/isolate_node.lock, so any consumer that only tests for that directory's presence keeps working regardless of the mode. The mode itself is recorded as a single word in the file /var/run/omnicube/isolate_node.lock/mode.

The script sets an explicit, exported PATH of /usr/sbin:/sbin:/opt/ooce/sbin:/usr/bin:/usr/gnu/bin:/opt/ooce/bin because pfexec matches /etc/security/exec_attr.d/omnicube on the resolved path of the binary, so sqlite3 and gsed must resolve to the paths listed there. It sources /opt/omnicube/lib/common/utils.sh for logging, the node identity and the marker path.

Isolation modes

auto: Temporary, machine-entered isolation. It is created by pool_monitor.sh(8) when at least two remote iSCSI targets fail the discovery-address probe, and it is meant to be self-healing: as soon as the peer answers again, reconcile removes the marker and the ordinary monitors resume and re-online the pools by themselves.
manual: Real, operator-entered isolation. It is never cleared automatically. Only an explicit release removes it, and an auto trigger can never downgrade it back to auto: when the recorded mode is already manual, auto rewrites the mode file as manual and leaves the lock manual.

A marker directory whose mode file is missing or unreadable, for instance one left behind by an older version, is treated as manual. The safe default is to require a human to clear it rather than to remove something that may have been intentional.

Entering isolation

For both auto and manual, the marker directory is created with mkdir (atomic; an existing directory is not an error), the effective mode is written to the mode file, and one of Entering <mode> isolation mode, Isolation mode changed from <old> to <new> or Already in <mode> isolation mode is logged at info level. The isolation action itself then runs: for every pool except rpool and except pools whose name contains the local host's short name (the hostname truncated at the first hyphen), Isolate pool <name> is printed and every device of that pool whose LUN id is not in the local pfexec stmfadm list-lu output is taken offline with pfexec zpool offline. Locally-served LUNs are therefore left online; only the peer's are dropped. The pool candidate list is filtered with shell pattern matching, and the local-LUN comparison uses fixed-string whole-word matching, so neither a metacharacter in the hostname nor a short WWN that is a substring of a longer one can select the wrong devices.

Leaving isolation

reconcile and release both remove the mode file and the marker directory. Neither re-onlines anything: clearing the marker is what allows pool_monitor.sh(8) and the zone-management service to resume and bring the pools back through their existing repair paths.

reconcile first decides whether the peer is really back. It reuses the probe that caused the isolation: for every non-local target in the iscsi_sessions table of /opt/efitsrv/sysinfo.db it reads discover_addr and runs nc -vz addr 3260. The node counts as reachable only when at least one remote target was probed and every one of them answered, which is symmetric with the condition that triggered the isolation. Target names and addresses are validated against ^[a-zA-Z0-9.:_-]+$ and skipped otherwise, since they are external input flowing into an SQL string literal and into a command line.

Run-level guard

auto, manual and reconcile call abort_if_shutting_down and exit 0 silently when the host is at run level 0, 1, 5, 6, S or s, or when /etc/nologin exists; a node on its way down must not start isolating itself or churning isolation state. OC_IGNORE_RUNLEVEL set to a non-empty value overrides this. release is a deliberate operator action and is allowed even mid-shutdown, and status only reads state, so neither is gated.

Locking

This command takes no single-instance lock of its own. The marker directory is the shared state, and it is created with a single atomic mkdir, so two concurrent invocations converge instead of colliding. The parent directory /var/run/omnicube is created by utils.sh on every source with mode 1775 and group config/lock_group, so both root and RBAC operators can create the marker.

Alerting

All progress messages go through info() from utils.sh: stdout plus syslog user.info under the tag <site>isolate_node, which /etc/rsyslog.d/omnicube.conf routes to /var/log/misc.log. A failure to create the marker is reported with error(), which additionally mails oc-alert; warning()-level events would mail oc-warn. Both are throttled per (severity, message) pair to at most one mail per config/mail_throttle_window seconds, default 3600.

OPERANDS

auto: Enter temporary isolation. This is the default when no verb is given. If the current mode is already manual, the lock stays manual.
manual: Enter operator isolation. Overrides a current auto mode, and will not be cleared by reconcile.
reconcile: Clear the marker only if the recorded mode is auto and the peer's iSCSI discovery addresses all answer. A manual or unknown mode logs Isolation is <mode>; leaving it for manual intervention and changes nothing; an auto mode with an unreachable peer logs Remote node still unreachable, keeping temporary isolation .
release: Manual intervention: clear whatever lock is present, whatever its mode. Logs Manually released <mode> isolation, or Not in isolation mode, nothing to release when there is no marker.
status: Print isolated (auto) , isolated (manual) or not isolated on stdout and exit 0. Changes nothing.

Any other argument prints usage: isolate_node.sh [auto|manual|reconcile|release|status] on stderr and exits 2.

EXAMPLES

Example 1: take a node out of service before maintenance

pfexec /opt/omnicube/sbin/isolate_node.sh manual
/opt/omnicube/sbin/isolate_node.sh status
isolated (manual)

Example 2: put the node back into service afterwards

Nothing is re-onlined by the release itself; the next pool_monitor.sh pass does that.

pfexec /opt/omnicube/sbin/isolate_node.sh release
/opt/omnicube/sbin/isolate_node.sh status
not isolated

Example 3: ask whether a temporary isolation can heal now

/opt/omnicube/sbin/isolate_node.sh reconcile
tail /var/log/misc.log

Example 4: check isolation state from a script

if [ "$(/opt/omnicube/sbin/isolate_node.sh status)" != "not isolated" ]; then
    echo "node is isolated, skipping"
fi

EXIT STATUS

0: The requested verb completed, including the cases where it deliberately did nothing: reconcile against a manual lock, release with no lock present, and any status query. Also returned silently when the run-level guard trips.
1: The isolation marker /var/run/omnicube/isolate_node.lock could not be created and does not exist. Logged and mailed as isolate_node: cannot create isolation lock.
2: Usage error: an unrecognised verb was given.
78: config/site is not set on the SMF instance, so utils.sh refused to initialise.

FILES

/var/run/omnicube/isolate_node.lock: The isolation marker. Its mere existence means "this node is isolated"; it is tested by pool_monitor.sh(8) on every pass.
/var/run/omnicube/isolate_node.lock/mode: One line, either auto or manual. Missing or unreadable is interpreted as manual.
/opt/efitsrv/sysinfo.db: SQLite database whose iscsi_sessions table supplies the peer's discovery addresses for the reconcile probe.
/opt/omnicube/lib/common/utils.sh: Shared shell library; defines the marker path and the logging functions.
/var/log/misc.log: Default destination of the syslog records.
/etc/nologin: Presence means a shutdown is in progress; auto, manual and reconcile exit 0.

ENVIRONMENT

OC_SMF_INSTANCE: Selects the svc:/omnicube/sysadm instance whose config properties are read. Unset means default.
OC_IGNORE_RUNLEVEL: When non-empty, bypasses the run-level guard on auto, manual and reconcile.
PATH: Set and exported by the script itself; the inherited value is discarded.

SECURITY

zpool, stmfadm and sqlite3 are invoked through pfexec, which requires the OmniCube Administration profile from /etc/security/prof_attr.d/omnicube. That profile is root-equivalent as a whole. Target names and discovery addresses read out of /opt/efitsrv/sysinfo.db are validated before being interpolated into SQL literals or privileged command lines.

NOTES

Isolation is a node-wide, cluster-visible state. While the marker exists every remote pool is offline, so zones backed by the peer's LUNs cannot run, and pool_monitor.sh(8) stops performing repairs.

The asymmetry between the two modes is intentional and is the main thing to remember: automation may create and clear an auto lock at will, but a manual lock is a promise to a human. Nothing in the suite removes it, and an automatic auto trigger arriving while a manual lock is held leaves the lock manual. After maintenance, the node stays isolated until someone runs release.

Neither reconcile nor release re-onlines the devices that auto or manual offlined. Recovery is left to the next monitor pass; if the monitors are disabled, the pools must be re-onlined by hand with zpool online.