devfs_cleanup.sh(8)

NAME

devfs_cleanup.sh - remove dead disk devices when all storage paths are healthy

SYNOPSIS

/opt/omnicube/sbin/devfs_cleanup.sh [-n]

DESCRIPTION

devfs_cleanup.sh removes the disk devices of LUNs that no longer exist. Devices of deleted, unmapped or relocated LUNs otherwise accumulate, and format(8) opens every one of them and waits for each open to fail, so listing disks gets slower with every LUN the node has ever seen. Two steps do the work:

1.: On FC/SAN, a LUN the array no longer reports stays configured with condition unusable. Its /devices node still exists, so its links cannot be cleaned yet. cfgadm -c unconfigure -o unusable_SCSI_LUN is run on each remote-port attachment point (cN::wwn) that has such LUNs.
2.: On every node, devfsadm -C -c disk removes the /dev/dsk and /dev/rdsk links whose device node is gone: iSCSI LUNs, the FC LUNs of step 1, and removed local disks.

A path that is down at this moment is indistinguishable from a LUN that was deleted. The script therefore cleans only when it can establish that storage is settled, and a check that cannot be evaluated counts as a failed check. The pass is skipped, with an info() record and exit status 0, when any of the following holds:

the node is isolated, that is, /var/run/omnicube/isolate_node.lock exists (see isolate_node.sh(8));
svc:/network/iscsi/initiator:default is in a state other than online or disabled;
the initiator is online and iscsiadm list target -S fails, lists a target with Connections: 0, lists a LUN whose OS Device Name is Not Available, or lists no target at all although static or SendTargets discovery is enabled and configured;
an FC initiator port that must be online is not (see config/devfs_cleanup_fc_ports below), a listed port is not present, or fcinfo hba-port fails;
an FC LUN is in condition failing or failed, or LUNs are unusable behind a remote port that is no longer logged in, which means the target is down rather than the LUNs deleted;
mpathadm list lu fails, or reports an LU with fewer operational paths than total paths, or with no operational path;
zpool status -x reports anything other than all pools are healthy.

Each transport is checked only where it is in use, so the same script serves iSCSI nodes, FC/SAN nodes, mixed nodes and nodes with local disks only. iSCSI is in use when the initiator service is online and discovery is configured or a target is logged in; a disabled or absent initiator is not used. FC is in use when fcinfo(8) reports an initiator HBA port; target-mode ports are ignored. With MPxIO on, the path-count check covers both transports.

The checks pass, then the unusable FC LUNs are counted and a dry run (devfsadm -C -s -v -c disk) lists the links that would go. If there is nothing the script exits. If the distinct disks with dangling links plus the unusable FC LUNs exceed config/devfs_cleanup_max_devices, nothing is removed and a warning() lists them, since a pass that large is not routine LUN churn. Otherwise the checks are repeated, step 1 then step 2 run.

The script is meant to run weekly from root's crontab; a sample entry is shipped in /opt/omnicube/share/README. A skipped week costs nothing, so a skip is not mailed: the degraded state that caused it is reported by pool_monitor.sh(8) and sys_monitor(8).

OPTIONS

-n: Dry run. Perform every check and the devfsadm -s pass, report the unusable FC LUNs that would be unconfigured and the disks whose links would be removed, and change nothing.
-h: Print usage and exit 0.

EXAMPLES

Example 1: weekly cron entry, Sunday 04:52

52 4 * * 0 /opt/omnicube/sbin/devfs_cleanup.sh >/dev/null

Example 2: see what the next pass would remove

pfexec /opt/omnicube/sbin/devfs_cleanup.sh -n

Example 3: SAN node with a spare, unplugged HBA port

Only the two ports in use must be online:

svccfg -s svc:/omnicube/sysadm:default setprop \\
    config/devfs_cleanup_fc_ports = astring: '("2100000e1e000001" "2100000e1e000101")'
svcadm refresh svc:/omnicube/sysadm:default

Example 4: node with an FC HBA that is not used

svccfg -s svc:/omnicube/sysadm:default setprop \\
    config/devfs_cleanup_fc_ports = astring: none
svcadm refresh svc:/omnicube/sysadm:default

Example 5: review past passes, including the per-link devfsadm output

grep devfs_cleanup /var/log/misc.log

EXIT STATUS

0: Links removed, nothing to remove, pass skipped or refused by the cap, run-level guard, or another instance already running.
1: devfsadm failed, unconfiguring the unusable LUNs failed on at least one attachment point, or the lock directory could not be created.
2: Invalid usage.
78: config/site is not set on the SMF instance, so utils.sh refused to initialise.

FILES

/var/run/omnicube/devfs_cleanup.lock: Single-instance lock directory.
/var/run/omnicube/isolate_node.lock: Presence means the node is isolated; the pass is skipped.
/opt/omnicube/lib/common/utils.sh: Shared shell library, sourced at start-up.
/var/log/misc.log: Default destination of the syslog records. The raw devfsadm output is logged under the tag <site>devfs_cleanup-dryrun for the dry run and <site>devfs_cleanup for the real pass.

ENVIRONMENT

OC_SMF_INSTANCE: Selects the svc:/omnicube/sysadm instance whose config properties are read. Unset means default.
OC_IGNORE_RUNLEVEL: When non-empty, bypasses the run-level guard.
PATH: Set by the script itself; the inherited value is discarded.

SMF PROPERTIES

config/devfs_cleanup_max_devices: Largest number of distinct disks one pass may remove links for. Above it the pass is refused with a warning. 0 disables the cap. A non-numeric value is ignored. Default 32.
config/devfs_cleanup_fc_ports: FC initiator port WWNs, as fcinfo hba-port prints them (case does not matter), that must be online before a pass runs. Ports not listed are ignored. Unset means every initiator port, so a spare port without a link blocks every pass; the skip record says so. The single value none means FC is not used: no FC check and no unconfigure step. Default: unset.
config/mail_throttle_window: Alert-mail throttle window in seconds. Default 3600.

SECURITY

devfsadm, iscsiadm, mpathadm, fcinfo and cfgadm are run through ${PFEXEC}, which is empty for root and pfexec otherwise. A non-root caller needs the OmniCube Administration profile and the matching entries in /etc/security/exec_attr.d/omnicube. That profile is root-equivalent as a whole.

NOTES

Removed links are not lost for good: when a device attaches again the kernel event makes devfsadmd(8) recreate them. The guards exist because a pool import or a zone boot that runs before the path returns would not find them.

Only FC LUNs in condition unusable are unconfigured. Other stale nodes under /devices are left alone.