monitor_nvme.sh(8)

NAME

monitor_nvme.sh - check NVMe health log pages and device wear level

SYNOPSIS

/opt/omnicube/sbin/monitor_nvme.sh

monitor_nvme.sh

DESCRIPTION

monitor_nvme.sh inspects every NVMe device in the host and raises an alert when a device reports health-log anomalies or has consumed too much of its rated write endurance. It only reports; it never touches a device. The script takes no options and no operands and is intended for unattended periodic execution from root's crontab, alongside the other periodic jobs whose sample entries are shipped in /opt/omnicube/share/README. Running it by hand is equally valid and has no side effects beyond the log and mail it may produce.

The script sets PATH to /usr/sbin:/sbin:/opt/ooce/sbin:/usr/bin:/opt/ooce/bin before sourcing /opt/omnicube/lib/common/utils.sh, because utils.sh runs svcprop, mkdir and hostname at source time. GNU grep is called by absolute path as /usr/gnu/bin/grep, since the option -A is not available in the Illumos grep.

Run-level guard

abort_if_shutting_down is called before the first device is examined and again at the top of every device iteration, so the script exits 0 silently when the host is at run level 0, 1, 5, 6, S or s, or when /etc/nologin exists, and a run-level change part way through stops it before the next device. OC_IGNORE_RUNLEVEL set to a non-empty value overrides the guard.

Locking

This command takes no lock and does not enforce single-instance execution. Concurrent runs are harmless: the script keeps no state and issues only read-only nvmeadm queries. It does, however, create and normalise /var/run/omnicube and the mail-throttle directories as a side effect of sourcing utils.sh.

Checks performed

Devices are enumerated from pfexec nvmeadm list, taking the device name from every line that starts with nvme. For each device:

health log page anomalies: pfexec nvmeadm -v get-logpage <dev> health is read and the lines that do not report OK are counted, within the device stanza. More than two such lines is treated as an anomaly and the complete health page is reported through warning() as NVMe <dev> health page anomalies (<n> non-OK lines).
wear level: The Device Life Used value is read from the same health page, its trailing percent sign is stripped, and it is compared against the fixed alarm threshold of 90. A value that is not below 90 is reported through warning() as NVME <dev> Device Life Used reached alarm threshold 90 (current <n>). The threshold is hard-coded; there is no SMF property for it.

Alerting

warning() from utils.sh writes to stdout, logs to syslog as user.warn under the tag <site>monitor_nvme, which /etc/rsyslog.d/omnicube.conf routes to /var/log/misc.log, and mails oc-warn from $HOSTNAME@<config/mail_domain>. The verbose health dump is deliberately routed through warning() rather than left on stdout, so that it reaches syslog and the alert mail instead of only the cron mail. Mail is throttled per (severity, message) pair to one message per config/mail_throttle_window seconds, default 3600. monitor_nvme.sh never calls error(), so it never mails oc-alert.

OPERANDS

This command takes no options and no operands. Any argument given is ignored.

SMF PROPERTIES

monitor_nvme.sh reads no properties of its own. Through utils.sh it consumes, from svc:/omnicube/sysadm:default unless OC_SMF_INSTANCE selects another instance:

config/site: Mandatory. Prefix for the syslog tag. If unset, sourcing utils.sh fails and the script exits 78.
config/lock_group: Group applied to /var/run/omnicube (mode 1775). Default sysadmin.
config/mail_domain: Domain used in the envelope sender of alert mail.
config/mail_throttle_window: Alert-mail throttle window in seconds. Default 3600.

EXAMPLES

Example 1: hourly cron entry

11 * * * * /opt/omnicube/sbin/monitor_nvme.sh >/dev/null

Example 2: run one pass by hand

Every warning is also printed on stdout, prefixed [w].

pfexec /opt/omnicube/sbin/monitor_nvme.sh

Example 3: review past NVMe alerts

grep monitor_nvme /var/log/misc.log

EXIT STATUS

monitor_nvme.sh does not set an explicit exit status on its normal path; it returns the status left by the last wear-level comparison. Callers must therefore not read the exit status as a health verdict, and the status must not be used to gate anything.

0: Returned by the run-level guard, when no NVMe device is present, and when the last device examined was at or above the wear threshold.
1: Returned when the last device examined was below the wear threshold, that is, on the ordinary healthy path. The comparison that finds a device healthy is the last statement executed, and its false result becomes the script's status.
78: config/site is not set on the SMF instance, so utils.sh refused to initialise.

FILES

/var/run/omnicube/mail_throttle/<user>/: Per-user alert-mail throttle markers, mode 0700.
/opt/omnicube/lib/common/utils.sh: Shared shell library, sourced at start-up.
/var/log/misc.log: Default destination of the syslog records.
/etc/nologin: Presence means a shutdown is in progress; the script exits 0.

ENVIRONMENT

OC_SMF_INSTANCE: Selects the svc:/omnicube/sysadm instance whose config properties are read. Unset means default.
OC_IGNORE_RUNLEVEL: When non-empty, bypasses the run-level guard.
PATH: Set by the script itself; the inherited value is discarded.
HOSTNAME: Used in the envelope sender of alert mail.

SECURITY

nvmeadm is run through pfexec, which requires the OmniCube Administration profile from /etc/security/prof_attr.d/omnicube and the /usr/sbin/nvmeadm entry in /etc/security/exec_attr.d/omnicube. That profile is root-equivalent as a whole and must only be assigned to accounts trusted with root. ${PFEXEC} is not used here; the pfexec calls are unconditional and are harmless no-ops for a caller who is already root.

NOTES

The alert-mail throttle keys on the exact message text, and the anomaly warning embeds the whole health page. Counters and temperatures in that page drift from run to run, so consecutive anomaly reports hash to different throttle keys and may produce more than one mail per window. This is deliberate: the branch is only reached for a device that is already misbehaving and an operator wants to see the progression rather than have it silenced.

The wear-level alarm fires from 90 percent of rated life used and keeps firing on every run; there is no acknowledgement mechanism. A device that has crossed the threshold should be replaced or excluded from monitoring by disabling the cron entry, since the script has no exclusion property.