oc_log(3)

NAME

oc_log, info, warning, error, _send_mail, _should_send_mail - log OmniCube messages to syslog and send throttled alert mail

SYNOPSIS

amp;. /opt/omnicube/lib/common/utils.sh

info message ...
warning message ...
error message ...

_send_mail body subject [recipient]
_should_send_mail severity message

DESCRIPTION

These functions are the only sanctioned way for a script in the suite to say something. Each writes a human-readable line for the interactive or cron reader, a logger(1) entry tagged ${logtag} for long-term forensics, and, for the two non-informational severities, a mail to an operator alias, rate-limited so that a loop over 200 datasets on one broken backup host cannot mail 200 times.

All three severity functions take the message as $@, so an unquoted multi-word message is joined with single spaces. None of them returns a meaningful status; they are statements, not predicates.

info

Prints [-] message on standard output and logs it at user.info. No mail is ever sent for info(). Used for progress and for the run-level abort notice.

warning

Prints [w] message on standard output and logs it at user.warn. When _should_send_mail() agrees, mails the message to oc-warn with the subject

Used for degraded-but-continuing conditions: a zone that did not come back, a pool at a capacity threshold, an excluded dataset skipped.

error

Prints [E] message on standard error and logs it at user.error. When _should_send_mail() agrees, mails the message to oc-alert with the subject

Because error() writes to standard error rather than standard output, a caller may safely use it inside a function whose stdout is captured with $(). That property is not universal in the library: a function whose stdout is its return value must avoid error() altogether, since the mailx(1) child would also write to the captured stdout. See the discussion of _bkp_priv() in oc_policy(3).

_send_mail

Pipes body into

recipient defaults to ${USER}, and ${USER} itself is set to the literal system when it is empty, which is the normal case under cron and SMF. MAIL_DOMAIN comes from the SMF property config/mail_domain; if it is unset the envelope sender degrades to ${HOSTNAME}@, which most mailers reject, so the property is effectively required on any host that must alert.

_should_send_mail

Decides whether an alert for the pair (severity, message) may be sent now, and records the decision. It is called with warn by warning() and with err by error(): the severity is part of the key, so the same text at two severities throttles independently.

The key is

and the marker file is ${MAIL_THROTTLE_DIR}/<hash>. The window is ${MAIL_THROTTLE_WINDOW} seconds (SMF property config/mail_throttle_window, default 3600, non-numeric values ignored). Since the age test is performed with find -mmin, the window is rounded up to whole minutes and clamped to at least one minute. If the marker exists and is younger than that many minutes, the function returns 1 and no mail is sent; otherwise the marker is truncated to update its timestamp and the function returns 0.

The function is deliberately fail-open: it returns 0 (send) when ${MAIL_THROTTLE_DIR} does not exist, is a symbolic link, or is not owned by the calling uid, and also when digest(1) produces no hash. Marker names are a plain MD5 of values any local user can guess, so a marker directory that is shared or foreign-owned could be pre-populated to silence another account's alerts; refusing to consult such a directory turns that attack into at most one duplicate mail. This is why the markers live in a per-uid 0700 directory under a sticky 1777 base rather than in one shared directory, and why throttling is per uid: root's cron alerts and an operator's interactive alerts throttle separately.

RETURN VALUES

info(),warning(),error()

Return the status of the last command executed, i.e. logger(1) for info(), and mailx(1) when mail was sent or 0 when it was throttled for the other two. Do not test them. None of these functions calls exit.

_send_mail()

The status of the mailx(1) pipeline.

_should_send_mail()

0 if an alert for this (severity, message) pair should be sent now, 1 if an identical alert was already sent within the window.

FILES

/var/run/omnicube/mail_throttle

${MAIL_THROTTLE_BASE}, mode 1777 (sticky), created when the library is sourced.

/var/run/omnicube/mail_throttle/<user>

${MAIL_THROTTLE_DIR}, mode 0700, one per uid; contains one empty MD5-named marker per throttled message.

ENVIRONMENT

USER

Default mail recipient for _send_mail(); becomes system when empty.

HOSTNAME

Local part host of the envelope sender.

EXAMPLES

Example 1: the three severities

info "snapshotting ${DS}"
warning "zone ${zone} is in state ${state}, expected running"
error "cannot import ${pool} on ${curr_host}"

Example 2: reporting a failure and returning

if ! ${PFEXEC} zfs snapshot "${SNAP}"; then
    error "autosnap: cannot create ${SNAP}"
    return 1
fi

Example 3: inspecting what is currently throttled

ls -l /var/run/omnicube/mail_throttle/$(id -run)

SEE ALSO

omnicube_utils(3), oc_lock(3), oc_runlevel(3), oc_validate(3), oc_ssh(3), oc_policy(3), monitor_nvme.sh(8), pool_monitor.sh(8), omnicube(7).

NOTES

oc-warn and oc-alert are local mail aliases and must be defined on every node, otherwise every warning and error produces an undeliverable message instead of an alert. They are separate on purpose: warnings can go to a team mailbox, errors to whatever actually pages.

The marker directory lives under /var/run, which is tmpfs on illumos. All throttle state is therefore lost at boot, and the first occurrence of a recurring message after a reboot always mails.

Because the key includes the whole message text, a message that interpolates a timestamp, a byte count or a dataset name is a different key on every iteration and is not throttled at all. Alert texts should name the condition, not the measurement.

Throttling suppresses only the mail. Every occurrence still reaches syslog and the job's own output, so logadm rotated logs remain the authoritative record of how often a condition fired.