Operations guide — day-to-day features

This doc covers the day-to-day operator features the landing page markets but that don't have (or didn't have) their own detail page: FreePBX notification aggregation, fleet-wide bulk fwconsole, custom command catalog, interactive SSH terminal, and edge-triggered email alerts. All admin/operator role gated as noted; every action audit-logged.

If you're looking for something more specific, see: - NATIVE_UI_HTTPS.md — Passwordless SSO into FreePBX admin UI (iframe embed). - UCP_IMPERSONATION.md — Log in as any UCP user for helpdesk troubleshooting. - CRON.md — Manager-side cron scheduler (bulk fleet-wide + individual per-PBX jobs). - BACKUP.md + RECOVERY.md — Encrypted backup + Reclaim wizard. - PAIRING.md — Pair handshake protocol.

FreePBX notification aggregation

Shipped v0.5.25+. The manager aggregates every paired PBX's FreePBX notifications (pending module updates, tampered files, cert expiries, low-disk warnings, sysadmin alerts, everything the FreePBX dashboard's notification widget shows) into one fleet view. No more logging into each of 20+ PBXes to check which ones have pending security updates or FW_TAMPERED warnings.

Where you see it: - Sidebar (left of the manager UI) — every paired PBX gets a per-PBX severity dot and count badge. Colors: red (critical), amber (warning), grey (info), no dot (clean). Sidebar sort since v0.6.5 is by severity — PBXes with problems float to the top; alphabetical only within the same severity bucket. - PBX Status page — full notification list with severity + module + message + first-seen timestamp. Each entry links out to the relevant FreePBX admin page via the Native UI iframe (see NATIVE_UI_HTTPS.md). - Fleet page top strip — aggregate counts across the whole fleet.

How it's populated: the sidecar SSH-probes each PBX every 60 s and reads its notifications table. The full serialized snapshot goes into pbxes.last_status_json — the sidebar reads counts from there; the PBX Status page renders the full list. On PBX-side fwconsole reload, the customsso module (v1.0.0+) also pushes an HMAC-signed status snapshot to the manager so notification changes propagate faster than the 60 s polling floor.

Refresh cadence (v0.6.7+): - 60 s automatic per PBX. - Instant on any fwconsole reload on the PBX (push from customsso module). - Instant on any manager-dispatched fwconsole job completion (sidecar auto-nulls last_status_at when the job finishes). - Manual: PBX detail → Refresh status (single PBX) or Fleet page → ↻ Refresh all (whole fleet). - Force FreePBX to re-check its module catalog (for pending-update notifications that won't clear otherwise): PBX Status page → ↻ Refresh module list button.

Fleet-wide bulk fwconsole

Operator role gated (some destructive actions admin-gated). Runs a single fwconsole command (or Custom Command template) against every selected PBX in parallel. Every job SSH-audited end-to-end with per-PBX stdout/stderr transcript captured.

Where you see it: - Bulk actions form — top nav → Bulk actions. Multi-select PBXes on the left, pick a command from the dropdown on the right. Dropdown contains built-in CMD_MAP operations (fwconsole reload, module upgrades, fwconsole chown, GPG key refresh, module install/uninstall) plus every entry from the admin-defined Custom Commands catalog (see below). - Batch view — every bulk run creates a batch row. Watch progress live at Jobs → Batches — each PBX gets a status pill (pending / running / done / error) and clicking a row shows the full stdout/stderr transcript for that PBX. - Per-PBX fwconsole tab — same operations available for a single PBX (PBX detail → fwconsole tab) if you don't need the bulk fan-out.

Concurrency: default 30 concurrent SSH sessions global, 5 per hardware node (throttles to avoid hammering a shared physical host). Tunable via STATUS_PROBE_CONCURRENCY and per-node config (see DEPLOYMENT.md).

Audit: every dispatched command writes action=bulk.dispatch on the batch row + per-PBX action=job.run with command, exit code, and stdout hash. Failure transcripts are retained (default 90 days).

Placeholder tokens — some Custom Command templates use @MODNAME@ (a FreePBX module name) or @LINES@ (arbitrary text). The Bulk form prompts for these values before running; server-side validation refuses shell-injection characters before substitution.

Custom command catalog

Admin role gated. Reusable shell templates that appear in every PBX's fwconsole dropdown and the Bulk actions form. Adding a new operation to your operators' toolkit is a form fill, not a code change.

Where you see it: - Settings → Custom commands — list, create, edit, delete templates. - Bulk actions dropdown — templates appear under a Custom commands optgroup below the built-in CMD_MAP operations. - Per-PBX fwconsole tab dropdown — same templates surface here for single-PBX runs.

Template syntax: - @MODNAME@ — replaced with a FreePBX module name (validated against ^[a-z][a-z0-9_]*$). - @LINES@ — replaced with an arbitrary multi-line text value (validated: no shell metacharacters). - Plain shell otherwise — runs as root over SSH via the same audited channel the sidecar uses.

Example template:

# Name: Rotate outbound CID for all extensions
# Body:
asterisk -rx "database put OUT/CID @LINES@"
fwconsole reload

Role model: admin-only for create/edit/delete. Operators can run templates (via Bulk or per-PBX fwconsole tab) but can't add new ones. This bounds the blast radius of a compromised operator-role login — they can only run what admins have pre-vetted. See SECURITY.md #5 for the full model.

Warning: an admin-role user can define templates with arbitrary shell. Treat the admin role as equivalent to fleet-wide root. Consider multi-person review for templates that touch dialplans, extensions, or asterisk.conf.

Interactive SSH terminal (xterm.js)

Admin role gated since v0.6.4 (operator role lost the free-shell path in the security audit). Full interactive root shell on any paired PBX, rendered in the browser via xterm.js. Uses the same audited SSH channel the sidecar uses — no separate credential path, no reused keys.

Where you see it: - PBX detail → Terminal tab — one click opens the shell. First open on a PBX warms the WebSocket ticket + connects; subsequent opens within the ticket TTL are instant. - Resize — drag the browser window; the terminal follows. - Copy/paste — standard browser select + Ctrl-Shift-C / Ctrl-Shift-V.

Idle timeout: the WebSocket connection auto-closes after N minutes of no input. Default was 30 min pre-v0.5.6; raised to 2 hours in v0.5.6. Change in Settings → Advanced → Terminal idle timeout. Requires updating two config files in lockstep (see DEPLOYMENT.md).

Session audit: every terminal open writes action=terminal.open with user_id + PBX uuid + WebSocket ticket ID. Close writes action=terminal.close with session duration.

Requirements: Apache needs mod_proxy_wstunnel enabled (installer sets this automatically on both OL9 and Debian). If the terminal tab shows "Connection failed" immediately, check apachectl -M | grep wstunnel.

WebSocket ticket (ws_tickets table): each terminal open mints a short-TTL (default 60 s) one-shot ticket. The browser presents the ticket to the WebSocket endpoint; the endpoint validates + burns it, then upgrades the connection. Prevents WebSocket-hijack replay via cookie-only auth.

Edge-triggered email alerts

Superuser configured, sidecar delivered. Emails fire when a paired PBX transitions into a critical/warning state or crosses a disk threshold. Not on every probe — only on state edges — so you get one email when a PBX goes down and one when it comes back up, not one every 60 s while it's still down.

Where you configure it: - Settings → Notifications — form with: - Up to 4 recipient email addresses (hard cap; enough for oncall + backup + team-alias without turning into a mailing list). - Toggle per alert type: status transitions (PBX → offline / warn / ok), disk warnings (disk % over threshold), FreePBX notification critical count changes. - Disk threshold slider (default 85%).

Edge semantics: - ok → warn — one email. - warn → critical — one email. - critical → ok (recovery) — one email. - critical → critical on next probe — silent (no re-alert).

Delivery: shells out to /usr/bin/msmtp -t via the same /etc/customsso-manager/msmtprc config used for password recovery (see DEPLOYMENT.md § 5). One SMTP config drives both.

Test the send path: Settings → Notifications → Send test email. Delivers to all configured recipients with a distinct subject line so you can verify inbox routing / mail rules before waiting for a real alert.

Audit: every alert emit writes action=notification.email_sent with recipient list, PBX uuid, alert type, and edge (from_state → to_state). Delivery failures land in detail_json.mail_error.

Cross-references

• Refresh module list button + fleet-wide Refresh all — introduced v0.6.7, covered under FreePBX notification aggregation above and in RECOVERY.md § "Dashboard shows wrong status".
• install.sh downgrade guard — v0.6.5+, --allow-downgrade flag to override. See DEPLOYMENT.md § Upgrading.
• Auto-freshen on PBX detail-page view — v0.6.6, visiting any tab of a PBX auto-nulls last_status_at if it's older than 30 s. No operator action needed.