Recovery playbook

Specific failure modes + how to recover from each. For lifecycle backup procedures see BACKUP.md; for the underlying security model see SECURITY.md.

v0.6.4+ recovery paths at a glance

I lost / forgot a manager admin password

If you have at least one other enabled superuser who CAN log in:

1. Log in as the superuser
2. Settings → Users → pick the locked-out account → Reset password — set a new password directly (v0.3+ superuser-reset-any-user feature). Give it to the user out-of-band.

If the other logged-in admin is admin-role (not superuser), they can Settings → Users → disable the account and re-create it with a fresh password — but the reset-any-user button is superuser-only.

SQL fallback (if the UI is somehow unreachable — e.g. Apache is down but MariaDB is up):

ssh root@your-manager-host
. /etc/customsso-manager/db.creds
NEWHASH=$(php -r "echo password_hash('YourNewPassword123', PASSWORD_DEFAULT);")
mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" \
    -e "UPDATE users SET pwhash='$NEWHASH' WHERE username='locked_user'"

If you can't log in at ALL (no working admin):

ssh root@your-manager-host
. /etc/customsso-manager/db.creds
NEWHASH=$(php -r "echo password_hash('NewPassword123', PASSWORD_DEFAULT);")
mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -e "
  UPDATE users SET pwhash='$NEWHASH', enabled=1, role='superuser' WHERE id=1;
  SELECT id, username, role, enabled FROM users;
"

Sets the lowest-id user to superuser with the new password. Pick any username from the row that returns.

I lost the master key

The encrypted columns are unrecoverable. Every paired PBX must be re-paired from scratch.

1. On each PBX: sudo /usr/local/bin/customsso-unpair — reads the pair manifest and undoes exactly what the manager wrote
2. Reinstall the manager — generates a fresh master key
3. Re-pair each PBX via the manager's Add PBX form (you'll need each PBX's root password again, one time; the manager wipes it after)
4. Future-proof: see BACKUP.md for master-key escrow patterns. The key is 32 bytes — print as QR, store in a password manager, etc.

A pairing fell out of sync (rotation failure, manual edit, etc.)

Symptoms: dashboard shows the PBX as offline; status refresh reports ssh_auth_failed; bulk action skips it.

1. On the agent (PBX): sudo /usr/local/bin/customsso-unpair — reads the pair manifest and removes local state + the manager's SSH key from /root/.ssh/authorized_keys
2. On the manager: PBX detail → Unpair — the outbound SSH will fail since keys are out of sync, but the manager-side DELETE still runs. Confirm the PBX disappeared from the dashboard.
3. On the manager: Add PBX — re-enter the PBX's SSH host + port + root password (you'll need to re-enable password auth on the PBX temporarily if the previous pair had disabled it)
4. New HMAC secret + SSH keypair are auto-generated; status badge should flip to ok within 60s when the sidecar refreshes

Agent module installation failed mid-flight

Symptoms: customsso shows up in fwconsole ma list but with errors; tables missing; cron file missing; sudoers missing.

# On the PBX:
ssh root@your-pbx
fwconsole ma uninstall customsso         # clean teardown — drops tables, removes cron + sudoers
# If that fails:
fwconsole ma delete customsso            # force-remove from filesystem
mysql asterisk -e "DROP TABLE IF EXISTS customsso_audit, customsso_config, customsso_jobs, customsso_nonces"
rm -f /etc/sudoers.d/customsso /etc/cron.d/customsso /etc/apache2/conf-enabled/customsso-headers.conf /etc/apache2/conf-available/customsso-headers.conf
rm -rf /var/spool/customsso /var/lib/asterisk/customsso
systemctl reload apache2

# Then reinstall from the signed tarball (pick whichever version matches your manager — usually latest 1.0.x):
fwconsole ma downloadinstall https://repo.voip-stuff.net/packages/customsso/customsso-1.0.2.tar.gz
fwconsole reload

Or, if the PBX is paired to a manager running the same version, just click ↻ Refresh module list on the PBX Status page in the manager — FreePBX will see the current signed tarball on the manager's configured repo and offer the install through Module Admin.

Module Admin can't see any customsso update / ma listonline returns nothing

Symptoms: you know a newer customsso is out (e.g. 1.0.2), but Admin → Module Admin → Check Online shows the PBX still on the old version, and fwconsole ma listonline returns empty for customsso.

Most likely cause: broken MODULE_REPO URL. Some historical builds (customsso 1.0.1 in particular) baked a wrong repo path — https://voip-stuff.net/pbxrepo/repo (public) or https://private.voip-stuff.net/pbxrepo/private (private) — into install.php. Both 404, so FreePBX can't discover any newer version, so the auto-fix in newer install.php never gets a chance to run. Chicken-and-egg.

Diagnose:

fwconsole setting MODULE_REPO
# Bad values contain /pbxrepo/... anywhere

Manual one-shot fix — pick the line matching your PBX's repo:

# Public repo:
fwconsole setting MODULE_REPO 'https://repo.voip-stuff.net,https://mirror.freepbx.org'

# OR private repo:
fwconsole setting MODULE_REPO 'https://private.voip-stuff.net,https://mirror.freepbx.org'

Then refresh + install:

fwconsole ma refreshsignatures
fwconsole ma listonline | grep customsso     # confirm 1.0.2 (or newer) now surfaces
fwconsole ma downloadinstall customsso
fwconsole reload

The new install.php includes the strip-and-prepend logic, so from this point forward MODULE_REPO stays clean across future upgrades. The manual step is only needed to break the cycle once.

Manager → PBX SSH suddenly stops working (other channels OK)

Likely causes (in order of probability):

1. PBX's /root/.ssh/authorized_keys was edited/rewritten by another tool removing the customsso-manager-key marker line. Fix: re-run the SSH key rotation from the PBX detail page (Per-PBX keys → Rotate SSH keypair only) — installs the manager's CURRENT pubkey freshly.
2. Manager IP changed (host moved networks, NAT changed, etc.). The agent's from="<old-ip>" rule no longer matches. Fix: revoke + re-pair so the agent records the new manager IP.
3. SSH daemon config changed on the PBX (port, AllowUsers, root-login disabled). Check /etc/ssh/sshd_config — root login + key auth must be permitted from the manager's IP at minimum.

Job stuck "running" forever

Sidecar marks jobs timeout after JOB_TIMEOUT_SEC=600 (10 minutes), but if the sidecar itself is stuck or restarted mid-job, jobs can be left in running state without progress.

Force-clean stuck jobs:

ssh root@your-manager-host
. /etc/customsso-manager/db.creds
mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -e "
  UPDATE jobs SET status='timeout', error='manual cleanup', finished_at=NOW()
  WHERE status='running' AND timeout_at < NOW() - INTERVAL 1 HOUR
"
systemctl restart customsso-sidecar

Dashboard shows wrong status / out of date

The sidecar refreshes each PBX every ~60 s (parallel probes since v0.6.7: STATUS_PROBE_CONCURRENCY=8, so a 30-PBX fleet clears in 15-20 s once triggered). Most refresh-needed cases have UI buttons now:

• Single PBX status only — PBX detail → Refresh status button
• Single PBX + force FreePBX to re-check module catalog — PBX detail → ↻ Refresh module list (v0.6.7+). This is what you want when you fixed a FreePBX warning (module update installed, tampered file cleared) on the PBX itself but the manager sidebar still shows the notification badge — FreePBX's own notifications table only clears when Check Online runs, which this button triggers.
• Whole fleet — Fleet page → ↻ Refresh all button, top right next to + Add PBX (v0.6.7+, operator role gated). Nulls last_status_at on every row; sidecar re-probes each PBX in parallel; sidebar badges converge live via the 60 s fragment poll.
• After any fwconsole action from the manager (module install/uninstall/update, fwconsole reload, notification clears) — automatic. The sidecar auto-nulls last_status_at on job completion since v0.6.7, so notification counts converge within ~60 s without operator action.

SQL fallback (only if the UI is unreachable):

. /etc/customsso-manager/db.creds
mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -e "UPDATE pbxes SET last_status_at=NULL"
# Sidecar will pick them up on its next tick (~5 s).

Apache won't start after deploying a new conf

Likely the customsso-headers.conf has a stale directive or a referenced module isn't enabled.

ssh root@your-pbx
apache2ctl -t                            # see the actual syntax error
# Fix or temporarily disable:
a2disconf customsso-headers
systemctl restart apache2                # now Apache is back; lost SSO + manager iframe support
# Then patch the snippet + re-enable:
a2enconf customsso-headers
apache2ctl -t && systemctl reload apache2

The install.php validates the snippet with apache2ctl -t before reload, so this is unlikely from a fresh install. Most common: third-party config in another conf-enabled file collides.

"Forbidden" on /admin pages after SSO

We added an Apache override that allows /admin traffic with the customsso_authed cookie. If you're getting Forbidden, check:

1. The cookie WAS set: in browser dev tools → Application → Cookies → look for customsso_authed=1 on the PBX origin
2. If yes: maybe there's a sticker rule (referer, mod_security, etc.) blocking. apache2ctl -t then check error log: tail -f /var/log/apache2/error.log
3. If no: SSO didn't complete. Check /customsso/audit on the agent for sso.redeem rows — failures are logged with reason

Common: the agent's SSO redemption fails because the token has expired (30s TTL). Just retry — click Open native UI ↗ again from the manager.

I changed the PBX's hostname/IP and now nothing works

The pair handshake recorded the manager's IP-at-time-of-pair (TOFU). If the manager moved, the agent's source-IP gate rejects everything.

Fix: revoke + re-pair (see "A pairing fell out of sync" above).

If many PBXes are affected (manager IP changed for the whole fleet):

# On each agent — fastest path:
mysql asterisk -e "UPDATE customsso_config SET config_value='<new-manager-ip>' WHERE config_key='manager_ip'"
# No service restart needed; the agent reads the value on every request

You'd still need to update the manager's outbound IP (firewall rules, etc.) for SSH to work — see "SSH suddenly stops" above.

Common error messages

Locked out because mail is broken (2FA can't deliver code)

Fresh install: /etc/customsso-manager/allow-2fa-bypass was created by install.sh and is single-use — the very first superuser login consumes it. Post-first-login, if a superuser gets locked out because msmtprc broke:

ssh root@your-manager-host
touch /etc/customsso-manager/allow-2fa-bypass
chmod 640 /etc/customsso-manager/allow-2fa-bypass
chown root:apache /etc/customsso-manager/allow-2fa-bypass    # or root:www-data on Debian

Next superuser login skips 2FA once, marker auto-deletes. Fix mail (Settings → Mail), then normal 2FA resumes.

Marker is root-owned + web-user-readable but only root-writable — a web-user shell compromise can't recreate it to bypass again.

Accidentally installed an older version

install.sh --update refuses to install a tarball whose VERSION is lower than the currently-installed one (v0.6.5+ feature). Symptom:

ERROR: refusing to downgrade 0.6.5 → 0.5.25
The tarball you're running is OLDER than what's already installed.
This is usually the wrong thing (running an install.sh from a stale
latest.tar.gz, or from a downloaded tarball you forgot to update).
If you deliberately want to roll back, re-run with --allow-downgrade

If you deliberately want to roll back:

sudo bash /path/to/customsso-manager-vX.Y.Z/install.sh --update --allow-downgrade

The guard only helps for downgrades TO versions that already have the guard (v0.6.5+). Older tarballs don't refuse downgrades because their install.sh predates the check.

A pairing failed auto-reclaim during restore (unpaired PBX)

After restoring a .csso-backup bundle, any PBX that was unreachable at restore time enters "soft-pair" state: DB row preserved, SSH + HMAC ciphertext wiped. The manager UI surfaces this three ways: a red Unpaired ⚠ (N) tab under Settings, a global banner on every page, and the Settings → Reclaim wizard.

To recover each unpaired PBX:

1. Get the PBX back online (network, power, whatever caused the reachability failure).
2. Manager → Settings → Reclaim.
3. Wizard shows each unpaired PBX with a reachability probe (TCP connect to ssh_host:ssh_port). If reachable, a blue Re-pair → button appears.
4. Click Re-pair, enter the root password, submit.
5. Sidecar's pair provisioner runs with force_repair=true + the existing pbx_uuid — the DB row is UPDATEd (not new-inserted), so label / notes / hardware node / descriptions survive. The tab + banner disappear when the last unpaired row is resolved.

If the PBX is genuinely gone (retired, replaced), just delete the pbxes row directly via SQL or via a future "Delete pbx" UI action.

Last-resort: nuke the manager and start over

If you've completely lost the state and don't have backups:

1. Each PBX continues to work normally as a standalone FreePBX — the customsso module is opt-in management, not a runtime dependency for telephony
2. From each PBX: Admin → SSO Pairing → Revoke pairing to clean up the orphaned state
3. Reinstall the manager fresh (./install.sh)
4. Pair each PBX again

You've lost the audit log + any custom user accounts on the manager, but no PBX-side state is lost.