Recovery playbook

Specific failure modes + how to recover from each. For lifecycle backup procedures see BACKUP.md; for the underlying security model see SECURITY.md.

I lost / forgot a manager admin password

If you have at least one other enabled admin (any role) who CAN log in:

1. Log in as the other admin
2. Settings → Admin users → disable the locked-out account, then re-enable + create a new one (you can't change others' passwords from the UI yet — v0.3 adds that). For now: directly UPDATE the pwhash:

ssh root@your-manager-host
. /etc/customsso-manager/db.creds
NEWHASH=$(php -r "echo password_hash('YourNewPassword123', PASSWORD_DEFAULT);")
mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" \
    -e "UPDATE users SET pwhash='$NEWHASH' WHERE username='locked_user'"

If you can't log in at ALL (no working admin):

ssh root@your-manager-host
. /etc/customsso-manager/db.creds
NEWHASH=$(php -r "echo password_hash('NewPassword123', PASSWORD_DEFAULT);")
mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -e "
  UPDATE users SET pwhash='$NEWHASH', enabled=1, role='superuser' WHERE id=1;
  SELECT id, username, role, enabled FROM users;
"

Sets the lowest-id user to superuser with the new password. Pick any username from the row that returns.

I lost the master key

The encrypted columns are unrecoverable. Every paired PBX must be re-paired from scratch.

1. On each PBX: sudo /usr/local/bin/customsso-unpair — reads the pair manifest and undoes exactly what the manager wrote
2. Reinstall the manager — generates a fresh master key
3. Re-pair each PBX via the manager's Add PBX form (you'll need each PBX's root password again, one time; the manager wipes it after)
4. Future-proof: see BACKUP.md for master-key escrow patterns. The key is 32 bytes — print as QR, store in a password manager, etc.

A pairing fell out of sync (rotation failure, manual edit, etc.)

Symptoms: dashboard shows the PBX as offline; status refresh reports ssh_auth_failed; bulk action skips it.

1. On the agent (PBX): sudo /usr/local/bin/customsso-unpair — reads the pair manifest and removes local state + the manager's SSH key from /root/.ssh/authorized_keys
2. On the manager: PBX detail → Unpair — the outbound SSH will fail since keys are out of sync, but the manager-side DELETE still runs. Confirm the PBX disappeared from the dashboard.
3. On the manager: Add PBX — re-enter the PBX's SSH host + port + root password (you'll need to re-enable password auth on the PBX temporarily if the previous pair had disabled it)
4. New HMAC secret + SSH keypair are auto-generated; status badge should flip to ok within 60s when the sidecar refreshes

Agent module installation failed mid-flight

Symptoms: customsso shows up in fwconsole ma list but with errors; tables missing; cron file missing; sudoers missing.

# On the PBX:
ssh root@your-pbx
fwconsole ma uninstall customsso         # clean teardown — drops tables, removes cron + sudoers
# If that fails:
fwconsole ma delete customsso            # force-remove from filesystem
mysql asterisk -e "DROP TABLE IF EXISTS customsso_audit, customsso_config, customsso_jobs, customsso_nonces"
rm -f /etc/sudoers.d/customsso /etc/cron.d/customsso /etc/apache2/conf-enabled/customsso-headers.conf /etc/apache2/conf-available/customsso-headers.conf
rm -rf /var/spool/customsso /var/lib/asterisk/customsso
systemctl reload apache2

# Then reinstall from the signed tarball:
fwconsole ma downloadinstall https://repo.voip-stuff.net/packages/customsso/customsso-0.1.0.tar.gz
fwconsole reload

Manager → PBX SSH suddenly stops working (other channels OK)

Likely causes (in order of probability):

1. PBX's /root/.ssh/authorized_keys was edited/rewritten by another tool removing the customsso-manager-key marker line. Fix: re-run the SSH key rotation from the PBX detail page (Per-PBX keys → Rotate SSH keypair only) — installs the manager's CURRENT pubkey freshly.
2. Manager IP changed (host moved networks, NAT changed, etc.). The agent's from="<old-ip>" rule no longer matches. Fix: revoke + re-pair so the agent records the new manager IP.
3. SSH daemon config changed on the PBX (port, AllowUsers, root-login disabled). Check /etc/ssh/sshd_config — root login + key auth must be permitted from the manager's IP at minimum.

Job stuck "running" forever

Sidecar marks jobs timeout after JOB_TIMEOUT_SEC=600 (10 minutes), but if the sidecar itself is stuck or restarted mid-job, jobs can be left in running state without progress.

Force-clean stuck jobs:

ssh root@your-manager-host
. /etc/customsso-manager/db.creds
mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -e "
  UPDATE jobs SET status='timeout', error='manual cleanup', finished_at=NOW()
  WHERE status='running' AND timeout_at < NOW() - INTERVAL 1 HOUR
"
systemctl restart customsso-sidecar

Dashboard shows wrong status / out of date

The sidecar refreshes one PBX per loop iteration (every 5s base), so each PBX gets a real status check every ~60s × fleet size. Manual force-refresh:

• Single PBX: PBX detail → Refresh status button
• All PBXes: SQL-level force on the manager:

. /etc/customsso-manager/db.creds
mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -e "UPDATE pbxes SET last_status_at=NULL"
# Sidecar will pick them up within seconds

Apache won't start after deploying a new conf

Likely the customsso-headers.conf has a stale directive or a referenced module isn't enabled.

ssh root@your-pbx
apache2ctl -t                            # see the actual syntax error
# Fix or temporarily disable:
a2disconf customsso-headers
systemctl restart apache2                # now Apache is back; lost SSO + manager iframe support
# Then patch the snippet + re-enable:
a2enconf customsso-headers
apache2ctl -t && systemctl reload apache2

The install.php validates the snippet with apache2ctl -t before reload, so this is unlikely from a fresh install. Most common: third-party config in another conf-enabled file collides.

"Forbidden" on /admin pages after SSO

We added an Apache override that allows /admin traffic with the customsso_authed cookie. If you're getting Forbidden, check:

1. The cookie WAS set: in browser dev tools → Application → Cookies → look for customsso_authed=1 on the PBX origin
2. If yes: maybe there's a sticker rule (referer, mod_security, etc.) blocking. apache2ctl -t then check error log: tail -f /var/log/apache2/error.log
3. If no: SSO didn't complete. Check /customsso/audit on the agent for sso.redeem rows — failures are logged with reason

Common: the agent's SSO redemption fails because the token has expired (30s TTL). Just retry — click Open native UI ↗ again from the manager.

I changed the PBX's hostname/IP and now nothing works

The pair handshake recorded the manager's IP-at-time-of-pair (TOFU). If the manager moved, the agent's source-IP gate rejects everything.

Fix: revoke + re-pair (see "A pairing fell out of sync" above).

If many PBXes are affected (manager IP changed for the whole fleet):

# On each agent — fastest path:
mysql asterisk -e "UPDATE customsso_config SET config_value='<new-manager-ip>' WHERE config_key='manager_ip'"
# No service restart needed; the agent reads the value on every request

You'd still need to update the manager's outbound IP (firewall rules, etc.) for SSH to work — see "SSH suddenly stops" above.

Common error messages

Last-resort: nuke the manager and start over

If you've completely lost the state and don't have backups:

1. Each PBX continues to work normally as a standalone FreePBX — the customsso module is opt-in management, not a runtime dependency for telephony
2. From each PBX: Admin → SSO Pairing → Revoke pairing to clean up the orphaned state
3. Reinstall the manager fresh (./install.sh)
4. Pair each PBX again

You've lost the audit log + any custom user accounts on the manager, but no PBX-side state is lost.