Manager-side cron scheduler

Manager-side cron for FreePBX fleets — customsso-manager v1.0.3+ adds fleet-wide scheduled command dispatch. Create bulk cron jobs from the manager UI that automatically run on every paired FreePBX server and auto-extend to newly-added ones, or per-PBX individual jobs for one-off automations. Schedule with standard 5-field Linux cron (0 2 * * 0), execute any shell command up to 512 characters over SSH, and pause any bulk job on a specific PBX without touching the rest. Every run captures exit code, stdout, stderr, and duration — searchable audit log built in. No per-PBX crontab entries; unpair a PBX and its scheduled work stops cleanly. Zero-runtime-dep, self-hosted, MIT.

Available in v1.0.3+. Requires no PBX-side changes; runs entirely from the manager over the existing pair-time SSH channel.

Two job kinds

Manager-side cron distinguishes two job origins:

The distinction is captured in a single origin column on scheduled_jobs. Bulk-origin rows share a group_id across their fleet-wide siblings; individual-origin rows carry a unique group_id (group of one).

Pick bulk when the same job should run everywhere (fwconsole reload, log rotations, module-list refresh, cert renewal). Pick individual when the schedule is unique to one PBX (customer-specific backup, timezone-shifted maintenance, one-off cleanup).

Bulk cron jobs

Creating

Bulk actions → Cron tab → + Add Bulk Cron Job.

The form shows a PBX picker on the left (checkbox list with search) and the job fields on the right. Every checkbox is checked by default; the checkboxes control the per-PBX enabled state at creation time, not membership. A bulk job always creates a row for every currently-paired PBX — unchecking a PBX just means the row is created with enabled=0 (present but paused).

Fields:

• Schedule — 5-field Linux cron expression (minute hour day-of-month month day-of-week). Validated server-side against the correct per-field ranges before persist.
• Command — the shell command to run, up to 512 characters. Executes as the pair-time SSH user (root on the PBX).
• Notes — optional free-text explaining why this job exists. Shown in the bulk list and the per-PBX Cron tab.

On save, one row per paired PBX lands in scheduled_jobs. The bulk list shows the job with an N/M enabled badge (N = number of PBXes with enabled=1, M = total PBXes in the group).

Per-PBX enable-at-creation

Uncheck any PBX in the picker to have the row created disabled on that PBX. The row still exists (so the group is intact), the job just doesn't fire there until someone re-enables it. Two ways to re-enable later:

1. Bulk view → Edit the group → check the box in the picker.
2. Per-PBX Cron tab → find the bulk row in the "Bulk cron jobs" section → click Enable.

Both do the same thing (flip the row's enabled flag).

Editing

Bulk view → Edit next to any group. The form is the same as create; picker checkboxes reflect current per-PBX enabled state. Save updates:

• Schedule / command / notes uniformly across every row in the group.
• Per-PBX enabled state for each row — checkbox on = enabled, checkbox off = disabled.

Content changes propagate to every PBX in the group. There is no per-PBX schedule/command divergence — that would be an individual job, not a bulk one.

Deleting

Bulk view → Edit → Delete bulk job. Removes every row of the group across the whole fleet atomically (one DELETE ... WHERE group_id = ?). Confirmation dialog gates the click.

Auto-replicate on new pair

When pair_provisioner successfully pairs a new PBX, the sidecar iterates every existing bulk group and inserts a fresh row for the new PBX with enabled=1. This happens before the pair-complete audit row lands, so any job created "for the fleet" today automatically covers every PBX added tomorrow — no operator action needed.

Skipped on the soft-pair repair path: repair preserves the existing pbxes.uuid FK reference, so already-inserted scheduled_jobs rows stay valid.

Individual cron jobs

Per-PBX Cron tab → + New Individual Cron. The form drawer opens inline — same schedule / command / notes fields as the bulk form, no PBX picker (the tab you're on IS the target).

Individual jobs are standalone. Editing changes only this one PBX's row. Deleting removes only this row. They're invisible from the bulk Cron list — that view shows bulk-origin groups only.

Per-PBX Cron tab

/pbx/{uuid}/cron — operator role required. Two stacked sections:

1. Individual cron jobs (top) — this PBX's individual-origin rows. Each row has Enable/Disable + Edit + Delete. The + New Individual Cron button opens the inline drawer.
2. Bulk cron jobs (bottom) — this PBX's assignment for every bulk group. Read-only except for the Enable/Disable toggle (which flips this PBX's row without touching the rest of the group). No Edit, no Delete — content changes require the bulk view; removing a bulk job requires deleting the whole group from the bulk view.

Below both sections, a Recent runs table shows the last 20 executions on this PBX (both origins mixed, ordered by start time). Each row has:

• Started-at timestamp
• Exit code (0 = green badge, non-zero = red)
• Duration in ms
• Command (truncated to first line; full text expandable if it fit under 512 chars)
• stdout and stderr as collapsible <details> panels
• Origin badge (bulk / individual)

Permission matrix at a glance:

Cron expression reference

Standard 5-field Vixie-cron format:

* * * * *  command
│ │ │ │ │
│ │ │ │ └── day-of-week   (0-7,   0 or 7 = Sunday)
│ │ │ └──── month         (1-12)
│ │ └────── day-of-month  (1-31)
│ └──────── hour          (0-23)
└────────── minute        (0-59)

Supported syntax in each field:

• Wildcard — * = every value in range
• Number — 5 = exactly this value
• Range — 1-5 = 1 through 5 inclusive
• List — 1,3,5 = these specific values
• Step — */15 = every 15 units starting from range low, 1-30/5 = every 5th value in the range

Range validation is enforced server-side (both PHP form validator and the sidecar Python parser). 60 * * * * is rejected before persist; * 24 * * * is rejected; 0 0 32 * * is rejected.

DOM/DOW OR-semantics

When both day-of-month and day-of-week are restricted (neither is *), Vixie-cron fires when either matches — not both. For example, 0 2 15 * 1 fires at 02:00 on the 15th of any month or on any Monday. If you want strict AND, use * in one of the two fields and filter with a wrapper.

Wildcard-in-one field falls through to standard AND semantics: 0 2 * * 1 fires at 02:00 every Monday (day-of-month is *, day-of-week matches).

Timezone

Schedules are interpreted in the manager's system timezone, not the PBX's. Set the manager's timezone during install (see TIMEZONE.md); it's pinned across the OS, PHP, and MariaDB so cron fires consistently.

For multi-tenant operators running PBXes in mixed timezones, this matters — a bulk job scheduled 0 2 * * * in a manager on America/Chicago fires at 02:00 CT, which is a different wall-clock time on a PBX in Europe/Warsaw. Consider adding a note on the job explaining the intended fire time in each region's local time.

Common recipes

Execution model

• Sidecar ticker cadence — customsso-sidecar.py's main loop runs every 5 seconds. Once per iteration it calls cron_ticker.tick() which evaluates every enabled row's schedule against the current wall-clock minute (floored to seconds). Due rows dispatch via the existing SSH runner.
• 60-second resolution — cron expressions have minute-precision granularity. A 0 2 * * * schedule fires within the 02:00-to-02:00:59 window, not at exactly 02:00:00. Nothing cron-based ever offers sub-minute precision; if you need sub-minute cadence, don't use cron.
• Concurrency cap — CRON_CONCURRENCY = 8 dispatches in flight per tick. Cron uses its own thread pool, separate from the bulk-command pool (BULK_CONCURRENCY_GLOBAL = 30, BULK_CONCURRENCY_PER_NODE = 5). Both pools share the same SSH runner underneath, so they compete for network / target-PBX CPU but not for slot allocation.
• Per-dispatch timeout — 300 seconds (5 min). A cron command that hasn't returned by then is marked exit_code = -1 and the log entry closed. The paramiko session on the PBX may still be running until the OS reaps it; nothing forcibly kills it.
• Dedup — each row tracks last_run_at at second precision, updated synchronously when the tick decides to fire. Subsequent ticks skip rows whose last_run_at is in the current minute-floor. Guarantees exactly-one dispatch per minute even though the tick fires every 5 s.
• No missed-fire back-fill — if the sidecar was down at the scheduled time, the job doesn't run for that fire. It fires again at its next scheduled time. Same semantics as system cron.
• PBX offline — if the SSH connect fails (PBX offline, network partition, host-key mismatch), the run finishes with exit_code = -1 and the connect error in stderr. Next scheduled fire tries again.
• Manager reboot — schedules survive; sidecar picks up on service start and evaluates the current minute against every row.
• PBX reboot — no effect on scheduling. The manager reaches into the PBX on next fire; if PBX is still booting, that fire fails with a connect error and the next one succeeds.

Logging + audit

Three places to look:

1. scheduled_job_runs DB table — every dispatch inserts a row with started_at (set synchronously when the tick fires), then UPDATEs with finished_at, exit_code, stdout, stderr (truncated to 4 KB each), and duration_ms on completion. Truncated stdout/stderr are shown in the per-PBX Cron tab's "Recent runs" table under <details> collapsibles.

2. Sidecar log (/var/log/customsso-manager/sidecar.log) — per-dispatch lines in the same format as bulk-command dispatches:

2026-08-17 18:36:01,731 INFO cron: 1 due at 2026-08-17 18:36
2026-08-17 18:36:01,764 INFO cron: dispatch job=6 pbx=pbx107test cmd='uptime'
2026-08-17 18:36:01,947 INFO cron: done    job=6 pbx=pbx107test exit=0 took=170ms

tail -f /var/log/customsso-manager/sidecar.log | grep cron: gives a live feed.

3. audit table — every operator action writes an audit row (visible in Settings → Audit log):

Each row carries user_id (who did it), pbx_uuid (which PBX if applicable), target (group_id or row id), success (bool), and a detail JSON with schedule + command excerpt + counts.

Failure modes + troubleshooting

Job never fires. Check in this order:

1. Row is enabled — SELECT id, schedule, enabled, last_run_at FROM scheduled_jobs WHERE id = ?. enabled=0 = paused. Toggle via per-PBX Cron tab.
2. Schedule is valid — try re-saving from the UI; the validator rejects bad expressions on save.
3. Sidecar is running — systemctl status customsso-sidecar.
4. Sidecar log shows tick activity — grep "cron:" /var/log/customsso-manager/sidecar.log | tail. If you see cron: N due at ... but not your job, the schedule doesn't match the current time.
5. Timezone confusion — sidecar tick uses manager's system TZ. Compare date on the manager to what you expect the schedule to fire.

Job fires but returns non-zero exit. Open the per-PBX Cron tab → Recent runs → expand stderr under the offending row. Common causes:

• Command references a path or binary that doesn't exist on the PBX — PATH inside SSH-exec is minimal; use absolute paths (/usr/sbin/fwconsole reload not just fwconsole reload).
• Command depends on a shell feature (globs, subshells, redirects) — wrap in bash -c '…' if you need bash-specific behavior.
• Command needs a specific working directory — SSH-exec lands in /root; cd /path && command if needed.
• Command needs interactive TTY (sudo with password) — cron-dispatched commands run with no TTY; use passwordless sudo or run as root directly.

Job fires but stdout/stderr are empty. The command probably closed both descriptors before writing (e.g. command > /dev/null 2>&1). Remove the redirect if you want the output captured.

"Bulk job" button greyed out on per-PBX tab. Expected — bulk jobs are edited/deleted from the bulk view only. Per-PBX view can only Enable/Disable bulk rows.

Sidecar was down at the scheduled fire time. No back-fill; the job simply doesn't run for that window. Same as system cron. If you need back-fill semantics, use anacron on the PBX itself instead.

scheduled_job_runs filling up. Currently no vacuum. If you have high-frequency jobs (* * * * *) accumulating over months, run manually:

DELETE FROM scheduled_job_runs WHERE started_at < NOW() - INTERVAL 90 DAY;

A vacuum step is planned for a future release.

Security

• Runs as the pair-time SSH user. That's root on the PBX (per the pair-provisioner authorized_keys line). Every cron dispatch has full root access on the target PBX — same trust boundary as bulk commands.
• No shell templating / expansion. The command field is passed verbatim to ssh_executor.run. @MODNAME@ / @LINES@ placeholders from the fwconsole custom-command catalog are NOT expanded in cron commands. Substitute values yourself before saving.
• Operator-role gated for create/edit/delete on both origins (bulk and individual). Same role gate as bulk commands. Anyone with operator can schedule arbitrary root commands on any paired PBX — treat this role as fleet-wide root by design.
• CSRF token required on every state-changing POST (form field auto-injected by Session::csrfField()).
• Command length capped at 512 chars. Enforced by the model (MAX_COMMAND_LEN), validated before persist, and reinforced by the DB column type.
• HMAC / TLS out-of-scope. Cron dispatches over the same SSH channel as status probes and bulk commands — paramiko host-key pinned (see SECURITY.md #3), per-PBX SSH private key encrypted at rest under the master key.
• Audit trail is exhaustive. Every write action (create, update, delete, enable/disable toggle, auto-replicate on new pair) writes to the audit table with attribution. Read the audit table under Settings → Audit log or via SQL.

FAQ

Written question-first for search engines + AI answer boxes. Every answer is standalone (no "see above").

Can I schedule a command to run on all FreePBX servers at once?

Yes. Create a bulk cron job from the Bulk actions → Cron tab in customsso-manager v1.0.3+. The job automatically runs on every currently-paired PBX and auto-extends to any newly-added PBX. Uncheck specific PBXes in the create form to have the row created disabled on those PBXes (still there, just paused; can be re-enabled later).

Can I schedule a command per PBX independently?

Yes. Open the PBX's detail page → Cron tab → + New Individual Cron. The job exists only for that PBX, has no siblings, and can be edited or deleted from the same tab. Individual jobs never appear in the bulk view.

What happens to my scheduled jobs when I add a new PBX to the manager?

Bulk cron jobs automatically extend to the new PBX. The pair-provisioner hook inserts one row per existing bulk group with enabled=1 before returning success from the pair operation. Individual cron jobs stay pinned to their original PBX and don't replicate.

Can I run a cron command every 5 seconds?

No. Manager-side cron has 60-second minimum resolution — the sidecar evaluates schedules at minute precision, so * * * * * fires once per minute. If you need sub-minute cadence, use systemd.timer or a dedicated daemon on the PBX itself.

What timezone does the schedule use?

The manager's system timezone, not the PBX's. If your manager runs America/Chicago and the PBX runs Europe/Warsaw, a schedule of 0 2 * * * fires at 02:00 CT — which is a different wall-clock time on the PBX. See TIMEZONE.md for how to pin the manager's TZ across OS + PHP + MariaDB.

How do I see if my cron job ran?

Three places. Per-PBX Cron tab has a "Recent runs" table at the bottom showing the last 20 executions with exit code, duration, stdout, stderr. Sidecar log (/var/log/customsso-manager/sidecar.log) has cron: dispatch / cron: done lines per fire. Audit table (Settings → Audit log) records every operator action but not every fire — for per-fire history use the runs table.

How do I temporarily pause a bulk cron job on one PBX?

Open that PBX's Cron tab → find the job in the "Bulk cron jobs" section → click Disable. This flips the enabled flag on this PBX's row only; other PBXes in the group keep running the job. Click Enable to resume.

How does this compare to per-PBX crontab entries?

Manager-side cron dispatches from the manager over SSH. No /etc/crontab or /var/spool/cron/* entries land on the PBX. Consequences: (a) unpairing a PBX cleanly stops its scheduled work — no orphan cron entries surviving on the PBX; (b) the manager's system timezone is what matters, not the PBX's; (c) if the manager is down at fire time, the job doesn't run — traditional system cron on the PBX would still run. Pick manager-side cron for fleet consistency + clean unpair semantics. Pick PBX-side cron for jobs that must run even when the manager is offline.

Does this survive manager restarts / crashes?

Yes. Schedules live in MariaDB (scheduled_jobs table); they persist across sidecar restarts, manager reboots, and OS crashes. The sidecar picks up the current state on start and evaluates every enabled row against the current wall-clock minute on its next tick.

Does this survive PBX restarts?

Yes. There's nothing installed on the PBX for the cron feature — no crontab entries, no local state. The next scheduled fire attempts an SSH connect; if the PBX is booting, that fire logs a connect error and the next scheduled fire tries again.

Can I back up / migrate my cron jobs?

Yes — they're rows in the manager's MariaDB scheduled_jobs and scheduled_job_runs tables. The encrypted-bundle backup (BACKUP.md) captures the whole database including these tables. Restore + Reclaim on a fresh manager brings the schedules back and the sidecar's next tick resumes firing them.

How do I run a fwconsole command every night on every PBX?

Bulk actions → Cron → + Add Bulk Cron Job. Fill:

• Schedule: 0 2 * * * (every day at 02:00 manager time)
• Command: /usr/sbin/fwconsole reload (use absolute path)
• Notes: nightly reload
• Leave all PBXes checked.

Save. The sidecar will dispatch /usr/sbin/fwconsole reload to every paired PBX at 02:00 manager time from tonight onwards. Any PBX added later automatically starts running this job on its 02:00.