Daemon supervision and alerting plan for rehearsals and live

Date: 29 August 2026 Status: RUNBOOK/PENDING OWNER · FIX-FIRST · alerting does not exist today Source status: VERIFIED · Date: 29 August 2026 · checked-out docs commit 9e5a130; external status reports #347 merged and #348 merged pending main CI, but those merges are not present in this branch history.

1 — Host and positive case

• STATUS: CONDITIONAL RECOMMENDATION · DATE: 29 August 2026. Run exactly 1 daemon and notify-only scheduler on the current Mac under launchd only if the host-safety checks and owner decisions below pass. The code's authority is local: mode-0700 state, ledger/artifacts, heartbeat, PID, lease, and account fence must share one canonical host/directory. Fly/VM custody is unsupported today; that makes the local host the current bounded proposal, not an accepted or uniquely credible path.

2 — Supervision and restart semantics

• STATUS: P0 REQUIRED · DATE: 3 September 2026. Add scripts/live-lane-supervisor.mjs; it spawns the actual daemon, forwards SIGINT/SIGTERM, reads heartbeat/ledger every 15 seconds, and classifies final JSON. On an unexpected crash whose last decision was continue, prove the old PID dead, wait 95 seconds—greater than the daemon’s 3×30-second lease—then retry once; subsequent retryable failure waits 5 minutes, then 15 minutes, maximum 3 attempts/hour. Any non-continuing heartbeat or pending incident stops restarts and pages. A healthy 30 minutes resets backoff.
• STATUS: SEALED IN CODE · DATE: 29 August 2026. Restart must never delete/replace heartbeat or lease, reuse a fencing token, reset seq/killPrioritySeq, skip pending cleanup, re-arm, or invoke plan/submit. Startup resumes hard actions before the loop (daemon/index.ts:626–646); each process acquires a fresh token and renews to 3×interval (:606–620,653); takeover requires expiry plus a dead same-host PID (lease/index.ts:135–146,320–326); hard action persists a non-continuing heartbeat before disarm/cancel/flatten (daemon/index.ts:806–839).
• STATUS: HAZARD · DATE: 29 August 2026. The same host and state directory refuse daemon #2. A different directory or host bypasses both file leases; cross-host recovery is explicitly unsupported. Ban foreground starts and never run a Fly/VM copy concurrently.

3 — Paging and stale-heartbeat detector

• STATUS: MUST-FIX BEFORE 3 SEPTEMBER 2026. Resend approval helper only — not live alerting. The existing helper and Resend-specific approval infrastructure are forbidden for live alerts. A separate strict, delivery-observable non-Resend alert channel is required. Page on every lastDecision !== 'continue', including soft_review; missing/malformed or >60-second heartbeat; invalid lease/PID; status.daemon.fresh:false; child exit; pending/incomplete cleanup; broker outage; unmatched reconciliation; and every wrapper refusal.
• STATUS: EXACT CONDITIONS · DATE: 3 September 2026. Page on hard_action_pending, flatten_incomplete, any refusal/soft review/disarm/hard kill, broker/policy/calendar failure, and every wrapped plan/submit/reconcile non-zero exit. IMPLEMENTABILITY_REFUSED is receipt-v3 backed only after prospective policy-v2 authority is sealed; an intentional flat plan is applicable:false, and kill/flatten bypasses the gate, so neither creates a false implementability page. Deduplicate by condition+event/receipt hash, repeat unresolved pages every 15 minutes, and send recovery.

4 — launchd skeleton

• STATUS: PROPOSED · DATE: 3 September 2026. Secrets stay in a mode-0600 env file, never the plist.

<plist version="1.0"><dict>
<key>Label</key><string>com.tradegg.live-lane</string>
<key>ProgramArguments</key><array>
<string>/bin/zsh</string><string>-lc</string>
<string>set -a; source &lt;OPERATOR_CONFIG_DIR&gt;/live-lane.env; set +a; exec &lt;NODE_BIN&gt; &lt;REPO_ROOT&gt;/scripts/live-lane-supervisor.mjs --ledger &lt;LEDGER_ABS&gt; --out &lt;ARTIFACTS_ABS&gt; --interval 30</string>
</array>
<key>WorkingDirectory</key><string>&lt;REPO_ROOT&gt;</string>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><dict><key>SuccessfulExit</key><false/></dict>
<key>ThrottleInterval</key><integer>60</integer>
<key>StandardOutPath</key><string>&lt;OPERATOR_LOG_DIR&gt;/live-lane.out.log</string>
<key>StandardErrorPath</key><string>&lt;OPERATOR_LOG_DIR&gt;/live-lane.err.log</string>
</dict></plist>

• STATUS: MUST-FIX BEFORE 3 SEPTEMBER 2026. KeepAlive.SuccessfulExit=false is valid only with the reviewer's terminal semantics: the supervisor stays alive while it classifies child state and applies its own backoff; a deliberate terminal/refusal state makes the supervisor exit zero, so launchd does not relaunch it; only an unexpected supervisor failure exits nonzero and may be relaunched. The notify-only scheduler is separate and cannot invoke plan, submit, reconcile, arm, graduation, or hard actions.

5 — Operator runbook

• STATUS: RUNBOOK · DATE: 3 September 2026. Start: plutil -lint "$PLIST"; launchctl bootstrap "gui/$(id -u)" "$PLIST"; launchctl kickstart -k "gui/$(id -u)/com.tradegg.live-lane". Verify: launchctl print "gui/$(id -u)/com.tradegg.live-lane"; pgrep -fl live-lane-daemon.mjs returns exactly 1; node scripts/live-lane.mjs status --json --ledger "$LEDGER" --out "$ARTIFACTS". Stop: launchctl bootout "gui/$(id -u)" "$PLIST"; wait for forwarded SIGTERM and lease release. Logs: the 2 plist paths.
• STATUS: MORNING CHECK · DATE: 4, 8, 9, 10, 11 September 2026. Before open verify 1 PID, heartbeat age ≤60s and fresh:true, pending hard actions 0, expected policy/pin/account/origin/stage/arm, broker reachable, refresh/calendar horizon current, expected exposure, clean stderr, and a successful alert canary. After open require matched reconciliation.
• STATUS: REHEARSAL DEPENDENCY · DATE: before 4 September 2026. LL-4c must close both production races: single-read registry-lock contention and historical reconciliation authority after disarm, with deterministic no-sleep tests. LL-4d is fix-first before Q13–Q16: exact ranks 1..D, canary go/no-go, current ledger/PBO bindings, split event/hash domains, and v3 rehearsal compatibility must be reviewed before owner ratification. If Q13–Q16 then authorize amendment v1.2-r2, land the rank-proof receipt, canary lifecycle, exact prefix-N refusal/unwind semantics, final policy-v2 hash, and fail-closed executor hash before any nonzero rehearsal. Otherwise Q13/Q14 remain dormant and no canary/prefix order is scheduled.

6 — Owners and dates

• STATUS: DUE · DATE: 3 September 2026 · OWNER: orchestrator. Deliver supervisor, new strict live-alert adapter, notify-only scheduler, command wrapper, stable refusal codes, log rotation and tests.
• STATUS: DUE · DATE: 3 September 2026 · OWNER: owner. Seal absolute paths, mode-0700 state/mode-0600 env, prevent sleep/logout, configure recipient, and prove delivery.
• STATUS: DUE · DATE: 4, 8, 9, 10, 11 September 2026 · OWNER: owner/orchestrator. owner runs pre-open check and commands; orchestrator audits post-open chain and alerts.
• STATUS: DUE · DATE: 11–12 September 2026 · OWNER: orchestrator/owner. orchestrator verifies 5 matched chains, ≥95% heartbeat coverage and zero incidents; owner alone approves rehearsal/graduation.
• STATUS: DUE · DATE: 15 September 2026 · OWNER: owner/orchestrator. Stop paper job, switch the single authority to live, refresh/start once, independently verify, then owner alone submits.

7 — Falsifiers, risks, unsolved

• STATUS: OPEN · DATE: 29 August 2026. Reject this host if it cannot remain powered, awake, logged in, online and backed up. Same-host paging cannot report total host/network loss; the live alert channel is unselected; log rotation and planned-maintenance suppression are absent. An external dead-man and cross-host fence remain unsolved.
• STATUS: MUST-FIX BEFORE 4 SEPTEMBER 2026. LL-4c is a pre-rehearsal blocker: fix the root-caused registry-lock EEXIST→ENOENT single-read race and historical reconciliation-authority defect without sleeps, then pass the scoped adversarial tests. Supervision must not mask the race with restarts.

Review amendments — controlling

STATUS: FIX-FIRST · DATE: 29 August 2026. Do not install the plist or treat this as an operational runbook until these corrections have scoped tests and the owner decisions close.

1. STATUS: REQUIRED · DATE: 3 September 2026. The supervisor remains alive while classifying child terminal states and backoff. If it exits after a deliberate terminal child state, it exits zero so KeepAlive.SuccessfulExit=false does not bypass the stop decision. Only an unexpected supervisor failure exits nonzero for launchd relaunch.
2. STATUS: REQUIRED · DATE: 3 September 2026. The command scheduler is notify-only: it derives the five rehearsal/open windows from the sealed exchange calendar, runs read-only status/canary checks, and pages the operator. It never invokes plan, submit, reconcile, arm, graduation, or hard-action commands; owner runs those through the audited wrapper.
3. STATUS: BOUNDARY · DATE: 3 September 2026. Host fencing proves single authority only for one account, one canonical state directory, and one host. A different state directory or host bypasses it and is forbidden.
4. STATUS: EXACT ALERT SET · DATE: 3 September 2026. Page on every lastDecision !== 'continue', including soft_review; malformed/missing/stale heartbeat; lease/PID mismatch; child exit; pending or incomplete cleanup; broker outage; unmatched reconciliation; and every wrapper refusal. The existing approval-email helper is forbidden because it swallows failures.
5. STATUS: HOST SAFETY · DATE: 3 September 2026. Prove sleep/lid/logout behavior, prevent sleep during the window, create the log directory, rotate logs, validate recovery after wake, and document planned-maintenance suppression. Failure of any check rejects this host.
6. STATUS: PENDING OWNER · DEADLINE: 3 September 2026. Accept Mac/network-loss blindness or authorize a Healthchecks.io dead-man. If authorized, ping only after each complete healthy check with curl -fsS -m 10 --retry 5 -o /dev/null "$HEALTHCHECKS_PING_URL". The endpoint is mode-0600 secret configuration, never printed or committed. If blindness is accepted, seal that residual risk before rehearsal; same-host email does not cover host loss.
7. STATUS: FALSIFIER · DATE: 4 September 2026. A launchd relaunch after a terminal refusal, an alert gap, two live daemon PIDs, a missed external heartbeat when that detector is selected, or a wrapper capable of autonomous submit refuses rehearsal.

Pending owner

1. STATUS: OPEN · DATE: 29 August 2026. Accept Mac host-loss blindness or authorize an external dead-man.
2. STATUS: OPEN · DATE: 29 August 2026. Approve the retry budget.
3. STATUS: OPEN · DATE: 29 August 2026. Choose canonical paper and live state/artifact paths.
4. STATUS: PENDING OWNER · DEADLINE: 3 September 2026. Choose and authorize a strict non-Resend live-alert channel, sender, recipient, acknowledgement, and secret-config path. Resend remains approval-helper-only.
5. STATUS: OPEN · DATE: 29 August 2026. Define and authorize IMPLEMENTABILITY_REFUSED.
6. STATUS: OPEN · DATE: 29 August 2026. Approve maintenance suppression.

Critic round 1 — applied

• STATUS: APPLIED · DATE: 29 August 2026. Removed the unsupported unique-host claim, aligned all paging with lastDecision !== 'continue', and named LL-4c as a pre-rehearsal blocker.
• STATUS: APPLIED · DATE: 29 August 2026. Replaced ambiguous launchd restart guidance with deliberate-zero/unexpected-nonzero supervisor semantics and retained a notify-only scheduler.
• STATUS: APPLIED · DATE: 29 August 2026. Kept Resend approval-helper-only, required a separate strict non-Resend live channel, and added the owner-contingent Healthchecks.io ping without exposing its endpoint.

Round 3 — applied

• STATUS: APPLIED · DATE: 29 August 2026. Added LL-4c as a hard pre-rehearsal dependency and made LL-4d conditional on sealed Q13–Q16 plus final policy/executor hashes.
• STATUS: APPLIED · DATE: 29 August 2026. Kept terminal launchd semantics, non-Resend live alerting, and the Healthchecks.io dead-man decision unchanged; no private host path or endpoint is published.