fix(server): observe persisted sandbox watch updates

[elezar]

Summary

Make WatchSandbox status-follow streams observe sandbox changes persisted by another gateway process. The existing in-memory notification bus remains the fast path; a bounded store reconciliation closes the cross-process consistency gap.

Related Issue

Related to #1868. This is a generally applicable consistency fix extracted from the HA review.

Changes

• Poll the authoritative sandbox record once per second only while follow_status is enabled.
• Delay the first poll until one interval after the initial snapshot and skip missed ticks to avoid redundant reads and catch-up bursts.
• Emit updates only when the persisted resource_version changes.
• Keep streams alive across transient poll failures, logging only the first failure in each consecutive failure period.
• Add a focused test that persists a status change without notifying the local in-memory bus.

Testing

• mise run pre-commit passes (run inside nix develop)
• Unit tests added/updated
• E2E tests added/updated (not applicable)

Focused regression test:

• nix develop -c cargo test -p openshell-server watch_sandbox_observes_store_update_without_local_notification --lib

Operational tradeoff

Each active WatchSandbox stream with follow_status=true performs one indexed single-record store read per second. This intentionally favors a small, isolated consistency fix over adding a database-specific notification mechanism. Streams that only follow logs or platform events do not poll. A future shared-store watch abstraction could coalesce reads if sustained watcher volume makes this material.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; no public interface, configuration, or stable subsystem boundary changed)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.