fix(broker): keep the shared socket reactor alive when one association's readiness publication fails - #1194
Conversation
…n's readiness publication fails The Linux userland broker runs a single epoll reactor thread that owns every guest socket across every session. On the asynchronous event path, a per-object readiness *publication* failure was fatal to the whole reactor: `handle_socket_event` / `complete_connect` propagated the publish error via `?`, `Reactor::run()` mapped it to `ReactorFailure::Broker`, and the reactor thread responded by calling `fail_all_sockets()` — tearing down every other session's sockets and permanently disconnecting the reactor. One guest whose readiness sink cannot accept a notification could therefore disconnect all other guests. Make the reactor's asynchronous readiness publication best-effort, so a single association's publication failure no longer fails the shared reactor. This mirrors the UDP endpoint handler, which already publishes with `let _ =` for exactly this reason. `update_snapshot` commits the cached snapshot readiness before publishing, so the cached snapshot remains authoritative when the out-of-band notification is dropped. Scope is limited to publication. Genuine host-syscall failures and internal-consistency errors (`tcp_state()?`, wrong socket kind / impossible connection status) still propagate and remain fatal. The synchronous guest-command paths are unchanged: their publication errors are returned to the guest through the command response and never reach `fail_all_sockets()`, and the guest-to-guest UDP enqueue keeps its transactional rollback-on-publication-failure contract. Note on production reachability: an admitted socket's publication does not actually fail. The broker bounds live readiness registrations to the sink's capacity at association setup, and the sink only rejects publication when registering a new object beyond that capacity. This change therefore hardens the reactor against a fault that admission control already prevents, and strictly reduces the blast radius of any such fault from "every session's sockets" to "one dropped notification, cached snapshot still authoritative". Add `native_tcp_readiness_failure_does_not_fail_shared_reactor`, the TCP analogue of the existing UDP reactor test: it fails the reactor's publication when peer data makes the socket readable and asserts the socket is still serviceable afterwards. The test fails without this fix (the reactor dies and the command channel is gone) and passes with it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1494e366-b3cf-4196-91a0-1430cb9d5cc8
|
AI-authored, human-reviewed. Before opening, this change was put through two independent static reviewers — GPT-5.6 Sol (xhigh) and Claude Opus 5 (xhigh). Both converged on a single substantive point, with different severity calls:
The shared concern — lost wakeup on a dropped async publication. Because publication is now best-effort and Resolution (not production-reachable for sockets). The production sink only rejects publication with |
|
🤖 SemverChecks 🤖 No breaking API changes detected Note: this does not mean API is unchanged, or even that there are no breaking changes; simply, none of the detections triggered. |
Problem
The Linux userland broker runs a single epoll reactor thread that owns every guest socket across every session. On the asynchronous event path, a per-object readiness publication failure was fatal to the whole reactor:
handle_socket_event/complete_connectpropagate a publish error via?,Reactor::run()maps it toReactorFailure::Broker,fail_all_sockets()— forcing every socket in every session toERROR, clearing the socket/session maps, and exiting the thread.So one guest whose readiness sink cannot accept a notification could tear down every other guest's sockets and permanently disconnect the reactor.
Fix
Make the reactor's asynchronous readiness publication best-effort, so a single association's publication failure no longer fails the shared reactor. This mirrors the UDP endpoint handler (
handle_udp_endpoint_event), which already publishes withlet _ =for exactly this reason.update_snapshotcommits the cached snapshot readiness before publishing, so the cached snapshot remains authoritative when the out-of-band notification is dropped.Scope is limited to publication:
tcp_state()?, wrong socket kind / impossible connection status) still propagate and remain fatal to the reactor.fail_all_sockets()), and the guest-to-guest UDP enqueue keeps its transactional rollback-on-publication-failure contract.Production reachability
An admitted socket's publication does not actually fail in production. The broker bounds live readiness registrations to the sink's capacity at association setup (
litebox_broker_hostreadiness admission), and the sink only rejects publication when registering a new object beyond that capacity. This change therefore hardens the reactor against a fault that admission control already prevents, and strictly reduces the blast radius of any such fault from "every session's sockets" to "one dropped notification, cached snapshot still authoritative".Test
Adds
native_tcp_readiness_failure_does_not_fail_shared_reactor, the TCP analogue of the existing UDP reactor test (native_udp_readiness_failure_does_not_fail_shared_reactor). It fails the reactor's publication when peer data makes the socket readable, then asserts the socket is still serviceable afterwards. The test fails without this change (the reactor dies and the command channel is gone, so the follow-up receive returnsErr(Internal)) and passes with it.