Skip to content

fix(broker): keep the shared socket reactor alive when one association's readiness publication fails - #1194

Open
Will Portnoy (willportnoy) wants to merge 1 commit into
uliteboxfrom
wportnoy/ulitebox-reactor-isolation
Open

fix(broker): keep the shared socket reactor alive when one association's readiness publication fails#1194
Will Portnoy (willportnoy) wants to merge 1 commit into
uliteboxfrom
wportnoy/ulitebox-reactor-isolation

Conversation

@willportnoy

Copy link
Copy Markdown
Member

Problem

The Linux userland broker runs a single epoll reactor thread that owns every guest socket across every session. On the asynchronous event path, a per-object readiness publication failure was fatal to the whole reactor:

  • handle_socket_event / complete_connect propagate a publish error via ?,
  • Reactor::run() maps it to ReactorFailure::Broker,
  • the reactor thread responds by calling fail_all_sockets() — forcing every socket in every session to ERROR, clearing the socket/session maps, and exiting the thread.

So one guest whose readiness sink cannot accept a notification could tear down every other guest's sockets and permanently disconnect the reactor.

Fix

Make the reactor's asynchronous readiness publication best-effort, so a single association's publication failure no longer fails the shared reactor. This mirrors the UDP endpoint handler (handle_udp_endpoint_event), which already publishes with let _ = for exactly this reason. update_snapshot commits the cached snapshot readiness before publishing, so the cached snapshot remains authoritative when the out-of-band notification is dropped.

Scope is limited to publication:

  • Unchanged / still fatal: genuine host-syscall failures and internal-consistency errors (tcp_state()?, wrong socket kind / impossible connection status) still propagate and remain fatal to the reactor.
  • Unchanged synchronous paths: synchronous guest-command handlers still return publication errors to the guest through the command response (they never reach fail_all_sockets()), and the guest-to-guest UDP enqueue keeps its transactional rollback-on-publication-failure contract.

Production reachability

An admitted socket's publication does not actually fail in production. The broker bounds live readiness registrations to the sink's capacity at association setup (litebox_broker_host readiness admission), and the sink only rejects publication when registering a new object beyond that capacity. This change therefore hardens the reactor against a fault that admission control already prevents, and strictly reduces the blast radius of any such fault from "every session's sockets" to "one dropped notification, cached snapshot still authoritative".

Test

Adds native_tcp_readiness_failure_does_not_fail_shared_reactor, the TCP analogue of the existing UDP reactor test (native_udp_readiness_failure_does_not_fail_shared_reactor). It fails the reactor's publication when peer data makes the socket readable, then asserts the socket is still serviceable afterwards. The test fails without this change (the reactor dies and the command channel is gone, so the follow-up receive returns Err(Internal)) and passes with it.

…n's readiness publication fails

The Linux userland broker runs a single epoll reactor thread that owns
every guest socket across every session. On the asynchronous event path,
a per-object readiness *publication* failure was fatal to the whole
reactor: `handle_socket_event` / `complete_connect` propagated the
publish error via `?`, `Reactor::run()` mapped it to
`ReactorFailure::Broker`, and the reactor thread responded by calling
`fail_all_sockets()` — tearing down every other session's sockets and
permanently disconnecting the reactor. One guest whose readiness sink
cannot accept a notification could therefore disconnect all other guests.

Make the reactor's asynchronous readiness publication best-effort, so a
single association's publication failure no longer fails the shared
reactor. This mirrors the UDP endpoint handler, which already publishes
with `let _ =` for exactly this reason. `update_snapshot` commits the
cached snapshot readiness before publishing, so the cached snapshot
remains authoritative when the out-of-band notification is dropped.

Scope is limited to publication. Genuine host-syscall failures and
internal-consistency errors (`tcp_state()?`, wrong socket kind /
impossible connection status) still propagate and remain fatal. The
synchronous guest-command paths are unchanged: their publication errors
are returned to the guest through the command response and never reach
`fail_all_sockets()`, and the guest-to-guest UDP enqueue keeps its
transactional rollback-on-publication-failure contract.

Note on production reachability: an admitted socket's publication does
not actually fail. The broker bounds live readiness registrations to the
sink's capacity at association setup, and the sink only rejects
publication when registering a new object beyond that capacity. This
change therefore hardens the reactor against a fault that admission
control already prevents, and strictly reduces the blast radius of any
such fault from "every session's sockets" to "one dropped notification,
cached snapshot still authoritative".

Add `native_tcp_readiness_failure_does_not_fail_shared_reactor`, the TCP
analogue of the existing UDP reactor test: it fails the reactor's
publication when peer data makes the socket readable and asserts the
socket is still serviceable afterwards. The test fails without this fix
(the reactor dies and the command channel is gone) and passes with it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1494e366-b3cf-4196-91a0-1430cb9d5cc8
@willportnoy

Copy link
Copy Markdown
Member Author

AI-authored, human-reviewed.

Before opening, this change was put through two independent static reviewers — GPT-5.6 Sol (xhigh) and Claude Opus 5 (xhigh). Both converged on a single substantive point, with different severity calls:

  • Opus 5 — SHIP. Independently verified: (a) the only two ReactorFailure::Broker escalations are the two handler calls in run(), and process_commands / drain_wake / expire_deadlined_state cannot escalate a publication failure; (b) the async-best-effort vs. synchronous-transactional split is correct — the guest-to-guest UDP enqueue rollback is untouched; (c) the new test genuinely exercises the async handle_socket_event path and is not flaky (the armed predicate cannot be consumed early, and the wait is deadline-asserted).
  • Sol — one High note: the shared concern below.

The shared concern — lost wakeup on a dropped async publication. Because publication is now best-effort and update_snapshot only re-publishes on change, a dropped notification is not retried; combined with edge-triggered host epoll, a guest blocked purely on that notification could stall on that one socket.

Resolution (not production-reachable for sockets). The production sink only rejects publication with TooManyObjects, i.e. when registering a new object beyond capacity. Broker admission bounds live readiness registrations to the sink capacity at association setup (litebox_broker_host readiness admission), so an already-admitted socket's publication does not fail. This PR therefore hardens the reactor against a fault admission control already prevents, and is strictly better than the prior behavior (which forced every session's sockets to ERROR and exited the reactor for the same input). The same tradeoff is already accepted in the merged UDP handler. The reasoning is captured in the handle_socket_event doc comment and the commit body. If sink failure were ever made genuinely reachable, the follow-up is to keep the object dirty on a failed publish (track last-published separately) rather than re-escalate — noted for the timerfd rebase, where readiness admission accounting for timer registrations is a separate item.

@github-actions

Copy link
Copy Markdown

🤖 SemverChecks 🤖 No breaking API changes detected

Note: this does not mean API is unchanged, or even that there are no breaking changes; simply, none of the detections triggered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant