fix: bound stalled responses and surface token parse failures - #1765
fix: bound stalled responses and surface token parse failures#1765james-bg wants to merge 1 commit into
Conversation
Once the first packet of a response arrived, the request timer was cleared and nothing bounded the remainder of the response: a response that stopped making progress mid-message left the request pending forever, and a token parse failure surfaced as an unhandled 'error' event on the parser's internal stream, crashing the process. Re-arm the request timer on every chunk of incoming socket data while in SentClientRequest, so that requestTimeout bounds inactivity rather than only time-to-first-packet. Legitimately long result streams are unaffected as long as data keeps flowing; a jammed incoming pipeline backpressures the socket and is caught the same way. requestTimeout: 0 still disables the timer entirely. Re-emit stream errors from the token parser and route them into the existing socket error path in SentClientRequest, failing the active request instead of crashing the process.
|
Thanks for the thorough analysis — the diagnosis is right, and the unbounded window you identified is real. I looked at how other drivers handle this before answering your open questions, and it also shaped how I'd like to sequence this. Budget semantics: agreed, This is a breaking change, though. Under the default 15s, a query that goes silent mid-result-set (blocked on a lock between rows, slow per-row computation) and previously succeeded will now fail with Sequencing — I'd like to split this PR and stage it:
If you're up for reworking the branch along those lines, I'm happy to drive the Generated by Claude Code |
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare` and `#unprepare` via a new
trailing options argument. The signal is scoped to that single execution
of the request - matching `fetch(url, { signal })` - so a `Request` that
is executed multiple times (e.g. the prepare/execute flow) can be given
a fresh signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The `abort`
listener is armed per execution and removed when the execution
completes, so a long-lived signal shared across many requests does not
accumulate listeners. The connection remains usable after an aborted
request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare` and `#unprepare` via a new
trailing options argument. The signal is scoped to that single execution
of the request - matching `fetch(url, { signal })` - so a `Request` that
is executed multiple times (e.g. the prepare/execute flow) can be given
a fresh signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The `abort`
listener is armed per execution and removed when the execution
completes, so a long-lived signal shared across many requests does not
accumulate listeners. The connection remains usable after an aborted
request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare` and the
transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The `abort`
listener is armed per execution and removed when the execution
completes, so a long-lived signal shared across many requests does not
accumulate listeners. The connection remains usable after an aborted
request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare` and the
transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The `abort`
listener is armed per execution and removed when the execution
completes, so a long-lived signal shared across many requests does not
accumulate listeners. The connection remains usable after an aborted
request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.
Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.
A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.
This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Once a query's response starts arriving, tedious currently has no timer and no error path left for the remainder of that response:
SentClientRequest, theawait this.messageIo.readMessage()is bounded by the request timer, but the moment the first packet arrives the timer is cleared (// request timer is stopped on first data package) and nothing bounds the token-parsing consumption of the rest of the message.requestTimeouttherefore only bounds time to first packet — not what its documentation leads users to expect, and not what other drivers (e.g. Microsoft.Data.SqlClient) do.'error'listener.Parserbuilds its internal stream withReadable.from(StreamParser.parseTokens(...))and subscribes onlydata/drain/end, so a parse failure is an unhandled'error'event on a stream nobody listens to — a process-level crash rather than a request error.Between those two facts, the window from "first response packet arrived" to "message fully parsed" is the only wait in the request path that is unbounded and unerrored. Additionally, the incoming pipeline can jam legitimately and permanently with the socket healthy:
IncomingMessageStreamwithholds its transform callback until the current message is fully consumed, so a single stalled consumer freezes the connection's entire incoming pipeline while TCP keep-alive keeps the socket ESTABLISHED indefinitely.Production impact
We hit this in production (via node-mssql 11 / TypeORM on Node 22, against an on-prem SQL Server): an awaited query — byte-identical to ~200,000 executions that succeeded around it — never settled, for 17+ hours until a process restart. No exception anywhere (our
uncaughtException/unhandledRejectionhandlers never fired), no timeout despiterequestTimeout: 600000. Server side, the query had completed and the session satsleeping; client side, the TCP connection was ESTABLISHED with empty rx/tx queues and zero retransmits — the response bytes had been consumed into Node userspace before the stall. The stalled connection was permanently leaked as a busy pool resource (tarn never reapsusedresources), and process shutdown hung on it until SIGKILL. The trigger is a rare race we have not isolated (order of 1-in-10⁵ in our workload); this PR is about bounding the window, not explaining that race.We reviewed the adjacent hang fixes in 19.2.2/20.0.x (#1736, #1738, #1739) — none matches this fingerprint (ours is post-login, no cancel in flight), but they suggest this defect family is in scope.
The fix
requestTimeoutas an inactivity timeout during response consumption. Instead of clearing the request timer at the first packet, re-arm it on every chunk of incoming socket data while inSentClientRequest, and stop when the response message ends (state exit still clears it). This bounds silence without bounding legitimately long result streams — data that keeps flowing is never interrupted (our own workload includes 600-second streaming reads, so a total-response-time bound would be unacceptable; this deliberately isn't one). A jammed incoming pipeline backpressures the socket within onehighWaterMark, so it is caught the same way.requestTimeout: 0still disables the timer entirely. On expiry, the existing timeout path runs (cancel → attention, itself bounded bycancelTimeout), so the total worst case isrequestTimeout + cancelTimeout.Parserre-emits its internal stream's'error', andSentClientRequestroutes it through the existing socket-error path — a parse failure leaves the connection at an undefined position in the TDS stream, so failing the request and closing the connection mirrors how other fatal transport errors are handled, instead of crashing the process.Tests
test/unit/connection-request-inactivity-test.ts(fake-server pattern fromconnection-cancel-test.ts):ETIMEOUTafter the inactivity budget — on current master this test hangs until the mocha timeout;requestTimeout— the regression a total-time interpretation would cause;Unknown typestream error.Full unit suite and lint pass locally.
Open questions for maintainers
requestTimeoutas the inactivity budget (making it mean "maximum silence" rather than "time to first packet", which is closer to what users assume). If you'd prefer a separate option (default-on or default-off), happy to rework.pause()d for longer than the budget will eventually backpressure the socket and time out. If that use case matters, the timer could be suspended while the request is paused — opinions welcome.