Skip to content

fix: bound stalled responses and surface token parse failures - #1765

Open
james-bg wants to merge 1 commit into
tediousjs:masterfrom
james-bg:fix/request-inactivity-timeout
Open

fix: bound stalled responses and surface token parse failures#1765
james-bg wants to merge 1 commit into
tediousjs:masterfrom
james-bg:fix/request-inactivity-timeout

Conversation

@james-bg

Copy link
Copy Markdown

Once a query's response starts arriving, tedious currently has no timer and no error path left for the remainder of that response:

  1. The request timer is cleared on the first data packet. In SentClientRequest, the await this.messageIo.readMessage() is bounded by the request timer, but the moment the first packet arrives the timer is cleared (// request timer is stopped on first data package) and nothing bounds the token-parsing consumption of the rest of the message. requestTimeout therefore only bounds time to first packet — not what its documentation leads users to expect, and not what other drivers (e.g. Microsoft.Data.SqlClient) do.
  2. The token parser's stream has no 'error' listener. Parser builds its internal stream with Readable.from(StreamParser.parseTokens(...)) and subscribes only data/drain/end, so a parse failure is an unhandled 'error' event on a stream nobody listens to — a process-level crash rather than a request error.

Between those two facts, the window from "first response packet arrived" to "message fully parsed" is the only wait in the request path that is unbounded and unerrored. Additionally, the incoming pipeline can jam legitimately and permanently with the socket healthy: IncomingMessageStream withholds its transform callback until the current message is fully consumed, so a single stalled consumer freezes the connection's entire incoming pipeline while TCP keep-alive keeps the socket ESTABLISHED indefinitely.

Production impact

We hit this in production (via node-mssql 11 / TypeORM on Node 22, against an on-prem SQL Server): an awaited query — byte-identical to ~200,000 executions that succeeded around it — never settled, for 17+ hours until a process restart. No exception anywhere (our uncaughtException/unhandledRejection handlers never fired), no timeout despite requestTimeout: 600000. Server side, the query had completed and the session sat sleeping; client side, the TCP connection was ESTABLISHED with empty rx/tx queues and zero retransmits — the response bytes had been consumed into Node userspace before the stall. The stalled connection was permanently leaked as a busy pool resource (tarn never reaps used resources), and process shutdown hung on it until SIGKILL. The trigger is a rare race we have not isolated (order of 1-in-10⁵ in our workload); this PR is about bounding the window, not explaining that race.

We reviewed the adjacent hang fixes in 19.2.2/20.0.x (#1736, #1738, #1739) — none matches this fingerprint (ours is post-login, no cancel in flight), but they suggest this defect family is in scope.

The fix

  1. Treat requestTimeout as an inactivity timeout during response consumption. Instead of clearing the request timer at the first packet, re-arm it on every chunk of incoming socket data while in SentClientRequest, and stop when the response message ends (state exit still clears it). This bounds silence without bounding legitimately long result streams — data that keeps flowing is never interrupted (our own workload includes 600-second streaming reads, so a total-response-time bound would be unacceptable; this deliberately isn't one). A jammed incoming pipeline backpressures the socket within one highWaterMark, so it is caught the same way. requestTimeout: 0 still disables the timer entirely. On expiry, the existing timeout path runs (cancel → attention, itself bounded by cancelTimeout), so the total worst case is requestTimeout + cancelTimeout.
  2. Route token-parser stream errors into the request. Parser re-emits its internal stream's 'error', and SentClientRequest routes it through the existing socket-error path — a parse failure leaves the connection at an undefined position in the TDS stream, so failing the request and closing the connection mirrors how other fatal transport errors are handled, instead of crashing the process.

Tests

test/unit/connection-request-inactivity-test.ts (fake-server pattern from connection-cancel-test.ts):

  • a response that stalls after its first packet errors with ETIMEOUT after the inactivity budget — on current master this test hangs until the mocha timeout;
  • a slow multi-chunk response whose per-chunk gaps stay under the budget completes normally, even though its total time exceeds requestTimeout — the regression a total-time interpretation would cause;
  • a token parse failure surfaces as a request error with the process alive — on current master this dies with an unhandled Unknown type stream error.

Full unit suite and lint pass locally.

Open questions for maintainers

  • Budget semantics/default: this PR reuses requestTimeout as the inactivity budget (making it mean "maximum silence" rather than "time to first packet", which is closer to what users assume). If you'd prefer a separate option (default-on or default-off), happy to rework.
  • Paused requests: a request deliberately pause()d for longer than the budget will eventually backpressure the socket and time out. If that use case matters, the timer could be suspended while the request is paused — opinions welcome.

Once the first packet of a response arrived, the request timer was
cleared and nothing bounded the remainder of the response: a response
that stopped making progress mid-message left the request pending
forever, and a token parse failure surfaced as an unhandled 'error'
event on the parser's internal stream, crashing the process.

Re-arm the request timer on every chunk of incoming socket data while
in SentClientRequest, so that requestTimeout bounds inactivity rather
than only time-to-first-packet. Legitimately long result streams are
unaffected as long as data keeps flowing; a jammed incoming pipeline
backpressures the socket and is caught the same way. requestTimeout: 0
still disables the timer entirely.

Re-emit stream errors from the token parser and route them into the
existing socket error path in SentClientRequest, failing the active
request instead of crashing the process.

Copy link
Copy Markdown
Collaborator

Thanks for the thorough analysis — the diagnosis is right, and the unbounded window you identified is real. I looked at how other drivers handle this before answering your open questions, and it also shaped how I'd like to sequence this.

Budget semantics: agreed, requestTimeout should be an inactivity bound. That's what the reference drivers do: SqlClient's CommandTimeout is documented as cumulative network-read time with a fresh budget per Read() ("a time-out can still occur after the first row is returned, and does not include user processing time, only network read time"), ODBC's query timeout applies per SQLFetch (HYT00 is a documented fetch outcome), and undici's bodyTimeout is the same shape in Node's own HTTP stack. Our "time to first packet" behavior is the outlier. And to preempt a likely question: TCP keepalive can't cover this — we already enable it, and as your own report shows, the socket stays ESTABLISHED through the hang because the peer's kernel answers the probes; only the driver can know a request has stopped progressing. TDS has no application-level heartbeat, so a driver-side inactivity timer is the only mechanism that exists.

This is a breaking change, though. Under the default 15s, a query that goes silent mid-result-set (blocked on a lock between rows, slow per-row computation) and previously succeeded will now fail with ETIMEOUT. That's SqlClient's documented behavior too, but it's new for us, so this needs to land marked as breaking, and it will go into the next major — there isn't one pending right now, so the timer change will wait until we cut v21 (likely batched with other planned breaking work), while parts 1 and 2 below can ship on v20 in the meantime.

Sequencing — I'd like to split this PR and stage it:

  1. The token-parser error routing is a pure bugfix (crash → request error) and I'll take it right away. Could you split it into its own PR? refactor: iterate login response tokens directly #1759 fixes the same unhandled-'error' hole for the login path, so this completes the picture.
  2. Before the timer change, we'll add AbortSignal support to Request — abort maps to the existing graceful cancel path (attention, bounded by cancelTimeout), with the request erroring with signal.reason so AbortSignal.timeout() is distinguishable. That covers total-time bounds, deadlines, and caller cancellation, which is why I don't want a second timeout option — and it gives users the migration target the changelog can point to. The internals are already moving to signal-based plumbing (refactor: add MessageIO.writeMessage/readMessage and use them for the PRELOGIN exchange #1756), so this is a thin mapping.
  3. Then the inactivity semantics land on top, where the timer becomes just another trigger of the same cancellation mechanism. Two asks for that revision: suspend the timer while the request is pause()d — SqlClient explicitly excludes user processing time, and a paused request is our equivalent of not calling Read() — with a test that a request paused longer than the budget survives; and update the requestTimeout documentation in the same PR.

If you're up for reworking the branch along those lines, I'm happy to drive the AbortSignal piece so you're not blocked on it.


Generated by Claude Code

arthurschreiber pushed a commit that referenced this pull request Aug 29, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare` and `#unprepare` via a new
trailing options argument. The signal is scoped to that single execution
of the request - matching `fetch(url, { signal })` - so a `Request` that
is executed multiple times (e.g. the prepare/execute flow) can be given
a fresh signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The `abort`
listener is armed per execution and removed when the execution
completes, so a long-lived signal shared across many requests does not
accumulate listeners. The connection remains usable after an aborted
request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
arthurschreiber pushed a commit that referenced this pull request Aug 29, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare` and `#unprepare` via a new
trailing options argument. The signal is scoped to that single execution
of the request - matching `fetch(url, { signal })` - so a `Request` that
is executed multiple times (e.g. the prepare/execute flow) can be given
a fresh signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The `abort`
listener is armed per execution and removed when the execution
completes, so a long-lived signal shared across many requests does not
accumulate listeners. The connection remains usable after an aborted
request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
arthurschreiber pushed a commit that referenced this pull request Aug 29, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare` and the
transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The `abort`
listener is armed per execution and removed when the execution
completes, so a long-lived signal shared across many requests does not
accumulate listeners. The connection remains usable after an aborted
request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
arthurschreiber pushed a commit that referenced this pull request Aug 29, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare` and the
transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The `abort`
listener is armed per execution and removed when the execution
completes, so a long-lived signal shared across many requests does not
accumulate listeners. The connection remains usable after an aborted
request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
arthurschreiber pushed a commit that referenced this pull request Aug 29, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
arthurschreiber pushed a commit that referenced this pull request Aug 29, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
arthurschreiber pushed a commit that referenced this pull request Aug 29, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
arthurschreiber pushed a commit that referenced this pull request Aug 29, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCKxVCmEPBmVzW9T8ksVnW
arthurschreiber pushed a commit that referenced this pull request Aug 30, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arthurschreiber added a commit that referenced this pull request Aug 30, 2026
Allow passing an `AbortSignal` to `Connection#execSql`, `#execSqlBatch`,
`#execute`, `#callProcedure`, `#prepare`, `#unprepare`, `#execBulkLoad`
and the transaction methods (`#beginTransaction`, `#commitTransaction`,
`#rollbackTransaction`, `#saveTransaction`) via a new trailing options
argument. The signal is scoped to that single execution of the request -
matching `fetch(url, { signal })` - so a `Request` that is executed
multiple times (e.g. the prepare/execute flow) can be given a fresh
signal per execution.

Aborting the signal cancels the request through the existing graceful
cancellation mechanism (terminating the request message with the
`IGNORE` bit, or sending an attention message), but the request
completes with the signal's abort reason instead of a generic `ECANCEL`
error - so a `TimeoutError` from `AbortSignal.timeout()` is
distinguishable from a manual cancellation. A non-`Error` abort reason
is wrapped in a `RequestError` with code `EABORT`. For a bulk load, the
signal covers both the `insert bulk` statement and the bulk load
message itself.

A signal that is already aborted when the request is executed fails the
request immediately, without sending anything to the server. The signal
and its `abort` listener are tracked on the connection alongside the
request and cancel timers, armed per execution and cleaned up at the
same points as those timers - so a long-lived signal shared across many
requests does not accumulate listeners. The connection remains usable
after an aborted request.

This provides a caller-controlled way to express total-time bounds,
deadlines, and linked cancellation (via `AbortSignal.timeout()` and
`AbortSignal.any()`), as discussed in #1765 - instead of introducing a
second driver-level timeout option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants