Skip to content

Fix SessionPool blocking indefinitely after a server outage 修复服务端断开后 SessionPool 永久阻塞的问题 - #62

Merged
HTHou merged 3 commits into
apache:mainfrom
CritasWang:fix/pool-wait-timeout-and-slot-leak
Jul 31, 2026
Merged

Fix SessionPool blocking indefinitely after a server outage 修复服务端断开后 SessionPool 永久阻塞的问题#62
HTHou merged 3 commits into
apache:mainfrom
CritasWang:fix/pool-wait-timeout-and-slot-leak

Conversation

@CritasWang

Copy link
Copy Markdown
Contributor

Fixes #61

What this fixes / 修复内容

A SessionPool that outlived a server outage became permanently unusable: callers blocked forever with no
exception, and the pool could not recover even after the server came back. Two independent defects combined
to produce that.

服务端断开后,SessionPool 会变得永久不可用:调用方永久阻塞且不抛异常,服务端恢复后也无法自愈。这是
两个独立缺陷叠加的结果。

1. Pool wait timeout was written in ms and read in seconds / 池等待超时的单位错误

Open() assigned _clients.Timeout = _timeout * 5 (a millisecond value) while
ConcurrentClientQueue.Take() consumed it via TimeSpan.FromSeconds. The 500 ms default connection timeout
therefore produced a ~41 minute block before SessionPoolDepletedException surfaced; the legacy
timeout: 10000 constructor argument produced 13.9 hours.

Open() 以毫秒赋值而 Take() 按秒消费,500ms 的默认连接超时因此变成约 41 分钟 的阻塞,旧构造函数
timeout: 10000 时更是 13.9 小时

  • ConcurrentClientQueue now exposes TimeoutInMs and waits with TimeSpan.FromMilliseconds.
    ConcurrentClientQueue 改用 TimeoutInMs,并以 TimeSpan.FromMilliseconds 等待。
  • The old seconds-based Timeout property is kept as an [Obsolete] shim so existing callers still compile
    and behave as they intended.
    原秒级 Timeout 属性保留为 [Obsolete] 兼容包装,既有调用方仍可编译且语义符合预期。
  • The pool wait budget is no longer derived from the socket timeout. It is configurable via
    SetPoolWaitTimeoutInMs on both builders, defaulting to 10 s.
    池等待时间不再由 socket 超时推导,改为两个 Builder 上的 SetPoolWaitTimeoutInMs,默认 10 秒。

2. Pool slots leaked on every failed reconnection / 重连失败导致槽位泄漏

When reconnection failed, the dead client was discarded without being returned and nothing replenished the
pool, so capacity shrank by one per failure. After PoolSize failures every caller blocked on an empty queue
that nobody would ever feed, and the pool stayed dead after the server recovered.

重连失败时失效连接被丢弃且不归还,池无补充机制,每失败一次容量减一;PoolSize 次失败后所有调用都阻塞在
无人投喂的空队列上,服务端恢复后连接池依然是死的。

  • Vacant slots are now tracked (_vacantSlots, surfaced as SessionPool.VacantSlots) instead of being
    silently dropped, so capacity stays at PoolSize.
    改为记录空槽(_vacantSlots,通过 SessionPool.VacantSlots 暴露),容量保持为 PoolSize
  • AcquireClientAsync re-materializes a vacant slot when the queue is empty, so the pool heals itself on the
    next operation once the server is reachable — no Close() + Open() cycle needed.
    队列为空时 AcquireClientAsync 会就地重建空槽,服务端恢复后下一次操作即可自愈,无需 Close() + Open()
  • Reconnect() now accepts a null original client so the self-heal path also works with nodeUrls
    (multi-node) configurations.
    Reconnect() 允许原始连接为 null,使 nodeUrls(多节点)配置下的自愈路径同样可用。

3. IsOpen() semantics documented / 明确 IsOpen() 语义

IsOpen() is a lifecycle flag, not a connectivity probe — the client keeps no heartbeat, so it stays true
after the server goes down. Behaviour is unchanged (and matches the Java client); this PR only documents it,
because the guard if (pool.IsOpen()) return; is a common way to accidentally prevent a pool from ever being
rebuilt.

IsOpen() 是生命周期标志而非连通性探针——客户端无心跳,服务端断开后它仍为 true。行为未变(与 Java
客户端一致),本 PR 仅补充文档说明,因为 if (pool.IsOpen()) return; 这种守卫很容易让连接池永远不被重建。

Compatibility / 兼容性

No breaking changes. New constructor overloads take the pool wait timeout as an extra parameter and the
existing overloads delegate to them with the default; ConcurrentClientQueue.Timeout still compiles and now
converts to/from TimeoutInMs.

无破坏性变更。新增的构造函数重载多接收一个池等待超时参数,原有重载以默认值委托到新重载;
ConcurrentClientQueue.Timeout 仍可编译,并与 TimeoutInMs 相互换算。

Behavioural change worth calling out: a depleted pool now fails after 10 s by default instead of tens of
minutes. Anyone who (unintentionally) relied on the long wait should set SetPoolWaitTimeoutInMs explicitly.

需要注意的行为变化:池耗尽时现在默认 10 秒后失败,而不是数十分钟。若有人(无意中)依赖了原来的长等待,
请显式设置 SetPoolWaitTimeoutInMs

Tests / 测试

  • ConcurrentClientQueueTests — timeout is honoured in milliseconds (the direct regression guard), the
    message reports the unit, Return() wakes a blocked Take(), a ready client is returned without waiting,
    and the obsolete Timeout shim converts correctly.
    ConcurrentClientQueueTests —— 超时按毫秒生效(针对该 bug 的直接回归测试)、异常消息标注单位、
    Return() 能唤醒阻塞的 Take()、有可用连接时立即返回、过时的 Timeout 兼容属性换算正确。
  • SessionPoolConfigurationTests — the pool wait timeout is independent of the connection timeout and is
    propagated by both SessionPool.Builder (host/port and nodeUrls paths) and TableSessionPool.Builder.
    SessionPoolConfigurationTests —— 池等待超时与连接超时相互独立,且能被 SessionPool.Builder
    (host/port 与 nodeUrls 两条路径)和 TableSessionPool.Builder 正确传递。

dotnet test tests/Apache.IoTDB.Tests — 33 passed, 0 failed. dotnet build Apache.IoTDB.sln — 0 errors.
dotnet format --verify-no-changes — clean.

Note: the unit tests were executed locally against net8.0 because no .NET 5 arm64 runtime is available on
this machine; the committed TargetFramework is unchanged at net5.0.
说明:本机没有 .NET 5 的 arm64 运行时,单元测试在本地以 net8.0 执行;提交的 TargetFramework 仍为 net5.0

Docs / 文档

  • docs/SessionPool_Exception_Handling.md — new sections on IsOpen() semantics and the pool wait timeout,
    VacantSlots added to the health metrics table, self-healing behaviour described, and the stale builder
    API names in the examples corrected (.Host().SetHost(), .Timeout().SetConnectionTimeoutInMs(),
    etc., which did not compile as written).
    docs/SessionPool_Exception_Handling.md —— 新增 IsOpen() 语义与池等待超时章节,健康指标表补充
    VacantSlots,说明自愈行为,并修正示例中过时的 Builder API 名(原样无法编译)。
  • docs/API.md — clarified the IsOpen row.
    docs/API.md —— 澄清 IsOpen 一行的说明。

修复服务端断开后 SessionPool 永久阻塞的问题

Two defects made a pool that survived a server outage unusable, with callers
blocking forever instead of receiving an error.
两个缺陷叠加导致服务端断开后连接池不可用,调用方永久阻塞且不抛异常。

- Fix the pool wait timeout being interpreted as seconds while callers assigned
  a millisecond value. Open() set `_clients.Timeout = _timeout * 5`, and Take()
  read it back through `TimeSpan.FromSeconds`, so the 500ms default connection
  timeout produced a ~41 minute block (13.9 hours for the legacy 10000ms
  constructor argument) before SessionPoolDepletedException surfaced.
  ConcurrentClientQueue now exposes TimeoutInMs; the seconds-based Timeout
  property is kept as an obsolete compatibility shim.
  修复池等待超时的单位错误:Open() 以毫秒赋值,Take() 却按秒解释,导致 500ms 的
  默认连接超时变成约 41 分钟的阻塞(旧构造函数传 10000ms 时为 13.9 小时)。
  ConcurrentClientQueue 改用 TimeoutInMs,原秒级 Timeout 属性保留为过时兼容包装。

- Fix pool slots being lost permanently when reconnection failed. The dead
  client was discarded without being returned and nothing replenished the pool,
  so capacity shrank by one per failure until the pool was empty; from then on
  every caller blocked on a queue no one would ever feed, and the pool could not
  recover even after the server came back. Vacant slots are now tracked and
  rebuilt lazily on the next acquisition, keeping capacity constant and letting
  the pool heal itself.
  修复重连失败导致的槽位永久丢失:失效连接被丢弃且不归还,池也无补充机制,每失败
  一次容量就减一,耗尽后所有调用都阻塞在无人投喂的队列上,且服务恢复后无法自愈。
  现在改为记录空槽并在下次获取时按需重建,池容量保持恒定并可自行恢复。

- Add SetPoolWaitTimeoutInMs to SessionPool.Builder and TableSessionPool.Builder
  (default 10s), independent of the socket-level SetConnectionTimeoutInMs.
  为 SessionPool.Builder 和 TableSessionPool.Builder 新增 SetPoolWaitTimeoutInMs
  (默认 10 秒),与 socket 级的 SetConnectionTimeoutInMs 相互独立。

- Expose SessionPool.VacantSlots and document that IsOpen() is a lifecycle flag,
  not a connectivity probe - it stays true after the server goes down because
  the client keeps no heartbeat and reconnects lazily.
  暴露 SessionPool.VacantSlots,并明确 IsOpen() 是生命周期标志而非连通性探针:
  客户端无心跳、按需重连,服务端断开后该标志仍为 true。

- Allow Reconnect() to run without an original client so the self-heal path works
  in multi-node deployments.
  允许 Reconnect() 在无原始连接时运行,使多节点部署下的自愈路径可用。

- Add regression tests for the timeout unit and builder wiring; update
  docs/SessionPool_Exception_Handling.md and docs/API.md accordingly.
  补充超时单位与 Builder 接线的回归测试;同步更新相关文档。
Comment thread src/Apache.IoTDB/ConcurrentClientQueue.cs Outdated
Comment thread src/Apache.IoTDB/SessionPool.cs
Comment thread src/Apache.IoTDB/SessionPool.cs Outdated
…pacity

处理评审意见:整体超时预算、Close() 生命周期、按需容量语义

[P1] Preserve one overall deadline across wake-ups in Take().
[P1] Take() 在多次唤醒之间保持单一的整体超时预算。

Return() uses PulseAll, so every waiter wakes while only one can dequeue. The
losing waiters looped and re-armed the full TimeoutInMs, so under steady churn a
waiter could exceed its configured bound indefinitely - the original "blocks
forever" failure mode was still reachable under contention. The budget is now
captured once and the remaining duration recomputed on each wake-up.
Return() 使用 PulseAll,所有等待者都会被唤醒但只有一个能出队。落败者会重新进入循环
并再次以完整的 TimeoutInMs 等待,因此在持续周转下等待者可能无限超出配置上限——
原本"永久阻塞"的失效模式在竞争条件下依然可达。现在预算只捕获一次,每次唤醒重新
计算剩余时长。

[P2] Transition lifecycle state independently of queued clients in Close().
[P2] Close() 的生命周期状态转换不再依赖队列中的连接。

_isClose was assigned only inside the foreach over queued clients. Once every
connection had become a vacant slot the queue was empty, the loop ran zero times,
and Close() returned while IsOpen() stayed true, leaving the rebuild path armed.
_isClose is now set before the loop, vacant slots are cleared, and
AcquireClientAsync no longer materializes capacity on a closed pool.
_isClose 原先只在遍历队列连接的 foreach 内部赋值。当所有连接都变成空槽后队列为空,
循环零次执行,Close() 返回时 IsOpen() 仍为 true,重建路径依然处于激活状态。现在
_isClose 在循环前设置,空槽被清零,且 AcquireClientAsync 不再在已关闭的池上创建容量。

[P2] Define slot refill as demand-driven capacity and correct the metric docs.
[P2] 将槽位补充明确定义为按需容量,并修正指标文档。

A slot is only materialized when an acquisition finds the idle queue empty, so
under sequential or light load one connection serves every request and VacantSlots
legitimately stays above zero after the server has recovered. The previous
documentation claimed a steady non-zero value meant the server was still
unreachable, which was wrong. VacantSlots is now documented as unrealized capacity,
and the alerting guidance points at FailedReconnections instead.
只有当获取连接时发现空闲队列为空才会实例化槽位,因此在串行或轻负载下一个连接即可
服务所有请求,服务端恢复后 VacantSlots 仍会合理地保持非零。原文档称持续非零表示
服务端仍不可达,这是错误的。现在 VacantSlots 被定义为未实例化的容量,告警指引改为
基于 FailedReconnections。

- Add multi-waiter and repeated-wake-up regression tests for the deadline, and
  Close()-with-empty-queue regression tests for the lifecycle flag.
  新增多等待者与重复唤醒的超时预算回归测试,以及空队列下 Close() 的生命周期回归测试。
将 VacantSlots 重命名为 UnrealizedCapacity

"Vacant slots" reads as "something is broken", which is exactly the wrong
intuition now that refill is documented as demand-driven: a steady non-zero value
under light load is normal, not a fault. UnrealizedCapacity states what the number
actually measures - configured capacity that has not been materialized yet.
"Vacant slots" 读起来像"出了故障",而在补充策略被明确定义为按需之后这恰恰是错误的
直觉:轻负载下持续非零是正常现象而非故障。UnrealizedCapacity 准确表达了这个数字实际
衡量的内容——尚未实例化的已配置容量。

No compatibility impact: the property is introduced by this PR and has never
shipped, so no released API is affected.
无兼容性影响:该属性由本 PR 引入,从未发布,不影响任何已发布的 API。

- SessionPool.VacantSlots -> SessionPool.UnrealizedCapacity
- _vacantSlots -> _unrealizedCapacity, TryReserveVacantSlot -> TryReserveUnrealizedCapacity
- Update the metrics table, prose and tests to match.
  同步更新指标表、说明文字与测试。

@HTHou HTHou left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 3ad458f. The timeout deadline, empty-queue close handling, and demand-driven capacity semantics have all been addressed. The remaining concurrent close/return behavior is consistent with the Java SessionPool and is not blocking this PR. LGTM.

@HTHou
HTHou merged commit 178ebb0 into apache:main Jul 31, 2026
6 checks passed
@CritasWang
CritasWang deleted the fix/pool-wait-timeout-and-slot-leak branch July 31, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SessionPool blocks indefinitely after a server outage and never recovers 服务端断开后 SessionPool 永久阻塞且无法自愈

2 participants