Fix SessionPool blocking indefinitely after a server outage 修复服务端断开后 SessionPool 永久阻塞的问题 - #62
Merged
HTHou merged 3 commits intoJul 31, 2026
Conversation
修复服务端断开后 SessionPool 永久阻塞的问题 Two defects made a pool that survived a server outage unusable, with callers blocking forever instead of receiving an error. 两个缺陷叠加导致服务端断开后连接池不可用,调用方永久阻塞且不抛异常。 - Fix the pool wait timeout being interpreted as seconds while callers assigned a millisecond value. Open() set `_clients.Timeout = _timeout * 5`, and Take() read it back through `TimeSpan.FromSeconds`, so the 500ms default connection timeout produced a ~41 minute block (13.9 hours for the legacy 10000ms constructor argument) before SessionPoolDepletedException surfaced. ConcurrentClientQueue now exposes TimeoutInMs; the seconds-based Timeout property is kept as an obsolete compatibility shim. 修复池等待超时的单位错误:Open() 以毫秒赋值,Take() 却按秒解释,导致 500ms 的 默认连接超时变成约 41 分钟的阻塞(旧构造函数传 10000ms 时为 13.9 小时)。 ConcurrentClientQueue 改用 TimeoutInMs,原秒级 Timeout 属性保留为过时兼容包装。 - Fix pool slots being lost permanently when reconnection failed. The dead client was discarded without being returned and nothing replenished the pool, so capacity shrank by one per failure until the pool was empty; from then on every caller blocked on a queue no one would ever feed, and the pool could not recover even after the server came back. Vacant slots are now tracked and rebuilt lazily on the next acquisition, keeping capacity constant and letting the pool heal itself. 修复重连失败导致的槽位永久丢失:失效连接被丢弃且不归还,池也无补充机制,每失败 一次容量就减一,耗尽后所有调用都阻塞在无人投喂的队列上,且服务恢复后无法自愈。 现在改为记录空槽并在下次获取时按需重建,池容量保持恒定并可自行恢复。 - Add SetPoolWaitTimeoutInMs to SessionPool.Builder and TableSessionPool.Builder (default 10s), independent of the socket-level SetConnectionTimeoutInMs. 为 SessionPool.Builder 和 TableSessionPool.Builder 新增 SetPoolWaitTimeoutInMs (默认 10 秒),与 socket 级的 SetConnectionTimeoutInMs 相互独立。 - Expose SessionPool.VacantSlots and document that IsOpen() is a lifecycle flag, not a connectivity probe - it stays true after the server goes down because the client keeps no heartbeat and reconnects lazily. 暴露 SessionPool.VacantSlots,并明确 IsOpen() 是生命周期标志而非连通性探针: 客户端无心跳、按需重连,服务端断开后该标志仍为 true。 - Allow Reconnect() to run without an original client so the self-heal path works in multi-node deployments. 允许 Reconnect() 在无原始连接时运行,使多节点部署下的自愈路径可用。 - Add regression tests for the timeout unit and builder wiring; update docs/SessionPool_Exception_Handling.md and docs/API.md accordingly. 补充超时单位与 Builder 接线的回归测试;同步更新相关文档。
HTHou
reviewed
Jul 31, 2026
HTHou
reviewed
Jul 31, 2026
HTHou
reviewed
Jul 31, 2026
…pacity 处理评审意见:整体超时预算、Close() 生命周期、按需容量语义 [P1] Preserve one overall deadline across wake-ups in Take(). [P1] Take() 在多次唤醒之间保持单一的整体超时预算。 Return() uses PulseAll, so every waiter wakes while only one can dequeue. The losing waiters looped and re-armed the full TimeoutInMs, so under steady churn a waiter could exceed its configured bound indefinitely - the original "blocks forever" failure mode was still reachable under contention. The budget is now captured once and the remaining duration recomputed on each wake-up. Return() 使用 PulseAll,所有等待者都会被唤醒但只有一个能出队。落败者会重新进入循环 并再次以完整的 TimeoutInMs 等待,因此在持续周转下等待者可能无限超出配置上限—— 原本"永久阻塞"的失效模式在竞争条件下依然可达。现在预算只捕获一次,每次唤醒重新 计算剩余时长。 [P2] Transition lifecycle state independently of queued clients in Close(). [P2] Close() 的生命周期状态转换不再依赖队列中的连接。 _isClose was assigned only inside the foreach over queued clients. Once every connection had become a vacant slot the queue was empty, the loop ran zero times, and Close() returned while IsOpen() stayed true, leaving the rebuild path armed. _isClose is now set before the loop, vacant slots are cleared, and AcquireClientAsync no longer materializes capacity on a closed pool. _isClose 原先只在遍历队列连接的 foreach 内部赋值。当所有连接都变成空槽后队列为空, 循环零次执行,Close() 返回时 IsOpen() 仍为 true,重建路径依然处于激活状态。现在 _isClose 在循环前设置,空槽被清零,且 AcquireClientAsync 不再在已关闭的池上创建容量。 [P2] Define slot refill as demand-driven capacity and correct the metric docs. [P2] 将槽位补充明确定义为按需容量,并修正指标文档。 A slot is only materialized when an acquisition finds the idle queue empty, so under sequential or light load one connection serves every request and VacantSlots legitimately stays above zero after the server has recovered. The previous documentation claimed a steady non-zero value meant the server was still unreachable, which was wrong. VacantSlots is now documented as unrealized capacity, and the alerting guidance points at FailedReconnections instead. 只有当获取连接时发现空闲队列为空才会实例化槽位,因此在串行或轻负载下一个连接即可 服务所有请求,服务端恢复后 VacantSlots 仍会合理地保持非零。原文档称持续非零表示 服务端仍不可达,这是错误的。现在 VacantSlots 被定义为未实例化的容量,告警指引改为 基于 FailedReconnections。 - Add multi-waiter and repeated-wake-up regression tests for the deadline, and Close()-with-empty-queue regression tests for the lifecycle flag. 新增多等待者与重复唤醒的超时预算回归测试,以及空队列下 Close() 的生命周期回归测试。
将 VacantSlots 重命名为 UnrealizedCapacity "Vacant slots" reads as "something is broken", which is exactly the wrong intuition now that refill is documented as demand-driven: a steady non-zero value under light load is normal, not a fault. UnrealizedCapacity states what the number actually measures - configured capacity that has not been materialized yet. "Vacant slots" 读起来像"出了故障",而在补充策略被明确定义为按需之后这恰恰是错误的 直觉:轻负载下持续非零是正常现象而非故障。UnrealizedCapacity 准确表达了这个数字实际 衡量的内容——尚未实例化的已配置容量。 No compatibility impact: the property is introduced by this PR and has never shipped, so no released API is affected. 无兼容性影响:该属性由本 PR 引入,从未发布,不影响任何已发布的 API。 - SessionPool.VacantSlots -> SessionPool.UnrealizedCapacity - _vacantSlots -> _unrealizedCapacity, TryReserveVacantSlot -> TryReserveUnrealizedCapacity - Update the metrics table, prose and tests to match. 同步更新指标表、说明文字与测试。
HTHou
approved these changes
Jul 31, 2026
HTHou
left a comment
Contributor
There was a problem hiding this comment.
Re-reviewed at 3ad458f. The timeout deadline, empty-queue close handling, and demand-driven capacity semantics have all been addressed. The remaining concurrent close/return behavior is consistent with the Java SessionPool and is not blocking this PR. LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #61
What this fixes / 修复内容
A
SessionPoolthat outlived a server outage became permanently unusable: callers blocked forever with noexception, and the pool could not recover even after the server came back. Two independent defects combined
to produce that.
服务端断开后,
SessionPool会变得永久不可用:调用方永久阻塞且不抛异常,服务端恢复后也无法自愈。这是两个独立缺陷叠加的结果。
1. Pool wait timeout was written in ms and read in seconds / 池等待超时的单位错误
Open()assigned_clients.Timeout = _timeout * 5(a millisecond value) whileConcurrentClientQueue.Take()consumed it viaTimeSpan.FromSeconds. The 500 ms default connection timeouttherefore produced a ~41 minute block before
SessionPoolDepletedExceptionsurfaced; the legacytimeout: 10000constructor argument produced 13.9 hours.Open()以毫秒赋值而Take()按秒消费,500ms 的默认连接超时因此变成约 41 分钟 的阻塞,旧构造函数传
timeout: 10000时更是 13.9 小时。ConcurrentClientQueuenow exposesTimeoutInMsand waits withTimeSpan.FromMilliseconds.ConcurrentClientQueue改用TimeoutInMs,并以TimeSpan.FromMilliseconds等待。Timeoutproperty is kept as an[Obsolete]shim so existing callers still compileand behave as they intended.
原秒级
Timeout属性保留为[Obsolete]兼容包装,既有调用方仍可编译且语义符合预期。SetPoolWaitTimeoutInMson both builders, defaulting to 10 s.池等待时间不再由 socket 超时推导,改为两个 Builder 上的
SetPoolWaitTimeoutInMs,默认 10 秒。2. Pool slots leaked on every failed reconnection / 重连失败导致槽位泄漏
When reconnection failed, the dead client was discarded without being returned and nothing replenished the
pool, so capacity shrank by one per failure. After
PoolSizefailures every caller blocked on an empty queuethat nobody would ever feed, and the pool stayed dead after the server recovered.
重连失败时失效连接被丢弃且不归还,池无补充机制,每失败一次容量减一;
PoolSize次失败后所有调用都阻塞在无人投喂的空队列上,服务端恢复后连接池依然是死的。
_vacantSlots, surfaced asSessionPool.VacantSlots) instead of beingsilently dropped, so capacity stays at
PoolSize.改为记录空槽(
_vacantSlots,通过SessionPool.VacantSlots暴露),容量保持为PoolSize。AcquireClientAsyncre-materializes a vacant slot when the queue is empty, so the pool heals itself on thenext operation once the server is reachable — no
Close()+Open()cycle needed.队列为空时
AcquireClientAsync会就地重建空槽,服务端恢复后下一次操作即可自愈,无需Close()+Open()。Reconnect()now accepts a null original client so the self-heal path also works withnodeUrls(multi-node) configurations.
Reconnect()允许原始连接为 null,使nodeUrls(多节点)配置下的自愈路径同样可用。3.
IsOpen()semantics documented / 明确IsOpen()语义IsOpen()is a lifecycle flag, not a connectivity probe — the client keeps no heartbeat, so it staystrueafter the server goes down. Behaviour is unchanged (and matches the Java client); this PR only documents it,
because the guard
if (pool.IsOpen()) return;is a common way to accidentally prevent a pool from ever beingrebuilt.
IsOpen()是生命周期标志而非连通性探针——客户端无心跳,服务端断开后它仍为true。行为未变(与 Java客户端一致),本 PR 仅补充文档说明,因为
if (pool.IsOpen()) return;这种守卫很容易让连接池永远不被重建。Compatibility / 兼容性
No breaking changes. New constructor overloads take the pool wait timeout as an extra parameter and the
existing overloads delegate to them with the default;
ConcurrentClientQueue.Timeoutstill compiles and nowconverts to/from
TimeoutInMs.无破坏性变更。新增的构造函数重载多接收一个池等待超时参数,原有重载以默认值委托到新重载;
ConcurrentClientQueue.Timeout仍可编译,并与TimeoutInMs相互换算。Behavioural change worth calling out: a depleted pool now fails after 10 s by default instead of tens of
minutes. Anyone who (unintentionally) relied on the long wait should set
SetPoolWaitTimeoutInMsexplicitly.需要注意的行为变化:池耗尽时现在默认 10 秒后失败,而不是数十分钟。若有人(无意中)依赖了原来的长等待,
请显式设置
SetPoolWaitTimeoutInMs。Tests / 测试
ConcurrentClientQueueTests— timeout is honoured in milliseconds (the direct regression guard), themessage reports the unit,
Return()wakes a blockedTake(), a ready client is returned without waiting,and the obsolete
Timeoutshim converts correctly.ConcurrentClientQueueTests—— 超时按毫秒生效(针对该 bug 的直接回归测试)、异常消息标注单位、Return()能唤醒阻塞的Take()、有可用连接时立即返回、过时的Timeout兼容属性换算正确。SessionPoolConfigurationTests— the pool wait timeout is independent of the connection timeout and ispropagated by both
SessionPool.Builder(host/port and nodeUrls paths) andTableSessionPool.Builder.SessionPoolConfigurationTests—— 池等待超时与连接超时相互独立,且能被SessionPool.Builder(host/port 与 nodeUrls 两条路径)和
TableSessionPool.Builder正确传递。dotnet test tests/Apache.IoTDB.Tests— 33 passed, 0 failed.dotnet build Apache.IoTDB.sln— 0 errors.dotnet format --verify-no-changes— clean.Note: the unit tests were executed locally against
net8.0because no .NET 5 arm64 runtime is available onthis machine; the committed
TargetFrameworkis unchanged atnet5.0.说明:本机没有 .NET 5 的 arm64 运行时,单元测试在本地以
net8.0执行;提交的TargetFramework仍为net5.0。Docs / 文档
docs/SessionPool_Exception_Handling.md— new sections onIsOpen()semantics and the pool wait timeout,VacantSlotsadded to the health metrics table, self-healing behaviour described, and the stale builderAPI names in the examples corrected (
.Host()→.SetHost(),.Timeout()→.SetConnectionTimeoutInMs(),etc., which did not compile as written).
docs/SessionPool_Exception_Handling.md—— 新增IsOpen()语义与池等待超时章节,健康指标表补充VacantSlots,说明自愈行为,并修正示例中过时的 Builder API 名(原样无法编译)。docs/API.md— clarified theIsOpenrow.docs/API.md—— 澄清IsOpen一行的说明。