Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ var tablet =
| -------------- | ------------------------- | ------------------------ | ----------------------------- |
| Open | bool | open session | session_pool.Open(false) |
| Close | null | close session | session_pool.Close() |
| IsOpen | null | check if the pool was opened and not yet closed by the caller. It is a lifecycle flag, **not** a connectivity probe: it stays `true` after the server goes down, because the client keeps no heartbeat and reconnects on demand instead. | session_pool.IsOpen() |
| IsOpen | null | check whether the pool was opened and not yet closed by the caller. It is a lifecycle flag, **not** a connectivity probe: it stays `true` after the server goes down, because the client keeps no heartbeat and reconnects on demand instead. For connectivity use `CheckHealthAsync` | session_pool.IsOpen() |
| CheckHealthAsync | CancellationToken=default | probe server connectivity on an idle pooled connection; returns a `SessionPoolHealth` snapshot | await session_pool.CheckHealthAsync() |
| OpenDebugMode | LoggingConfiguration=null | open debug mode | session_pool.OpenDebugMode() |
| CloseDebugMode | null | close debug mode | session_pool.CloseDebugMode() |
| SetTimeZone | string | set time zone | session_pool.GetTimeZone() |
Expand Down
78 changes: 62 additions & 16 deletions docs/SessionPool_Exception_Handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,24 +54,62 @@ catch (SessionPoolDepletedException ex)
}
```

## `IsOpen()` is a lifecycle flag, not a health check
## Checking connectivity: `CheckHealthAsync` vs `IsOpen`

`SessionPool.IsOpen()` reports whether **you** have opened the pool and not yet closed it. It is not a
connectivity probe:
`SessionPool.IsOpen()` reports whether **you** have opened the pool and not yet closed it. It is a
lifecycle flag, not a connectivity probe:

- It becomes `true` after a successful `Open()` and only returns to `false` when you call `Close()`.
- The client runs no heartbeat, so a server that goes down does **not** flip it back to `false`.
Reconnection happens lazily, on the next operation.

This means the following common guard never re-opens the pool, because the flag stays `true` forever:
This makes the following common guard a trap - it short-circuits forever, so the pool is never rebuilt:

```csharp
// Anti-pattern: this short-circuits even while every connection is dead
// Anti-pattern: IsOpen() stays true even while every connection is dead
if (_pool != null && _pool.IsOpen()) return;
```

To reason about actual availability, use the health metrics below, or simply let an operation throw
`SessionPoolDepletedException` and handle it.
Use `CheckHealthAsync` when you need to know whether the server is actually reachable. It issues one
lightweight request on an idle pooled connection and returns a `SessionPoolHealth` snapshot:

```csharp
var health = await sessionPool.CheckHealthAsync();

if (!health.IsHealthy)
{
Console.WriteLine(health); // e.g. "Unhealthy: The server did not answer the probe: ..."
Console.WriteLine(health.Status); // NotOpen | Healthy | Degraded | Unhealthy
Console.WriteLine(health.Error); // the underlying exception, when Status is Unhealthy
}
```

### Status values

| Status | Meaning |
| ----------- | ------------------------------------------------------------------------------------------- |
| `NotOpen` | The pool has not been opened yet, or has already been closed. No network call is attempted. |
| `Healthy` | The server answered the probe. |
| `Degraded` | The pool is open but no connection was idle to probe with. Says nothing about the server. |
| `Unhealthy` | A connection was available but the server did not answer. See `Error` for the cause. |

### Behaviour worth knowing

- **The probe never blocks.** If every client is busy, it returns `Degraded` immediately instead of
queueing behind ordinary work, so a health endpoint cannot be starved by application load. It is also
unaffected by the pool wait timeout described below.
- **The borrowed connection is always returned** to the pool, and a failed probe does not close it. The
regular reconnect-on-use path still applies to the next operation.
- **`Degraded` is not a server verdict.** Under high concurrency it simply means the pool was saturated at
that instant. Treat a persistent `Degraded` as a sizing signal, not an outage.
- **Bound the probe yourself if you need to.** Pass a cancellation token:

```csharp
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2));
var health = await sessionPool.CheckHealthAsync(cts.Token);
```

`TableSessionPool` exposes the same `CheckHealthAsync` method with identical semantics.

## Pool Wait Timeout

Expand All @@ -94,7 +132,9 @@ var sessionPool = new SessionPool.Builder()

When the wait budget is exhausted, the operation throws `SessionPoolDepletedException` with the reason
`Connection pool is empty and wait time out(...ms)`. Raise `SetPoolWaitTimeoutInMs` if your workload
legitimately queues behind long operations; lower it if you would rather fail fast and retry.
legitimately queues behind long operations; lower it if you would rather fail fast and retry. The budget is
a single deadline for the whole call: waiters woken by a returned connection that they lose the race for do
not restart it.

> **Note:** before this setting existed, the wait budget was derived from the connection timeout and then
> misinterpreted as seconds, which turned the 500 ms default into a ~41 minute block. If you are upgrading
Expand Down Expand Up @@ -333,6 +373,8 @@ var sessionPool = new SessionPool.Builder()

### Health Check Implementation

The simplest health check delegates to the built-in probe and only adds your own policy on top:

```csharp
public class SessionPoolHealthCheck
{
Expand All @@ -343,34 +385,35 @@ public class SessionPoolHealthCheck
_pool = pool;
}

public HealthStatus CheckHealth()
public async Task<HealthStatus> CheckHealthAsync()
{
var availableRatio = (double)_pool.AvailableClients / _pool.TotalPoolSize;
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2));
var health = await _pool.CheckHealthAsync(cts.Token);

if (_pool.FailedReconnections > 10)
if (health.Status == SessionPoolHealthStatus.Unhealthy)
{
return new HealthStatus
{
Status = "Critical",
Message = $"High reconnection failures: {_pool.FailedReconnections}",
Message = health.Message,
Recommendation = "Check IoTDB server availability"
};
}

if (availableRatio < 0.25)
if (health.Status == SessionPoolHealthStatus.Degraded)
{
return new HealthStatus
{
Status = "Warning",
Message = $"Low available clients: {_pool.AvailableClients}/{_pool.TotalPoolSize}",
Message = $"No idle connection to probe ({health.AvailableClients}/{health.TotalPoolSize} available)",
Recommendation = "Consider increasing pool size"
};
}

return new HealthStatus
{
Status = "Healthy",
Message = $"Pool healthy: {_pool.AvailableClients}/{_pool.TotalPoolSize} available"
Message = health.Message
};
}
}
Expand All @@ -383,6 +426,9 @@ public class HealthStatus
}
```

If you would rather not issue a network call on every scrape, the metrics below can be sampled on their
own - just remember they describe the pool, not the server.

### Metrics Collection for Monitoring Systems

```csharp
Expand Down Expand Up @@ -559,7 +605,7 @@ public class ProductionSessionPoolManager
The SessionPool exception handling and health monitoring features provide comprehensive tools for building robust IoTDB applications:

- Use `SessionPoolDepletedException` to understand and react to pool issues
- Treat `IsOpen()` as a lifecycle flag, never as a connectivity check
- Use `CheckHealthAsync` for connectivity checks; `IsOpen()` is a lifecycle flag only
- Tune `SetPoolWaitTimeoutInMs` separately from `SetConnectionTimeoutInMs`
- Monitor `AvailableClients`, `TotalPoolSize`, and `FailedReconnections`; read `UnrealizedCapacity` as
capacity not yet materialized rather than as an outage signal
Expand Down
7 changes: 6 additions & 1 deletion src/Apache.IoTDB/ConcurrentClientQueue.cs
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@ public void Return(Client client)
public void AddRef() => Interlocked.Increment(ref _ref);
public int GetRef() => Volatile.Read(ref _ref);
public void RemoveRef() => Interlocked.Decrement(ref _ref);

/// <summary>
/// The maximum time, in milliseconds, that <see cref="Take"/> waits for a client to be
/// returned to the pool before throwing. Defaults to 10000 (10 seconds).
Expand All @@ -79,6 +78,12 @@ public int Timeout
set => TimeoutInMs = value * 1000;
}

/// <summary>
/// Attempts to take a client without ever blocking. Returns false when no client is idle.
/// Use this for probes and diagnostics, which must not queue behind ordinary work.
/// </summary>
public bool TryTake(out Client client) => ClientQueue.TryDequeue(out client);

public Client Take()
{
Client client = null;
Expand Down
56 changes: 53 additions & 3 deletions src/Apache.IoTDB/SessionPool.cs
Original file line number Diff line number Diff line change
Expand Up @@ -471,12 +471,62 @@ public async Task<Client> Reconnect(Client originalClient = null, CancellationTo
/// <remarks>
/// This reflects the lifecycle of the pool object only - it is NOT a server-connectivity probe.
/// The client performs no heartbeat, so a server going down does not flip this back to false;
/// it stays true until <see cref="Close"/> is called. To reason about connectivity, use
/// <see cref="AvailableClients"/>, <see cref="UnrealizedCapacity"/> and <see cref="FailedReconnections"/>,
/// or simply let an operation throw <see cref="SessionPoolDepletedException"/>.
/// it stays true until <see cref="Close"/> is called. Use <see cref="CheckHealthAsync"/> when you
/// need to know whether the server is actually reachable, or read <see cref="AvailableClients"/>,
/// <see cref="UnrealizedCapacity"/> and <see cref="FailedReconnections"/> for pool state.
/// </remarks>
public bool IsOpen() => !_isClose;

/// <summary>
/// Probes server connectivity by issuing one lightweight request on an idle pooled connection,
/// and returns a snapshot of the result. This is the connectivity check that <see cref="IsOpen"/>
/// is often mistaken for.
/// </summary>
/// <remarks>
/// The probe never blocks waiting for a connection: if every client is busy, the call returns
/// <see cref="SessionPoolHealthStatus.Degraded"/> immediately rather than queueing behind ordinary
/// work. The borrowed connection is always returned to the pool, and a failed probe does not close
/// it - the regular reconnect-on-use path still applies to the next operation.
/// Pass a cancellation token if you want to bound how long the probe may take.
/// </remarks>
public async Task<SessionPoolHealth> CheckHealthAsync(CancellationToken cancellationToken = default)
{
if (_isClose || _clients == null)
{
return new SessionPoolHealth(SessionPoolHealthStatus.NotOpen, 0, TotalPoolSize, FailedReconnections,
"The pool has not been opened yet, or it has already been closed.");
}

if (!_clients.TryTake(out var client))
{
return new SessionPoolHealth(SessionPoolHealthStatus.Degraded, 0, TotalPoolSize, FailedReconnections,
"No idle connection was available to probe. The pool is either saturated by concurrent work or has lost its connections.");
}

SessionPoolHealthStatus status;
string message;
Exception error = null;
try
{
await client.ServiceClient.getTimeZoneAsync(client.SessionId, cancellationToken);
status = SessionPoolHealthStatus.Healthy;
message = "The server answered the probe.";
}
catch (Exception e)
{
status = SessionPoolHealthStatus.Unhealthy;
message = $"The server did not answer the probe: {e.Message}";
error = e;
_logger?.LogWarning(e, "Health probe failed for session pool");
}
finally
{
_clients.Add(client);
}

return new SessionPoolHealth(status, AvailableClients, TotalPoolSize, FailedReconnections, message, error);
}

public async Task Close()
{
if (_isClose)
Expand Down
107 changes: 107 additions & 0 deletions src/Apache.IoTDB/SessionPoolHealth.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

using System;

namespace Apache.IoTDB
{
/// <summary>
/// Outcome of a <see cref="SessionPool.CheckHealthAsync"/> probe.
/// </summary>
public enum SessionPoolHealthStatus
{
/// <summary>
/// The pool has not been opened yet, or it has already been closed.
/// </summary>
NotOpen,

/// <summary>
/// The server answered the probe on a pooled connection.
/// </summary>
Healthy,

/// <summary>
/// The pool is open but could not be probed, because no connection was idle at that moment.
/// This says nothing about the server: the pool may simply be saturated by concurrent work.
/// </summary>
Degraded,

/// <summary>
/// A connection was available but the server did not answer the probe.
/// </summary>
Unhealthy
}

/// <summary>
/// A point-in-time snapshot of pool connectivity, returned by <see cref="SessionPool.CheckHealthAsync"/>.
/// Unlike <see cref="SessionPool.IsOpen"/> - which only reports whether the caller has opened the pool -
/// this reflects whether the server actually answered just now.
/// </summary>
public class SessionPoolHealth
{
/// <summary>
/// The probe outcome.
/// </summary>
public SessionPoolHealthStatus Status { get; }

/// <summary>
/// True only when <see cref="Status"/> is <see cref="SessionPoolHealthStatus.Healthy"/>.
/// </summary>
public bool IsHealthy => Status == SessionPoolHealthStatus.Healthy;

/// <summary>
/// Idle clients in the pool at the time the snapshot was taken.
/// </summary>
public int AvailableClients { get; }

/// <summary>
/// Configured maximum capacity of the pool.
/// </summary>
public int TotalPoolSize { get; }

/// <summary>
/// Cumulative tally of reconnection failures since the pool was opened.
/// </summary>
public int FailedReconnections { get; }

/// <summary>
/// Human-readable explanation of <see cref="Status"/>.
/// </summary>
public string Message { get; }

/// <summary>
/// The exception that caused an <see cref="SessionPoolHealthStatus.Unhealthy"/> result, if any.
/// </summary>
public Exception Error { get; }

public SessionPoolHealth(SessionPoolHealthStatus status, int availableClients, int totalPoolSize,
int failedReconnections, string message, Exception error = null)
{
Status = status;
AvailableClients = availableClients;
TotalPoolSize = totalPoolSize;
FailedReconnections = failedReconnections;
Message = message;
Error = error;
}

public override string ToString()
=> $"{Status}: {Message} (available {AvailableClients}/{TotalPoolSize}, failed reconnections {FailedReconnections})";
}
}
9 changes: 9 additions & 0 deletions src/Apache.IoTDB/TableSessionPool.cs
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,15 @@ public void OpenDebugMode(Action<ILoggingBuilder> configure)
sessionPool.OpenDebugMode(configure);
}

/// <summary>
/// Probes server connectivity on an idle pooled connection. See
/// <see cref="SessionPool.CheckHealthAsync"/> for the semantics.
/// </summary>
public async Task<SessionPoolHealth> CheckHealthAsync(CancellationToken cancellationToken = default)
{
return await sessionPool.CheckHealthAsync(cancellationToken);
}

public async Task Close()
{
await sessionPool.Close();
Expand Down
Loading
Loading