feat: add application-level keepalive to workflow runtime channel - #1795
Merged
Conversation
…timeouts AWS ALBs do not forward HTTP/2 PING frames, causing idle gRPC connections to be closed. This adds a background loop that periodically calls the existing Hello RPC as application-level traffic to keep the connection alive through L7 load balancers. Signed-off-by: joshvanl <me@joshvanl.dev>
… opt-in property Relocate the hello() keepalive out of DurableTaskGrpcWorker (reverted to master) into sdk-workflows, per review feedback: the keepalive is a channel concern, so it now lives with the classes that create workflow channels. - New internal GrpcChannelKeepalive: pings hello() on a daemon thread, catches Throwable so a ping failure can never cancel the periodic task, applies the 5s deadline per ping. - WorkflowRuntimeBuilder creates it on the worker channel; WorkflowRuntime owns and closes it (new optional constructor arg). - DaprWorkflowClient wires it on the client channel, protecting long waitForInstanceCompletion calls through L7 load balancers too. - Gated behind dapr.workflows.app.keep.alive.enabled (default false) with dapr.workflows.app.keep.alive.interval.seconds (default 30), following the existing opt-in convention of dapr.grpc.enable.keep.alive. - Tests: in-process gRPC server coverage for ping cadence, failure tolerance, and close semantics; gating tests for both channel owners. Signed-off-by: Javier Aliaga <javier@diagrid.io>
Restore fix-by-default for the worker channel: the keepalive now defaults to on, but only pings between WorkflowRuntime.start() and close() - a started runtime is by definition talking to a sidecar, so idle apps and built-but-unstarted runtimes send nothing. - GrpcChannelKeepalive lifecycle is explicit: inert constructor, idempotent start() and close(). - dapr.workflows.app.keep.alive.enabled (opt-in) is replaced by dapr.workflows.runtime.app.keep.alive.enabled (default true, opt-out). - Drop the client-channel keepalive: a dead idle client connection reconnects on the next call, so it isn't worth the extra surface; DaprWorkflowClient reverts to master. Signed-off-by: Javier Aliaga <javier@diagrid.io>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1795 +/- ##
============================================
+ Coverage 76.85% 77.10% +0.25%
- Complexity 2300 2317 +17
============================================
Files 244 245 +1
Lines 7145 7186 +41
Branches 748 750 +2
============================================
+ Hits 5491 5541 +50
+ Misses 1288 1284 -4
+ Partials 366 361 -5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
siri-varma
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an application-level keepalive to the workflow runtime's gRPC channel: while the runtime is started, the SDK pings the sidecar's
helloRPC every 30s (5s deadline) on a daemon thread. Some L7 load balancers (e.g. AWS ALB) do not treat HTTP/2 PING frames as connection activity, so an idlegetWorkItemsstream gets killed at the idle timeout and workflow processing silently stops; a real RPC keeps the connection alive.WorkflowRuntime.start()/close()— nothing pings before the runtime starts or after it closes.dapr.workflows.runtime.app.keep.alive.enabled; interval tunable viadapr.workflows.app.keep.alive.interval.seconds.DaprWorkflowClientis untouched.Checklist