ateapi: return ResourceExhausted when no worker is free - #908
Open
ILLUM1N0X wants to merge 1 commit into
Open
Conversation
ResumeActor returned FailedPrecondition both when the fleet had no free worker and when the actor was not in a resumable state. Only the former is fixable by suspending the actor to drop its node pinning, and a client driving that recovery could not tell the two apart. `assignWorkerAttempt()` now returns `ResourceExhausted` for `scheduling.ErrNoCapacity`. `FailedPrecondition` was the de-facto capacity signal, so three router consumers move with it: - `retryable()`: `ResourceExhausted` is parked, so saturation behaves as before. `FailedPrecondition` stays retryable on purpose. It no longer carries saturation, but it does cover states a concurrent operation can move the actor out of. - `mapResumeError()`: `ResourceExhausted` now preserves the gRPC description, keeping saturation at 503 as today. Its former 429 "rate limited" branch was written speculatively; nothing in substrate produced the code until now, and the fleet being full is not the caller sending too many requests. - `classifyOutcome()`: `no_capacity` moves to `ResourceExhausted`, and `FailedPrecondition` gets its own label so the capacity bucket stops collecting non-capacity errors. `TestResumeActor_RelocatesAfterSuspendFromPaused` covers the recovery flow this signal exists for: a PAUSED actor pinned to a full node cannot resume, and after a suspend clears the pinning it schedules onto a worker on another node. Part of agent-substrate#660.
ILLUM1N0X
requested review from
Haven Xia (HavenXia) and
Julian Gutierrez Oschmann (juli4n)
August 13, 2026 11:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ResumeActor returned
FailedPreconditionboth when the fleet had no free worker and when the actor was not in a resumable state. Only the former is fixable by suspending the actor to drop its node pinning, and a client driving that recovery could not tell the two apart.assignWorkerAttempt()now returnsResourceExhaustedforscheduling.ErrNoCapacity.FailedPreconditionwas the de-facto capacity signal, so three router consumers move with it:retryable():ResourceExhaustedis parked, so saturation behaves as before.FailedPreconditionstays retryable on purpose. It no longer carries saturation, but it does cover states a concurrent operation can move the actor out of.mapResumeError():ResourceExhaustednow preserves the gRPC description, keeping saturation at 503 as today. Its former 429 "rate limited" branch was written speculatively as nothing in substrate produced the code until now, and the fleet being full is not the caller sending too many requests.classifyOutcome():no_capacitymoves toResourceExhausted, andFailedPreconditiongets its own label so the capacity bucket stops collecting non-capacity errors.TestResumeActor_RelocatesAfterSuspendFromPausedcovers the recovery flow this signal exists for: a PAUSED actor pinned to a full node cannot resume, and after a suspend clears the pinning it schedules onto a worker on another node.Fixes #660
make testclean (except the pre-existing macOS-onlyinternal/atunnelfailures, untouched by this PR)make fmtcleango vet ./cmd/...clean