Add in-place update for gateway replicas#4014
Conversation
Support scaling gateways via in-place update of the `replicas` configuration property. For now, this feature is only intended for internal testing and is hidden behind the `GATEWAY_SCALING` flag, since it has limited applicability at the current stage — without a state sync mechanism, newly started gateway replicas are not aware of existing services.
| async with get_session_ctx() as session: | ||
| for gateway_compute_model in result.gateway_compute_models: | ||
| session.add(gateway_compute_model) | ||
| await _apply_replica_scaling(session, result.scale_result) |
There was a problem hiding this comment.
I think there is an issue with _apply_replica_scaling() being called before lock_token guard. It can lead to a stale worker committing new replicas without updating the gateway. E.g. a stale submitted gateway can create replicas. Then, it will be re-processed with duplicate replicas committed again. All related db changes / side effects should be done after the lock_token guard.
There was a problem hiding this comment.
If a gateway replica never provisions (e.g. there is some cloud issue, backend availability, etc), the reconcile loop never terminates, re-creating replicas forever. We need some backoff logic and a limit. For fleets, we have _MAX_CONSOLIDATION_ATTEMPTS and _CONSOLIDATION_RETRY_DELAYS.
| for num in itertools.count(): | ||
| if num not in used_nums: | ||
| yield num | ||
| assert False, "unreachable" |
There was a problem hiding this comment.
itertools.count() is infinite. why assert False, "unreachable"?
2 participants
Support scaling gateways via in-place update of
the
replicasconfiguration property. For now,this feature is only intended for internal testing
and is hidden behind the
GATEWAY_SCALINGflag,since it has limited applicability at the current
stage — without a state sync mechanism, newly
started gateway replicas are not aware of existing
services.
#3959