Fix HA-enabled KVM VMs stuck in Running after out-of-band stop - #13738
Open
poddm wants to merge 1 commit into
Open
Fix HA-enabled KVM VMs stuck in Running after out-of-band stop#13738poddm wants to merge 1 commit into
poddm wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes HA-enabled KVM instances getting stuck in the
Runningstate after anout-of-band stop (e.g. the QEMU process is OOM-killed or crashes on the host).
Actual behaviour (before):
When the power-state sync detected that a running, HA-enabled KVM instance was no longer
present on the host, the management server called
HighAvailabilityManager.scheduleRestart(vm, true)fromhandlePowerOffReportWithNoPendingJobsOnVM(). That path invokesadvanceStop(), whichsubmits a VM work job and blocks waiting for it to complete. Because this handler runs
on the
AgentManager-Handlerthread — a context that does not dispatch VM work jobs — thecall blocks indefinitely. The instance was left in
Running, and the follow-up HAinvestigation (host still up, domain already gone) concluded "no need to restart".
Additionally, a libvirt
VIR_DOMAIN_CRASHEDstate was not mapped to a power state, so acrashed domain was reported as
PowerUnknownand only detected later via the "missing VM"threshold, delaying recovery.
Expected behaviour (after):
An HA-enabled KVM instance that stops out-of-band is transitioned to
Stoppedand an HArestart is scheduled promptly, without blocking the AgentManager thread.
Changes:
Management server —
engine/orchestration/.../VirtualMachineManagerImpl.javaIn the HA out-of-band-stop branch of
handlePowerOffReportWithNoPendingJobsOnVM(),replace the blocking
scheduleRestart(vm, true)with a non-blocking path:Stopped(FollowAgentPowerOffReport),op_ha_workwithStep.Scheduledfor the HAworker to pick up asynchronously.
A guard ensures a valid
host_idis present before enqueuing HA work (avoids violatingthe FK constraint), and existing pending HA work is not duplicated.
KVM agent —
plugins/hypervisors/kvm/.../LibvirtComputingResource.javaMap
DomainState.VIR_DOMAIN_CRASHEDtoPowerState.PowerOffso a crashed domain isreported as off immediately instead of being omitted from the report (which delayed
detection via the missing-VM threshold).
The power-state report already confirms the instance is off, so re-investigation is
unnecessary in this path.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
N/A
How Has This Been Tested?
RunningtoStopped, a row is created inop_ha_workwithStep.Scheduled, and the HA worker restarts the instance.AgentManager-Handlerthread no longer blocks (no stuck power-off handler),and that
management-server.logno longer shows the instance stuck inRunningafter theinvestigation.
VIR_DOMAIN_CRASHEDdomain is now reported asPowerOffand triggers the samerecovery path without waiting for the missing-VM threshold.
How did you try to break this feature and the system with this change?
instance is still recovered rather than being deemed "alive" by the investigator.
host_id/last_host_idis unavailable: HA work is not enqueued(guarded) to avoid violating the
op_ha_workFK constraint.instance.
the branch is guarded by
vm.getState() == State.Runningand the hypervisor-type checks.