CKS: Fix node stuck in Starting due to missing qemu-guest-agent by goal86sg · Pull Request #13715 · apache/cloudstack

goal86sg · 2026-07-27T03:21:52Z

Backport: Add qemu-guest-agent to REQUIRED_PACKAGES in validate-cks-node script so template registration fails fast with a clear error when the package is missing (rather than letting the VM hang at Starting).

Add a 120-second hard timeout with progress messages to the guest-ping loop in patch.sh. On timeout, print a clear error pointing to the missing qemu-guest-agent package. Also bound guest-sync sanity check to 30s with a clear error.

Fixes #13471

Description

This PR...

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Backport: Add qemu-guest-agent to REQUIRED_PACKAGES in validate-cks-node script so template registration fails fast with a clear error when the package is missing (rather than letting the VM hang at Starting). Add a 120-second hard timeout with progress messages to the guest-ping loop in patch.sh. On timeout, print a clear error pointing to the missing qemu-guest-agent package. Also bound guest-sync sanity check to 30s with a clear error. Fixes apache#13471

codecov · 2026-07-27T08:23:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 19.64%. Comparing base (4f11707) to head (68cf637).

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13715      +/-   ##
============================================
- Coverage     19.65%   19.64%   -0.01%     
+ Complexity    19792    19790       -2     
============================================
  Files          6368     6368              
  Lines        574881   574881              
  Branches      70351    70351              
============================================
- Hits         112970   112962       -8     
- Misses       449639   449648       +9     
+ Partials      12272    12271       -1

Flag	Coverage Δ
uitests	`3.41% <ø> (ø)`
unittests	`20.92% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR backports safeguards for CloudStack Kubernetes Service (CKS) on KVM so nodes don’t hang indefinitely in Starting when the VM template lacks a functioning qemu-guest-agent. It aims to fail fast during template validation and to add bounded waiting with clearer diagnostics during the KVM guest-agent handshake.

Changes:

Add qemu-guest-agent to the CKS node template REQUIRED_PACKAGES validation list so missing dependencies are detected early.
Add a 120s hard timeout (with periodic progress output) to the guest-ping wait loop in the KVM patch.sh flow, and a 30s bound for the guest-sync sanity check.
Improve operator-facing error messages to point to the missing/disabled guest agent and relevant documentation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
scripts/vm/hypervisor/kvm/patch.sh	Adds bounded guest-agent wait loops with progress and clearer timeout errors to prevent indefinite hangs.
plugins/integrations/kubernetes-service/src/main/resources/script/validate-cks-node	Adds `qemu-guest-agent` to required packages for node-template validation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

DaanHoogland · 2026-07-27T15:34:07Z

+    if [ $((GUEST_AGENT_WAIT_TICK % 100)) -eq 0 ]; then
+        echo "Waiting for qemu-guest-agent to respond... (${GUEST_AGENT_WAIT_TICK}/1200 ticks, ~$((GUEST_AGENT_WAIT_TICK / 10))s elapsed)"
+    fi
+    if [ $GUEST_AGENT_WAIT_TICK -ge $GUEST_AGENT_MAX_TICKS ]; then
+        echo "ERROR: qemu-guest-agent not responding after 120 seconds."


this one makes sense to me @goal86sg . Would you apply?

DaanHoogland · 2026-07-27T15:34:37Z

    sleep 0.1
+    GUEST_SYNC_TICK=$((GUEST_SYNC_TICK + 1))
+    if [ $GUEST_SYNC_TICK -ge $GUEST_SYNC_MAX_TICKS ]; then
+        echo "ERROR: qemu-guest-agent sanity check (guest-sync) failed after 30 seconds."


This one as well?

 REQUIRED_PACKAGES=(cloud-init cloud-guest-utils conntrack apt-transport-https ca-certificates curl gnupg gnupg-agent \
-                    software-properties-common gnupg lsb-release python3-json-pointer python3-jsonschema cloud-init containerd.io)
+                    software-properties-common lsb-release python3-json-pointer python3-jsonschema containerd.io \
+                    qemu-guest-agent)
 declare -a MISSING_PACKAGES
 if [[ $OS == *"Ubuntu"* || $OS == *"Debian"* ]]; then


DaanHoogland

code looks good, some changes can be applied

DaanHoogland · 2026-07-27T15:32:20Z

+# Fixed: Added qemu-guest-agent to REQUIRED_PACKAGES (GitHub Issue #13471)
+# Also removes duplicate cloud-init and gnupg entries
+


I don’t think we need this comment here

Suggested change

# Fixed: Added qemu-guest-agent to REQUIRED_PACKAGES (GitHub Issue #13471)

# Also removes duplicate cloud-init and gnupg entries

DaanHoogland · 2026-07-27T15:33:18Z


-# Wait for the guest agent to come online
+# Wait for the guest agent to come online (max 120s to avoid indefinite hang)
+# FIX: Added timeout + clear error message (GitHub Issue #13471)


Suggested change

# FIX: Added timeout + clear error message (GitHub Issue #13471)

I think we don’t need fix comments when code is fixed.

DaanHoogland · 2026-07-27T15:34:07Z

+    if [ $((GUEST_AGENT_WAIT_TICK % 100)) -eq 0 ]; then
+        echo "Waiting for qemu-guest-agent to respond... (${GUEST_AGENT_WAIT_TICK}/1200 ticks, ~$((GUEST_AGENT_WAIT_TICK / 10))s elapsed)"
+    fi
+    if [ $GUEST_AGENT_WAIT_TICK -ge $GUEST_AGENT_MAX_TICKS ]; then
+        echo "ERROR: qemu-guest-agent not responding after 120 seconds."


this one makes sense to me @goal86sg . Would you apply?

DaanHoogland · 2026-07-27T15:34:37Z

    sleep 0.1
+    GUEST_SYNC_TICK=$((GUEST_SYNC_TICK + 1))
+    if [ $GUEST_SYNC_TICK -ge $GUEST_SYNC_MAX_TICKS ]; then
+        echo "ERROR: qemu-guest-agent sanity check (guest-sync) failed after 30 seconds."


This one as well?

Pearl1594 · 2026-07-27T15:59:11Z

-while [ "$(virsh qemu-agent-command $name '{"execute":"guest-sync","arguments":{"id":1234567890}}' 2>/dev/null)" != '{"return":1234567890}' ]
-do
+# Test guest agent sanity (bounded to 30s)
+# FIX: Added timeout (GitHub Issue #13471)


worth removing this comment as well.

Suggested change

# FIX: Added timeout (GitHub Issue #13471)

boring-cyborg Bot added component:kubernetes component:kvm labels Jul 27, 2026

Merge branch 'main' into fix/cks-qemu-guest-agent-v2

68cf637

DaanHoogland requested review from Pearl1594, Copilot and kiranchavala July 27, 2026 08:13

Copilot started reviewing on behalf of DaanHoogland July 27, 2026 08:22 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

DaanHoogland approved these changes Jul 27, 2026

View reviewed changes

Pearl1594 reviewed Jul 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CKS: Fix node stuck in Starting due to missing qemu-guest-agent#13715

CKS: Fix node stuck in Starting due to missing qemu-guest-agent#13715
goal86sg wants to merge 2 commits into
apache:mainfrom
goal86sg:fix/cks-qemu-guest-agent-v2

goal86sg commented Jul 27, 2026

Uh oh!

codecov Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

DaanHoogland Jul 27, 2026

Uh oh!

DaanHoogland Jul 27, 2026

Uh oh!

DaanHoogland left a comment

Uh oh!

DaanHoogland Jul 27, 2026

Uh oh!

DaanHoogland Jul 27, 2026

Uh oh!

DaanHoogland Jul 27, 2026

Uh oh!

DaanHoogland Jul 27, 2026

Uh oh!

DaanHoogland Jul 27, 2026

Uh oh!

Pearl1594 Jul 27, 2026

Uh oh!

DaanHoogland Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		# Fixed: Added qemu-guest-agent to REQUIRED_PACKAGES (GitHub Issue #13471)
		# Also removes duplicate cloud-init and gnupg entries

Uh oh!

Conversation

goal86sg commented Jul 27, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Jul 27, 2026 •

edited

Loading