I get error decoding CKS cluster installation after upgrade from 4.19 into 4.21

[ahmadamirahmadi007]

2025-11-08 14:59:24,445 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-93:[ctx-714cf072, job-60993/job-60995, ctx-b9d9e8f7]) (logid:eb39427a) Trying to deploy VM [error decoding VM instance {"id":3648,"instanceName":"i-2-3648-VM","state":"Starting","type":"User","uuid":"2ac6d25d-2f3d-40d8-bc0c-acd599154496"}] and details: Plan [error decoding {"_clusterId":null,"_dcId":1,"_physicalNetworkId":null,"_podId":null,"_poolId":null,"migrationPlan":false}]; avoid list [{}] and planner: [null].

creating instances separately is ok, but when I create a kubernetes cluster, it doesn't start worker or control node instances inside it.
Is there a problem somewhere in the database?

[vishesh92]

@ahmadamirahmadi007 Can you share more logs? It's most likely some configuration issue. You can ignore the "error decoding" part. I think it's a logging issue and not related to your problem.

[kiranchavala]

@ahmadamirahmadi007

Are you launching cks in an existing network or a new network

Could you please try restarting the network with cleanup option if its a existing network and check again by deploying a new cks cluser

[ahmadamirahmadi007]

@kiranchavala @vishesh92
I have used the new network and the problem is still there. But what I have noticed is that this problem is happening to the admin user who is in the root domain. When I create a new user who is a member of domain-admin, the Kuber cluster installs and starts up without any problems. I am getting the following errors while installing the Kuber cluster with the admin user.

2025-11-15 12:39:42,438 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-66:[ctx-c1a8d6b5, job-65305/job-65307, ctx-3fc067ea]) (logid:509bf6b1) Trying to deploy VM [error decoding VM instance {"id":3705,"instanceName":"i-2-3705-VM","state":"Starting","type":"User","uuid":"57ef2be5-ea4c-431f-a1fc-d0f3c47ff91d"}] and details: Plan [error decoding {"_clusterId":null,"_dcId":1,"_physicalNetworkId":null,"_podId":null,"_poolId":null,"migrationPlan":false}]; avoid list [{}] and planner: [null].

2025-11-15 12:39:42,636 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-66:[ctx-c1a8d6b5, job-65305/job-65307, ctx-3fc067ea]) (logid:509bf6b1) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3705,"instanceName":"i-2-3705-VM","state":"Starting","type":"User","uuid":"57ef2be5-ea4c-431f-a1fc-d0f3c47ff91d"}Scope=interface com.cloud.dc.DataCenter; id=1

2025-11-15 12:39:42,636 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-66:[ctx-c1a8d6b5, job-65305/job-65307]) (logid:509bf6b1) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkStart","cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAOeXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA","cmdVersion":0,"completeMsid":null,"created":"Sat Nov 15 12:39:40 UTC 2025","id":65307,"initMsid":345051312967,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"fa395beb-ae36-4459-bb8b-f8febb758bb4"}, job origin: 65305 com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3705,"instanceName":"i-2-3705-VM","state":"Starting","type":"User","uuid":"57ef2be5-ea4c-431f-a1fc-d0f3c47ff91d"}Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1451)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at jdk.internal.reflect.GeneratedMethodAccessor1145.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2025-11-15 12:39:42,895 ERROR [c.c.k.c.a.KubernetesClusterStartWorker] (API-Job-Executor-80:[ctx-617b3ff2, job-65305, ctx-9845117c]) (logid:509bf6b1) Provisioning the control VM failed in the Kubernetes cluster : admin-kuber com.cloud.exception.ManagementServerException: Failed to start VM in the Kubernetes cluster : admin-kuber
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.startKubernetesVM(KubernetesClusterResourceModifierActionWorker.java:343)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.provisionKubernetesClusterControlVm(KubernetesClusterStartWorker.java:520)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startKubernetesClusterOnCreate(KubernetesClusterStartWorker.java:790)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1790)

[DaanHoogland]

I have used the new network and the problem is still there. But what I have noticed is that this problem is happening to the admin user who is in the root domain. When I create a new user who is a member of domain-admin, the Kuber cluster installs and starts up without any problems. I am getting the following errors while installing the Kuber cluster with the admin user.

Can you check the limits for either user @ahmadamirahmadi007 ?

[prashanthr2]

@ahmadamirahmadi007 Can you share full management server log that covers the time of the issue

[ahmadamirahmadi007]

@prashanthr2
https://gist.github.com/ahmadamirahmadi007/c715bb07b9af62314e2dc595bc6a6279

[prashanthr2]

@ahmadamirahmadi007 We see below in the logs, Can you confirm if you have any host dedication in place?

2025-11-18 07:44:02,347 DEBUG [o.a.c.a.ExplicitDedicationProcessor] (Work-Job-Executor-138:[ctx-eac1c46f, job-65538/job-65540, ctx-441af89c]) (logid:34b8889f) Processing affinity group AffinityGroup {"id":1,"name":"DedicatedGrp-domain-ROOT","uuid":"52f4b352-69fc-40d8-adb5-d378123d5f93"} of type 'ExplicitDedication' for VM: VM instance {"id":3713,"instanceName":"i-2-3713-VM","state":"Starting","type":"User","uuid":"bfaef7a2-f32a-4044-bb93-c18b71f7e8b2"}
2025-11-18 07:44:02,349 DEBUG [o.a.c.a.ExplicitDedicationProcessor] (Work-Job-Executor-138:[ctx-eac1c46f, job-65538/job-65540, ctx-441af89c]) (logid:34b8889f) No dedicated resources available for this domain or account under this group
2025-11-18 07:44:02,349 DEBUG [o.a.c.a.ExplicitDedicationProcessor] (Work-Job-Executor-138:[ctx-eac1c46f, job-65538/job-65540, ctx-441af89c]) (logid:34b8889f) ExplicitDedicationProcessor returns Avoid List as: Deploy avoids pods: [], clusters: [], hosts: null
2025-11-18 07:44:02,352 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:[ctx-eac1c46f, job-65538/job-65540, ctx-441af89c]) (logid:34b8889f) DeploymentPlan [DataCenterDeployment] has not specified host. Trying to find another destination to deploy VM [VM instance {"id":3713,"instanceName":"i-2-3713-VM","state":"Starting","type":"User","uuid":"bfaef7a2-f32a-4044-bb93-c18b71f7e8b2"}], avoiding pods [], clusters [] and hosts [null].
2025-11-18 07:44:02,353 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:[ctx-eac1c46f, job-65538/job-65540, ctx-441af89c]) (logid:34b8889f) Deploy avoids pods: [], clusters: [], hosts: null.
2025-11-18 07:44:02,353 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:[ctx-eac1c46f, job-65538/job-65540, ctx-441af89c]) (logid:34b8889f) Deploy hosts with priorities {}, hosts have NORMAL priority by default
2025-11-18 07:44:02,353 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:[ctx-eac1c46f, job-65538/job-65540, ctx-441af89c]) (logid:34b8889f) DataCenter = 'Zone {"id": "1", "name": "DC-02", "uuid": "b7b84374-f17b-4dc2-9d16-ef9858e7e232"}' provided is in avoid set, DeploymentPlanner cannot allocate the VM, returning.
2025-11-18 07:44:02,428 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:[ctx-eac1c46f, job-65538/job-65540, ctx-441af89c]) (logid:34b8889f) VM instance {"id":3713,"instanceName":"i-2-3713-VM","state":"Stopped","type":"User","uuid":"bfaef7a2-f32a-4044-bb93-c18b71f7e8b2"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: null, new host: null, host before state transition: null

cc: @Pearl1594 @nvazquez

[ahmadamirahmadi007]

You are right, I noticed that such an affinity group exists, but I did not create it. Even the explicitdedication type is not among its types when created.Removing it fixed the problem. Why did this happen?
@prashanthr2 @DaanHoogland

[prashanthr2]

@ahmadamirahmadi007 Looks like you have a host dedicated to an account or domain which seems to have cause it.

We tried to reproduce this issue ( CKS deployment) with dedication in place ourselves but couldn't. Let me know if you have exact steps to reproduce, we can try it and file an issue if needed. If you cannot reproduce the issue and is resolved now , then you can mark this as resolved.

cc: @Pearl1594 @nvazquez

[ahmadamirahmadi007]

There was a configuration in the affinity group section that I had not created and I don't know why it existed, but eventually removing it solved the cluster creation problem. Thank you very much for your follow-up.