VM on Xen - weird ISO issue

[tdtmusic2]

Hi everyone. Got a strange error today and it doesn't make sense as it's not related to the actual action that I want to do, or at least I don't think so, maybe someone else has an idea. The setup is simple, a xen host added in ACS 4.19.2, with a GPU in passtrough mode, a Windows VM deployed with a W11 iso attached and Windows PV as OS version. Setup goes all the way, although I noticed it's MUCH slower than deploying the same way but with Windows 11 as OS version instead of PV (and manually adding the virtio drivers for the disk controller in the setup). After the setup I wanted to detach the ISO to attach another one and I got this error. The only way to detach it was to stop the VM and do it. I tested with other iso's and it's the same behavior. The logs bellow don't seem to reveal much about the issue either. Any thoughts?

2025-03-17 16:13:19,472 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:8eec4d00) Add job-337 into job monitoring
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Executing AsyncJobVO: {id:337, usene, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47a8-a302-d84a51df8d84",uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d43-a8e1-47a8-a302-d84"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: null, lastUpdated: null}
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) submit async job-337, details: AsanceType: VirtualMachine, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47Id":"2","sessionkey":"uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d":"2","cmdEventType":"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: nd: null, removed: null}
2025-03-17 16:13:19,474 DEBUG [c.c.a.ApiServlet] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) ===END===  10.84.84.1 -- GET  virtualmachineid=0chIso&response=json&sessionkey=uZWE79gkLJARV7XWYGQoTHFX7Cg
2025-03-17 16:13:19,487 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) template 204 is alr
2025-03-17 16:13:19,492 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) Host 11 does not have requioup of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
2025-03-17 16:13:19,495 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Unexpected exception while executing ohIsoCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Host 11 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL
        at com.cloud.resource.ResourceManagerImpl.getGPUDevice(ResourceManagerImpl.java:3469)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy200.getGPUDevice(Unknown Source)
        at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:285)
        at com.cloud.hypervisor.XenServerGuru.implement(XenServerGuru.java:93)
        at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1296)
        at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1314)
        at com.cloud.template.TemplateManagerImpl.detachIso(TemplateManagerImpl.java:1175)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy236.detachIso(Unknown Source)
        at org.apache.cloudstack.api.command.user.iso.DetachIsoCmd.execute(DetachIsoCmd.java:109)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-17 16:13:19,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Complete async job-337, jobStatus:udstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Host 11 does not have required GPU device or out of capacity. GPU grT4] GPUs, vGPU Type: passthrough"}
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Publish async job-337 complete on
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs related to job-337
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Update db status for job-337
2025-03-17 16:13:19,499 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs joined with job-337 a
2025-03-17 16:13:19,500 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Done executing org.apache.cloudsta job-337
2025-03-17 16:13:19,500 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Remove job-337 from job monitoring

[DaanHoogland]

@tdtmusic2 , i cannot give you a direct answer. Can you tell me the exact version which that stacktrace came from? I'll have a look.

[tdtmusic2]

@tdtmusic2 , i cannot give you a direct answer. Can you tell me the exact version which that stacktrace came from? I'll have a look.

Hi Daan. You mean CS version? It's 4.19.2.0.

[DaanHoogland]

@tdtmusic2 , can you check the contents of your host_gpu_groups table? The code is expecting to find content in it and doesn't.

In the logs above I see ...does not have requioup of... , the word requioup does not exist and does not occur anywhere in the code.

2025-03-17 16:13:19,492 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) Host 11 does not have requioup of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough

the line should read something like:

"Host " + hostId + " does not have required GPU device or out of capacity. GPU group: " + groupName + ", vGPU Type: " + vgpuType;

You might also want to try restarting the management servers.

[tdtmusic2]

This is what's in the sql table, I guess it's as it should be. Just tried to deploy another instance now, with gpu passthrough, and while powered on I tried to detach the iso, got the same error. I restarted the management service before this.

What I don't undestand is what's the connection between available gpus (in my case there is only one, so I guess it IS out of capacity after being attached to the new vm) and ejecting a iso image.

[DaanHoogland]

So there is no host 11 in that table and this is what the system is looking for. I am not sure (yet) if this is a configuration error or whether the system should handle this.

[tdtmusic2]

So there is no host 11 in that table and this is what the system is looking for. I am not sure (yet) if this is a configuration error or whether the system should handle this.

Host 11 was removed since then, now it's 12 and it's the same error message about it:

2025-03-27 12:22:14,670 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-3:ctx-77c18f69 job-415 ctx-8f3c75db) (logid:50f8449f) Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
2025-03-27 12:22:14,672 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-3:ctx-77c18f69 job-415) (logid:50f8449f) Unexpected exception while executing org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
        at com.cloud.resource.ResourceManagerImpl.getGPUDevice(ResourceManagerImpl.java:3469)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy200.getGPUDevice(Unknown Source)
        at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:285)
        at com.cloud.hypervisor.XenServerGuru.implement(XenServerGuru.java:93)
        at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1296)
        at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1314)
        at com.cloud.template.TemplateManagerImpl.detachIso(TemplateManagerImpl.java:1175)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy236.detachIso(Unknown Source)
        at org.apache.cloudstack.api.command.user.iso.DetachIsoCmd.execute(DetachIsoCmd.java:109)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-27 12:22:14,674 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-77c18f69 job-415) (logid:50f8449f) Complete async job-415, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough"}

[harikrishna-patnala]

@tdtmusic2 what I can see from the error in the logs is, CS is calling the below sql query, the output of it seems to be empty. Please check if that can give you any hint

SELECT hgg.*
FROM host_gpu_groups hgg
INNER JOIN vgpu_types vt ON hgg.id = vt.gpu_group_id
WHERE hgg.host_id = 12
AND hgg.group_name = 'Group of Nvidia Corporation TU104GL [Tesla T4] GPUs'
AND vt.vgpu_type = 'passthrough'
AND vt.remaining_capacity > 0
ORDER BY vt.remaining_capacity DESC, hgg.id;

[DaanHoogland]

@tdtmusic2 , are you still looking at this?

[DaanHoogland]

assuming this is no longer an issue @tdtmusic2 . please comment if you think it is.