failed live migration to a different host on cloudstack 4.19.1

[Chih80]

problem

Live migration keeps on failing when trying to move VM to another host. Same result when tried on multiple host.

versions

cloudstack version 4.19.1 on ubuntu 22.04
kvm host on ubuntu 20,04
NFS on secondary storage

The steps to reproduce the bug

1. Selected instance
2. click migration to another host
3. selected host and migrate with storage on same host
4. After a few seconds, error comes back with:
5. Migration failed.
   Failed to migrate VM [VM instance {"id":103,"instanceName":"i-5-103-VM","type":"User","uuid":"5edeefc5-462d-4dda-b082-4358ec49486d"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "102", from: "1", to:"10"}] and VM to host [{vm: "103", from: "1", to:"27"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to stat /var/lib/libvirt/images/870ee0f4-212c-4116-8bf1-99b98985b9a7: No such file or directory].].
   ...

What to do about it?

No response

[DaanHoogland]

@Chih80 , can you investigate the MS - and agent logs for around the time of the error?

Are there any ERROR logs or stack traces preceding the error above?

edit: also please check the versions of libvirt qemu and the cloudstack agents on the hosts, to see if these match.

[Chih80]

restarting any VM that encountered this issue solves it. I was able to live migrate after shutting down and restarting the VM. Not ideal, but an acceptable work around.

[rajujith]

@Chih80 based on the error '/var/lib/libvirt/images/870ee0f4-212c-4116-8bf1-99b98985b9a7', the VM seems to be using local storage?

[weizhouapache]

vm live migration between local storage pools should work.

I suspect the /var/lib/libvirt/images/870ee0f4-212c-4116-8bf1-99b98985b9a7 is the base image (template) which is not copied, or removed

it would be good to upload more logs

[DaanHoogland]

@Chih80 , can you update us?

[Chih80]

Yes it is using local storage. Work around was to power cycle the VM, so I'm satisfied with it.