Ceph cluster full, how to live migrate a VM and one of its volumes to a different host and pool?

[Rubueno]

We are running into a full ceph cluster. How can we live migrate a VM to a different host and one of its volumes to a different pool? Currently we cannot migrate just the volume to a different pool because it's not possible to update the volume details in KVM without migrating a VM. Migrating a VM to a different host and also specifying a different pool results in:

2025-09-02 10:17:37,418 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-103:ctx-769ddfe9 job-602372/job-602373 ctx-fc1a004e) (logid:93387da3) Failed to migrate VM [VM instance {"id":20704,"instanceName":"i-960-20704-VM","state":"Migrating","type":"User","uuid":"75ccad2e-59b3-4114-8366-c2e0b5808d70"}] along with its volumes due to [Can't find strategy to move data. Source Host: hv-138-f14-21.ams02.my.org, Destination Host: hv-138-a05-30.ams02.my.org, Volume UUIDs: 5b7c9988-8c8b-44f3-a672-3c5f32947ec3,ed3feacd-995d-4777-b3bb-f73f3e8b8cff].

[weizhouapache]

@Rubueno
in my opinion, similar to some other storage providers, we should migrate the volume (between ceph pools) instead of vm
however, volume migration (on ceph) of running VMs is not implemented yet, AFAIK. maybe it works when vm is stopped, have you tested it ? @Rubueno

@wido
do you migrate volumes between ceph pools ? could you share some experience ?

[Rubueno]

Yes, of course it works when the VM is offline (in that case we use migrateVolume), but this is running production VMs which will impact many thousands of customers, which cannot be taken offline. So the information that it is not implemented yet is good to know.

[wido]

This is not implemented indeed. This functionality is only present in more recent Qemu versions (last 2y) and that would require updating the CloudStack code.

Technically it can be done probably by doing this by hand and then updating the CS database. "Just" use raw Qemu commands.