Live-Migration of QEMU/KVM VMs with libvirt: Command Cheat Sheet and Tips

Since recently, I am working with libvirt to perform live-migration VMs with QEMU/KVM VMs (QEMU as VMM and KVM as hypervisor). A few things were not very clear nor well documented. Therefore, I’d like to provide you with this basic cheat sheet and a few tips and tricks. Likely I’m going to extend the list when I get more insights.

Everything I’m telling here focuses on one of multiple use-cases of libvirt. libvirt knows different migration strategies and different VMMs/Hypervisors. I’m focusing on QEMU with the native QEMU migration via SSH. All my experiments were done with Linux 6.11.5, libvirt 10.5, and QEMU 9.1.1.

Command Cheat Sheet

Trigger a live migration (via native QEMU transport through an SSH channel).

This spawns a blocking process that waits until the migration is done. The flags can be adapted accordingly to fit your use-case.

sudo virsh migrate --domain $DOMAIN --desturi qemu+ssh://$USER@$HOST/system --migrateuri tcp://$HOST --live --auto-converge --verbose

Get Information and Statistics about an Ongoing Migration

This must happen on the sender VMM side:

sudo virsh domjobinfo --domain $DOMAIN.

The output looks something like this:

Job type:         Unbounded   
Operation:        Outgoing migration
Time elapsed:     280866       ms
Data processed:   26.086 GiB
Data remaining:   637.297 MiB
Data total:       16.009 GiB
Memory processed: 26.086 GiB
Memory remaining: 637.297 MiB
Memory total:     16.009 GiB
Memory bandwidth: 106.762 MiB/s
Dirty rate:       29459        pages/s
Page size:        4096         bytes
Iteration:        28          
Postcopy requests: 0           
Constant pages:   3782873     
Normal pages:     6815548     
Normal data:      25.999 GiB
Expected downtime: 9502         ms
Setup time:       64           ms
Compression cache: 64.000 MiB
Compressed data:  30.065 MiB
Compressed pages: 403027       
Compression cache misses: 6392597      
Compression overflows: 6739         
Auto converge throttle: 99    

If you frequently poll this data, you can get nice statistics. Especially, but not limited to, the vCPU throttling of the auto-converge feature and the impact on the memory dirty rate.

Get Information about a Completed Migration

This has to be invoked when the command from above gives no more output:

sudo virsh domjobinfo --domain $DOMAIN --completed [--keep-completed]

The output somehow looks like this:

Job type:         Completed
Operation:        Outgoing migration
... as above

Abort an Ongoing Migration

sudo virsh domjobabort --domain $DOMAIN

Tips & Tricks

• The VM that you want to migrate needs a CPU model configured that is compatible with both hosts. Configure something more compatible, such as “IvyBridge” in virt-manager rather than “host-passthrough” (except you have identical CPUs).
• The VM image must live in network storage. The corrsponding network storage pool (libvirt terminology) must be configured in all libvirt hosts manually, beforehand.
• Always specify --migrateuri in conjunction to --desturi! Otherwise, weird behavior can happen regarding the data transport/the migration! Set it for example like this: -desturi $argc_migration_scheme://$DST_HOST/system and --migrateuri tcp://$DST_HOST. Without migrateuri, I experienced situations where additional (reverse) DNS magic was done by the sender VM host, which caused the migration to use another network connection than the one it was supposed to use (specifically not the direct link that was attached to the machine next to the public internet)!