Live-Migration of QEMU/KVM VMs with libvirt: Command Cheat Sheet and Tips

Recently, I was working with libvirt to perform live-migration VMs with QEMU/KVM VMs (QEMU as VMM and KVM as hypervisor). A few things were not very clear nor well documented. Therefore, I’d like to provide you with this basic cheat sheet and a few tips and tricks. Likely I’m going to extend the list when I get more insights.

Everything I’m telling here focuses on one of multiple use-cases of libvirt. libvirt knows different migration strategies and different VMMs/Hypervisors. I’m focusing on QEMU with native QEMU via SSH migration. All my experiments were done with Linux 6.11.5, libvirt 10.5, and QEMU 9.11.

Command Cheat Sheet

Trigger a live migration (via native QEMU transport through an SSH channel).

This spawns a blocking process that waits until the migration is done. The flags can be adapted accordingly to fit your use-case.

sudo virsh migrate --domain $DOMAIN --desturi qemu+ssh://$USER@$HOST/system --migrateuri tcp://$HOST --live --auto-converge --verbose

Get Information and Statistics about an Ongoing Migration

This must happen on the sender VMM side:

sudo virsh domjobinfo --domain $DOMAIN.

The output looks something like this:

Job type:         Unbounded   
Operation:        Outgoing migration
Time elapsed:     280866       ms
Data processed:   26.086 GiB
Data remaining:   637.297 MiB
Data total:       16.009 GiB
Memory processed: 26.086 GiB
Memory remaining: 637.297 MiB
Memory total:     16.009 GiB
Memory bandwidth: 106.762 MiB/s
Dirty rate:       29459        pages/s
Page size:        4096         bytes
Iteration:        28          
Postcopy requests: 0           
Constant pages:   3782873     
Normal pages:     6815548     
Normal data:      25.999 GiB
Expected downtime: 9502         ms
Setup time:       64           ms
Compression cache: 64.000 MiB
Compressed data:  30.065 MiB
Compressed pages: 403027       
Compression cache misses: 6392597      
Compression overflows: 6739         
Auto converge throttle: 99    

If you frequently poll this data, you can get nice statistics. Especially, but not limited to, the vCPU throttling of the auto-converge feature and the impact on the memory dirty rate.

Get Information about a Completed Migration

This has to be invoked when the command from above gives no more output:

sudo virsh domjobinfo --domain $DOMAIN --completed [--keep-completed]

The output somehow looks like this:

Job type:         Completed
Operation:        Outgoing migration
... as above

Abort an Ongoing Migration

sudo virsh domjobabort --domain $DOMAIN

Tips & Tricks

• The VM that you want to migrate needs a CPU model configured that is compatible with both hosts. Configure something more compatible, such as “IvyBridge” in virt-manager rather than “host-passthrough” (except you have identical CPUs).
• The VM image must live in network storage. The corrsponding network storage pool (libvirt terminology) must be configured in all libvirt hosts manually, beforehand.
• Always specify --migrateuri in conjunction to --desturi! Otherwise, weird behavior happens! For example: -desturi $argc_migration_scheme://$DST_HOST/system and --migrateuri tcp://$DST_HOST. I experienced situations where additional (reverse) DNS magic was done by the main VM host, which caused it to turn another network connection than the one it was supposed to use!