Isolating CPU cores from VM host

It is possible to isolate the CPU cores/threads from the Host OS, so it will not use those cores/threads for anything.

This is done by using the isolcpus paramater to the grub CMDLINE

GRUB_CMDLINE_DEFAULT="... isolcpus=1,2,3,4,5,7,8,9,10,11"

The above command will isolate the CPU threads 1-5,7-11 preventing the Host OS from auto assigning tasks to those CPU threads.

Don’t forget to run update-grub after those changes.

Disable scheduler interrupting

Using the nohz_full paramater on the grub CMDLINE will tell the task scheduler on the Host OS to not interrupt the running task if it is only one task running on that CPU thread.

GRUB_CMDLINE_DEFAULT="... nohz_full=1,2,3,4,5,7,8,9,10,11"

This will make sure the qemu virtual cpu thread is not interrupted when it is running alone on a CPU thread.

And again don’t forget to run update-grub after those changes.

CPU Pinning

Now because we have isolated the CPU cores/threads from the Host OS, we have to explicity tell the OS to use them for qemu/kvm. This is done by pinning the virtual cpu threads to a CPU thread.

in the XML file for the virtual machine we need to add a few lines

<domain ...>
  ...
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='8'/>
  </cputune>
  ...
</domain>

Virtual machine XML this will pin the virtual cpu 0 vcpu='0' to the physical cpu thread 1 cpuset='1'

Identifying the two threads of a CPU core

When you CPU pin the virtual cpus to physical cpus you need to map them so each physical core maps to one virtual core. The layout for AMD and Intel cpus are different.

On my AMD Ryzen 3600 my 6 cores and 12 threads have this layout

So to map Core 1 to virtual machine Core 0 we pin CPU number 0 and 7 (core 0 thread 0 and 1)

Verify CPU pinning

You can verify that a running VM is using the intended CPU cores/threads you have pinned with the virsh vcpuinfo VMNAME

# virsh vcpuinfo win10
VCPU:           0
CPU:            1
State:          running
CPU time:       3245.0s
CPU Affinity:   -y----------

VCPU:           1
CPU:            7
State:          running
CPU time:       465.8s
CPU Affinity:   -------y----

VCPU:           2
CPU:            2
State:          running
CPU time:       439.8s
CPU Affinity:   --y---------

VCPU:           3
CPU:            8
State:          running
CPU time:       462.1s
CPU Affinity:   --------y---

The above output is printing the information for each VCPU and what CPU thread it is currently running on. The CPU Affinity: -y---------- line is saying what CPU core/thread the VCPU thread is allowed to run on.