![Page 1: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/1.jpg)
Virtualizing NUMA
Andre Przywara, AMD OSRC, Dresden
Virtualization developer (KVM and Xen)
Work areas:
NUMA
CPUID
Cross vendor migration
![Page 2: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/2.jpg)
NUMA architecture
Driven by integrated memory controllers
Performance optimization
ACPI based
Smaller guests scale well
Guests may exceed one node's resources
They should know!
Scheduling should be restricted
(or be very clever)
![Page 3: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/3.jpg)
State of integration
QEMU: can emulate in guest
KVM host binding patches pending
Xen: patches posted, but need more work
Proper topology emulation requiredNo. of Cores must match NUMA topology
Both HVM and PV targetted
![Page 4: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/4.jpg)
Numbers
Four-way AMD Opteron 6164
Contains 8 nodes, 6 cores each
Each node has 8 or 16GB of RAM
Kernbench:
different no. of VCPUs and RAM
Numactl'ed or not
Lmbench
Starting many instances in parallel
Helping scheduler or not
![Page 5: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/5.jpg)
Lmbench (rd) KVM unpinned
1 guest 4 guests 8 guests 16 guests
0
20
40
60
80
100
HighestAverageLowest
![Page 6: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/6.jpg)
Lmbench (rd) KVM numactl
1 guest 4 guests 8 guests 16 guests
0
20
40
60
80
100
HighestAverageLowest
![Page 7: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/7.jpg)
Lmbench (lat) KVM unpinned
1 guest 4 guests 8 guests 16 guests
0
20
40
60
80
100
HighestAverageLowest
![Page 8: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/8.jpg)
Lmbench (lat) KVM numactl
1 guest 4 guests 8 guests 16 guests
0
20
40
60
80
100
HighestAverageLowest
![Page 9: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/9.jpg)
Lmbench (rd) Xen numa=off
native 1 guest 7 guests 15 guests 23 guests
0
20
40
60
80
100
HighestAverageLowest
![Page 10: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/10.jpg)
Lmbench (rd) Xen numa=on
native 1 guest 7 guests 15 guests 23 guests
0
20
40
60
80
100
HighestAverageLowest
![Page 11: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/11.jpg)
Kernbench Xen, 1 vCPU
1 guest 7 guests 15 guests 23 guests
0
20
40
60
80
100
numa=onnuma=off
![Page 12: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/12.jpg)
Kernbench Xen, 2 vCPUs
1 guest 7 guests 15 guests 23 guests
0
20
40
60
80
100
numa=onnuma=off
![Page 13: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/13.jpg)
Kernbench KVM, 7 guest
numactl not pinned
0
20
40
60
80
100
highestaveragelowest
![Page 14: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/14.jpg)
Discussion items
Realization of KVM host NUMA binding
libnuma in QEMU
Externally by numactl or hugetlbfs
Marry topology and NUMA?
QEMU cmdline syntax for NUMA
Currently flexible, but hard to comprehend
Does it matter? (libvirt)
Unfortunate limits with comma
![Page 15: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/15.jpg)
Scheduler items
Avoid pinning (denies load balancing)
But avoid node migration
Schedule guests apart
Like Xen, but without pinning
Rebalancing with page migration?
Hot pages first, maybe temporary?
![Page 16: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/16.jpg)
Backup
![Page 17: Virtualizing NUMA - Indicowiki.linuxplumbersconf.org/_media/2010:05-lpc2010... · 2010. 11. 15. · Andre Przywara, AMD OSRC, Dresden Virtualization developer (KVM and Xen) Work areas:](https://reader036.vdocument.in/reader036/viewer/2022081621/611d795bd2792f5309539bb7/html5/thumbnails/17.jpg)
Kernbench Xen
1 guest 7 guests 15 guests 23 guests
0
20
40
60
80
100
numa=onnuma=off, 1vCPUnuma=off, 2 vCPUsnuma=on, 2 vCPUs