xpds14 - intel(r) virtualization technology for directed i/o (vt-d) posted interrupts - feng wu,...
DESCRIPTION
With the development of virtualization, there are more device assignment requirements. Based on VT-d interrupt remapping, Intel introduces VT-d interrupt posting as a more enhanced method to handle interrupts in the virtualization environment. The Posted Interrupts (PI) on CPU side has been already supported in Intel CPUs, with VT-d Posted Interrupt we can get some additional advantages, it can directly deliver external interrupts to running vCPUs without hypervisor involvement, decease the interrupt migration complexity, differentiate between urgent and non-urgent external interrupt, and avoid consuming host-vector for each interrupt to vCPU. In this presentation, Feng will talk about the mechanism of VT-d PI and its advantages, as well as some performance data of I/O intensive workload in Xen, which will show the performance gain after using VT-d PI.TRANSCRIPT
VT-d Posted Interrupts
Feng Wu, Jun Nakajima <Speaker>
Intel Corporation
2
Agenda
• Motivation • Difference btw CPU-based and VT-d Posted Interrupts
• Architecture
• Implementation Details • Performance
• Summary
3
• Interrupt virtualization efficiency
• Interrupt migration complexity
• Big requirement of host vector for different assigned devices
Motivation
4
CPU-based Posted-Interrupt in Xen
VCPU0 running on
Physical CPU 0
External interrupt
VCPU1 running on
physical CPU 1
Set bit for guest Vector in
Posted-interrupt requests (PIR) of VCPU1
• External interrupt handling
VM Exit
Guest IPI: Notification Event
- No need to kick VCPU1 - Notification Event is handled by
VCPU1 in non-root mode - vIRR is updated by H/W and
delivered by “virtual interrupt delivery” mechanism
- VM-Exit is eliminated
Xen
5
Key Data Structures for CPU-based Posted-Interrupt Processing
• Posted-interrupt notification vector: − Send virtual interrupts to guests w/o VM exit − If physical vector == Posted-interrupt
notification vector (VMCS field)
• PIR (Posted-interrupt requests) − Set bits for guest vectors in advance
• ON (Outstanding Notification) − If this bit is set, there is a notification outstanding
for one or more posted interrupts
Virtual-APIC Page
Posted-interrupt notification vector
PI Desc. Address
PIR
Posted-interrupt Descriptor
ON
5
VMCS
6
CPU-based Posted-Interrupt in Xen – cont’d
VCPU1 running on
physical CPU 1
Set bit for guest Vector in
Posted-interrupt requests (PIR) of VCPU1
• Virtual interrupts from QEMU
Guest IPI: Notification Event
Virtual Devices
QEMU Virtual interrupts - Virtual MSI - Virtual IOAPIC
Xen
- No need to kick VCPU1 - Notification Event is handled by
VCPU1 in non-root mode - vIRR is updated by H/W and
delivered by “virtual interrupt delivery” mechanism
- VM-Exit is eliminated
7
What’s new for VT-d Posted-Interrupts
VCPU1 running on
physical CPU 1
Direct-assigned Device
VT-d Engine with
Posted-Interrupt support
External Interrupts
Notification Event
- No need to kick VCPU1 - Notification Event is handled by
VCPU1 in non-root mode - vIRR is updated by H/W and
delivered by “virtual interrupt delivery” mechanism
- VM-Exit is eliminated
No VMM overhead AT ALL!
8
VT-d Posted-Interrupts Architecture Virtual Processor A
VMCS vAPIC Page
Posted Descriptor
CPU 0 CPU 1
Virtual Processor B
VMCS vAPIC Page
Virtual Processor C
VMCS Posted Descriptor
Interrupt Remapping Table
IRTE for Interrupt Z
IRTE for Interrupt Y
IRTE for Interrupt X
IRTE for Interrupt N
External Interrupts
Notification Event
Notification Event
Posted Descriptor
vAPIC Page
Notification Event
CPU 0 Host-Vectors
Other Vectors (e.g. Hyp. IPIs)
Notification Vector for vCPU B
Notification Vector for vCPU A
Other Vectors (e.g. Timer)
CPU 1 Host-Vectors
Other Vectors (e.g. Hyp. IPIs)
Notification Vector for vCPU C
Other Vectors (e.g. Timer)
9
Xen Implementation Details:
• Update IRET according to guest’s modification to the interrupt configuration (MSI address, data)
• Interrupt migration during VCPU scheduling
10
Xen Implementation: IRTE update
QEMU
Xen
VCPU
Interrupt configuration update (e.g. MSI/MSIx)
Hypercall XEN_DOMCTL_bind_pt_irq!
Update the guest information in IRTE - Guest vector - Posted-interrupt descriptor address
No changes needed for this part
Changes happen here
11
Xen Implementation: VCPU Scheduling
RUNSTATE_runnable RUNSTATE_blocked
RUNSTATE_running
‘SN’ – Set ‘NV’ –Notification vector
‘SN’ – Clear ‘NDST’ – New physical CPU of the VCPU
‘SN’ – Set ‘NV’ –Wakeup vector
‘SN’ – Set ‘NV’ –Notification vector
‘SN’ – Set ‘NV’ –Wakeup vector
PIR [0 - 255]
Posted-Interrupt Descriptor
NV SN
NV – Host vector for Notification Event SN – If set, suppress Notification Event
12
• VT-d Posted-interrupts advantages – External interrupts from direct-assigned devices are delivered to guest
running in non-root mode directly – Improve Interrupt virtualization efficiency, e.g. Less VM-Exits. – Simplify interrupt migration – Consume less physical interrupts
• Performance
• The Specification will be published very soon – Can be found in Intel website
Summary
14
Back up
15
• Interrupt-remap-table-entry (IRTE) enhanced as follows: - An existing reserved bit claimed to indicated Posted-interrupt (PST) - Software may choose to “remap” or “post” each interrupt independently
• IRTEs with ‘PST’ set are interpreted per below format
- New Fields • Descriptor Address: the address of the posted-interrupt descriptor • Virtual Vector: the guest vector of the interrupt • URG: indicates if the interrupt is urgent
- Other fields continue to have the same meaning
VT-d Posted-Interrupts Support
16
Posted-interrupt Descriptor
17
VT-d: Steps for Interrupt Posting Read IRTE
Interrupt Remapping
‘PST’ == 1
Read Posted-interrupt descriptor (locking the cache-line)
N
Set guest virtual in PIR
(‘ON’ == 0) &&
(‘URG’ == 1 || ‘SN’ == 0)
Y
Set ‘ON’ and Generate notification event
Unlock cache-line N
Unlock cache-line
Y
Y