Download - Track A-Shmuel Panijel, Windriver
Developing Embedded Developing Embedded
Systems with Multicore Systems with Multicore
and Virtualizationand VirtualizationSamuel PanijelSamuel Panijel
May 4, 2011 1
Samuel Panijel
Regional Engineering Manager
Wind River Q2-2011
Samuel Panijel
Regional Engineering Manager
Wind River Q2-2011
Context• Multicore and virtualization are creating a disruption
in the embedded industry
– Multicore provides more processing power for less power
– Virtualization provides partitioning on single and multicore
silicon
• Wind River provides a broad portfolio of technology
to assist the migration to multicore
May 4, 2011 2| © 2010 Wind River. All Rights Reserved.2
to assist the migration to multicore
– Runtimes, Middleware, Virtualization and Tooling
• This presentation will focus on the use of
virtualization
– For single and multicore
– Based on the Wind River Hypervisor for Embedded
Agenda
• Multicore and virtualization overview
– An introduction to AMP, SMP and Virtualization
• Virtualization capabilities
• Virtualization details
May 4, 2011 3| © 2010 Wind River. All Rights Reserved.3
– Modes of virtualization
– The mechanisms behind virtualization
– Virtualizing memory, cores, devices, interrupts
– Inter-board communication
How to design your next generation of devices
Primary Multicore Software
Configurations
Single CoreCPU
OS
“Traditional” Virtualization
CPU
OS OS
Hypervisor
May 4, 2011 4| © 2010 Wind River. All Rights Reserved.4
Multicore
OS: Could be VxWorks, Wind River Linux, or other executive or OS
Arbitrary combinations of these primary configurations can be used to create more
advanced configurations.
OS
Core 2Core 1
OS
Supervisor
Supervised AMP (sAMP)SMP
OS
Core 1 Core 2
OS
Core 2Core 1
OS
Unsupervised AMP
Symmetric Multiprocessing (SMP)
• A lot of experience
• Works well for parallel algorithms
• OS performs scheduling and load balancing
• A viable solution, depending on your needs
May 4, 2011 5| © 2010 Wind River. All Rights Reserved.5
• A viable solution, depending on your needs
– Does not always scale with many cores
• Linux and VxWorks SMP
SMP
OS
Core 1 Core 2
Asymmetric Multiprocessing (AMP)
• One operating system per core
• Questions raised in regard to configuring and
run-time
– Who “owns” devices?
– How is memory split?
May 4, 2011 6
| © 2010 Wind River. All Rights Reserved.6
– How is memory split?
– How is memory protected?
– How is this configured?
– How is this built into the kernel?
• Mixed operating systems?
OS
Core 2Core 1
OS
Unsupervised AMP
Choosing Supervised AMP
What Is sAMP?
– AMP, but guarded by a supervisor
When to Use sAMP
– Memory and device partitioning (protection, robustness)
May 4, 2011 9| © 2010 Wind River. All Rights Reserved.9
– Flexibility in the mapping of hardware to software
– Memory virtualization (each OS starts at 0x0)
– Reuse of the same image multiple times
– Out-of-the-box life cycle management
OS
Core 2Core 1
OS
Supervisor
Supervised AMP (sAMP)
Agenda
• Multicore and virtualization overview
– An introduction to AMP, SMP and Virtualization
• Virtualization capabilities
• Virtualization details
– Modes of virtualization
May 4, 2011 10
| © 2010 Wind River. All Rights Reserved.10
– Modes of virtualization
– The mechanisms behind virtualization
– Virtualizing memory, cores, devices, interrupts
– Inter-board communication
How to design your next generation of devices
Virtualization
• Partitioning is a benefit of virtualization
• Virtualization partitions the hardware into virtual boards
• The virtualization layer gives the board access to
– Virtualized processing cores (optional)
– Virtualized memory
– Virtualized devices and interrupts
May 4, 2011 11
| © 2010 Wind River. All Rights Reserved.11
– Virtualized devices and interrupts
• Virtualization can be done in a scalable fashion
| © 2009 Wind River. All Rights Reserved.
Core Virtualization Guest Protection
Resource Partitioning Boot Life Cycle Management Debugging
Core Protection InterruptsVirtual Memory and Devices
Hypervisor
Protected Supervisor
Supervisor
An Example Virtualization Scenario
Virtual Board 3Virtual Board 1
VCore Memory Timer
Ethernet
VxWorks
Application
Virtual Board 2
VCore Memory Timer
IDE Serial
Wind River Linux
Application
VCore Memory Timer
Application
Ethernet
MIPC
May 4, 2011 12
| © 2010 Wind River. All Rights Reserved.12
VCore Memory Timer VCore Memory Timer VCore Memory Timer
Wind River Hypervisor
Physical Board
IDE Ethernet Memory CoreTimer Serial Core
Configuration
Summary of the Example
• Virtual boards
– Virtual board scheduling if required
• Virtualized memory
• Virtualized processing cores
• Virtualized devices
May 4, 2011 13| © 2010 Wind River. All Rights Reserved.13
• Virtualized devices
• Inter-board communication
• Configuration and debuggingVirtualization provides new and flexible design capabilities
for single and multicore systems
Virtualization Performance
• Virtualization needs to have minimal cost
• Wind River and customer benchmarking
– sAMP: Overall cost is less than 1%
– Core virtualized: With significant load only a few percent,
similar to running multiple processes
– Some additional latency to interrupt handling
May 4, 2011 14
| © 2010 Wind River. All Rights Reserved.14
– Some additional latency to interrupt handling
• Actual performance depends on
– Processor, board, application
– Multicore configuration
• The additional design capabilities and flexibility
typically outweigh any potential overhead
Agenda
• Multicore and virtualization overview
– An introduction to AMP, SMP and Virtualization
• Virtualization capabilities
• Virtualization details
– Modes of virtualization
May 4, 2011 15| © 2010 Wind River. All Rights Reserved.15
– Modes of virtualization
– The mechanisms behind virtualization
– Virtualizing memory, cores, devices, interrupts
– Inter-board communication
How to design your next generation of devices
Virtualization Principles
The hypervisor adds an additional layer of indirection
to between the OS and the physical hardware
The hypervisor virtualizes the hardware and executes
guest operating systems. The level of virtualization can
be configured based on device needs
May 4, 2011 16| © 2010 Wind River. All Rights Reserved.16
be configured based on device needs
Hypervisor
Guest Guest Guest
Hypervisor Mechanics
• OS privilege modes
• Emulation vs. paravirtualization vs. hardware
assist
• Different types of hypervisors
May 4, 2011 17| © 2010 Wind River. All Rights Reserved.17
• Memory
• Devices, interrupts
• Inter board communication
Privilege Modes
Operating systems work in privilege modes:
� The OS kernel itself can do anything with the
hardware
� Device drivers are more restricted
� User applications/components are least privileged
May 4, 2011 18| © 2010 Wind River. All Rights Reserved.18
Intel privilege rings source: Wikipedia
Ring 3
Ring 2
Ring 1
Ring 0
Kernel
Device Drivers
Device Drivers
Applications
Least Privileged
Most Privileged
Example based on Intel privilege rings, other processors have similar
concepts
Hypervisor Is The Most Powerful
Reserve the highest privilege mode for the hypervisor.
Lower operating system privilege modes through
– Emulation
– ParavirtualizationRing 3
Ring 2 Least Privileged
May 4, 2011 20| © 2010 Wind River. All Rights Reserved.20
– Paravirtualization
– Hardware assist
Hypervisor
Ring 1
Ring 0
Kernel
Device Drivers
Device Drivers
Applications
Most Privileged
Approaches to Virtualization
Type 2 – Hosted
Hardware Emulation
System Virtualization
Hardware Virtualization
Type 1 – Native
May 4, 2011 21
| © 2010 Wind River. All Rights Reserved.21
Paravirtualized Full Virtualization
(Transparent)
Dynamic Binary
Translation
Hardware
Assisted
Privilege Modes – Hardware Assist• Privilege modes are processor specific. You cannot just “add”
another mode. Only the silicon vendor can:
– Freescale e500mc
– Intel VTx
– Virtualization-aware instruction set (aka hardware-assisted
virtualization)
• If not available, paravirtualization will be more for the OS
May 4, 2011 22| © 2010 Wind River. All Rights Reserved.22
• If not available, paravirtualization will be more for the OS
developer.
Often the truth is in the middle: a mix of software and
hardware assist.
Privilege Modes – Paravirtualization
• Modify the guest operating system, to collaborate
with the hypervisor.
• Reduce the privilege level during design time.
• Provide a collaborative system approach.
• Use for embedded.
May 4, 2011 23| © 2010 Wind River. All Rights Reserved.23
• Use for embedded.
• Handle devices:
– In the hypervisor (impact to hypervisor size and stability)
– In a specially crafted guest (guest running in privilege)
– Directly assigned to guest
Mixing Technologies
Mixing of Paravirtualization and Hardware Assist
• Use the best of both worlds for embedded
– Paravirtualization collaboration of OS with
hypervisor (avoid privilege code and call
hypervisor instead)
May 4, 2011 24
hypervisor instead)
– Hardware assist using silicon (great for address
translation)
Wind River hypervisor uses a mix of paravirtualization
and hardware assist, if the processor supports it.
24
| © 2010 Wind River. All Rights Reserved.
Type 1 vs. Type 2 Hypervisor
A type 1 hypervisor runs directly on the hardware,
managing resources, and providing services to guests
A type 2 hypervisor is hosted in another operating
system (the host)
May 4, 2011 25| © 2010 Wind River. All Rights Reserved.25
Guest
Hypervisor
Type 1
Host
Hypervisor
Guest
Type 2
The difference between type 1 and type 2 is not as simple and useful as it seems
Embedded vs. IT Hypervisors
Guest
Guest
Hypervisor
Guest
Guest GuestDom0
Driver
Emu Driver
Driver
Driver EmuDriver
May 4, 2011 28| © 2010 Wind River. All Rights Reserved.28
Slim and Fat provide more IT-level capabilities (VM migration, load balancing, full
device virtualization) and hence carry an overhead cost. Wind River Hypervisor is
custom built for embedded and provides better performance
Embedded
Hypervisor
IT – Slim
Hypervisor
IT – Fat
Hypervisor
Driver
Manage Memory
• Every guest expects memory from 0x0 – MAXMEM.
• Hypervisor provides translation and protection.
• It uses flat or guest virtual memory.
May 4, 2011 30| © 2010 Wind River. All Rights Reserved.30
Guest
0x0 MAXMEM
0x0 MAXPHYSMEM
Guest Guest
Virtual
Physical
• Possible to provide noncontinuous mapping
• Possible to provide shared memory
– Shared segment can be put anywhere in guest memory
Manage Memory – Special Mappings
May 4, 2011 31| © 2010 Wind River. All Rights Reserved.31
0x0
Physical
GuestGuest Guest
Virtual
Shared
Device SupportDriver
Direct
� One owner
� Others shareShared
Guest
Driver
Guest Guest
shDev Driver
� Fastest, native performance
� Recommended where possible
May 4, 2011 34
| © 2010 Wind River. All Rights Reserved.34
HypervisorDriver� Driver in hypervisor
� Stubs in the guestVirtualized
Guest
Stub
Hypervisor
Guest
Stub
Driver
Guest
Driver1
Hypervisor
Guest
Driver1
Driver1 Emulation
Driver2
� Driver in hypervisor
� Emulation in hypervisor
� Regular driver in guest
Emulated
Devices Access – Virtual Interrupts
Virtual Board
Virtual Interrupt Controller
Guest
Virtual Board
Pending
Interrupt
Guest
Virtual Interrupt Controller
Running
May 4, 2011 38
| © 2010 Wind River. All Rights Reserved.38
Wind River Hypervisor
Pending Interrupt
Pending Interrupt
Interrupt Controller
Core Virtualization – Schedulers
• Two different schedulers available
• Priority
– Each virtual board has a priority
– Highest priority board that is non-idle is on-processor
– Boards with the same priority are scheduled round-robin
• Time slice
May 4, 2011 40
| © 2010 Wind River. All Rights Reserved.40
• Time slice
– Each virtual board gets a designated time slice
– Only single core
• Scheduler is pluggable
– Other scheduling algorithms can be plugged into the
hypervisor
Inter-Board Communication
• Shared memory can be given to each board
– Easy way to share large amounts of data
– Synchronization through asynchronous messaging
• For signaling, or using zero-copy data sending
• MultiOS Inter Processor Communication (MIPC)
• Asynchronous messaging promotes parallel algorithms
May 4, 2011 42
| © 2010 Wind River. All Rights Reserved.42
• MultiOS Inter Processor Communication (MIPC)
– Fast communication for multicore and virtualization
– Socket-like API for easy migration
– Wind River standard API between OSes in any
configuration
– Uses shared memory and interrupts
Services on top of MIPC
• MIPC Network Device (MND)
– TCP/IP layered on top of MIPC
– Slower than MIPC, but offers TCP/IP layer
• MIPC Serial Device (MSD)
– Ability to use a single serial port with multiple
May 4, 2011 43| © 2010 Wind River. All Rights Reserved.43
– Ability to use a single serial port with multiple
guests
– For terminal access, as well as printf for example
for debugging
• MIPC File System (MFS)
The How – Tailored IPCs
• Shared memory
– High performance
• MIPC
– High performance, zero copy
– OS independent
– 20K footprint
– Hardware support
Physical Board
Physical Board
Core 2
Virtual Board
App 6
MIPC
Core 1
Virtual Board
App 5
MIPC
Virtual Board
App 4
Linux 2
MIPC
Linux 1VxWorks
May 4, 2011 44
| © 2010 Wind River. All Rights Reserved.44
– Hardware support
– Static topology
• TIPC
– 100K footprint
– Full-featured
– Transparent multiboard
– Dynamic topology
Physical Board
Physical
Board
Core
Virtual Board
TIPC
Ethernet
App
Ethernet
App
TIPC
Virtual Board
TIPC
App
Linux 1Linux 3Linux 2
Building a System (Standard
Approach)
Hypervisor ELFImage (hypervisor.elf)
VxWorks ELFImage (vxworks.elf)
Linux ELF
Boot Image(ELF format)
Linux
Packaging
May 4, 2011 45
Linux ELFImage (vmlinux.elf)
XML Configuration
Image Manifest
VxWorks
Hypervisor
PackagingTool
Booting and Lifecycle (Standard Approach)
VB1
VxWorks
App
Wind River Hypervisor
Boot Image(ELF format)
Image Manifest
VxWorks
Linux
Hypervisor
VB2
Linux
App
May 4, 2011 46
• Hypervisor boots from ROM file system
• Hypervisor first (very quickly), then the appropriate VB
• VBs can be stopped, reloaded and restarted from initial image
• VB could contain a bootloader
46
Hypervisor
Virtualization Details – Summary
• Modes of virtualization
• The mechanisms behind virtualization
• Virtualizing memory, cores, devices, interrupts
• Inter-board communication
May 4, 2011 47| © 2010 Wind River. All Rights Reserved.47
Ring 3
Ring 2
Ring 1
Ring 0
Kernel
Device Drivers
Device Drivers
Applications
Ring 3
Ring 2
Ring 1
Ring 0
Kernel
Device Drivers
Device Drivers
Applications
Ring 3
Ring 2
Ring 1
Ring 0
Kernel
Device Drivers
Device Drivers
Applications
Hypervisor
Core 0 Core 1
Benefits of Virtualization
� Partition a system
� Evolve the partitioning
� Protect partitions
� Reduced time to market
� Reduced risk
� Reduced BOM
Provides the ability to … Resulting in …
May 4, 2011 49
| © 2010 Wind River. All Rights Reserved.49
� Protect partitions
� Reuse legacy code
� Share devices
� Verify a configuration
� Reduced BOM
� Investment protection
� Future proofing
Multi-Context Debugging
with OCD & Wind River Hypervisor
Debug Context ViewVisibility to
physical cores
- registers- bit level details- memory
- cacheTarget Manager
Multiple virtual boards (VBs)
running VxWorks in hypervised environment
May 4, 2011 54
environmentFine-grainedbreakpointcontrol withassociationto multiplecontexts
Deep visibility and granular control to debug complex system issues:Examples: inter-core race lock conditions, synchronization, message passing.