dangers of out-of-order execution: meltdown•meltdown vulnerability is in cpu hardware •a...
TRANSCRIPT
![Page 1: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/1.jpg)
Dangers of Out-of-Order Execution: Meltdown
Brad KarpUCL CS
(with background CPU architecture content drawn from Randy Bryant, Dave O’Hallaron, and Phil Gibbons)
CS 013311th December 2019
![Page 2: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/2.jpg)
Agenda
• Meltdown context: vulnerable systems and consequences
• Background:– CPU caches and the memory hierarchy– Virtual memory for protection and isolation– CPU pipelines– Out-of-order execution
• How Meltdown works• Meltdown mitigations• Take-aways
![Page 3: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/3.jpg)
Before Meltdown:Guarantees We Thought We Had
• Modern OS runs multiple processes on CPU– Each process’s memory isolated from every
other’s; enforced strongly by hardware– OS’s memory isolated from each process’s
• So even if one process is malicious, it can’t read data from another; basis of user privacy
• Widely deployed modern tricks for running multiple services on one server (e.g., paravirtualization, Docker containers) rely on process-style isolation from hardware for privacy; crucial to cloud computing
![Page 4: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/4.jpg)
After Meltdown
• Modern OS runs multiple processes on CPU– Each process’s memory isolated from every
other’s; enforced strongly by hardware– OS’s memory isolated from each process’s
• So even if one process is malicious, it can’t read data from another; basis of user privacy
• Widely deployed modern tricks for running multiple services on one server (e.g., paravirtualization, Docker containers) rely on process-style isolation from hardware for privacy; crucial to cloud computing
![Page 5: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/5.jpg)
After Meltdown (if OS Unpatched)
• Meltdown vulnerability is in CPU hardware• A malicious program can read kernel memory• Kernel memory generally maps all physical
memory; malicious program can read all processes’ memory– e.g., one user exploits web server, gets
other user’s credit card number• Malicious programs within paravirtualized
virtual machines and Docker containers can read other VMs’/containers’ memory– important in cloud computing: these
techniques used to host services in cloud
![Page 6: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/6.jpg)
After Meltdown (if OS Unpatched)
• Meltdown vulnerability is in CPU hardware• A malicious program can read kernel memory• Kernel memory generally maps all physical
memory; malicious program can read all processes’ memory– e.g., one user exploits web server, gets
other user’s credit card number• Malicious programs within paravirtualized
virtual machines and Docker containers can read other VMs’/containers’ memory– important in cloud computing: these
techniques used to host services in cloud
Vulnerable: all modern-era Intel CPUs since 1995 P6; ARM Cortex A15, A57, A72, A75; AMD not vulnerable to original Meltdown;Linux, Windows, MacOS…
![Page 7: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/7.jpg)
Agenda
• Meltdown context: vulnerable systems and consequences
• Background:– CPU caches and the memory hierarchy– Virtual memory for protection and isolation– CPU pipelines– Out-of-order execution
• How Meltdown works• Meltdown mitigations• Take-aways
![Page 8: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/8.jpg)
8
Example Memory Hierarchy Regs
L1 cache (SRAM)
Main memory(DRAM)
Local secondary storage(local disks)
Larger, slower, and cheaper (per byte)storagedevices
Remote secondary storage(e.g., Web servers)
Local disks hold files retrieved from disks on remote servers.
L2 cache (SRAM)
L1 cache holds cache lines retrieved from the L2 cache.
CPU registers hold words retrieved from the L1 cache.
L2 cache holds cache linesretrieved from L3 cache.
L0:
L1:
L2:
L3:
L4:
L5:
Smaller,faster,and costlier(per byte)storage devices
L3 cache (SRAM)
L3 cache holds cache linesretrieved from main memory.
L6:
Main memory holds disk blocks retrieved from local disks.
![Page 9: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/9.jpg)
9
Examples of Caching in the Memory Hierarchy
Hardware MMU
0On-Chip TLBAddress translations
TLB
Web browser10,000,000Local diskWeb pagesBrowser cache
Web cache
Network buffer cache
Buffer cacheVirtual MemoryL2 cacheL1 cache
Registers
Cache Type
Web pages
Parts of files
Parts of files4-KB pages64-byte blocks64-byte blocks
4-8 byte words
What is Cached?
Web proxy server
1,000,000,000Remote server disks
OS100Main memory
Hardware4On-Chip L1Hardware10On-Chip L2
NFS client10,000,000Local disk
Hardware + OS100Main memory
Compiler0CPU core
Managed ByLatency (cycles)
Where is it Cached?
Disk cache Disk sectors Disk controller 100,000 Disk firmware
![Page 10: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/10.jpg)
10
General Cache Concepts
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAMLarger, slower, cheaper memoryviewed as partitioned into “blocks”
Smaller, faster, more expensivememory caches a subset ofthe blocks
![Page 11: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/11.jpg)
11
General Cache Concepts
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAMLarger, slower, cheaper memoryviewed as partitioned into “blocks”
Data is copied in block-sized transfer units
Smaller, faster, more expensivememory caches a subset ofthe blocks
![Page 12: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/12.jpg)
12
General Cache Concepts
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAMLarger, slower, cheaper memoryviewed as partitioned into “blocks”
Data is copied in block-sized transfer units
Smaller, faster, more expensivememory caches a subset ofthe blocks
4
![Page 13: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/13.jpg)
13
General Cache Concepts
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAMLarger, slower, cheaper memoryviewed as partitioned into “blocks”
Data is copied in block-sized transfer units
Smaller, faster, more expensivememory caches a subset ofthe blocks
4
4
![Page 14: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/14.jpg)
14
General Cache Concepts
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAMLarger, slower, cheaper memoryviewed as partitioned into “blocks”
Data is copied in block-sized transfer units
Smaller, faster, more expensivememory caches a subset ofthe blocks
4
4
![Page 15: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/15.jpg)
15
General Cache Concepts
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAMLarger, slower, cheaper memoryviewed as partitioned into “blocks”
Data is copied in block-sized transfer units
Smaller, faster, more expensivememory caches a subset ofthe blocks
4
4
10
![Page 16: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/16.jpg)
16
General Cache Concepts
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAMLarger, slower, cheaper memoryviewed as partitioned into “blocks”
Data is copied in block-sized transfer units
Smaller, faster, more expensivememory caches a subset ofthe blocks
4
4
10
10
![Page 17: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/17.jpg)
17
General Cache Concepts
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAMLarger, slower, cheaper memoryviewed as partitioned into “blocks”
Data is copied in block-sized transfer units
Smaller, faster, more expensivememory caches a subset ofthe blocks
4
4
10
10
![Page 18: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/18.jpg)
18
General Cache Concepts: Hit (FAST!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAM
![Page 19: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/19.jpg)
19
General Cache Concepts: Hit (FAST!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3
Data in block b is neededRequest: 14
CPU Cache
DRAM
![Page 20: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/20.jpg)
20
General Cache Concepts: Hit (FAST!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3
Data in block b is neededRequest: 14
14Block b is in cache:Hit!CPU Cache
DRAM
![Page 21: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/21.jpg)
21
General Cache Concepts: Miss (SLOW!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3CPU Cache
DRAM
![Page 22: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/22.jpg)
22
General Cache Concepts: Miss (SLOW!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3
Data in block b is neededRequest: 12
CPU Cache
DRAM
![Page 23: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/23.jpg)
23
General Cache Concepts: Miss (SLOW!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3
Data in block b is neededRequest: 12
Block b is not in cache:Miss!CPU Cache
DRAM
![Page 24: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/24.jpg)
24
General Cache Concepts: Miss (SLOW!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3
Data in block b is neededRequest: 12
Block b is not in cache:Miss!
Block b is fetched frommemoryRequest: 12
CPU Cache
DRAM
![Page 25: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/25.jpg)
25
General Cache Concepts: Miss (SLOW!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3
Data in block b is neededRequest: 12
Block b is not in cache:Miss!
Block b is fetched frommemoryRequest: 12
12
CPU Cache
DRAM
![Page 26: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/26.jpg)
26
General Cache Concepts: Miss (SLOW!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3
Data in block b is neededRequest: 12
Block b is not in cache:Miss!
Block b is fetched frommemoryRequest: 12
12
12
CPU Cache
DRAM
![Page 27: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/27.jpg)
27
General Cache Concepts: Miss (SLOW!)
0 1 2 34 5 6 78 9 10 1112 13 14 15
8 9 14 3
Data in block b is neededRequest: 12
Block b is not in cache:Miss!
Block b is fetched frommemoryRequest: 12
12
12
Block b is stored in cache•Placement policy:determines where b goes•Replacement policy:determines which blockgets evicted (victim)
CPU Cache
DRAM
![Page 28: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/28.jpg)
Agenda
• Meltdown context: vulnerable systems and consequences
• Background:– CPU caches and the memory hierarchy– Virtual memory for protection and isolation– CPU pipelines– Out-of-order execution
• How Meltdown works• Meltdown mitigations• Take-aways
![Page 29: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/29.jpg)
Memory Map of a Linux Process
• N.B. that every Linux process has kernel (OS) memory in its virtual address space, and that kernel memory maps all physical RAM in machine (so all processes’ memory)!
• Why: performance– avoids changing page tables, flushing TLB on
switches between user code and kernel code– convenient and fast for kernel always to map all
physical RAM (to access processes’ memory)
![Page 30: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/30.jpg)
Virtual Memory: Same Addresses for Each Process, Yet Isolated
Process 1
![Page 31: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/31.jpg)
Virtual Memory: Same Addresses for Each Process, Yet Isolated
Process 1 Process 2
![Page 32: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/32.jpg)
Virtual Memory: Same Addresses for Each Process, Yet Isolated
Process 1 Process 2 Process n
![Page 33: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/33.jpg)
Agenda
• Meltdown context: vulnerable systems and consequences
• Background:– CPU caches and the memory hierarchy– Virtual memory for protection and isolation– CPU pipelines– Out-of-order execution
• How Meltdown works• Meltdown mitigation• Take-aways
![Page 34: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/34.jpg)
Real-World Pipelines: Car Washes
![Page 35: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/35.jpg)
Real-World Pipelines: Car WashesSequential
![Page 36: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/36.jpg)
Real-World Pipelines: Car WashesSequential Parallel
![Page 37: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/37.jpg)
Real-World Pipelines: Car WashesSequential Parallel
Pipelined
![Page 38: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/38.jpg)
Real-World Pipelines: Car Washes
• Idea– Divide process into
independent stages– Move objects through
stages in sequence– At any given time, multiple
objects being processed
Sequential Parallel
Pipelined
![Page 39: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/39.jpg)
Pipelining for Fast CPUs
• Divide instruction execution into pipeline– short, sequentially arranged stages– stages operate concurrently as pipeline fills with
instructions– instructions advance to next stage each clock
cycle– CPU completes one instruction per clock cycle
(oversimplified)• Shorter pipeline stages, deeper pipelines à higher
clock rates, faster instruction completion rate• To go even faster, duplicate pipeline to execute
even more instructions concurrently (known as superscalar, multiple issue)
![Page 40: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/40.jpg)
A Simplified CPU Pipelineirmovq $1,%rax #I1
1 2 3 4 5 6 7 8 9
F D E MWirmovq $2,%rcx #I2 F D E M
W
irmovq $3,%rdx #I3 F D E M Wirmovq $4,%rbx #I4 F D E M Whalt #I5 F D E M W
Cycle 5
WI1
MI2
EI3
DI4
FI5
![Page 41: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/41.jpg)
A Simplified CPU Pipelineirmovq $1,%rax #I1
1 2 3 4 5 6 7 8 9
F D E MWirmovq $2,%rcx #I2 F D E M
W
irmovq $3,%rdx #I3 F D E M Wirmovq $4,%rbx #I4 F D E M Whalt #I5 F D E M W
Cycle 5
WI1
MI2
EI3
DI4
FI5
• Key challenge:– 5 cycles to finish any
instruction– What if subsequent
instruction needs result from prior one?
– Want to keep CPU’s transistors busy!
![Page 42: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/42.jpg)
Agenda
• Meltdown context: vulnerable systems and consequences
• Background:– CPU caches and the memory hierarchy– Virtual memory for protection and isolation– CPU pipelines– Out-of-order execution
• How Meltdown works• Meltdown mitigations• Take-aways
![Page 43: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/43.jpg)
Out-of-Order Execution, Or How to Keep Transistors Busy
• Central idea: if CPU functional units idle, fetch instructions from later in program; if their inputs are ready, execute them now
• N.B. means CPU hardware may execute instructions in order other than given in program– CPU hardware ensures only retires instructions
(writes back externally visible results, e.g., in registers) in program order
– CPU hardware ensures if instruction causes a hardware exception, later instructions in program order executed “early” are squashed—results not written back to registers
• Vital performance improvement for modern CPUs with multiple, deep pipelines
![Page 44: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/44.jpg)
Agenda
• Meltdown context: vulnerable systems and consequences
• Background• How Meltdown works:
– Building block: Flush+Reload side-channel cache attack
– Overview– Meltdown (byte-at-a-time)– Meltdown (bit-at-a-time)
• Meltdown mitigations• Take-aways
![Page 45: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/45.jpg)
Building Block: Flush+Reload
• Goal: determine whether process has accessed target address in process’s memory during some period
• Three phases:– Flush part of data cache that holds the target
address with clflush instruction– Wait to allow access to target address to occur– Time the latency of reading from target address
• Short latency: cache already filled, so program accessed address during wait
• Long latency: had to bring data into cache, so program didn’t access address during wait
• Side channel attack: timing reveals whether address written to previously (without observing address directly, e.g., in some register)
![Page 46: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/46.jpg)
Meltdown Attack: Overview
• Goal: in a user-level, non-root process, read data from another process’s virtual memory
• Approach:– Flush the CPU’s data cache for a range of valid
process memory addresses [Flush step]– Tell OS not to kill process on seg fault– Read from target kernel virtual memory address (will
cause seg fault once CPU detects invalid address!)– Read from valid in-process memory address derived
from value read from kernel virtual memory address (executes because of out-of-order execution)
– Time reads from all addresses prior step might have read from; one that completes more quickly than all others reveals kernel memory value [Reload step]
![Page 47: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/47.jpg)
Meltdown: Byte-at-a-Time• Cache flush and seg fault handling
already done at this point; attacker next executes ”core” of Meltdown at right
• Inputs:– %rcx holds target kernel memory
address where want to read one byte– %rbx holds base address of 256
pages in user space, used for Flush+Reload side channel
xorq %rax, %raxretry:movb (%rcx), %alshl %rax, $0xcjz retrymovq (%rbx,%rax,1), %rbx
![Page 48: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/48.jpg)
Meltdown: Byte-at-a-Time• Cache flush and seg fault handling
already done at this point; attacker next executes ”core” of Meltdown at right
• Inputs:– %rcx holds target kernel memory
address where want to read one byte– %rbx holds base address of 256
pages in user space, used for Flush+Reload side channel
xorq %rax, %raxretry:movb (%rcx), %alshl %rax, $0xcjz retrymovq (%rbx,%rax,1), %rbx
• movb reads one byte from target kernel address; will eventually cause seg fault, but CPU initially does load into %rcx
![Page 49: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/49.jpg)
Meltdown: Byte-at-a-Time• Cache flush and seg fault handling
already done at this point; attacker next executes ”core” of Meltdown at right
• Inputs:– %rcx holds target kernel memory
address where want to read one byte– %rbx holds base address of 256
pages in user space, used for Flush+Reload side channel
xorq %rax, %raxretry:movb (%rcx), %alshl %rax, $0xcjz retrymovq (%rbx,%rax,1), %rbx
• movb reads one byte from target kernel address; will eventually cause seg fault, but CPU initially does load into %rcx
• Instructions below movb execute because of out-of-order execution, before seg fault
• shl multiplies byte read from kernel by 4096 (page size); CPU doesn’t prefetch across page boundaries
![Page 50: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/50.jpg)
Meltdown: Byte-at-a-Time• Cache flush and seg fault handling
already done at this point; attacker next executes ”core” of Meltdown at right
• Inputs:– %rcx holds target kernel memory
address where want to read one byte– %rbx holds base address of 256
pages in user space, used for Flush+Reload side channel
xorq %rax, %raxretry:movb (%rcx), %alshl %rax, $0xcjz retrymovq (%rbx,%rax,1), %rbx
• movb reads one byte from target kernel address; will eventually cause seg fault, but CPU initially does load into %rcx
• Instructions below movb execute because of out-of-order execution, before seg fault
• shl multiplies byte read from kernel by 4096 (page size);CPU doesn’t prefetch across page boundaries
• movq reads from address %rbx + %rax, i.e., start of page N in attacker’s Flush+Reload region, where N is byte read from kernel mem
![Page 51: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/51.jpg)
Meltdown: Byte-at-a-Time• Cache flush and seg fault handling
already done at this point; attacker next executes ”core” of Meltdown at right
• Inputs:– %rcx holds target kernel memory
address where want to read one byte– %rbx holds base address of 256
pages in user space, used for Flush+Reload side channel
xorq %rax, %raxretry:movb (%rcx), %alshl %rax, $0xcjz retrymovq (%rbx,%rax,1), %rbx
• movb reads one byte from target kernel address; will eventually cause seg fault, but CPU initially does load into %rcx
• Instructions below movb execute because of out-of-order execution, before seg fault
• shl multiplies byte read from kernel by 4096 (page size);CPU doesn’t prefetch across page boundaries
• movq reads from address %rbx + %rax, i.e., start of page N in attacker’s Flush+Reload region, where N is byte read from kernel mem
• Occasionally %al contains zero because CPU doesn’t propagate kernel read result; jz retries in this case, and loop terminates either when non-zero value read or when seg fault delivered to application
![Page 52: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/52.jpg)
Read FromFlush+Reload Side Channel
• Before executing “core” Meltdown code on prior slide:– Allocate 256 pages (1 MB) in process user-level
memory– Flush cache for this entire region
• After executing “core” Meltdown code on prior slide:– Read from start of all 256 pages, timing latency for
each read– Much faster read for page i than all others indicates
value of byte read from target kernel memory address was i
– Exception: page 0. Because core code may see zero erroneously, cache hit in page 0 doesn’t mean zero read from kernel memory. Instead, absence of cache hit on any other page means zero truly read.
![Page 53: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/53.jpg)
Read FromFlush+Reload Side Channel (2)
0 1 2 3 … 84 … 254 255
4K page
![Page 54: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/54.jpg)
Read FromFlush+Reload Side Channel (2)
0 1 2 3 … 84 … 254 255
4K page
![Page 55: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/55.jpg)
Read FromFlush+Reload Side Channel (2)
0 1 2 3 … 84 … 254 255
4K page
![Page 56: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/56.jpg)
Meltdown: Bit-at-a-Time
• Byte-at-a-time Meltdown has to scan starts of 256 pages for Flush+Reload to exfiltrate one byte of kernel memory
• These scans far more expensive than execution of “core” Meltdown code
• Faster exfiltration: one bit at a time!• Add instructions to “core” Meltdown routine to
mask and shift single bit of the byte read from kernel memory; make versions for all 8 bits
• Now only need to Flush+Reload for 8 pages total per exfiltrated byte (once per exfiltrated bit):– Only need to scan page 1; if fast, received a 1
bit; if slow, received a 0 bit
![Page 57: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/57.jpg)
Why Does Meltdown Work?
• Shouldn’t read of kernel virtual address (no permission for user level to read) cause process to be halted?– Yes, but it takes CPU a long time to detect violation
because of pipeline– And out-of-order execution keeps executing more
program before CPU realizes earlier instruction accessed forbidden memory
– And process can catch SIGSEGV, rather than being killed by OS ;-)
• Doesn’t the CPU squash OoOE results that shouldn’t have been computed?– Yes, by restoring registers to their old values– But cache occupancy for allowed memory accesses
survives a squash!
![Page 58: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/58.jpg)
Meltdown Mitigation: Software
• Change kernel to not map kernel’s pages into user processes’ page tables (apart from a few pages needed for kernel entry points)
• KPTI (kernel page table isolation) patches already in Linux, Windows, MacOS
• Performance cost: when process makes system call, OS must first change active page table to map kernel memory, and reverse this before resuming user process
• Cost heavily workload-dependent (frequency of system calls); early reports range from “unnoticeable” to “30%+ reduction”
![Page 59: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/59.jpg)
Meltdown Mitigation: Hardware
• Intel announcements:– August 2018: Cascade Lake Xeon server CPUs,
Whiskey Lake notebook CPUs (due in late 2018) include hardware mitigation for Meltdown
– October 2018: Coffee Lake Refresh 9th
generation (i9-9900K, i7-9700K, i5-9600K) desktop CPUs include hardware mitigation for Meltdown
• Mitigation hardware internals not (yet) published• Mitigation causes memory reads to check whether
target address legal (and if not, deliver hardware exception) before OoOE of later instructions can modify cache
![Page 60: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/60.jpg)
Meltdown Mitigation: Hardware
• Intel announcements:– August 2018: Cascade Lake Xeon server CPUs,
Whiskey Lake notebook CPUs (due in late 2018) include hardware mitigation for Meltdown
– October 2018: Coffee Lake Refresh 9th
generation (i9-9900K, i7-9700K, i5-9600K) desktop CPUs include hardware mitigation for Meltdown
• Mitigation hardware internals not (yet) published• Mitigation causes memory reads to check whether
target address legal (and if not, deliver hardware exception) before OoOE of later instructions can modify cache
Alas, even this new hardware vulnerable to new “Fallout” Meltdown-like attack; targets CPU store buffer hardware, leaks kernel writes to user-level code [Canella et al., CCS 2019, Nov. 2019]For now, still need KPTI, CPU microcode update, and VERW instruction on context switch to flush store buffer explicitly!
![Page 61: Dangers of Out-of-Order Execution: Meltdown•Meltdown vulnerability is in CPU hardware •A malicious program can read kernel memory •Kernel memory generally maps all physical memory;](https://reader036.vdocument.in/reader036/viewer/2022071217/6049ba6f06c04f3460622162/html5/thumbnails/61.jpg)
Take-Aways
• Modern CPU architectures are rife with side channels: state held by the CPU (e.g., caches) that can leak information in subtle ways
• CPU vendors appear to have been largely oblivious to privacy risks caused by performance-improving optimizations (e.g., OoOE, speculative execution) since the mid 1990s
• Meltdown (and Spectre) unlikely to be the last CPU privacy vulnerabilities of this sort (already seen Foreshadow, L1 Terminal Fault, now Fallout, etc.)
• Those with strongest need for privacy should be vigilant about risks of running on shared hardware alongside users and/or code they don’t trust
• Considerable research effort in architecture community today on mitigating microarchitectural side channels without killing CPU performance