dynamic runtime testing for cycle-accurate simulators saša tomić, adrián cristal, osman unsal,...
TRANSCRIPT
Dynamic Runtime Testing for Cycle-Accurate Simulators
Saša Tomić, Adrián Cristal, Osman Unsal, Mateo Valero
Barcelona Supercomputing Center (BSC)Universitat Politecnica de Catalunya (UPC)
2
Can we trust the simulator-based evaluations?
• Typical simulator evaluation:Make a simulatorREPEAT {
DebugSimulate
} UNTIL: the results make sense (intuition!)• Discard and ignore the failed simulations• Are there any bugs left?
Verifying the simulators
• Verification is important!– Industry puts significant resources
• Testing and Verification 50-70% of the costs• Mission critical application even 90% of the costs
– Academia puts less resources• Why do we have bugs?
– Simulators are complex– Proposed extensions are often complex– The extensions may uncover existing bugs
3
Simulator bugs
• Timing bugs– Incorrect estimation of the execution time– Simulation terminates without obvious errors– Needs other types of testing and verification
• Functional bugs– Incorrect implementation of functional units– Simulations may or may not terminate correctly
4
Our target
Outline
• Examples of functional bugs• An overview of the Dynamic Simulator Testing
methodology• Use Cases of Dynamic Simulator Testing• Performance evaluation• Conclusions
5
Example: a bug in the cache coherence protocol
6
simulator ofmulti-level coherent caches
X=0
X=0
X=10 X=0 X=20
X=20
X += 10 X += 20
Bug: X should beX = 10+20 = 30shared memory
processor 1 processor 2
Proc 1X+=10
Proc 2X+=20
X=0
X=30
time
Example: a bug in the HTM
X = 0;Atomic { X += 10;}
7
HTM simulator
X=0
X=0 X=10
nothing?
Bug: not committedX = 10
processor
shared memory
X += 10
Detecting functional bugs
• The functionality of the simulators is often simple– Can be emulated with simple emulators– The emulators can be fast and stable
• Can we take an advantage of the emulators?
8
Dynamic Testing Methodology
• Add a simple, no-timing emulator• Execute each operation in the simulator and then in
the emulator• Compare the executions
– We compared only the memory accesses• The execution must be identical during entire simulation
9
An overview of dynamic testing
10
timing simulatorsimple
no-timing emulator
input input
input inputoutput output
output output
• Use the same input• Compare the outputs• Repeat for every operation!
Dynamic testing a cache coherence protocol
11
STL maptiming simulator of
multi-level coherent caches
X += 10 X += 20
shared memory
processor 1 processor 2
input
X=10X=10
input
X=0
output
X=0
output
X=0
output
X=20
input
X=20
outputoutput input
X=0 X=0
input
X=10
Check failed:should be X=10
X=0X=10
Dynamic testing of an HTM
12
timing simulator of an HTM
STL mapper TX
TX_Begin; X += 10;TX_Commit;
Check failed:should commit X=10
input
X=0
input
X=0
processor
shared memory
input
X=10
input
X=10
output
X=0 X=0
output
output
X=??? X=10
output
X=0X=10
Other Use Cases
• Out-Of-Order or pipelined processor– With a processor emulator, e.g., QEMU
• Complex memory hierarchy– With an STL map
• Incoherent multilevel memory hierarchy– W/ multiple STL maps, one per memory hierarchy
• System-On-Chip, Routing Protocols, etc.– Simple emulators of the functionalities
13
Performance Evaluation
• Implemented on 4 HTMs with lazy and eager version management
• Implemented for a directory-based cache-coherence protocol
• Baseline: M5 full-system simulator
14
Performance evaluationOS booting
15
1 2 4 8 16 320
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
OS booting with cache testing OS booting with HTM and cache testing
No
rma
lize
d e
xe
cu
tio
n t
ime
Performance evaluationapplications
16
baye
s
geno
me
intru
der
kmea
ns-h
i
kmea
ns-lo
w
laby
rinth
ssca
2
vaca
tion-
hi
vaca
tion-
low
yada
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Application with cache testing Application with HTM and cache testing
No
rma
lize
d e
xe
cu
tio
n t
ime
Our experience with Dynamic Testing
• Reduced the time spent on writing tests• Faster debugging
– Detects most bugs “in minutes”– Eliminating a bug takes tens of minutes instead of
hours/days/weeks/…• Shortened the total simulator development from
12-18 months to 3-4 months
17
Conclusions
• Presented the Dynamic Simulator Testing• Detects the functional bugs in Cycle-Accurate
Simulators• Modest reduction of simulator performance
18
Cache/HTM emulator implementation
20
STL map (dictionary)
addressaddressaddress
line dataline dataline data
address line data...
.
.
.