dynamic runtime testing for cycle-accurate simulators
DESCRIPTION
Dynamic Runtime Testing for Cycle-Accurate Simulators. Saša Tomić, Adri án Cristal, Osman Unsal , Mateo Valero. Barcelona Supercomputing Center (BSC) Universitat Polit ecnica de Catalunya (UPC). Can we trust the simulator-based evaluations?. Typical simulator evaluation: - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/1.jpg)
Dynamic Runtime Testing for Cycle-Accurate Simulators
Saša Tomić, Adrián Cristal, Osman Unsal, Mateo Valero
Barcelona Supercomputing Center (BSC)Universitat Politecnica de Catalunya (UPC)
![Page 2: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/2.jpg)
2
Can we trust the simulator-based evaluations?
• Typical simulator evaluation:Make a simulatorREPEAT {
DebugSimulate
} UNTIL: the results make sense (intuition!)• Discard and ignore the failed simulations• Are there any bugs left?
![Page 3: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/3.jpg)
Verifying the simulators• Verification is important!
– Industry puts significant resources• Testing and Verification 50-70% of the costs• Mission critical application even 90% of the costs
– Academia puts less resources• Why do we have bugs?
– Simulators are complex– Proposed extensions are often complex– The extensions may uncover existing bugs
3
![Page 4: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/4.jpg)
Simulator bugs• Timing bugs
– Incorrect estimation of the execution time– Simulation terminates without obvious errors– Needs other types of testing and verification
• Functional bugs– Incorrect implementation of functional units– Simulations may or may not terminate correctly
4
Our target
![Page 5: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/5.jpg)
Outline• Examples of functional bugs• An overview of the Dynamic Simulator Testing
methodology• Use Cases of Dynamic Simulator Testing• Performance evaluation• Conclusions
5
![Page 6: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/6.jpg)
Example: a bug in the cache coherence protocol
6
simulator ofmulti-level coherent caches
X=0
X=0
X=10 X=0 X=20
X=20
X += 10 X += 20
Bug: X should beX = 10+20 = 30shared memory
processor 1 processor 2
Proc 1X+=10
Proc 2X+=20
X=0
X=30
time
![Page 7: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/7.jpg)
Example: a bug in the HTMX = 0;Atomic { X += 10;}
7
HTM simulator
X=0
X=0 X=10
nothing?
Bug: not committedX = 10
processor
shared memory
X += 10
![Page 8: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/8.jpg)
Detecting functional bugs• The functionality of the simulators is often simple
– Can be emulated with simple emulators– The emulators can be fast and stable
• Can we take an advantage of the emulators?
8
![Page 9: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/9.jpg)
Dynamic Testing Methodology
• Add a simple, no-timing emulator• Execute each operation in the simulator and then in
the emulator• Compare the executions
– We compared only the memory accesses• The execution must be identical during entire simulation
9
![Page 10: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/10.jpg)
An overview of dynamic testing
10
timing simulatorsimple
no-timing emulator
input input
input inputoutput output
output output
• Use the same input• Compare the outputs• Repeat for every operation!
![Page 11: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/11.jpg)
Dynamic testing a cache coherence protocol
11
STL maptiming simulator ofmulti-level coherent caches
X += 10 X += 20
shared memory
processor 1 processor 2
inputX=10X=10
inputX=0
outputX=0
outputX=0
outputX=20input
X=20
outputoutput input
X=0 X=0
input
X=10
Check failed:should be X=10
X=0X=10
![Page 12: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/12.jpg)
Dynamic testing of an HTM
12
timing simulator of an HTM
STL mapper TX
TX_Begin; X += 10;TX_Commit;
Check failed:should commit X=10
input
X=0
input
X=0
processor
shared memory
input
X=10
input
X=10
output
X=0 X=0
output
output
X=??? X=10
output
X=0X=10
![Page 13: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/13.jpg)
Other Use Cases• Out-Of-Order or pipelined processor
– With a processor emulator, e.g., QEMU• Complex memory hierarchy
– With an STL map• Incoherent multilevel memory hierarchy
– W/ multiple STL maps, one per memory hierarchy• System-On-Chip, Routing Protocols, etc.
– Simple emulators of the functionalities
13
![Page 14: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/14.jpg)
Performance Evaluation• Implemented on 4 HTMs with lazy and eager
version management• Implemented for a directory-based cache-
coherence protocol• Baseline: M5 full-system simulator
14
![Page 15: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/15.jpg)
Performance evaluationOS booting
15
1 2 4 8 16 320
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
OS booting with cache testing OS booting with HTM and cache testing
Norm
aliz
ed e
xecu
tion
time
![Page 16: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/16.jpg)
Performance evaluationapplications
16
baye
s
geno
me
intrud
er
kmea
ns-hi
kmea
ns-lo
w
labyri
nthss
ca2
vaca
tion-h
i
vaca
tion-l
owya
da0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Application with cache testing Application with HTM and cache testing
Norm
aliz
ed e
xecu
tion
time
![Page 17: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/17.jpg)
Our experience with Dynamic Testing
• Reduced the time spent on writing tests• Faster debugging
– Detects most bugs “in minutes”– Eliminating a bug takes tens of minutes instead of
hours/days/weeks/…• Shortened the total simulator development from
12-18 months to 3-4 months
17
![Page 18: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/18.jpg)
Conclusions• Presented the Dynamic Simulator Testing• Detects the functional bugs in Cycle-Accurate
Simulators• Modest reduction of simulator performance
18
![Page 20: Dynamic Runtime Testing for Cycle-Accurate Simulators](https://reader036.vdocument.in/reader036/viewer/2022062310/56815f56550346895dce35f2/html5/thumbnails/20.jpg)
Cache/HTM emulator implementation
20
STL map (dictionary)
addressaddressaddress
line dataline dataline data
address line data...
.
.
.