![Page 1: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/1.jpg)
An Empirical Study of Optimizations in Yogi
Aditya V. Nori, Sriram K. Rajamani
Microsoft Research India
![Page 2: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/2.jpg)
What is Yogi? An industrial strength program verifier
Idea: Synergize verification and testing
Synergy [FSE ’06], Dash [ISSTA ‘08], SMASH [POPL ‘10] algorithms to perform scalable analysis
Engineered a number of optimizations for scalability
Integrated with Microsoft’s Static Driver Verifier (SDV) toolkit and used internally
![Page 3: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/3.jpg)
Motivation
Share our experiences in making Yogi robust, scalable and industrial strength
Several of the implemented optimizations are folklore Very difficult to design tools that are bug free evaluating
optimizations is hard! Our empirical evaluation gives tool builders information about
what gains can be realistically expected from optimizations
Vanilla implementation of algorithms: (flpydisk, CancelSpinLock) took 2 hours
Algorithms + engineering + optimizations: (flpydisk, CancelSpinLock) took less than 1 second!
![Page 4: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/4.jpg)
Outline
Overview of Yogi
Overview of optimizations
Evaluation setup
Empirical Results
Summary
![Page 5: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/5.jpg)
Property checking
void foo(){ *p = 4; *q = 5; if (condition) error();}
QuestionIs error() unreachable for all possible inputs?
Verification: can prove the absence of bugs, but can result in false errorsTesting: finds bugs, but can’t prove their absence
![Page 6: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/6.jpg)
The Yogi algorithm
no
no
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
τ = error path in abstraction f = frontier of error path
yes
no
yes
Proof! yes
Input:Program P
Property ψ
![Page 7: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/7.jpg)
Example: Abstraction & Tests
no
no
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
τ = error path in abstraction f = frontier of error path
yes
no
yes
Proof! yes
Input:Program P
Property ψ
void foo(int y){0: int x, lock = 0;1: do {2: lock = 1;3: x = y;4: if (*) {5: lock = 0;6: y = y+1; }7: } while (x != y);8: if (lock != 1)9: error();10:}
y = 101234
56
789
×
× ×
× ×
× ×
× ×
×
×
× ×
×
10×
Symbolic execution +
Theorem proving
![Page 8: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/8.jpg)
Example: Refinement
no
no
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
τ = error path in abstraction f = frontier of error path
yes
no
yes
Proof! yes
Input:Program P
Property ψ
void foo(int y){0: int x, lock = 0;1: do {2: lock = 1;3: x = y;4: if (*) {5: lock = 0;6: y = y+1; }7: } while (x != y);8: if (lock != 1)9: error();10:}
01234
56
78:ρ
9
×
× ×
× ×
× ×
× ×
×
×
× ×
×
10×
8:¬ρ×
![Page 9: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/9.jpg)
Example: Proof!
no
no
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
τ = error path in abstraction f = frontier of error path
yes
no
yes
Proof! yes
Input:Program P
Property ψ
void foo(int y){0: int x, lock = 0;1: do {2: lock = 1;3: x = y;4: if (*) {5: lock = 0;6: y = y+1; }7: } while (x != y);8: if (lock != 1)9: error();10:}
012
34:¬s5:¬s6:¬r
9
×
× ×
× ×××
××
×
×
7:¬q×
8:¬p×
4:s5:s6:r7:q8:p×
10
![Page 10: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/10.jpg)
Optimizations
Initial abstraction from property predicates
Relevance heuristics for predicate abstraction Suitable predicates (SP) Control dependence predicates (CD)
Interprocedural analysis Global modification analysis Summaries for procedures
Thresholds for tests
Fine tuning environment models
![Page 11: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/11.jpg)
Evaluation setup
Benchmarks: 30 WDM drivers and 83 properties (2490 runs) Anecdotal belief: most bugs in the tools are
usually caught with this test suite
Presentation methodology: Group optimizations logically such that related
optimizations are in the same group Total time taken, total number of defects found
for every possible choice of enabling/disabling each optimization in the group
![Page 12: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/12.jpg)
Initial abstraction
state { enum {Locked = 0, Unlocked = 1} state = Unlocked;}
KeAcquireCancelSpinlock.Entry { if (state != Locked) { state = Locked; } else abort;}
KeReleaseCancelSpinlock.Entry { if (state == Locked) { state = Unlocked; } else abort;}
01
(𝑠𝑡𝑎𝑡𝑒≠𝐿𝑜𝑐𝑘𝑒𝑑)
01
(𝑠𝑡𝑎𝑡𝑒=𝐿𝑜𝑐𝑘𝑒𝑑)
01𝑇
𝑇
![Page 13: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/13.jpg)
Empirical resultsAbstractio
n using SLIC predicates
Total time
(minutes)
#defects #timeouts
yes 2160 241 77no 2580 241 86
16%
![Page 14: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/14.jpg)
Relevance heuristics (SP)
Avoid irrelevant conjuncts
AC
𝑇
𝑇
B𝑇
D 𝛿
AC
𝑇
¬𝜌
B𝑇
D 𝛿
C 𝜌
𝑎𝑠𝑠𝑢𝑚𝑒(𝜙)
𝑎𝑠𝑠𝑢𝑚𝑒(𝜙)
Irrelevant?
![Page 15: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/15.jpg)
Relevance heuristics (CD)
Abstract assume statements that are not potentially relevant by skip statements
If Yogi proves that the program satisfies property, we are done.
Otherwise, validate the error trace
and refine the abstraction by putting back assume statements, if the error trace is spurious
![Page 16: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/16.jpg)
Example: SP heuristic
int x;void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock();}
AC
𝑇
𝑇
B𝑇
D 𝑠𝑡𝑎𝑡𝑒=𝑙𝑜𝑐𝑘𝑒𝑑
AC
𝑇
¬𝜌
B𝑇
D 𝑠𝑡𝑎𝑡𝑒=𝑙𝑜𝑐𝑘𝑒𝑑
C 𝜌
𝑎𝑠𝑠𝑢𝑚𝑒( 𝑖>1000)
𝑎𝑠𝑠𝑢𝑚𝑒( 𝑖>1000)
𝜌=(state=Locked )∧( 𝑖>1000)
![Page 17: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/17.jpg)
Example: SP heuristic
int x;void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock();}
AC
𝑇
𝑇
B𝑇
D 𝑠𝑡𝑎𝑡𝑒=𝑙𝑜𝑐𝑘𝑒𝑑
AC
𝑇
¬𝜌
B𝑇
D 𝑠𝑡𝑎𝑡𝑒=𝑙𝑜𝑐𝑘𝑒𝑑
C 𝜌
𝑎𝑠𝑠𝑢𝑚𝑒( 𝑖>1000)
𝑎𝑠𝑠𝑢𝑚𝑒( 𝑖>1000)
𝜌=(state=Locked )
![Page 18: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/18.jpg)
Example: CD heuristic
int x;void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock();}
![Page 19: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/19.jpg)
Empirical resultsSP
heuristic
CD heurist
ic
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2580 239 91no yes 2400 238 87no no 2894 235 174
10%
![Page 20: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/20.jpg)
Empirical resultsSP
heuristic
CD heurist
ic
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2580 239 91no yes 2400 238 87no no 2894 235 174
16%
![Page 21: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/21.jpg)
Empirical resultsSP
heuristic
CD heurist
ic
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2580 239 91no yes 2400 238 87no no 2894 235 174
25%
![Page 22: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/22.jpg)
Interprocedural analysis
Yogi performs a compositional analysis : Is it possible to execute starting from
state and reach state ?
Global modification analysis
May-Must analysis (SMASH, POPL 2010)
![Page 23: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/23.jpg)
Example
AC
𝑇
𝑇
B𝑇
D 𝑠𝑡𝑎𝑡𝑒=𝑙𝑜𝑐𝑘𝑒𝑑
AC
𝑇
¬𝜌
B𝑇
D 𝑠𝑡𝑎𝑡𝑒=𝑙𝑜𝑐𝑘𝑒𝑑
C 𝜌
𝑓𝑜𝑜(…)
foo(…)
⟨𝜙1 , 𝑓𝑜𝑜 (…) ,𝜙2 ⟩
![Page 24: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/24.jpg)
Empirical resultsModificat
ion analysis
Summaries
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2760 239 109no yes 3180 237 134no no 3780 236 165
32%
![Page 25: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/25.jpg)
Empirical resultsModificat
ion analysis
Summaries
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2760 239 109no yes 3180 237 134no no 3780 236 165
28%
![Page 26: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/26.jpg)
Empirical resultsModificat
ion analysis
Summaries
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2760 239 109no yes 3180 237 134no no 3780 236 165
42%
![Page 27: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/27.jpg)
Testing
Yogi relies on tests for “cheap” reachability
Long tests avoiding several potential reachability
queries results in too many states and thus
memory consumption
Test thresholds: time vs. space tradeoff
![Page 28: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/28.jpg)
Empirical evaluation
Test threshol
d
Total time
(minutes)
#defects
#timeouts
250 2600 236 92500 2160 241 771000 2359 240 881500 2400 239 89
![Page 29: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/29.jpg)
Modeling the environment
if (DestinationString) { DestinationString->Buffer = SourceString;
// DestinationString->Length should be set to the // length of SourceString. The line below is missing // from the original stub SDV function DestinationString->Length = strlen(SourceString);}
if (SourceString == NULL){ DestinationString->Length = 0; DestinationString->MaximumLength = 0;}
Issue type #issues
Integers used as pointers
8Uninitialized
variables15
Type inconsistencies 9
![Page 30: Aditya V. Nori, Sriram K. Rajamani Microsoft Research India](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed95503460f94be8086/html5/thumbnails/30.jpg)
Summary
Described optimizations implemented in Yogi Evaluated optimizations on the WDM test suite
Empirical data used to decide which optimizations to include in Yogi
We believe that this detailed empirical study of optimizations will enable tool builders to decide which optimizations to include and how to engineer their tools
http://research.microsoft.com/yogi