automatically inferring temporal properties for program evolution jinlin yang and david evans 15 th...
TRANSCRIPT
Automatically Inferring Temporal Properties for Program Evolution
Jinlin Yang and David Evans15th IEEE International Symposium on Software Reliability Engineering 5 November 2004Saint-Malo, FranceUniversity of
VirginiaComputer
Science
www.cs.virginia.edu/ipa 2
Temporal Properties• Constrain ordering of events
– Essential for program correctness (file open before read)
• Reveal important differences between programs: S P not a chef
• But, hard for humans to document correctly [Holzmann’s FSE 2002 keynote]
• Can we infer useful temporal properties automatically?This talk argues: Yes!
www.cs.virginia.edu/ipa 3
Dynamically Inferring Properties
• Inherently unsound: guessing properties of all executions by looking at a small number
• Value-based invariants– Daikon [Ernst, TSE, Feb 01]
• Temporal properties– Mining specification [Ammons, POPL 02]– Extracting component interfaces [Whaley, ISSTA
02]
• We focus on relationship between 2 or 3 events: automation, scalability
www.cs.virginia.edu/ipa 4
System Overview
Inferred Properties 1
DifferentProperties
Diff
ere
nce A
naly
zer
Inferred Properties 2
SharedPropertiesProgram
Version 1
Program Version 2
Dyn
am
ic In
fere
nce
… …
www.cs.virginia.edu/ipa 5
Property Inference
Inferred Properties 1
DifferentProperties
Diff
ere
nce A
naly
zer
Inferred Properties 2
SharedPropertiesProgram
Version 1
Program Version 2
Dyn
am
ic In
fere
nce
… …
ProgramInstrumented
Program
Instru
menta
tion
Test Suite
Execution Traces
Testin
g
Inferred Properties
CandidatePropertyPatterns
Infe
ren
ce
www.cs.virginia.edu/ipa 6
Candidate Property Patterns
• Response pattern [Dwyer, ICSE ‘99]
[-P]* (P [-S]* S [-P]*)*– Filtered version (all non P/S events
removed): S*(P+S+)*
SPPSPS SPSP • Too weak for our purposes
www.cs.virginia.edu/ipa 7
Partial Order of Patterns
MultiEffect
(PS+)*
MultiCause
(P+S)*
EffectFirst
S*(PS)*
Alternating
(PS)*
OneCause
S*(PS+)*
OneEffect
S*(P+S)*
CauseFirst
(P+S+)*
ResponseS*(P+S+)*
Str
icte
r
ΛΛΛ
Λ
3 P
rimitiv
ePatte
rns
4 D
eriv
ed
Patte
rns
www.cs.virginia.edu/ipa 8
Inferring Properties
• For all pairs of events, P and S:– Check which (if any) of
OneCause, CauseFirst and OneEffect are satisfied
– Infer the strictest property satisfied if more than one primitive is
MultiEffect(PS+)*
MultiCause
(P+S)*
EffectFirst
S*(PS)*
Alternating
(PS)*
OneCauseS*(PS+)*
OneEffect
S*(P+S)*
CauseFirst(P+S+)*
ΛΛΛ
Λ
www.cs.virginia.edu/ipa 9
Traces Example
Trace 1: PSPS Trace 2: PPS All Traces
CauseFirst
(P+S+)*
OneCauseS*(PS+)*
OneEffectS*(P+S)*
For any two events, determine the strictest pattern satisfied by all traces
www.cs.virginia.edu/ipa 10
Traces Example
Trace 1: PSPS Trace 2: PPS All Traces
CauseFirst
(P+S+)*
OneCauseS*(PS+)*
OneEffectS*(P+S)*
For any two events, determine the strictest pattern satisfied by all traces
www.cs.virginia.edu/ipa 11
Traces Example
Trace 1: PSPS Trace 2: PPS All Traces
CauseFirst
(P+S+)*
OneCauseS*(PS+)*
OneEffectS*(P+S)*
For any two events, determine the strictest pattern satisfied by all traces
CauseFirst OneEffect MultiCause
www.cs.virginia.edu/ipa 12
Implementation
ProgramInstrumented
Program
Instru
menta
tion
Test Suite
Execution Traces
Testin
g
Inferred Properties
CandidatePropertyPatterns
Infe
ren
ce
Automated for Java using JRat Method entry/exit eventsManually for C code or output (today’s examples)
Prototype (described in paper) 900 lines of PerlNew implementation 8K lines of Java Improved performance Analyses, ranking heuristics
www.cs.virginia.edu/ipa 13
Experiments
• Hypotheses– We can automatically extract interesting
temporal properties– Differences in inferred temporal properties
among multiple versions can reveal interesting things
• Target programs– Tour bus simulator (8 student submissions)– OpenSSL (0.9.6, 0.9.7-0.9.7d)
www.cs.virginia.edu/ipa 14
Tour Bus Simulator
• Bus and each passenger are a separate thread
• Assignment in Fall 2003 graduate-level course (before we started this project)
• 8 submissions from the instructor (all believed to be correct)
www.cs.virginia.edu/ipa 15
Testing
> cville_bus –N 2 –C 1 –T 2Bus waiting for trip 1Passenger 0 boardsBus drives around CharlottesvillePassenger 0 exitsBus waiting for trip 2Passenger 1 boardsBus drives around CharlottesvillePassenger 1 exitsBus stops for the day
N, the number of peopleC, the capacity of the busT, the number of trips
Executed each submission with 100 randomly generated inputs, where
20 < C ≤ 40C+1 ≤ N ≤ 2C1 ≤ T ≤ 10
Event traces extracted fromprogram output (no need to instrument program)
www.cs.virginia.edu/ipa 16
Differences Reveal Problems
Pattern7 Correct Versions
1 Faulty Version
Alternating
waitdrives
MultiEffect
drivesboardswaitexitswaitboards
waitdrives
waitboards
MultiCause
boardsdrives boardsdrives
CauseFirst
boardsexits boardsexitsdrivesexitswaitexits
– waitdrives not Alternating• Bus drives around before all passengers exit• Bug in locking code (misplaced synchronization)
– drivesexits, waitexits not MultiEffect• Bus drives around twice before letting passengers exit• Missing synchronization
www.cs.virginia.edu/ipa 17
OpenSSL
• Widely used implementation of the Secure Socket Layer protocol
• 6 versions [0.9.6, 0.9.7, 0.9.7a-d] between Sept 2000 and March 2004
• We focus on the handshake protocol– 38 different event types
www.cs.virginia.edu/ipa 18
SR_CLNT_HELLO SW_SRVR_HELLO
SW_CERT SW_KEY_EXCH SW_CERT_REQ
SW_SRVR_DONE
SR_CERT SR_KEY_EXCH
SR_CERT_VRFY SR_FINISHED
SW_CHANGE SW_FINISHED
BEFORE+ACCEPT
SW_FLUSH OK
SW_FLUSH
Client Server
www.cs.virginia.edu/ipa 19
Testing
• Manually instrumented server to record handshake events
• Executed each version of server with 1000 randomly generated clients– Client modified advance to a
randomly selected state with 5% probability
www.cs.virginia.edu/ipa 20
Inferred Alternating Patterns
0.9.6 0.9.7 0.9.7a
0.9.7b
0.9.7c
0.9.7d
SR_KEY_EXCHSR_CERT_VRFY
SW_CERTSW_KEY_EXCH
SW_SRVR_DONESR_CERT
Documented change:
ignore unrequeste
d client certificates
Fixed bug
causing server
crashes
Race condition
(present in all
versions)7 alternating patterns same for all versions
www.cs.virginia.edu/ipa 21
Partitioning Traces
All Traces
Correct Clients (never jump to random state) Faulty Clients
No Server Error Server Error
Follows SSL specification
Missing Alternating patterns:allows handshake cycle
Segmentation fault (<0.9.7d)
www.cs.virginia.edu/ipa 22
Performance• Prototype implementation (described in
paper)– Up to 3-4 minutes for examples (Pentium 4, 3GHz,
1G RAM)– Too slow (several days) for larger programs– Scales approximately as trace length events2
• New implementation– Scales approximately as trace length events– Analyze these experiments in < .5s
• Bus: 102 events, 100 traces, 222 events/trace: 0.4s• OpenSSL: 35 events, 1000 traces, 18 events/trace: 0.3s
– Feasible to analyze large systems• Thousands of different events• Traces with millions of events• Preliminary results from experiments with JBoss
www.cs.virginia.edu/ipa 23
Summary of Experiments• Useful in program evolution
– Reveal interesting changes in OpenSSL– Identify unexpected differences
• Revealed bugs– Tour bus: identified faulty implementation
• Multiple implementations are rare, but multiple representations are common (i.e. design, model, code)
– OpenSSL: • Differences between versions revealed• Flaws found by partitioning traces
www.cs.virginia.edu/ipa 24
Future Work/Research Questions
• Can we make this feasible in practice?– Automatically identify interesting events– Heuristics to identify important patterns– Scalability and automation
• Can we make it more useful?– More expressive patterns: more events,
combine with data-flow– Understand impact of different testing
strategies
www.cs.virginia.edu/ipa 25
Conclusion
• Automatically inferring temporal properties is feasible
• Even very simple property patterns reveal interesting program properties
www.cs.virginia.edu/ipa 26
Questions?
http://www.cs.virginia.edu/ipa
This work is funded in part by the National Science Foundation.
Thanks: Marty Humphrey for providing the student submissions, Chengdu Huang for help with OpenSSL, Joel Winstead, the anonymous reviewers, and anonymous CS650 students for writing buggy code.