automatically inferring temporal properties for program evolution jinlin yang and david evans 15 th...

Automatically Inferring Temporal Properties for Program Evolution

Jinlin Yang and David Evans15th IEEE International Symposium on Software Reliability Engineering 5 November 2004Saint-Malo, FranceUniversity of

VirginiaComputer

Science

www.cs.virginia.edu/ipa 2

Temporal Properties• Constrain ordering of events

– Essential for program correctness (file open before read)

• Reveal important differences between programs: S P not a chef

• But, hard for humans to document correctly [Holzmann’s FSE 2002 keynote]

• Can we infer useful temporal properties automatically?This talk argues: Yes!

www.cs.virginia.edu/ipa 3

Dynamically Inferring Properties

• Inherently unsound: guessing properties of all executions by looking at a small number

• Value-based invariants– Daikon [Ernst, TSE, Feb 01]

• Temporal properties– Mining specification [Ammons, POPL 02]– Extracting component interfaces [Whaley, ISSTA

02]

• We focus on relationship between 2 or 3 events: automation, scalability

www.cs.virginia.edu/ipa 4

System Overview

Inferred Properties 1

DifferentProperties

Diff

ere

nce A

naly

zer

Inferred Properties 2

SharedPropertiesProgram

Version 1

Program Version 2

Dyn

am

ic In

fere

nce

… …

www.cs.virginia.edu/ipa 5

Property Inference

Inferred Properties 1

DifferentProperties

Diff

ere

nce A

naly

zer

Inferred Properties 2

SharedPropertiesProgram

Version 1

Program Version 2

Dyn

am

ic In

fere

nce

… …

ProgramInstrumented

Program

Instru

menta

tion

Test Suite

Execution Traces

Testin

g

Inferred Properties

CandidatePropertyPatterns

Infe

ren

ce

www.cs.virginia.edu/ipa 6

Candidate Property Patterns

• Response pattern [Dwyer, ICSE ‘99]

[-P]* (P [-S]* S [-P]*)*– Filtered version (all non P/S events

removed): S*(P+S+)*

SPPSPS SPSP • Too weak for our purposes

www.cs.virginia.edu/ipa 7

Partial Order of Patterns

MultiEffect

(PS+)*

MultiCause

(P+S)*

EffectFirst

S*(PS)*

Alternating

(PS)*

OneCause

S*(PS+)*

OneEffect

S*(P+S)*

CauseFirst

(P+S+)*

ResponseS*(P+S+)*

Str

icte

r

ΛΛΛ

Λ

3 P

rimitiv

ePatte

rns

4 D

eriv

ed

Patte

rns

www.cs.virginia.edu/ipa 8

Inferring Properties

• For all pairs of events, P and S:– Check which (if any) of

OneCause, CauseFirst and OneEffect are satisfied

– Infer the strictest property satisfied if more than one primitive is

MultiEffect(PS+)*

MultiCause

(P+S)*

EffectFirst

S*(PS)*

Alternating

(PS)*

OneCauseS*(PS+)*

OneEffect

S*(P+S)*

CauseFirst(P+S+)*

ΛΛΛ

Λ

www.cs.virginia.edu/ipa 9

Traces Example

Trace 1: PSPS Trace 2: PPS All Traces

CauseFirst

(P+S+)*

OneCauseS*(PS+)*

OneEffectS*(P+S)*

For any two events, determine the strictest pattern satisfied by all traces

www.cs.virginia.edu/ipa 10

Traces Example

Trace 1: PSPS Trace 2: PPS All Traces

CauseFirst

(P+S+)*

OneCauseS*(PS+)*

OneEffectS*(P+S)*

For any two events, determine the strictest pattern satisfied by all traces

www.cs.virginia.edu/ipa 11

Traces Example

Trace 1: PSPS Trace 2: PPS All Traces

CauseFirst

(P+S+)*

OneCauseS*(PS+)*

OneEffectS*(P+S)*

For any two events, determine the strictest pattern satisfied by all traces

CauseFirst OneEffect MultiCause

www.cs.virginia.edu/ipa 12

Implementation

ProgramInstrumented

Program

Instru

menta

tion

Test Suite

Execution Traces

Testin

g

Inferred Properties

CandidatePropertyPatterns

Infe

ren

ce

Automated for Java using JRat Method entry/exit eventsManually for C code or output (today’s examples)

Prototype (described in paper) 900 lines of PerlNew implementation 8K lines of Java Improved performance Analyses, ranking heuristics

www.cs.virginia.edu/ipa 13

Experiments

• Hypotheses– We can automatically extract interesting

temporal properties– Differences in inferred temporal properties

among multiple versions can reveal interesting things

• Target programs– Tour bus simulator (8 student submissions)– OpenSSL (0.9.6, 0.9.7-0.9.7d)

www.cs.virginia.edu/ipa 14

Tour Bus Simulator

• Bus and each passenger are a separate thread

• Assignment in Fall 2003 graduate-level course (before we started this project)

• 8 submissions from the instructor (all believed to be correct)

www.cs.virginia.edu/ipa 15

Testing

> cville_bus –N 2 –C 1 –T 2Bus waiting for trip 1Passenger 0 boardsBus drives around CharlottesvillePassenger 0 exitsBus waiting for trip 2Passenger 1 boardsBus drives around CharlottesvillePassenger 1 exitsBus stops for the day

N, the number of peopleC, the capacity of the busT, the number of trips

Executed each submission with 100 randomly generated inputs, where

20 < C ≤ 40C+1 ≤ N ≤ 2C1 ≤ T ≤ 10

Event traces extracted fromprogram output (no need to instrument program)

www.cs.virginia.edu/ipa 16

Differences Reveal Problems

Pattern7 Correct Versions

1 Faulty Version

Alternating

waitdrives

MultiEffect

drivesboardswaitexitswaitboards

waitdrives

waitboards

MultiCause

boardsdrives boardsdrives

CauseFirst

boardsexits boardsexitsdrivesexitswaitexits

– waitdrives not Alternating• Bus drives around before all passengers exit• Bug in locking code (misplaced synchronization)

– drivesexits, waitexits not MultiEffect• Bus drives around twice before letting passengers exit• Missing synchronization

www.cs.virginia.edu/ipa 17

OpenSSL

• Widely used implementation of the Secure Socket Layer protocol

• 6 versions [0.9.6, 0.9.7, 0.9.7a-d] between Sept 2000 and March 2004

• We focus on the handshake protocol– 38 different event types

www.cs.virginia.edu/ipa 18

SR_CLNT_HELLO SW_SRVR_HELLO

SW_CERT SW_KEY_EXCH SW_CERT_REQ

SW_SRVR_DONE

SR_CERT SR_KEY_EXCH

SR_CERT_VRFY SR_FINISHED

SW_CHANGE SW_FINISHED

BEFORE+ACCEPT

SW_FLUSH OK

SW_FLUSH

Client Server

www.cs.virginia.edu/ipa 19

Testing

• Manually instrumented server to record handshake events

• Executed each version of server with 1000 randomly generated clients– Client modified advance to a

randomly selected state with 5% probability

www.cs.virginia.edu/ipa 20

Inferred Alternating Patterns

0.9.6 0.9.7 0.9.7a

0.9.7b

0.9.7c

0.9.7d

SR_KEY_EXCHSR_CERT_VRFY

SW_CERTSW_KEY_EXCH

SW_SRVR_DONESR_CERT

Documented change:

ignore unrequeste

d client certificates

Fixed bug

causing server

crashes

Race condition

(present in all

versions)7 alternating patterns same for all versions

www.cs.virginia.edu/ipa 21

Partitioning Traces

All Traces

Correct Clients (never jump to random state) Faulty Clients

No Server Error Server Error

Follows SSL specification

Missing Alternating patterns:allows handshake cycle

Segmentation fault (<0.9.7d)

www.cs.virginia.edu/ipa 22

Performance• Prototype implementation (described in

paper)– Up to 3-4 minutes for examples (Pentium 4, 3GHz,

1G RAM)– Too slow (several days) for larger programs– Scales approximately as trace length events2

• New implementation– Scales approximately as trace length events– Analyze these experiments in < .5s

• Bus: 102 events, 100 traces, 222 events/trace: 0.4s• OpenSSL: 35 events, 1000 traces, 18 events/trace: 0.3s

– Feasible to analyze large systems• Thousands of different events• Traces with millions of events• Preliminary results from experiments with JBoss

www.cs.virginia.edu/ipa 23

Summary of Experiments• Useful in program evolution

– Reveal interesting changes in OpenSSL– Identify unexpected differences

• Revealed bugs– Tour bus: identified faulty implementation

• Multiple implementations are rare, but multiple representations are common (i.e. design, model, code)

– OpenSSL: • Differences between versions revealed• Flaws found by partitioning traces

www.cs.virginia.edu/ipa 24

Future Work/Research Questions

• Can we make this feasible in practice?– Automatically identify interesting events– Heuristics to identify important patterns– Scalability and automation

• Can we make it more useful?– More expressive patterns: more events,

combine with data-flow– Understand impact of different testing

strategies

www.cs.virginia.edu/ipa 25

Conclusion

• Automatically inferring temporal properties is feasible

• Even very simple property patterns reveal interesting program properties

www.cs.virginia.edu/ipa 26

Questions?

http://www.cs.virginia.edu/ipa

This work is funded in part by the National Science Foundation.

Thanks: Marty Humphrey for providing the student submissions, Chengdu Huang for help with OpenSSL, Joel Winstead, the anonymous reviewers, and anonymous CS650 students for writing buggy code.