CS590 Z

Matching Program Versions

Xiangyu Zhang

CS590Z

Problem Statement

Suppose a program P’ is created by modifying P. Determine the difference between P and P’. For an artifact c’ in P’, decide if c’ belongs to the difference, if not, find the correspondence of c’ in P.

• Static mapping• Non-trivial

Name comparison? What if

• Clone analysis, comparison checking

CS590Z

Motivations

Validate compiler transformations

Facilitate regression testing

Reverse obfuscation

Information propagation

Debugging

Code plagiarism detection

Information Assurance

CS590Z

Approaches

Static Approaches• Entity name based• String based (MOSS)• AST based (DECKARD)• CFG based (JDIFF)• PDG based (PDIFF)• Binary based (BMAT)• Log based (editor plugin, comparison checking)

Dynamic Approaches (not today)

CS590Z

Static Approaches

Entity name matching• Model a function/field as tuples• Coarse grained matching

String matching• Diff (CVS, Subservion)• Longest common subsequence (LCS)

Available operations are addition and deletion Matched pairs can not cross one another Programs are far more complicated than strings

Copy, paste, move• CP-Miner (scale to linux kernel clone detection)

Frequent subsequence mining

CS590Z

MOSS

Code plagiarism detection• It also handles other digital contents

Challenges• White space (variable name)• Noise (“the”, “int i”);• Order scrambling (paragraph reorders)

Problem statement• Given a set of documents, identify substring matches that

satisfy two properties: If there is a substring match at least as long as the guarantee threshold

t, then this match is detected; Do not detect any matches shorter than the noise threshold, k.

CS590Z

MOSS

k-gram• A continuous substring of length k

CS590Z

MOSS

Incremental hashing• Hashing strings of length k is expensive for large k.• “rolling” hash function

The (i+1)th k-gram hash = F (the ith k-gram hash, …)

CS590Z

MOSS

Fingerprint selection• A subset of hash values

• Our goals: find all matching substrings >t; ignore matchings <k)

• One of every tth hash values• 0 mod p

CS590Z

MOSS

Winnowing• Observation: given a sequence of hashes h1,…hn, if n>t-k,

then at least one of the hi must be chosen• Have a sliding window with size w=t-k+1• In each window select the minimum hash value, break ties

by select the rightmost occurrence.

CS590Z

MOSS

Algorithm• Build an index mapping fingerprints to locations for all

documents.• Each document is fingerprinted a second time and the

selected fingerprints are looked up in the index; this gives the list of all matching fingerprints for each document.

• Sort (d,d1,fx), (d, d2,fy) by the first two elements. • Matches between documents are rank-ordered by size

(number of fingerprints)

CS590Z

MOSS

Advantages• Guarantee to detect any >t substring matches

Limitations• Minor edits fail MOSS.

x= a*b + c vs. z= c + a*b• Insertion, deletion

CS590Z

AST based matching

[YANG, 1991, Software Practice and Experience]• Given two functions, build the ASTs• Match the roots• If so, apply LCS to align subtrees• Continue recursively

Fragile

CS590Z

DECKARD (ICSE 2007)

CS590Z

DECKARD

Advantages• Scalability• Insensitive to minor structural changes such as reordering,

insertion, deletion

Limitations• Structural similarity only• Insertion that incurs structure change.

CS590Z

CFG matching

Hammock graph (JDIFF ,ASE 2004)• Match classes by names• Match fields by types• Match methods by signatures• Match instruction in methods by hammock graphs

A hammock is a single entry single exit subgraph of a CFG.

CS590Z

CFG matching

Pros• Orthogonal

Can be combined with other matching techniques• Simple

Cons• Coarse grained matching only

Not good at clone detection• In case of code transformation

CS590Z

Semantic Based Matched

Using PDG (SAS’01)

CS590Z

Semantic Based

CS590Z

Semantic Based

Pros• Non-contiguous, intertwined, reordered• Insensitive to code transformations.

Cons• Scalability

Points-to analysis• Starting from a matching pair seems to be a problem

CS590Z

Wrap Up

For clone detection• Maybe structural / text similarity is a good idea

For whole program matching / method matching with code transformations

• Semantic based is more appropriate

Scalability • PDG < CFG | AST < STRING < NAME