An Implicit Feedback-based Approach to the Evaluation of Text Analysis

Techniques for Software Engineering

Kostadin Damevski, David Shepherd, Lori Pollock

Evaluating Code Search…

3

Common Approach: Eliminate developers from the evaluation

4

Common Approach: Eliminate developers from the evaluation

The problem with “gold sets”…• Time consuming to

create• Created by

researchers, not developers

• Subjective and context dependent

5

JBidMouse.DoSave(Component)JBidMouse.DoAction(Object, String, AuctionEntry)AuctionServer.registerAuction(AuctionEntry) AuctionsManager.saveAuctions()AuctionsManager.backupByDate(String, File) AuctionsManager.preserveFiles(String)AuctionsManager.ensureDirectories(String) AuctionsManager.buildSaveBuffer(XMLElement, XMLElement) AuctionsManager.needSwapSaves(String) AuctionsManager.makeBackupFilename(String String)

LET’S CREATE A GOLD SET for “save auctions”

from jBidWatcher (183 classes)

6

Our Approach: Humans “Taste-Test” Competing techniques*

*Used to evaluate web search engines

7

A B

A Taste Test for Code Search

8

A Taste Test for Code Search

The interface does not betray which FLT (A or B) the results Originated from

9

A Taste Test for Code Search

Count clicks to determine a statistically significant preference

Next Steps

Verify approach works

Widespread data collection*

Use paired interleaving to compare lexical approach to IR-based approach, knowing that the IR-approach should perform better.

Compare variations on approaches, such as use of different splitters, via widespread distribution and data collection.

* Use our Sando Search tool (http://sando.codeplex.com)