![Page 1: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/1.jpg)
An Implicit Feedback-based Approach to the Evaluation of Text Analysis
Techniques for Software Engineering
Kostadin Damevski, David Shepherd, Lori Pollock
![Page 2: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/2.jpg)
Evaluating Code Search…
![Page 3: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/3.jpg)
3
Common Approach: Eliminate developers from the evaluation
![Page 4: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/4.jpg)
4
Common Approach: Eliminate developers from the evaluation
The problem with “gold sets”…• Time consuming to
create• Created by
researchers, not developers
• Subjective and context dependent
![Page 5: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/5.jpg)
5
JBidMouse.DoSave(Component)JBidMouse.DoAction(Object, String, AuctionEntry)AuctionServer.registerAuction(AuctionEntry) AuctionsManager.saveAuctions()AuctionsManager.backupByDate(String, File) AuctionsManager.preserveFiles(String)AuctionsManager.ensureDirectories(String) AuctionsManager.buildSaveBuffer(XMLElement, XMLElement) AuctionsManager.needSwapSaves(String) AuctionsManager.makeBackupFilename(String String)
LET’S CREATE A GOLD SET for “save auctions”
from jBidWatcher (183 classes)
![Page 6: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/6.jpg)
6
Our Approach: Humans “Taste-Test” Competing techniques*
*Used to evaluate web search engines
![Page 7: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/7.jpg)
7
A B
A Taste Test for Code Search
![Page 8: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/8.jpg)
8
A Taste Test for Code Search
The interface does not betray which FLT (A or B) the results Originated from
![Page 9: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/9.jpg)
9
A Taste Test for Code Search
Count clicks to determine a statistically significant preference
![Page 10: An Implicit Feedback-based Approach to the Evaluation of Text Analysis Techniques for Software Engineering Kostadin Damevski, David Shepherd, Lori Pollock](https://reader036.vdocument.in/reader036/viewer/2022081514/5697bfac1a28abf838c9bc17/html5/thumbnails/10.jpg)
Next Steps
Verify approach works
Widespread data collection*
Use paired interleaving to compare lexical approach to IR-based approach, knowing that the IR-approach should perform better.
Compare variations on approaches, such as use of different splitters, via widespread distribution and data collection.
* Use our Sando Search tool (http://sando.codeplex.com)