Lund University - cs.lth.se/markus_borg/

Recommendation Systems for Issue ManagementMARKUS BORG, SOFTWARE ENGINEERING RESEARCH GROUP

Lund University - cs.lth.se/markus_borg/

• PhD student 2010-• Software engineering• Requirements-test alignment• Member of SWELL

• Supervisor: Per Runeson

Markus Borg

Per Runeson

• MSc CS and Eng. 2007• Developer, ABB (2007-2010)

• Compiler and editor development• Safety-critical systems• Embedded development

Lund University - cs.lth.se/markus_borg/

Agenda

• Background– Bug report inflow in large projects– Recommendation systems

• Overview of some ongoing bug research– Duplicate detection– Team assignment– Effort prediction– Change impact analysis

Lund University - cs.lth.se/markus_borg/

There Will Be Bugs!

Lund University - cs.lth.se/markus_borg/

Issue Reports as Batons in the Development “Relay Race”

Release

Change Board

Project managerTester

Issue handover

Developer

Developer Tester

Developer

Architect

Analyst

Designer

Lund University - cs.lth.se/markus_borg/

Challenges in the Bug Overflow

• 10 000s of bug reports in large projects– Continous inflow of bug reports– Limited resources to deal with it

• Q1. Which issues do we already know about?• Q2. Who should investigate the issue?• Q3. How long time will it take to correct this bug?• Q4. If we make a corrective change, what is the impact?

Lund University - cs.lth.se/markus_borg/

Recommendation System

• General definition

“a software application that aim to support users in their decision-making while interacting with large information spaces”

– Amazon, Netflix, RateBeer, Filmtipset...

• In software engineering”a software application that provides information items estimated to be valuable for a software engineering task in a given context”

Lund University - cs.lth.se/markus_borg/

Two main approaches to recommendations

Bug example

Lund University - cs.lth.se/markus_borg/

Q1. Which issues do we already know about?

Duplicate Detection

Lund University - cs.lth.se/markus_borg/

Issue Duplicate Detection

Lund University - cs.lth.se/markus_borg/

Network Analysis of Bug Reports

Lund University - cs.lth.se/markus_borg/

Subgraph Analysis in Bug Networks

Bug star

One central bug report pointing at several others

Dense ringMost bug reports are connected

Lund University - cs.lth.se/markus_borg/

Textual Similarity Analysis

1. Study natural language content

2. Calculate overlap

3. Recommend duplicates

Lund University - cs.lth.se/markus_borg/

Calculating Textual Similarity

Issue reports

Represent

Calculate angles in vector space

Lund University - cs.lth.se/markus_borg/

Reported Results

• Results typically reported as:– ”% found among top 10 recommendations”

• Sony Ericsson 40%• Firefox 60%, 90%• Eclipse 40%, 70%• Open Office 60%, 65%• Mozilla Foundation 45%, 65%• Android 60% (Jens?)

• Are these good results? Good question.

Lund University - cs.lth.se/markus_borg/

Q2. Who should investigate the issue?

Team assignment

Lund University - cs.lth.se/markus_borg/

Bug Assignment

Bug tossing!

Lund University - cs.lth.se/markus_borg/

Machine Learning to Find Patterns

• Bugs found in India late in testing phase are severe?• Bugs reports containing term ”termination” are costly?• Bugs reported on fridays with attached stack traces tend to

be deferred to later releases?

Lund University - cs.lth.se/markus_borg/

Automated Bug Assignment – Supervised Machine Learning

Machine learning

Train

Recommend team

Lund University - cs.lth.se/markus_borg/

How to Represent a Bug Report?

• Function• Severity• System Version• Submit Date• Submitter Location

etc.• … And the text! Title and description.

Lund University - cs.lth.se/markus_borg/

Experimental Setup

Ericsson

Company A

Machine learning

4 x

Pre-processing & feature selection

25-35 teams to choose between!

Lund University - cs.lth.se/markus_borg/

Results

• Prediction accuracy using state-of-the-art machine learning:– Ericsson 30-70%– Company A 30%

• At Ericsson– In line with accuracy of human assignment– Deploying tool in selected projects

• Textual features improve predictions, but training times increase a lot!– Working on finding a good balance

Lund University - cs.lth.se/markus_borg/

Q3. How long time will it take to correct this bug?

Effort prediction

Lund University - cs.lth.se/markus_borg/

Bug Report Clustering – Unsupervised Machine Learning

Automatically cluster bug reports based on natural language in report

Lund University - cs.lth.se/markus_borg/

Prediction of Bug Resolution Time

1. A new bug is submitted

2. Put it in the right cluster

3. Predict its resolution time

Lund University - cs.lth.se/markus_borg/

Results

• Clustered bug reports based on textual content– Android– Eclipse– Company A

• Average resolution times in bug clusters are different– Statistically significant

• Is it a good way to predict resolution times?– No, at least not yet…

Lund University - cs.lth.se/markus_borg/

Q4. If we make a corrective change, what is the impact?

Change impact analysis

Lund University - cs.lth.se/markus_borg/

Recommendations for Impact Analysis

Mandated by safety standards!

Reqs.Tests

Lund University - cs.lth.se/markus_borg/

Networks of Impacted Artifacts

Data mining in archive of previous impact analysis reports

Lund University - cs.lth.se/markus_borg/

Identification of Key Artifacts

Lund University - cs.lth.se/markus_borg/

Recommend Possible Impact

1. Textual similarity analysis

2. Follow links in network

Lund University - cs.lth.se/markus_borg/

Recommend Possible Impact

3. Rank recommendations– Network statistics– Textual similarity

Lund University - cs.lth.se/markus_borg/

Lund University - cs.lth.se/markus_borg/

Results

• Experiments in a laboratory setting– 30% of previous impact among top 5 recommendations– 40% among top 10 recommendations

• Prototype evaluation in industrial setting ongoing...

Lund University - cs.lth.se/markus_borg/

Summary

Lund University - cs.lth.se/markus_borg/

Summary

• Issue inflow in large projects can be daunting

• On the other hand, issue reports contain important information!

• With the right tools, the information overflow can be tamed to recommendations!