The Role of Patch Review in Software Evolution: An Analysis of the Mozilla

FirefoxIWPSE-EVOL 2009, Amsterdam, The Netherlands

Mehrdad NurolahzadeSeyed Mehdi Nasehi Shahedul Huq Khandkar Shreya Rawal

Patch Review

• The process of incrementally submitting and integrating patches (fix for a bug report) into a software system is a core activity related to software evolution.

• It act as a basic mechanism for validating the design and implementation of patches.

• It happens in both, commercial and FLOSS projects.

2

Research Questions

• What are the roles of various people involved in the patch review process?

• What is the process of conducting reviews? • When are reviews performed? • What do reviewers look at and what do they

miss?

3

Mozilla Firefox

• One of the successful FLOSS projects• Second most popular web browser

• Has both open source and commercial characteristics • 37% of code is contributed by the community

• 40 developers in Mozilla Corporation Firefox development team

• 100 daily contributors (approx)• 1000 contributors (approx)• 20-30 million users (approx)• 100,000 lines of code (approx)

4Source: Mike Beltzner, Mozilla Corporation, Web 2.0 Expo 2007

Software Evolution in Mozilla

5

Mozilla Development Process

Check-in

Review

Develop Patch

Discuss

File a bug

6

review?, super-review?, ui-review?

review+, super-review+, ui-review+

negative review

backed out by QA

Overview of Research Approach

7

Codes:ReviewNegative ReviewRubber StampMerciful reviewer

Open Coding

Bug Reports

Tool for Qualitative Analysis

Observations and Patterns

Themes

Quantification

Database

Online Mozilla Bugzilla Documentation

Process

Data Collection

8

History (Status Changes)

Attachments (Patches)

Discussion and Comments

Bug Report

Tool for Qualitative Analysis

9

Data Set

10

• Bug Reports (112) (2002-2009)–66 Bugs–38 Enhancements–8 New Features

• Patches (310)–Comments (1842)• 318 Reviews

Findings

Patterns• Patchy Patcher• Merciful Reviewer• Doubtful reviewer

Observations• Module Owner Reviews• Peer Reviews• Types of Feedback• Undiscovered Errors

11

Patterns: Patchy Patcher

12

• Submits work-in-progress patches– To get early feedback

“incomplete, still waiting for Feedback” (Bug # 343172)

– To show involvement “New patch coming soon…” (Bug

# 456214)“wip:0.1 works, but I need to fix

…” (Bug # 390614)

Patterns: Merciful Reviewer

• When reviewers are reluctant to give a review minus (review-).• 56 negative reviews• 42 review-• Where did the 14 reviews go?

• After the review comments the reviewer changed the status from ‘review?’ to nothing.

Hypothesis: Module Owners refrain from giving review- to Newcomers

13

Pattern: Doubtful Reviewer

• There has not been enough early feedback from community.

• A change to functionality is being made.

“Let's put this in for beta, and make sure we blog about the change a little and see the reaction” (Bug # 412862)

14

Observation: Module Owner Reviews

• Every completed patch has to be reviewed by Module Owners unless somebody finds a flaw in it.

• 38 Module Owners• 198 Code or UI inspections• Module owners do not consider patches

based on bug report priority (but developers do).

15

Observation: Peer Reviews

• 66 Peer Developers• Most patches (76%) do not get any Peer

Review• 80% of the comments by peers are negative or

partly negative• On an average Peer conducts review before

Module Owners

16

Observation: Types of Reviewer Feedback

# Module Owner Comments # Peer Comments

On Implementation 63 45

On Functionality and Usability 6 31

On Documentation 9 1

On Coding Standards 7 3

17

Hypothesis: While peers are interested in functionality and usability, Module Owners are also interested in long-term maintainability of the product.

Observation: Undiscovered Errors

• 8.4% of checked-in patches were backed out.• Performance and regression issues that remain

unidentified during the review.• It would be interesting to look if there a common

pattern in occurrence of undiscovered errors? "Based on the site breakage, re-opening and suggesting we back out

this change.“ (Bug # 412862)

18

Conclusion

Peers play a key role in the process by providing ideas before a patch is developed and reviewing developed patches before module owners do.

19

Conclusion

Module owners are more inclined toward the quality and consistency of developed patches.

20

Take Away

Module owners and peer developers are

complementing each other in reviewing patches. They refine the developed solution based on their own interests and concerns.

21

22

Debate Questions

• How quantitative/qualitative analysis of software engineering data can be facilitated?

• How empirical findings can be cross-validated with the FLOSS development communities?