predictors of customer perceived software quality
Post on 25-Feb-2016
42 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Predictors of customer perceived software quality
Paul Luo Li (ISRI – CMU) Audris Mockus (Avaya Research)Ping Zhang (Avaya Research)
2
Need to View Quality from the Customer’s Perspective
… We translate these advanced technologies into value for our customers …
-IBM (#9 on the Fortune 500)
… Our strategy is to offer products, services and solutions that are high tech, low cost and deliver the
best customer experience. -HP (#11 on the Fortune 500)
… We deliver unparalleled value to our customers. Only by serving our customers well do we justify our existence as a business
-Avaya (#401 on the Fortune 500)
3
What Would be Ideal
Predict customer perceived qualityUsing customer characteristicsFor each customer
Key idea: Focus on the customer
4
Possible Applications of Predictions
How do I plan deployment to meet the quality expectations of the customer?
How do I target improvement efforts?
How do I allocate the right resources to deal with customer problems
Predict customer experience for each customer
Identify possible causes of problems
Predict customer interactions
5
Solutions for Software Producers
How do I plan deployment to meet the quality expectations of the customer?
How do I target improvement efforts?
How do I allocate the right resources to deal with customer problems
Predict customer experience for each customer
Identify possible causes of problems
Predict customer interactions
6
To Improve Customer Perceived Quality
How do I plan deployment to meet the quality expectations of the customer?
How do I target improvement efforts?
How do I allocate the right resources to deal with customer problems
Predict customer experience for each customer
Identify possible causes of problems
Predict customer interactions
7
Gaps in Current Research
Prior work examined: Software defect prediction for a single
customer (Musa et al. 1987, Lyu et al. 1996)
Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996)
Is not scalable
8
Not Focused on Customers
Prior work examined: Software defect prediction for a single
customer (Musa et al. 1987, Lyu et al. 1996)
Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996)
Tell us nothing about a specific customer
9
Does not Capture other Aspects of Customer Perceived Quality Prior work examined:
Software defect prediction for a single customer (Musa et al. 1987, Lyu et al. 1996)
Software defect prediction for modules or features (Jones et al. 1999, Khoshgoftaar et al. 1996)
Does not predict other aspects of customer perceived quality that are not code related.
10
Research Contributions
Predict software defects for each customer in a cost effective manner
Predict other aspects of customer perceived quality for each customer
Empirically validate deployment, usage, software, and hardware predictors
11
Rest of This Talk
The setting Customer interactions (outputs) Customer characteristics (inputs) Results Conclusion
12
Empirical Results from a Real World Software System Avaya telephone call processing software
system7 million+ lines of C/C++Fixed release schedule
Process improvement efforts Tens of thousands of customers
90% of Fortune 500 companies use it Professional support organization
13
Data Used are Commonly Available
Customer issue tracking system Trouble ticket database
The equipment database Change management
Sablime database
Data collected as a part of everyday operations
Data sources available at other organizations e.g.
IBM and HP
14
Data collected as a part of everyday operations
Data sources available at other organizations e.g.
IBM and HP
At Other Organizations Customer issue tracking system
Trouble ticket database The equipment database Change management
Sablime database
15
Customer Interactions (Outputs) We assume customer interaction == customer
perceived quality Five customer interaction (Chulani et al. 2001,
Buckley and Chillarege 1995) within 3 month of deployment Software defects: high impact problem System outages: high impact problem Technician dispatches Calls Automated alarms
Important for Avaya and likely for other organizations as well
16
Examine Customer Installations
Months after general availability
Num
ber o
f dep
loym
ents
1
5
17
Capture Characteristics of Each Installation
Months after general availability
Num
ber o
f dep
loym
ents
1
5
Customer 1: Deployed first month, a Large system, Linux…Customer 2: Deployed first month, a Small system, Windows…Customer 3: Deployed first month, a Large system, Proprietary Os…Customer 4: Deployed first month, a Small system, Linux…Customer 5: Deployed first month, a Large system, Linux…
18
Analyze Using Statistical Analysis
Months after general availability
Num
ber o
f dep
loym
ents
1
5
Customer 1: Deployed first month, a Large system, Linux…Customer 2: Deployed first month, a Small system, Windows…Customer 3: Deployed first month, a Large system, Proprietary Os…Customer 4: Deployed first month, a Small system, Linux…Customer 5: Deployed first month, a Large system, Linux…
Similarities Differences
19
Category of Predictors (Kinds of Inputs) We examine:
Deployment issues Usage patterns Software platform Hardware configurations
Prior work examines: Software product Development process
Common sense issues, but lack empirical validation
20
Category of Predictors (Kinds of Inputs) We examine:
Deployment issues Usage patterns Software platform Hardware configurations
Prior work examines: Software product Development process
Key idea: From the customer’s perspective, they are not good predictors (i.e. do not vary for a single release)
21
Specific Predictors (Inputs) Total deployment time
deployment issues Operating system
software platform, hardware configurations System size
hardware configurations, software platform, usage patterns
Ports usage pattern, hardware configurations
Software upgrades deployment issue
22
Recap
Predict for each customer (outputs): Software defects System outages Technician dispatches Calls Automated alarms
Using Logistic regression and Linear regression
Using predictors (inputs): Total deployment time Operating system System size Ports Software upgrades
For a real world software system
23
Example: Field Defect Predictions
24
Predictors
25
Nuisance Variables
26
All Predictors are Important
27
The Most Important Predictor
Total deployment time (deployment issue) Systems deployed half way into our observational
period are 13 to 25 times less likely to experience a software defect
28
May Enable Deployment Adjustments
Total deployment time (deployment issue) Systems deployed half way into our observational
period are 13 to 25 times less likely to experience a software defect
May be due to software patching, better tools, more experienced technicians
29
Another Important Predictor
Total deployment time (deployment issue) Operating system (software platform, hardware
configurations) Systems running on the proprietary OS are 3 times
less likely to experience a software defect compared with systems on running the open OS (Linux)
Systems running on the commercial OS (Windows) are 3 times more likely to experience a software defect compared with systems running on the open OS (Linux)
30
May Allow for Targeted Improvement or Improved Testing Total deployment time (deployment issue) Operating system (software platform, hardware
configurations) May be due to familiarity with the operating system May be due to operating system complexity
31
More Results in Paper
The complete results and analyses for field defects
Predictions for other customer interactions
32
Validation of Results and Method
We accounted for data reporting differences Included indicator variables in the models to identify
populations (e.g. US or international customers) We independently validated the data collection
process Independently extracted data and performed analyses
We interviewed personnel to validate findings Programmers Field technicians
33
Summary: Identified Predictors of Customer Perceived Quality We identified and quantified characteristics,
like time of deployment, that can affect customer perceived quality by more than an order of magnitude
34
Summary: Modeled Customer Interactions
We identified and quantified characteristics , like time of deployment, that can affect customer perceived quality by more than an order of magnitude
We created models that can predict various customer interactions and found that predictors have consistent effect across interactions
35
Summary: Deployment is Important for High Reliability We identified and quantified characteristics ,
like time of deployment, that can affect customer perceived quality by more than an order of magnitude
We created models that can predict various customer interactions and found that predictors have consistent effect across interactions
We learned that controlled deployment may be the key for high reliability systems
36
Improve Customer’s Experiences
You can target improvement efforts You can allocate the right resources to deal
with customer reported problems You can adjust deployment to meet the quality
expectations of your customers
37
Predictors of customer perceived software quality
Paul Luo Li (paul.li@cs.cmu.edu) Audris Mockus (Avaya Research)Ping Zhang (Avaya Research)
38
Predicted Number of Calls Match Actual Number of Calls
Calls for the next release
Calls
Time
Predictions are made here
top related