msc software maintenance ms viðhald hugbúnaðar

20/04/23 Dr Andy Brooks 1

MSc Software MaintenanceMS Viðhald hugbúnaðar

Fyrirlestrar 3 & 4

Measurements to Manage Software Maintenance

mælingar


Case StudyDæmisaga

MWSSSThe Missile Warning and Space Surveillance Sensors program.

ReferenceMeasurements to Manage Software Maintenance, George E Stark,CROSSTALK The Journal of Defense Software Engineering, July 1997http://stsc.hill.af.mil/crosstalk/about.html


In 1994

• The MWSSS program management office took over the maintenance of seven systems.

• Software running in 10 locations worldwide.• 8.5 million source lines of code.• Software written in 22 different languages.• Latest system became operational in 1992.• To manage and understand the software

maintenance effort, a measurement program was started based on Basili´s Goal-Question-Metric paradigm.


Goal-Question-Metric

Goal Question Metric(s)

Maximize Customer Satisfaction

How many problems affect the customer?

Current Change Backlog

Software Reliability

How long does it take to fix an Emergency or Urgent problem?

Change Cycle Time from Date Approved and from Date Written


Goal-Question-Metric

Goal Question Metric(s)

Minimize Cost

How much does a software maintenance delivery cost?

Cost per delivery

How are the costs allocated? Cost per activity

What kinds of changes are being made?

Number of Changes by Type

How much effort is expended per change type?

Staff Days Expended/Change Type

How many invalid change requests are evaluated?

Percentage of Invalid Change Requests Closed Each Quarter


Goal-Question-MetricGoal Question Metric(s)

Minimize Schedule

How difficult is the delivery?

Complexity Assessment

Software Maintainability*

Computer Resource Utilization*

How many changes are made to the planned delivery content?

Percentage Content Changes by Delivery

Are we meeting our delivery schedules?

Percentage of On-time Deliveries

Also addresses the goal of maximizing customer satisfaction.


Using metrics information

• Attention can be directed to problems that need to be addressed.

• Supports making decisions about how to solve problems.

• Keeps everyone informed of progress– “how are they doing?”


MWSSS Software Maintenance Process

Overview from the CROSSTALK article.


Activities in the Maintenance Process

• An analyst at the user´s location checks problem reports for completeness and duplications, and that the user has properly understood the system.

• Further, this analyst categorizes the problem as either a software, hardware, or communications equipment problem.

• For software problems, a software change form (SCF) is created:– 18 items including proposed change, justification, and

resource estimates.



• A software maintenance engineer independently evaluates each SCF:– estimates effort based on taxonomy of change types– estimates computer resources required– estimates impact on quality of service

• If the independent evaluation of effort is 20% greater or less than the original estimate, the user and engineer meet to resolve the difference.

• SCFs can be generated for systems by teams who are not directly responsible for these systems but who are otherwise dependent on them.

• The User Configuration Control Board categorizes an SCF as either a modification or as a fix (fault correction).



• The User Configuration Control Board recommends a priority to the SCF:– emergency to avoid downtime or meet high-

priority mission requirements– urgent if needed in next delivery or to fix a

problem arising from a change in operations– routine e.g. unit conversions, default value

changes, fixing printouts



• Version content for next release is negotiated involving the reviewing of release complexity, software reliability, software maintainability, and computer resource utilization.

• Maintenance Configuration Control Board reviews the release plan and schedules the release.

• The software engineering team completes the design, code, test, installation, and quality assurance of the release.


Metric: Current Change Backlog


Metric: Current Change Backlog

• “Managers use this chart to allocate computer and staff resources, plan release content, and track the effect of new tools or other process improvement programs over time.”

• It is better to be in a state of equilibrium i.e. incoming change requests are closed in the next release (zero backlog)

• Chart indicates that several releases were on schedule.

• Chart based on data for one project.


Metric: Software Reliability



• Downtime incidents as a result of a software failure are counted (as reported in monthly maintenance logs).

• Chart shows failure rate for the last nine releases of one product.

• If the failure rate is below 4 per 1,000 hours of operation, it might be decided to incorporate more difficult changes in the next release.

• If the failure rate is above 4 per 1,000 hours of operation, it might be decided to revert to a previous version or only work on fault correction– postpone making enhancements



• Failure rate can be used to determine the probability of the software supporting a complete mission.– the probability of no failures over a week is

exp (-168 hrs/wk * .002 fails/hr) = 0.71• From the Poisson distribution

• Historical failure rate can be used as a quality requirement when trading off the cost and schedule of a major upgrade.


Metric: Change Cycle Time

From the time the user writes the change request

From the time the requirement has been approved.


Metric: Change Cycle Time

• The horizontal distance between the two lines on the chart is the time taken to approve the change.– Chart average for this in-process time is 54 days.

• 80% of priority change requests are ready within 90 days of user board approval.

• 80% of priority change requests are ready within 170 days from when they were written.


Metric: Cost Per Delivery

Chart used for long term budget planning.


Metric: Cost Per Activity• Cost categories:

– 1. software development activities– 2. configuration management– 3. quality assurance– 4. security– 5. administrative support– 6. travel– 7. project management– 8. system engineering– 9. hardware system maintenance– 10. system management– 11. finance

• 1. & 9. accounted for 88% of the cost of a typical release i.e. most of the money is going into productive work.


Software Change Taxonomy• Computational

– Incorrect operand in equation– Incorrect use of parentheses– Incorrect/inaccurate equation– Rounding or truncation error

• Logic– Incorrect operand in logical

expression– Logic out of sequence– Wrong variable being checked– Missing logic or condition test– Loop iterated incorrect number of

times• Input

– Incorrect format– Input read from incorrect location– End-of-file missing or

encountered prematurely

• Data Handling– Data file not available– Data referenced out-of-bounds– Data initialization– Variable used as flag or index

not set properly– Data not properly

defined/dimensioned– Subscripting error

• Output– Data written to different

location– Incorrect format– Incomplete or missing output– Output garbled or misleading


Software Change Taxonomy

• Interface– Software/hardware interface– Software/user interface– Software/database interface– Software/software interface

• Operations– COTS/GOTS software change– Configuration control

• Performance– Time limit exceeded– Storage limit exceeded– Code or design inefficient– Network efficiency

• Specification– System/system interface– Specification

incorrect/inadequate– Requirements specification

incorrect/inadequate– User manual/training

inadequate• Improvement

– Improve existing function– Improve interface

GOTS: Government Off The Shelf


Metric: Number Of Changes By Type

Pareto Diagram

Data for lasteight releases.

67 modifications110 fixes


Metric: Staff Days Expended/Change Type


Metric: Staff Days Expended/Change Type

• Changes in requirements or interface specifications account for 42% of the total effort.

• By categorising change requests, estimates can be made of the staff effort required to design, code, and test individual changes.– Average effort for an interface specification change is

36 days with a standard deviation of 43 days.– Average effort for a requirements change is 22 days

with a standard deviation of 24 days.

• Taxonomy and cost information is updated at the completion of a release.


Metric: Percentage of Invalid Change Requests

On average, 72 change requests are evaluated per quarter. On average, 8% are withdrawn. Estimated loss of $7,500 per year.


Metric: Complexity Measurement

• Spreadsheet-based tool used to calculate release complexity on a scale of 0 to 1.

• Set of objective and subjective data contribute to the complexity measure:– Product characteristics (e.g. age and size).– Management processes (e.g. V&V)– Staff experience (e.g. group dynamics)– Environment (e.g. tools)

• Aim to keep complexity below 0.5.


Metric: Complexity Measurement

• For each release proposal, steps can be taken to reduce complexity and manage the risks associated with complexity.– The best team can be assigned to the most

complex work.– An improved process might be introduced.– Tools or better tools might be acquired.


Metric: Percentage Content Changes By Delivery


Metric: Percentage Content Changes By Delivery

• Once a delivery plan has been agreed, requirement volatility becomes an important factor.

• A customer at a preliminary design review might decide to:– Add to the delivery content– Delete some delivery content– Change the scope of some requirements


Quantifying impact of requirement volatility

25.0%


Quantifying impact of requirement volatility

• SQRT to decrease the contribution of large values.

• 100% means the schedule was met. Values greater than 100% indicate late delivery.

• Least squares fit to develop a predictive model of schedule volatility (combining chart data with delivery effort data).

• Estimate goes up even if changes are all deletions – some disagreement here regarding model validity.


Application of schedule volatility model

• In one case, a version contained 15 planned requirements for delivery in 91 calendar days.

• At preliminary design, the customer wanted to drop two planned requirements and change the scope of a third.

• Using the model, a schedule slip of 18% or 16 days was predicted.

• The customer decided the schedule slip was unacceptable and decided to ask only for the scope change to be undertaken.

• The model provided a basis for objective communication about release plans and so helped with customer relations.


Limitations

1. No way found as yet to measure maintainability.

2. No method as yet to predict computer resource impacts.

• Systems are old and require constant capacity planning to keep the CPU less than 100% busy and memory below 98% full.

3. No account taken of the variation in size of individual changes.

• SLOC or function points could be used.


Objective Communication

If you do not have a measurement program, can you really hope to have informed and friendly discussions between the user organisation and the organisation doing the maintenance?

msc software maintenance ms viðhald hugbúnaðar

Documents