scram: schedule compliance risk assessment methodology
Post on 31-Jan-2016
60 Views
Preview:
DESCRIPTION
TRANSCRIPT
SCRAM: Schedule Compliance Risk Assessment Methodology
CSSE ARR - March 2012Los Angeles USA
Adrian PitmanDirector Materiel Acquisition Improvement
Defence Materiel Organisation Australian Dept. of Defence
What does SCRAM mean?
Shoo - Go away!
Secure Continuous Remote Alcohol Monitoring As modeled here by
Lindsay Lohan
Schedule Compliance Risk Assessment Methodology
SCRAM
Collaborative effort: Australian Department of Defence
- Defence Materiel Organisation
Systems and Software Quality Institute, Brisbane, Australia
Software Metrics Inc., Haymarket, VA
ScheduleComplianceRiskAssessmentMethodology
DMO SCRAM Usage
SCRAM has been sponsored by the Australian Defence Materiel Organisation (DMO) To improve our Project Schedule Performance in response to
Government concern
DMO equips and sustains the Australian Defence Force (ADF) Manages 230+ Concurrent Major Capital Equipment Projects &
100 Minor (<$20M) defence projects
DMO SCRAM Usage (cont.)
SCRAM has evolved from our reviews of troubled programs:
Schedule is almost always the primary concern of program stakeholders
SCRAM is a technical Schedule Risk Analysis methodology that can identify and quantify risk to schedule compliance (focus risk mitigation)
SCRAM is also used to identify root cause of schedule slippage to allow remediation action
What SCRAM is Not
Not a technical assessment of design feasibility
Not an assessment of process capability However, process problems may be identified and treated as an
issue if process performance is identified as contributing to schedule slippage
Topics
Overview of SCRAM SCRAM Components
Process Reference and Assessment Model (ISO 15504 compliant) Root Cause Analysis of Schedule Slippage (RCASS) Model SCRAM Assessment Process
Experiences using SCRAM
Benefits of Using SCRAM
SCRAM Components
The SCRAM contains the following:
Root Cause Analysis of Schedule Slippage (RCASS) Model
SCRAM Process Reference/Assessment Model
SCRAM Assessment Process
Review Preparation & Project Awareness
Risk and Issue Identification
Schedule Health Check
Monte Carlo Analysis
Reporting
Reference Model and Assessor Training, Assessors Guidebook (Currently under development)
SCRAM PR/AM Model
Developed by the SCRAM Team ISO 15504 Compliant (Levels 1 and 2) Contains Processes, Practices, implementation evidence All PR/AM Processes are traceable to a Category within
the RCASS Mode;
Model in public domain. Download from: www.scramsite.org
9
•Rework Planning•Rework Management•Technical Debt Mgt. (To be added in next update)
SCRAM-RCASSStakeholders
Functional Assets
Requirements
Workload
Staffing & Effort
Schedule & Duration
Schedule Execution
Rework
Management & Infrastructure
Subcontractors
Adapted from Integrated Analysis Model in McGarry et al., Practical Software Measurement: Objective
Information for Decision Makers
Root Cause Analysis of Schedule Slippage (RCASS)
Model evolved with experience on SCRAM assessments Used as guidance for:
Questions during assessments Categorizing the wealth of data and details gathered during an
assessment Highlighting missing information Determining the root causes of slippage slippage Recommending a going-forward plan Recommending measures to serve as leading indicators
For visibility and tracking in those areas where there are risks and problems
Similar to the use of the Structured Analysis Model in PSM to guide categorization of issues and risks via issue identification workshops
SCRAM Assessment Process
1.0 Assessment Preparation
2.0 Project Awareness
3.0 Project Risk / Issue
Identification
4.0 Project Schedule Validation
7.0 Observation & Reporting
Schedule Compliance
Risk Quantified
5.0 Data Consolidation &
Validation
6.0 Schedule Compliance
Risk Analysis
SCRAM Key Principles
Minimal Disruption Artefact Review (plans, procedures, model evidence) conducted
offline Information is collected one person at a time Interviews typically last an hour
Independent Review team members are organizationally independent of the
program under review
Non-advocate All significant issues and concerns are considered and reported
regardless of origin or source (Customer and/or Contractor). Some SCRAM reviews have been joint contractor/customer
team – facilitates joint commitment to resolve review outcomes
SCRAM Key Principles (cont.)
Non-attribution Information obtained is not attributed to any individual Focus is on identifying and mitigating the issues/risk
Corroboration of Evidence Significant Findings and Observations based on at least two
independent sources of corroboration
Rapid turn-around One to two weeks spent on-site Executive out-briefing presented at end of second week Written Report (including team recommendations) after a
following two weeks
SCRAM Team Composition
Assessment conducted by a small team including:
Engineer Assessors to Validate the Technical Baseline
Validate WBS, engineering-related basis of estimates (BoEs), work load estimates, assess technical risk, identify technical debt
Scheduler (experienced in the project schedule tool)
Validates schedule construction
Performs the schedule health checks
Run Monte Carlo
Other domain specialists as needed
E.g. – Flight Test Engineer to assess airworthiness regulatory certification and adequacy of Flight Test DT&E program
Schedule Review and Health Checks To evaluate schedule construction and logic
Includes analyses of task duration, dependencies, logic, constraints, schedule float (+ or –ve), successors, etc.
Government, Prime, and Subcontractor schedule integration / alignment is reviewed
Ensure external dependencies are included and linked in the schedule logic Interface Availability, Specialist Resources, Government Furnished
Equipment (GFE) delivery, Integration & Test facilities, Test Assets etc
Assess Critical Path Is there contingency in the schedule if risks are realized?
Any Rework contingency? Or is the schedule so tight that nothing can go wrong?
Monte Carlo Analysis
Based on the SCRAM risk analysis allocate three point estimates to tasks/work packages on critical and near-critical path based on identified risks from RCASS and run Monte Carlo optimistic, pessimistic & most likely task duration
Example
Topics
Overview of SCRAM SCRAM Components
RCASS Model Assessment Process
Experiences using SCRAM
Benefits of Using SCRAM
Some Statistics Number of SCRAM Assessments to date
10 DMO Project assessments conducted using SCRAM or SCRAM principles (i.e. before SCRAM was formalised)
Number of years First review conducted in 2007
Time on Site Typically two weeks – ONE WEEK INTERVIEWS/DATA COLLECTION – ONE
WEEK DATA CONSOLIDATION, ANALYSIS AND OUT-BRIEF
Typical Assessment Team size: SCRAM Lead plus 4 or 5 assessors
Team experience: typically 150 - 200 years Project experience
19
What Underpins Good Project Schedule Performance
20
Baseline & Execute
Schedule
Requirements
Schedule and Duration
Staffing & Effort
WorkloadEstimates
Customers
Rework
Construct Schedule Execute & Manage
Status and Report
Monitor & Control
Assess & Mitigate Risk
Forecast & Replan
Communicate to Stakeholders
Execute Schedule
What Causes Projects to Slip?
There are often multiple causes of schedule slippage:
poor planning and schedule construction
issues that arise during schedule execution
Once risk and root causes have been identified, they can be mitigated/remediated
21
Construction Stage Causes • Ambiguous, misunderstood requirements• Inadequate planning• Poor schedule construction • Poor schedule estimation− Overly optimistic estimates − Underestimate technical challenges− IPDP not factored• Inadequate SI&T Planning• Unplanned dependencies
Execution Stage Causes•‘Actual’ productivity below estimated•Requirements volatility•Schedule not used as a communication or project management tool•Poor Risk Management/Mitigation•Technical Debt + Interest•Rework (not scheduled, no contingency)•Inadequate resources (inc. staff and skills)•Stakeholder involvement•External unforeseen factors
SCRAM-RCASSStakeholders
Functional Assets
Requirements
Workload
Staffing & Effort
Schedule & Duration
Schedule Execution
Rework
Management & Infrastructure
Subcontractors
Adapted from Integrated Analysis Model in McGarry et al., Practical Software Measurement: Objective
Information for Decision Makers
Stakeholders
“Our stakeholders are like a 100-headed hydra – everyone can say ‘no’ and no one can say ‘yes’.”
Experiences Critical stakeholder (customer) added one condition for
acceptance that removed months from the development schedule
Failed organisational relationship, key stakeholders were not talking to each other (even though they were in the same facility)
Requirements
What was that thing you wanted?
Experiences Misinterpretation of a communication standard led to an
additional 3,000 requirements to implement the standard. A large ERP project had two system specifications – one with
the sponsor/customer and a different specification under contract with the developer – would this be a problem?
.
Subcontractor
Experiences Subcontractor omitting processes in order to make delivery
deadlines led to integration problems with other system components
Prime and sub-contractor schedules not aligned
If the subcontractor doesn’t perform, additional work required by the Prime Subcontractor Management
essential
Functional Assets
Commercial-off-the-shelf (COTS) products that do not work as advertised, resulting in additional work.
Experiences COTS product required a “technology refresh” as the project was years
late (cost the project $8M) IP issues with the COTS product resulted in additional development
effort required Over 200 instantiations of Open Source code resulted in breach the
indemnity requirements
Workload
Overly Optimistic estimates Inadequate BoEs
Source lines of code grossly underestimated
Contract data deliverables (CDRLs) workload often underestimated by both contractor and customer Experiences
Identical estimates in four different areas of software development (Cut & Paste estimation)
Re-plan based on twice the historic productivity with no basis for improvement
Five delivery iterations before CDRL approval
Staffing & Effort
High turnover, especially among experienced staff
Experiences Generally, most project had a lack of human resources Parent company sacking 119 associated workers
Remaining workers went on a “Go Slow” Scheduling staff for 12 hours (to recover schedule)
Schedule & Duration
Area of primary interest
Experiences No effective integrated master schedule to provide an overall
understanding of the completion date of the project 13 subordinate schedules
Critical Path went subterranean!
Schedule Execution
Using the schedule to communicate project status and plans
Experiences The schedule was not available to program staff or stakeholders
Undergoing a schedule tool transition for approx 2 years
Rework
Often underestimated or not planned for (e.g. defect correction during integration and test)
Experiences No contingency built into the schedule for rework Clarification of a requirements resulted in an additional 7
software releases (not originally planned or scheduled)
Management & Infrastructure
Contention for critical resources
Processes Risk Management, Configuration
Management
Experiences Lack of adequate SI&T facilities (in terms of fidelity or capacity)
concertinaed the schedule causing major blow out
Topics
Overview of SCRAM SCRAM Components
RCASS Model Assessment Process
Experiences using SCRAM
Benefits of Using SCRAM
SCRAM Benefits
SCRAM root-cause analysis model (RCASS) useful in communicating the status of programs to all key stakeholders Particularly senior executives (They get it!)
Identifies Root Causes of schedule slippage and permits remediation action
Identifies Risks and focuses Risk Mitigation action
Provides guidance for collection of measures Provides visibility and tracking for those areas where there is
residual risk
Provides greater confidence in the schedule
SCRAM - Benefit
SCRAM can be used to validate a schedule before project execution (based on SCRAM identified systemic schedule management issues)
Widely applicable SCRAM can be applied at any point in the program life cycle SCRAM can be applied to any major system engineering activity or
phase
Examples System Integration & Test (often – as this is typically when the sea of
green turns to a sea of red) Aircraft Flight Testing Installation/integration of systems on ship Logistics Enterprise Resource Planning (ERP) application roll out
readiness
SCRAM – Can be repeated to measure effectiveness
36
3. Enter risk parameters.
Min. Max.
Most Likely
4. Run schedule simulation & quantify impact of risk
on schedule.
5. Analyze schedule results.%
Time
6. Document results.
7. Present position to program office. Develop risk reduction actions.
The impact of project risk is assessed and critical risks fed back for risk reduction
2. Identify critical milestones for risk
quantification.
1. Develop a complete critical path network and
prepare for SRA.
Schedule
Incorporate into Risk Management Processes
Chart: Courtesy NAVAIR 4.2
SCRAM
QUESTIONS
For further information on SCRAM contact: www.scramsite.org
Govt/Govt to Govt: - Adrian Pitman: adrian.pitman@defence.gov.auIn Australia: - Angela Tuffley: a.tuffley@ssqi.org.auIn USA: - Betsy Clark: betsy@software-metrics.com In USA: - Brad Clark: brad@software-metrics.com
Acronyms ANAO – Australian National Audit Office BoE – Basis of Estimate COTS/MOTS – Commercial off the Shelf/Modified or Military off the Shelf DMO – Defence Materiel Organisation (Australia) GFE – Government Furnished Equipment ISO/IEC – International Organization for Standardization/International
Electrotechnical Commission ISO/IEC 15504 – Information Technology – Process Assessment
Framework RCASS – Root Cause Analysis of Schedule Slippage SCRAM – Schedule Compliance Risk Assessment Methodology SMI – Software Metrics Inc. (United States) SSQi – Systems & Software Quality Institute (Australia)
top related