system engineering metrics · 2017. 8. 2. · system engineering metrics 26 oct 05 james c. miller...
TRANSCRIPT
![Page 2: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/2.jpg)
• When performance is measured …performance improves
• When performance is measured andreported … the rate of performance improves
• When performance is measured, reported,and compared … the rate of performancecontinues to improve
Why Measure Systems Engineering?Why Measure Systems Engineering?
![Page 3: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/3.jpg)
ProblemProblem
• Sys Eng Scope is Huge, So …– What tenets should be measured?– What are the key characteristics?– How can it apply across different programs and
organizations?
• Sys Eng Important, But …– No accepted, standard metrics– No measure of sys eng current status– No metrics for both PM and upper management
![Page 4: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/4.jpg)
• Must Measure Major Components of Sys Eng• Must Be Targeted for Management• Must Be Few in Number• Must Describe Current Status, Not Lagging• Must Allow For Comparison Between Programs,
Organizations, and Time• Must Be Cumulative (Ability to Roll-Up)• Must Avoid Extensive Data Collection Efforts
Sys Eng Metrics Key CharacteristicsSys Eng Metrics Key Characteristics
![Page 5: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/5.jpg)
Solution: Sys Eng “Dashboard”Solution: Sys Eng “Dashboard”
• Measure Five Key Areas of Sys Eng:– Requirements Management– Risk Management– Incentivizing Contractors– Robustness/LCC– Process Management
• Used on All Programs• Regularly Shown at Organization Staff Meetings
![Page 6: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/6.jpg)
1. Requirements Management Metric1. Requirements Management Metric
• Most Important Area• Quantify, quantify, quantify• Level of Detail
– Appropriate to Life Cycle– Examples
• Objective Review• Agreement & Understanding
– User– Contractor– Program Manager
• Sources
![Page 7: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/7.jpg)
0102030405060708090
TotalRequirements
RequirementsReviewed
RequirementsQuantified
RequirementsChanged
Requirements Management Metric
Goal
Goal
![Page 8: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/8.jpg)
2. Risk Management Metric2. Risk Management Metric
• Proactive• Dynamic• Reviewed Regularly• Tangible Reduction Plan• Tracked
![Page 9: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/9.jpg)
Basic Risk Rating Chart
ConsequenceConsequence
Like
lihoo
dLi
kelih
ood
Low
Low
Med.
Med.
High
High
RISK ASSESSMENTRISK ASSESSMENT
HIGHHIGH -- Unacceptable.Unacceptable.Major disruption likely.Major disruption likely.Different approachDifferent approachrequired.required.
MODERATEMODERATE -- SomeSomedisruption. Differentdisruption. Differentapproach may be required.approach may be required.
LOWLOW -- Minimum impact.Minimum impact.Minimum oversight neededMinimum oversight neededto ensure risk remains low.to ensure risk remains low.
![Page 10: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/10.jpg)
Risk Assessment Metric
ConsequenceConsequence
Like
lihoo
dLi
kelih
ood
4
4
Low
Low
Med.
Med.
High
High
0 2
16
3 4 3
10
4
13
# of Risks
![Page 11: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/11.jpg)
Risk Management Metric
% With Plan
ConsequenceConsequence
Like
lihoo
dLi
kelih
ood
Low
Low
Med.
Med.
High
High
42 210
61 10 43
324231
%50
%30
%60
![Page 12: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/12.jpg)
3. Robustness/LCC Metric3. Robustness/LCC Metric
• Hard to Measure• Measures More the “Attempt” or Effort• Can Include Underlying Processes
– Example: Type of paint or the paint application process• Need “Toolbox” Vice One Approved Way
– Lean processes– Trade studies– Benchmarks– Combining components– COTS– Paredo Charts– Etc.
![Page 13: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/13.jpg)
Total Components
Reviewed
Changed
Time
Com
pone
nts
Goal
Robustness/LCC Metric
![Page 14: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/14.jpg)
4. Incentivizing Contractors Metric4. Incentivizing Contractors Metric
• Required for USAF by Policy– Policy Memo 03A-005, 9 Apr 03– Subject: “Incentivizing Contractors for Better Systems
Engineering”– Signed by Marvin R. Sambour, Assistant Secretary of the Air
Force (Acquisition)
• “A more robust SE environment can only be achievedthrough joint cooperative efforts with our contractors.”
• “…incentivize your contractors to perform robust SE…”
![Page 15: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/15.jpg)
Incentivizing Contractors MetricIncentivizing Contractors Metric
% of Contracts withSys Eng Incentives
Goal
![Page 16: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/16.jpg)
5. Process Management Metric5. Process Management Metric
• List Program’s Key Processes– Configuration Management– Waivers– Quality– Aircraft Structural Integrity Program– Deficiency Reviews– Etc.
• Each Program Does Own Processes• For Each Process, 4 “Steps”
– Define & Document– Lean, Improve or Refine– Keep Current by Periodic Reviews– Measure the Process
![Page 17: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/17.jpg)
Process Management MetricProcess Management Metric
Waivers
Measured
Current
Lean/Improve
Documented/Defined C
M Quality
ASIP
DR
s
...
Not Done
Done
![Page 18: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/18.jpg)
Program Sys Eng DashboardProgram Sys Eng Dashboard
• Developed Individual Metrics for the Five KeyAreas of Systems Engineering:– Requirements Management– Risk Management– Incentivizing Contractors– Robustness/LCC– Process Management
• Now Put it All Together For the ProposedProgram’s Sys Eng Dashboard…
![Page 19: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/19.jpg)
0102030405060708090
Program Sys Eng DashboardProgram Sys Eng Dashboard
Low
Low
Med.
Med.
High
High
42 210
61 10 43
324231
Meas.
Curr.
Lean
Doc.
Requirements LCC/Robust
RiskProcesses
Contracts
90
Qua
ntifi
edRev
iew
ed
Tota
l Req
uire
men
ts
40Reviewed
Changed
Total
![Page 20: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/20.jpg)
How to Roll-Up from Program to OrganizationHow to Roll-Up from Program to Organization
• Requirements Management– Convert each program to a percentage– Display average (each program has equal weight)
• Risk Management– Convert each program “square” to percentage– Display average “square’s” percentage (equal weight)
• Incentivizing Contractors– Bottom number equals sum of contracts– Depict percentage of contracts (program independent)
• Robustness/LCC– Calculate reveiwed/changed as a percentage– Display avg percentage (equal weight)
• Process Management– Depict overall percentage for each category (process/program
independent)
![Page 21: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/21.jpg)
0102030405060708090
100TotalRequirements
RequirementsReviewed
RequirementsQuantified
RequirementsChanged
Organization Requirements Metric (%)
Goal
Goal
![Page 22: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/22.jpg)
Organization Risk Metric (%)
ConsequenceConsequence
Like
lihoo
dLi
kelih
ood
Low
Low
Med.
Med.
High
High
40 8020
5 50 60
40510
%36
%7
% w/ Plan
%60
![Page 23: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/23.jpg)
Reviewed
Changed%of
Com
pone
nts
Time
Goal100%
50%
Organization Requirements Metric (%)
![Page 24: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/24.jpg)
% of Contracts withSys Eng Incentives
Organization Incentivizing Contractors Metric
Goal
![Page 25: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/25.jpg)
50% 100%
Measured
Current
Lean/Improve
Documented/Defined
Goal
Organization Process Metric (%)
![Page 26: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/26.jpg)
0102030405060708090
Low
Low
Med.
Med.
High
High
40 8020
5 50 60
40510
Meas.
Curr.
Lean
Doc.
Requirements LCC/Robust
RiskProcesses
Organization Sys Eng DashboardOrganization Sys Eng Dashboard
Contracts
100%
100%50%
50%
Reviewed
Changed
Tota
l
Rev
iew
ed
Qua
ntifi
ed
%
![Page 27: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/27.jpg)
SummarySummary
• Sys Eng Important, but No Consistent Way toMeasure…Until Now
• Need Concurrent Metrics…Not Lagging• Metrics For Management…Essential to Drive Action• What to Measure…Sys Eng “Dashboard”• Means To Use…Regular Part of an Organization’s
Overall Management Indicators
• Allows Comparison…DrivesImprovement
![Page 28: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/28.jpg)
Questions?Questions?
![Page 29: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/29.jpg)
RISK ASSESSMENTRISK ASSESSMENT
HIGHHIGH -- Unacceptable.Unacceptable.Major disruption likely.Major disruption likely.Different approach required.Different approach required.Priority managementPriority managementattention required.attention required.
MODERATEMODERATE -- SomeSomedisruption. Differentdisruption. Differentapproach may be required.approach may be required.Additional managementAdditional managementattention may be needed.attention may be needed.
LOWLOW -- Minimum impact.Minimum impact.Minimum oversight neededMinimum oversight neededto ensure risk remains low.to ensure risk remains low.
Sample: 5 - Level Risk Rating Chart
1 Minimal or no impact Minimal or no impact Minimal or no impact None
2 Acceptable with some Additional resources required; < 5% Some impactreduction in margin able to meet need dates
3 Acceptable with Minor slip in key milestone; 5 - 7% Moderate impactsignificant reduction not able to meet need datesin margin
4 Acceptable, no Major slip in key milestone > 7 - 10% Major impactremaining margin or critical path impacted
5 Unacceptable Can’t achieve key team or > 10% Unacceptablemajor program milestone
a Remoteb Unlikelyc Likelyd Highly Likelye Near Certainty
Level What Is The LikelihoodThe Risk Will Happen?
LIKELIHOOD:
TechnicalLevel Performance Schedule Cost Impact on Other Teamsand/orand/or and/orand/or and/orand/or
Like
lihoo
dCONSEQUENCE:
Given The Risk Event is Realized, What is the Magnitude of the Impact?
edcba
1 2 3 4 5Consequence
ASSESSMENT GUIDE
![Page 30: System Engineering Metrics · 2017. 8. 2. · System Engineering Metrics 26 Oct 05 James C. Miller Chief Engineer 327th CLSG Phone: 736-4294 james.c.miller@tinker.af.mil • When](https://reader035.vdocument.in/reader035/viewer/2022071601/613d4f8e984e1626b65781f2/html5/thumbnails/30.jpg)
Risk Handling Plan - “Waterfall”Risk Handling Plan - “Waterfall”R
isk
Rat
ing
Ris
k R
atin
g
TimeTime
HighHigh
MediumMedium
LowLow
EVENTEVENT
EVENT
EVENT