Download - Developmental Test and Evaluation Committee
NDIA SE Division MeetingFebruary 13, 2013
1
Developmental Test and EvaluationCommittee
Beth Wilson, RaytheonSteve Scukanec, Northrop Grumman
Industry Co-Chairs
February 2013
NDIA SE Division MeetingFebruary 13, 2013
2
DT&E Committee Focus2013
Activity Plans for 2013 Status/PlansScientific Test and Analysis TechniquesSTAT for T&E
Examples of effective use of statistical approaches for test optimization (including DOE) for an implementation framework
Status: draft white paper reviewThread at T&E conference (cancelled)Plans: participate in May MORS conference, results at SE conference
Modeling &SimulationTest as a Stakeholder
Focus on test as stakeholder for M&S products (collaboration on M&S lifecycle sub-team project)
Collaboration with M&S CommitteeJoint meeting: TBD(possibly M&S, Architecture, DT&E)
ArchitectureTest as a Stakeholder
Develop T&E perspective for architecture efforts and products
Collaboration with Architecture WGEffort TBDPotential joint meeting with M&S
NDIA T&E Conf DT&E panel, STAT report Conference cancelled
NDIA SE Conf DT&E tracks Joint M&S, STAT report
Added Task: Metrics
Provide more information on leading indicators for T&E
Collaboration with System Performance Measurement WG
Added Task: Reliability
Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)
Effort TBD
NDIA SE Division MeetingFebruary 13, 2013
3
DT&E Committee - 2013 Task PlanProjects Working Group
Proposed 2013 Tasks:• Scientific Test and Analysis Techniques
(including DOE) • Joint Meeting with Modeling and Simulation• Test as a Stakeholder for ArchitectureAdditional 2013 Tasks:• Better test strategy for reliability (collaborate
with IEEE Reliability Society)• Provide more information on proposed
leading indicators
Deliverables/Products• Scientific Test and Analysis Techniques:
• White paper• Thread at T&E conference • Results at SE conference
• Joint Meetings with SED committees• Modeling and Simulation• Architecture
Schedule / Resources• STAT White Paper: weekly progress tag-up• Conference threads:
• T&E February 2013 [CANCELLED]• SE October 2013
• Joint meetings: TBD
• Resources: • DT&E committee members• STAT white paper sub-committee
Issues / Concerns:• Resource availability
NDIA SE Division MeetingFebruary 13, 2013
4
DT&EStatistical Test Optimization
2012 SE Conference summit/workshop thread- Tutorials on Monday 10/22- Presentations on Wednesday 10/24- Synthesis Panel on Wednesday 10/24
Define an implementation framework - Define attributes of successful application- Identify framework for implementation- Provide amalgamation of success examples
Activity Plans for 2013 Status/PlansScientific Test and Analysis TechniquesSTAT for T&E
Examples of effective use of statistical approaches for test optimization (including DOE) for an implementation framework
Status: draft white paper reviewThread at T&E conference (cancelled)Plans: participate in May MORS conference, results at SE conference
SE Conference Report
October 2013
T&E Conference CANCELLED
February 2013
NDIA SE Division MeetingFebruary 13, 2013
5
NDIA DT&E Committee Summary of Accomplishments
2006 – 2012
NDIA SE Division MeetingFebruary 13, 2013
6
T&E Themes
ForConferences
SEDCommittee
CollaborativeProjects
DT&E Committee Current Structure Since 2010
OSDSE
OSDOT&E/LFT&E
OSDDT&E
NDIA Systems Engineering Conference (Q4),
Collaborative Projects
NDIA Test and EvaluationAnnual Conference (Q1), Periodic T&E Committee-Focused
Conferences, Cooperation with other NDIA Divisions
IssuesIdentified
WhitePapers, Journal Articles & Workshops
NDIAIndustrial
Committee on
Test and
Evaluation (ICOTE)
DT&E Committee Focus: T&E initiatives aligned with SE, DT&E
T&E Themes & Issues
Project ResultsDT&E Themes
IssuesIdentified
Discussion
NDIATest and
Evaluation Division
NDIASystems
Engineering DivisionWhite
Papers
NDIADT&E Committee
NDIA SE Division MeetingFebruary 13, 2013
7
Summary ofDT&E Committee Efforts
Topic Activity
DT&E Committee
DoD T&E Policy Study 2006 – 2008: Workshops and Study ReportImproving T&E in the DoD Acquisition Process
Integrated Testing 2008 – 2010: Integrated Test Study NDIA presentations and tutorialITEA journal article
RFP Language for T&E 2010 – 2011: Comments for Update to OSD GuideIncorporating T&E into DoD Acquisition Contracts
Test Optimization 2012: Statistical Optimization Conference Thread
SE Collaboration
Software 2009: SW T&E Summit Recommendations
System of Systems 2010 – 2011: T&E for SoS Workshop and Initiatives2012: Final Report on Best Practices Model
Modeling and Simulation
2011: Effective Use of M&S for T&E Use of M&S2012: Distributed Model-Based Testing
Metrics 2012: Leading Indicators for T&E
Architecture Future: T&E Perspective for Architecture views
NDIA SE Division MeetingFebruary 13, 2013
8
DoD T&E Policy Study August 2006: DT&E Committee Kickoff
Policy Study:“Improving T&E in the DoD Acquisition Process”Industry T&E policy recommendations
Workshops:August 2007January 2008
Focus Areas:1. Earlier contractor and tester involvement2. Integrated DT/OT and DT operational relevance3. Suitability
April 2008: Report Summarized Results:10 Findings15 Recommendations
2006-2008
NDIA SE Division MeetingFebruary 13, 2013
9
Integrated Testing (CT/DT/OT)Implementation Framework
2008-2010
NDIA SE Division MeetingFebruary 13, 2013
10
Software Test and EvaluationSoftware Summit
RFP Language
How Much Testing
is Enough?
Lifecycle and
End-to-End Software T&E
Changing Paradigms
Training & Competency ModelPolicy, Guidance, and Standards
Tools, Automation, Methodologies, Process2009
NDIA SE Division MeetingFebruary 13, 2013
11
RFP LanguageIndustry
#
Master #
Reviewer Line# Comment and Rationale Recommended Input A/R/P
Industry Comments for Update:Incorporating Test and Evaluation
Into Department of DefenseAcquisition Contracts
Recommendations from SW Summit
2010-2011
NDIA SE Division MeetingFebruary 13, 2013
12
Test and Evaluation forSystems of Systems
2010: Workshop2009: “Sleepless Nights” List of Issues
2010: “Sominex” Resulting Initiatives
InitiateSoS
PlanSoS
Update
EvolveSoSArch
EvolveSoSArch
ImplementSoS
Update
PlanSoS
Update
ContinueSoS Analysis
ImplementSoS
Update
PlanSoS
Update
ContinueSoS Analysis
ConductSoS Analysis
ContinueSoS Analysis
ImplementSoS
Update
DevelopSoSArch
External Environment
2011: Best PracticesWave Model
2009-2012 2012: Final Report
NDIA SE Division MeetingFebruary 13, 2013
13
Effective Use of Modeling and Simulation for Test and Evaluation
Joint Meeting in August 2011• Distributed Testing, the Joint Mission
Environment Test Capability (JMETC) and the Test and Training Enabling Architecture (TENA)
• DoD M&S Community of Interest Data Management Working Group
• LVC Architecture Roadmap Implementation (LVCAR‐I) Gateways Effort Applicability to T&E
• OSD T&E Working Group • Raytheon Presentation on M&S for T&E • Potential Topics for November AMSWG
Meeting
2011
NDIA SE Division MeetingFebruary 13, 2013
14
Modeling & Simulation CollaborationDistributed Testing
Benefits:• Find integration issues earlier• Test to learn in ‘safe’ environment• Protect proprietary information• Facilitate DT to OT transition• Increase performance testing range in operating environments • Support end to end studies throughout the programBarriers:• Security• Lack of persistent network• Early consideration of technical issues• Perceived value• Disconnect between the communities (M&S and T&E)Recommendations:• Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)• Create a framework for reusing and repurposing M&S through the product model• Establish M&S as part of statistical test design
• Determine what tests are conducted to acquire data for model validation.• Fewer test events with better models.
• Recommend the use of M&S to do I&T • Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T
Joint TrackSE Conference October 2012
Joint MeetingAugust 2012
2012
NDIA SE Division MeetingFebruary 13, 2013
15
Metrics CollaborationLeading Indicators for T&E
http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Documents/Studies/NDIA%20System%20Develpopment%20Performance%20Measurement%20Report.pdf
Workshop October 22, 2012
Follow-on workshop with System Performance Measurement WG Focused on other information needs not addressed in first report
2012