NAME OF THE SYSTEM UNDER TEST

TAILORED INTEGRATED EVALUATION FRAMEWORK

COMOPTEVFOR 3980 (XXX-OT-XX)Ser XXX/XXX

DATE

The SCG dictates the distribution statement. If the SCG does not specify a distribution statement, COTF documents will use statement B or E.

DISTRIBUTION STATEMENT B. Distribution authorized to U.S. Government agencies only; test and evaluation document. Other requests for this document shall be referred to CNO (N84) or COMOPTEVFOR via DTIC using DTIC form 55.IEFs are disseminated through a web-based document repository. For access to the repository, contact the COMOPTEVFOR Policy Department.

(If there is no FOUO information in the CDR’s ltr and only your survey sheets have FOUO (when filled in), remove this statement.)

T

his document contains information exempt from mandatory disclosure under the FOIA. Exemption 5USC 552(b)(5) applies.

COMMANDER, OPERATIONAL TEST AND EVALUATION FORCENORFOLK, VIRGINIA

The tailored IEF is used for programs led by another OTA, JCTD or other nontraditional test program. It is tailored at the discretion of 01B. This is just one example.

The cover page will identify the system (increment) and include a picture of the system, date, serial number of the document, and distribution statements. It will also contain the COMOPTEVFOR logo and appropriate FOUO/Classification markings.

This is the signature date.

Adjust picture size to ensure all contents fits first page. This may mean expanding the picture to fill the full height of the page.

BLANK PAGE

Template changes:22Jun15 – Magenta direction and format of SUT paragraphs updated to incorporate in-scope, out-of-scope, and

focus of FOT&E (or similar continuum of test) on enhancements, new capabilities, and regression. Magenta direction for SOS similarly updated. Appendix A (definitions) removed. Appendix B and other appendices moved forward as appropriate. Data Requirements tables moved to appendix B. Optional appendix C added for TIEFs on Joint Programs. Magenta direction for critical measures section clarified to confirm that critical measure write-ups (not statistics) are required for TIEFs. Formatting errors fixed throughout.

COMOPTEVFOR TAILORED INTEGRATED EVALUATION FRAMEWORK (TIEF)

The tailored IEF is used for programs where COTF is not the lead OTA and nontraditional assessments (i.e., JCTDs). It is constructed by tailoring the execution of the MBTD steps as appropriate. The 01B and Warfare Division Directors agree on how the MBTD process is tailored on a case-by-case basis. At a minimum, it will entail executing steps 1 through 8 of the OTD IEF checklist to include IPR1. The purpose is to identify all the critical tasks, measures, conditions, and data that must be addressed in the relevant test plan. The tailored IEF will also be used for programs that are beginning the test planning process well in advance of MS B, when requirement documents have not been finalized and decisions with respect to capability have not been made. This includes supporting MS A TEMPs.

This is the TIEF for the System Under Test (SYSTEM ACRONYM). The document is intended to:

Document mission and capabilities analyses and test design conducted during the Mission-Based Test Design (MBTD) process. Define the System Under Test (SUT) and the associated System of Systems (SoS). Detail the missions and subtasks to be supported by the system, conditions under which tasks must be performed, measures that define success for task execution, and the data required to support performance evaluation. Identify the critical tasks and measures that will be key to evaluation of the SUT effectiveness and suitability across the operational environment. Provide the foundation for the Operational Test (OT) input to the Test and Evaluation Master Plan (TEMP) if applicable.

Enclosure (1) is provided for planning purposes. Updates to the TIEF will be prepared per reference (a), COMOPTEVFORINST 3980.2D, Operational Test Director’s (OTD) Manual, and released any time substantive program or requirements changes occur.

J. R. PENFIELD

For TIEFs for joint programs where COTF is the lead, coordination signatures may be requested by the other OTAs. Use the format below for the signature section (replacing the format containing just the admiral’s signature above).

1

Start defining acronyms here for first use. Do not need to be redefined in the enclosure.

(Modify bullets below as appropriate.)

J. R. PENFIELD, RADM, USNCommander, Operational Testand Evaluation Force

MRS. OTHER DIRECTORDirector, OTHER OTA

JAMES T. KIRK, COL, USAFCommander, USS Enterprise

2

BLANK PAGE

3

If the DCR’s letter is only one page, keep this blank page. Otherwise, delete.

NAME OF THE SYSTEM UNDER TEST

TAILORED INTEGRATED EVALUATION FRAMEWORK

DISTRIBUTION STATEMENT B. Distribution authorized to U.S. Government agencies only; test and evaluation document dated ____________. Other requests for this document shall be referred to CNO (N84) or COMOPTEVFOR via DTIC using DTIC form 55.IEFs are disseminated through a web-based document repository. For access to the repository, contact the COMOPTEVFOR Policy Department.

This document contains information exempt from mandatory disclosure under the FOIA. Exemption 5USC 552(b)(5) applies

(If there is no FOUO information in the enclosure and only your survey sheets have FOUO (when filled in), remove this statement.)

Enclosure (1)

ii

BLANK PAGE

III

CONTENTSSECTION 1 - INTRODUCTION 1-1

1.1 PURPOSE 1-11.2 SYSTEM DESCRIPTION 1-1

SECTION 2 - CRITICAL OPERATIONAL ISSUES (COI) 2-12.1 EFFECTIVENESS COIS 2-12.2 SUITABILITY COIS 2-12.3 JOINT COIS 2-22.4 CRITICAL TASKS AND MEASURES 2-2

APPENDIX A - MISSION AND CAPABILITIES ANALYSIS A-1APPENDIX B - DATA REQUIREMENTS B-1

B.1 MEASURES-TO-DATA REQUIREMENTS MATRIX B-1B.2 CONDITIONS-TO-DATA REQUIREMENTS MATRIX B-1

APPENDIX C - JOINT COI TO NAVY TASKS AND MEASURES C-1APPENDIX D - ACRONYMS AND ABBREVIATIONS D-1APPENDIX E - REFERENCES E-1

TABLESTable 2-3. AW Critical Measures 2-3Table A-1. Conditions Directory A-1Table A-2. Attribute Matrix A-1Table A-3. Orphaned Measures A-1Table A-4. Traceability Matrix A-1Table B-1. Measures-to-Data Requirements Matrix B-1Table B-2. Conditions-to-Data Requirements Matrix B-1Table C-1. JCOI to Navy Tasks and Measures C-1

FIGURESFigure 1-1. Submarines are Awesome 1-2Figure 1-2. Everybody Loves a Sub 1-4

TIEF for SUT Contents

To generate the TOC, Styles must be used in the section/appendix headings: Heading 1 (TOC 1), Heading 2 (TOC 2), Heading 7 (TOC 1), and Heading 8 (TOC 2).

IV

BLANK PAGE

TIEF for SUT Contents

Blank pages are only added if the section (without the blank page) ends on an odd numbered page. Each section will begin with an odd numbered page.

1

SECTION 1 - INTRODUCTION

1.1 PURPOSEDescribe the overarching purpose of the document. What is driving the creation of this TIEF? What test phases will be affected by this IEF? Any other unique programmatics affecting MBTD (IT Box for example) and the resulting effects on the IEF (which RDPs/CDs are covered) can also be described. If this is a revision to previous IEFs, describe the revisions made. The reason why we are doing a TIEF (vice a full IEF) should be apparent here. If this is a revision to previous TIEFs, describe the revisions made.EXAMPLE

The TIEF for SUT (SYSTEM ACRONYM) documents the results of the mission and requirements analysis phases of the MBTD process (steps 1 through 8). MBTD was executed for this program per reference (a). This TIEF supports input to the Air Force TEMP and to subsequent Air Force test plans, ensuring data required for Navy test objectives are collected.

1.2 SYSTEM DESCRIPTION

1.2.1 SUTThe SUT paragraphs will be used to help the reader understand the scope of test. The reader must understand what the system does before they can properly review MBTD products (tasks, conditions, measures, etc.) and the resulting test strategy. The SUT is the hardware and/or software being delivered/developed to meet the requirements set by the resource sponsor and to provide the capabilities needed by the Fleet.The SUT description may not remain the same throughout testing. As the SUT is upgraded, testing must focus on the impact of those upgrades. For frameworks written to support IOT&E or earlier, the SUT is the entire TEIN because everything is new, and being tested at IOT&E.For programs that have already undergone IOT&E (FOT&E, subsequent increments, limited remaining scope, etc.), the SUT is divided between In-Scope and Out-of-Scope. The In-Scope SUT definition is limited to the impacts of the upgrade being purchased this time through. Upgrades to the system include hardware, software, and/or tactics changes providing/affecting new capabilities, enhancements, and regression confirmation. The Out-of-Scope SUT includes legacy functions/capabilities not impacted by the upgrade, and hardware/software not included in the In-Scope SUT. These legacy components and functions are still part of the program, so they must be SUT.Start here (directly below SUT header) with a short/simple SUT introduction. If needed, explain how your SUT scope has changed, including the fact that some is no longer in-scope. Provide greater detail on the In-Scope SUT in the first three subparagraphs (Components, Capabilities, and CONOPS). Only include the fourth subparagraph if there is Out-of-Scope SUT.EXAMPLE

The SYSTEM ACRONYM SUT Increment 1 completed IOT&E in 2012. Testing of Increment II as planned through this IEF will focus on the In-Scope SUT, as described by paragraphs 1.2.1.1, 1.2.1.2, and 1.2.1.3 below. The SUT components and functions not impacted by Increment II are Out-of-Scope for this IEF. Testing will not be expanded to exercise these capabilities, but valid test

TIEF for SUT Section 1

Use style Heading 1 for all SECTION headings. Type the title (and classification, if required). No numbering, spacing, or tabbing required. Refer to the Quick Style Gallery.

Use style Heading 2 for all section level 2 headings.

Use style Heading 3 for all paragraph level 3 headings.

2

observations may still be made. The Out-of-Scope SUT is described in paragraph 1.2.1.4.

1.2.1.1 Fielding ConfigurationThis subparagraph is always used. Describe the expected configuration of the In-Scope SUT to be deployed to the Fleet. Production-representative configuration must be understood to know if IT data can be used for OT conclusions. List all the hardware and software components, and provide drawings/pictures of the In-Scope SUT. The reader must be able to understand where the SUT stops (SUT boundary) and the SoS begins.For FOT&E (and the like), be certain this paragraph only contains the hardware and software components that are associated with the new capabilities, enhancements, and legacy functions requiring regression test.

The In-Scope SYSTEM ACRONYM SUT…

Figure 1-1. Submarines are Awesome

TIEF for SUT Section 1

Use style Heading 4 for all paragraph level 4 headings.

3

1.2.1.2 CapabilitiesThis subparagraph is always used. Identify the key/overarching In-Scope SUT functions that will support mission accomplishment. Be clear how the new system is meant to improve over existing performance (task execution, reliability, etc.). A bulletized list is a good idea. Follow each bullet with the measures for testing those functions.For FOT&E (and the like), this paragraph should be further broken into three subparagraphs: 1.2.1.2.1 New Capabilities (completely new functions) 1.2.1.2.2 Enhancements (improvements to existing functions tested in prior OT) 1.2.1.2.3 Regression Confirmation (functions tested in prior OT, not upgraded this time,

but potentially impacted by latest upgrade).Emphasize how performance should improve from prior test. Follow each bullet with measures.EXAMPLE

The capabilities delivered by the upgraded SYSTEM ACRONYM that are in-scope for testing are detailed below as new capabilities, enhancements, and regression confirmations:

1.2.1.2.1 New CapabilitiesThe upgraded SUT will deliver the following new capabilities: Capability 1 (M4, M5-10) Capability 2 (M2)

1.2.1.2.2 EnhancementsThe upgraded SUT incorporates the following capability enhancements: Capability 3 (M1) Improved Reliability (M68-69)

1.2.1.2.3 Regression ConfirmationsThe upgraded SUT requires regression confirmation of the following capabilities: Capability 4 (M11-15) Capability 5 (M20)

1.2.1.3 Concept of Operations (CONOPS)This subparagraph is always used. Describe the CONOPS for the SUT, including the status of the official acquisition CONOPS being approved. Describe the end user and the environment in which it is intended to operate. The OV-1 is best included here. This discussion must include the maintenance and logistics concepts for the SUT. Also, include details that explain the choices made regarding Cybersecurity (PIT, PITI, etc.). For multi-mission systems, it may be appropriate to create subparagraphs to describe each separate mission CONOPS.For FOT&E (and the like), describe any impact to the CONOPS by the upgraded SUT. Descriptions of maintenance, logistics, and Cybersecurity also focus on the upgraded SUT.

The SYSTEM ACRONYM SUT…

TIEF for SUT Section 1

Use style Heading 5 for all paragraph level 5 headings.

4

1.2.1.4 Out-of-Scope SUTThis subparagraph is only written for FOT&E (and the like), when part of the SUT is Out-of-Scope. The boundary between in-scope and out-of-scope should be clear above. The boundary between SUT and SoS must be clear here. For larger TEINs, subparagraphs to describe out-of-scope components and out-of-scope functions may be needed. Missions that are not in-scope may call for a CONOPS subparagraph here.

The entire SYSTEM ACRONYM SUT is the…

1.2.2 System of Systems (SoOS)The SoS is the existing infrastructure and systems within which the SUT will function to support accomplishment of all missions detailed by the MBTD. The SoS write-up includes all SoS components and functions supporting the SUT that are required to fully exercise the SUT during test. This usually includes just direct connections, but may expand further if required.Determining SoS boundaries is not always intuitive. The reader must understand those SoS inputs to the SUT and SUT products provided to the SoS for mission accomplishment. SUT and SoS are completely separate for test deficiencies, COI resolution, etc. For test, it is vital to think of SoS as the SoS supporting the SUT.For FOT&E (and the like), the SoS may change. This must be explained. Changing test scope affects the SoS needed to support test. Even if In-Scope SUT changes little, the SoS may have changed from prior test. If SoS enhancements (or the interactions between the SoS and SUT enhancements) are significant to test results (i.e., regression analysis), explain that here.

The SoS supporting the SYSTEM ACRONYM (henceforth referred to as the SoS)…

Figure 1-2. Everybody Loves a SubFOR OFFICIAL USE ONLY

TIEF for SUT Section 1

This is how you label and classify figures (above). Use style heading COTF Figure so it shows up in the Table of Contents.

5

BLANK PAGE

TIEF for SUT Section 1

Blank pages are only added if the section (without the blank page) ends on an odd numbered page. Each section will begin with an odd numbered page.

1

SECTION 2 - CRITICAL OPERATIONAL ISSUES (COI)

2.1 EFFECTIVENESS COISBe certain that your COIs are consistent with the standard Navy missions and the associated default mission threads (located in the IEF database). If you have any nonstandard COIs, explain why. It is possible that not all your SUT COIs will be analyzed as a part of this IEF (i.e., the IEF update supports an FOT&E where only some COIs apply). If so, state that in the paragraph below.EXAMPLE

The following effectiveness COIs reflect the analysis of SUT capabilities and the missions it supports. The SUT is net enabled and Cybersecurity is included as an additional COI to support analysis and reporting on the program’s Cybersecurity capabilities.

2.1.1 E-1, CybersecurityWill SYSTEM Cybersecurity protect, detect, react, and restore capabilities support completion of its missions?

2.1.2 E-2, Intelligence (INT)Will the SYSTEM ACRONYM successfully support the INT mission?

2.1.3 E-3, Air Warfare (AW)Will the SYSTEM ACRONYM…?

2.2 SUITABILITY COISUse the four standard COIs; Reliability, Maintainability, Availability, and Logistic Supportability. If you add other suitability COIs or remove any of the standard four, explain why.EXAMPLE

The following suitability COIs reflect the standard COIs for COMOPTEVFOR, as set by reference (a).

2.2.1 S-1, ReliabilityWill SYSTEM reliability support mission accomplishment?

2.2.2 S-2, MaintainabilityWill the SYSTEM be maintainable by Fleet personnel?

2.2.3 S-3, AvailabilityWill SYSTEM availability support mission accomplishment?

2.2.4 S-4, Logistic SupportabilityWill the SYSTEM be logistically supportable?

TIEF for SUT Section 2

Cybersecurity (if used) should be the first COI of every new IEF.

2

2.3 JOINT COISFor programs led by other OTAs with established Joint COIs, include this paragraph to discuss the relationship between the COTF COIs and the Joint COIs used by the lead OTA. If you’re not Joint, don’t include this.EXAMPLE

These are the COIs identified in the MOT&E plan (reference (b)) to be used by the multiservice TBMCS integrated test team.

2.3.1 JCOI-1, Does TBMCS support Theater Air Campaign Planning? JCOI-1 addresses the capability of TBMCS to provide solutions to support target development, to support weaponeering, master air attack planning, and allocation and to support Joint Air Tasking Order development. The measures and conditions applicable to this Joint COI are captured under the USN CCC COI.

2.3.2 JCOI-2, Does TBMCS support Theater Air Campaign Execution?JCOI-2 addresses the capability to support force management and execution, and to support combat assessment. The measures and conditions applicable to this Joint COI are captured under the USN CCC COI.

2.3.3 JCOI-3, Does TBMCS support sustained Theater Air Operations?JCOI-3 addresses the capability to support sustained theater aerospace campaign operations, to support interoperability requirements, and to evaluate the impact of TBMCS on current TTPs. The measures and conditions applicable to this Joint COI are captured under the USN Effectiveness CS COI and Suitability COIs.

2.4 CRITICAL TASKS AND MEASURESIdentify the critical tasks and measures associated with each COI. Point the reader towards appendices A and B for additional details.For FOT&E updates to an IEF, it is important to distinguish between critical tasks/measures that directly apply to the enhancement or new capability and those that were retained from IOT&E to support regression testing. Do not simply reuse table 2-1 from IOT&E. In the FOT&E example, any measures retained as critical for the sole purpose of regression testing must be marked as note 1.

(U) Table A-4 illustrates how all the subtasks and measures of each COI relate. This section expands on critical tasks and critical measures that will be key to assessing SUT effectiveness/suitability in all mission areas. Further understanding of how each critical measure will be tested requires referencing the data requirements of appendix B.

TIEF for SUT Section 2

3

Every critical measure must be written up in section 2.3. All measure paragraphs, at minimum: define the measure, provide the threshold, and explain why the measure is considered critical.

2.4.1 E-1, CSTable 2-1… .

2.4.2 E-2, INTTable 2-2… .

2.4.3 E-3, AWTable 2-3 depicts the critical tasks associated with this SUT under the AW COI. Performance of these tasks will constitute a major portion of COI assessment. Also, table 2-3 lists the critical measures associated with those tasks.

The table below is required for every COI. Identify the critical tasks for each COI and the critical measures associated with those tasks. Mark KPPs as appropriate.For FOT&E updates to an IEF, it is important to distinguish between critical tasks/measures that directly apply to the enhancement or new capability of the SUT, and those that were retained from IOT&E to support regression testing. Do not simply reuse table 2-1 from IOT&E. In the FOT&E example, any measures retained as critical for the sole purpose of regression testing must be marked as note 1.

Table 2-3. AW Critical MeasuresTask Title Critical Measure

1.2.3.1 – Navigate (climb out) M35 Worldwide Navigation Database

1.2.3.2 – Aviate (climb out) M11 Primary Flight Information DisplayedM15 Caution and Warning Information Displayed

1.3.2.2 – Fly Assigned Route M13 Flight Management System (FMS) Navigation Assist

M35 Worldwide Navigation Database1.4.2 – Conduct Search Pattern

M111 Primary Flight Information DisplayedM15 Caution and Warning Information Displayed

M24 Terrain Awareness and Warning System (TAWS) Display

M71 Night Vision Goggle (NVG) Capable (KPP)Note 1. These critical measures specifically focus on regression testing.

2.4.4 S-1, Reliability

2.4.5 S-2, Maintainability

2.4.6 S-3, Availability

2.4.7 S-4, Logistic Supportability

TIEF for SUT Section 2

4

BLANK PAGE

TIEF for SUT Section 2

Blank pages are only added if the section (without the blank page) ends on an odd numbered page. Each section will begin with an odd numbered page.

1

APPENDIX A - MISSION AND CAPABILITIES ANALYSISThe following tables are provided:

Table A-1. Conditions Directory – a listing of conditions that are controlled or recorded to support post-test analysis.

Table A-2. Attribute Matrix – a listing of all measures used to assess effectiveness and suitability of the SUT.

Table A-3. Orphaned Measures – measures identified in the capabilities document that OT will not report on.

Table A-4. Traceability Matrix – a linkage of the operator tasks (for each COI) to the measures and conditions associated with those tasks that will be used to assess the performance of the SUT.

Critical tasks and measures are presented in red. Items that are highlighted in gray are retained for traceability but are out of scope of this TIEF. Definitions for acronyms used in this and all subsequent embedded excel workbook can be found in appendix X, Acronyms and Abbreviations.

Table A-1. Conditions DirectoryTable A-2. Attribute MatrixTable A-3. Orphaned MeasuresTable A-4. Traceability Matrix

The appendix A tables must be exported from the IEF database and embedded as a workbook (as shown above). Headers/footers and page numbers are automatically formulated and included in this workbook. The Footer SUT name will match the database name. Edits to this will be needed to make sure the footer matches the footer of the word document..Do not convert the excel output into MS Word format and paste into this file.For routing or review of a hardcopy, the individual tables can be printed directly from Excel and inserted after this page.Guidance for each of the tables is included in the example Excel workbook.These rules also apply to all other embedded workbooks.For any workbook (except AppB-B), column widths should be examined to confirm that content is displayed in an ideal manner. Often the tool exports can be widened and still fit one page.

TIEF for SUT Appendix A

Steps for embedding: Export file from tool. Save to location (use naming convention displayed above). In word, under INSERT menu, select OBJECT. In OBJECT box, select CREATE FROM FILE tab. Browse for the File. Check the box for DISPLAY AS ICON. Click OK.

Use style Heading 7 for all APPENDIX level headings.

2

BLANK PAGE

TIEF for SUT Appendix A

Blank pages are only added if the section (without the blank page) ends on an odd numbered page. Each section will begin with an odd numbered page.

1

APPENDIX B - DATA REQUIREMENTS

B.1 MEASURES-TO-DATA REQUIREMENTS MATRIXTable B-1 provided below shows the relationship of measures to the data requirements that must be collected to satisfactorily resolve each measure.

Table B-1. Measures-to-Data Requirements MatrixData requirements should direct test participants to observe and record specific items that are needed to confirm satisfactory subtask performance based on measures. Each measure must be fully confirmed by data. Listing data requirements by measure gives visibility to the adequacy of test. Data requirements are categorized to make gathering them easier. Suggested categories (manual, automated, DT, etc.) are provided in the IEF databased. The OTD can create others as needed. Data requirements will not appear in vignettes if they are not categorized. Gray measures are not required to have data requirements.

B.2 CONDITIONS-TO-DATA REQUIREMENTS MATRIXTable B-2 provided below shows the data requirements that must be collected to document conditions at the time of test. These data requirements are also required to satisfactorily resolve measures.

Table B-2. Conditions-to-Data Requirements MatrixEach condition must be recorded during test using these data requirements. Traceability of conditions to tasks will add these data requirements to vignettes (just like data requirements for measures). Usually all of these are categorized as Conditions Data.

TIEF for SUT Appendix B

Use style Heading 8 for all appendix level 2 headings.

Use style Heading 9 for all appendix level 3 headings.

2

BLANK PAGE

TIEF for SUT Appendix B

Blank pages are only added if the section (without the blank page) ends on an odd numbered page. Each section will begin with an odd numbered page.

1

APPENDIX C - JOINT COI TO NAVY TASKS AND MEASURESFor every JCOI, list and describe the associated Joint measures from the TEMP. Then show how the Navy MBTD products roll into these. The TIEF’s purpose in feeding a Joint TEMP is to ensure Navy data is collected. This mapping is part of the confirmation.

Table C-1 provided below shows a listing of AFOTEC’s JCOIs, MOE number, and MOE description mapped to COTF's COI, subtask, measure number, and description of the measure.

Table C-1. JCOI to Navy Tasks and Measures

AFOTEC JCOI AFOTEC MOE# AFOTEC MOE Description COTF

COI COTF Subtask COTF M# COTF Measure Description

1 - Does TBMCS Support Theater Air Campaign Planning?

2-1

Capability of TBMCS s1.1.3 MR2 provided solutions to support the development of theater targets.

E-1 1.3.41.3.5 1 Does TBMCS provide the capability to

support planning to a Target List?

2-2

Capability of TBMCS s1.1.3 MR2 provided solutions to support the development and dissemination of target nomination lists.

E-1

1.3.5 2Does TBMCS provide the capability to support the development of target nomination lists?

1.2.1.1 3Does TBMCS provide the capability to support the dissemination of target nomination lists?

3-1 Capability of TBMCS to support weaponeering. E-1 1.3.2 4 Does TBMCS support weaponeering?

3-2Capability of TBMCS to support allocation determination.

E-1 1.6.1 5 Does TBMCS support allocation determination?

3-3Capability of TBMCS to support the production of the MAAP.

E-1 1.6.1 6Time (in hours) to develop the MAAP during the ATO Production Cycle after receiving a JFACC approved JIPTL.

4-1 Capability of TBMCS to support the production and dissemination of ATOs.

E-1 1.2.1.11.2.1.21.3.11.6.11.6.21.6.31.7.3

7

As a Lead AOC, can an ATO (1,500 missions-3,000 sorties sized) be produced within 12 hours of JFACC approval of the MAAP? (time of ATO production commencement for 1,500 missions-3,000 sorties / time of ATO approval)

8

As Lead AOC - Time (in minutes) required to disseminate USMTF ATO (1,500 missions-3,000 sorties sized). (KPP)

10 As a Component, can an ATO (350 missions-500 sorties sized) be produced

TIEF for SUT Appendix C

This is an optional appendix. Only use it for Joint Programs where COTF is not the lead, and thus the COIs in the TEMP do not match the Navy COIs we have selected through MBTD. If this appendix is not used, make sure to renumber the later appendices. The division may choose not to use this appendix (approved at IPR-1). Discuss with your CTF.

2

Table C-1. JCOI to Navy Tasks and Measures

AFOTEC JCOI AFOTEC MOE# AFOTEC MOE Description COTF

COI COTF Subtask COTF M# COTF Measure Description

within 12 hours of JFACC approval of the MAAP? (time of ATO production commencement for 350 missions-500 sorties approved by JFACC for the MAAP/time of ATO approval)

12 Can TBMCS support generation of special instructions (SPINS)?

13Elapsed time required to import the integrated JIPTL into the ATO producing application.

14Elapsed time required to conduct mission approval of planned missions to the AODB.

17As a Component - Time (in minutes) required to disseminate USMTF ATO (350 missions-500 sorties sized).

19 Does TBMCS provide the capability to approve an ATO?

4-2

Tasked military agencies receive a USMTF ATO (dependent upon communications equipment and systems).

E-1

1.2.1.11.2.1.21.3.11.6.11.6.21.6.31.7.3

19 Does TBMCS provide the capability to approve an ATO?

4-3

Capability to receive, rank, manipulate and display preplanned air support requests originating from other services.

E-1 1.3.3 16 Are preplanned air support requests supported by TBMCS? (KPP)

4-4 Capability of TBMCS to support receiving airspace control measures requests (ACMREQ) and producing and disseminating ACOs.

E-1 1.3.31.6.1 11 Can airspace control measures requests

(ACMREQ) be received by TBMCS? (KPP)1.3.11.3.31.6.2

15 Can TBMCS support determination of airspace usage in the joint ABP?

1.3.1 18 Can airspace control measures requests (ACMREQ) be included in the ACO dissemination?

TIEF for SUT Appendix C

3

Table C-1. JCOI to Navy Tasks and Measures

AFOTEC JCOI AFOTEC MOE# AFOTEC MOE Description COTF

COI COTF Subtask COTF M# COTF Measure Description

4-5

Tasked military agencies receive a USMTF ACO (dependent upon communications equipment and systems).

E-1

1.2.1.11.7.3 9 As Lead AOC Time (in minutes) required

to disseminate USMTF ACO. (KPP)

1.6.3 20 Does TBMCS provide the capability to approve an ACO?

2 - Does TBMCS support Theater Air Campaign Execution?

5-1

Capability of TBMCS to maintain current situational and status displays.

E-1 1.4 21Does TBMCS maintain current Execution and Force status (ESTAT/FSTAT) displays? (KPP)

5-2

Capability of TBMCS to support the production and dissemination of intelligence updates.

E-1 1.2.1.21.3.6 22

Time (in minutes) required by TBMCS to produce intelligence updates once MISREP is imported?

5-3

Capability of TBMCS to support the directing and monitoring of ATO (ABP) execution to achieve JFC objectives.

E-1 1.5.21.6.2

23Does TBMCS provide the capability to process ATO`s (350 missions-500 sorties sized) to be incorporated into the ABP?

24

Does TBMCS provide the capability to process ATO`s (1,500 missions-3,000 sorties sized) to be incorporated into the ABP?

5-4

Capability of TBMCS to support the development and dissemination of ATO changes.

E-1

1.6.1 25

As Lead AOC - Time (in hours) required by TBMCS to develop ATO changes. (time ATO change received / time ATO change approved)

1.2.1.11.7.3 36

As Lead AOC - Time (in hours) required by TBMCS to disseminate ATO changes. (time ATO change approved / time ATO change transmitted)

5-5

Capability of TBMCS to support the processing immediate JTASRs (non-CAS) at the JAOC.

E-1 1.6.11.7.1 26 Does TBMCS support the processing of

immediate JTASRs (non-CAS) at the JAOC?

5-6

Capability of TBMCS to support the directing and monitoring of ACO execution to achieve JFC objectives.

E-1 1.5.21.6.2 27

Percent of accurate ACO data executed to achieve JFC objectives? (# of ACO's received with data discrepancies / # of ACO's received)

5-7

Capability of TBMCS to support the development and dissemination of ACO changes.

E-1 1.2.1.11.7.3 28

As Lead AOC - Time (in hours) required by TBMCS to disseminate ACO changes. (time ACO change approved / ACO change transmitted)

TIEF for SUT Appendix C

4

Table C-1. JCOI to Navy Tasks and Measures

AFOTEC JCOI AFOTEC MOE# AFOTEC MOE Description COTF

COI COTF Subtask COTF M# COTF Measure Description

1.6.1 35

As Lead AOC - Time (in hours) required by TBMCS to develop ACO changes. (time ACO change received / time ACO change approved)

5-8

Capability of TBMCS to accurately display missile launch information and support the processing of the information.

E-11.2.1.11.7.11.7.3

32 Time to receive CAS scramble order by tasked units.

5-9

Capability of TBMCS to support the processing of CAS immediate air requests. MOP 5-9-1 is a KPP pertaining to ASR processing rates.

E-1 1.6.11.7.1

29 Time to receive CAS scramble order by tasked units.

30 Number of CAS immediate air requests processed per hour.

5-10

Capability of TBMCS to support the monitoring of CAS assets (to include interdiction missions for TACC/DASC).

E-1 1.5.2

31Does TBMCS support the monitoring of CAS assets (to include interdiction missions for TACC/DASC)?

34

Does TBMCS provide the capability to monitor ATO's in ABP (1,500 missions-3,000 sorties sized)? (# of 1,500 missions-3,000 sorties monitored /# of 1,500 missions-3,000 sorties in the ABP)

6-1

Capability of TBMCS s1.1.3 provided solution to support the reporting of mission summary data.

E-1

1.3.61.5.11.5.21.9

37Does TBMCS provide solutions to support the reporting of mission summary data (ESTAT/FSTAT)?

6-2

Capability of TBMCS s1.1.3 provided solution to support operator assessment of the status of air and space operations.

E-1

1.2.1.1 33Time (in minutes) required by TBMCS to disseminate intelligence updates once MISREP is imported?

1.3.61.5.11.5.21.9

38

Does TBMCS provide warfighters the capability to use MISREPS, IPIRs and IIRs as information sources to develop Tactical Assessments on the status of air operations?

3Does TBMCS support sustained Theater Air

7-1 Capability to transfer JFACC responsibility and data between two geographically separated locations in two modes:

E-1 1.2.1.11.7.2

39 Time (in hours) required for TBMCS to transfer JFACC responsibility and data between two geographically separated locations in planned transfer mode. (KPP)

TIEF for SUT Appendix C

5

Table C-1. JCOI to Navy Tasks and Measures

AFOTEC JCOI AFOTEC MOE# AFOTEC MOE Description COTF

COI COTF Subtask COTF M# COTF Measure Description

Operations?planned transfer and emergency transfer. 40

Time (hh:mm:ss) for TBMCS transfer JFACC responsibility and data between two geographically separated locations in emergency transfer mode? (KPP)

7-9

Backward-Forward Version Compatibility. Capability of TBMCS s1.1.3 MR2 to send and receive data from previously fielded TBMCS versions.

E-1 1.2.1.1 44

Can TBMCS send and receive data from previously fielded TBMCS versions to ensure backward/forward version compatibility?

7-10

Installation and Configuration. Capability of TBMCS s1.1.3 MR2 to be installed and configured.

E-1 1.1.1 41 Can TBMCS be installed and configured?

7-11

Documentation. Adequacy of TBMCS s1.1.3 MR2 documentation and online help to support system operation and maintenance.

E-1 1.1.15 42

Does TBMCS have adequate documentation and online help to support system operation?

7-12

Manpower Supportability. Capability to operate and support TBMCS s1.1.3 MR2 at current manpower levels.

S-2 3 75 Can TBMCS be operated and maintained safely with current manning levels?

7-13 Protection. Capability of IA protection controls. E-2 6.1

53 Is TBMCS patch management adequate?

54What are the operational impacts of vulnerabilities and exploitations on the TBMCS system?

55 Are TBMCS layered protection mechanisms adequate?

56 Is TBMCS access control adequate?

57 Are TBMCS authentication measures adequate?

58 Is TBMCS IA configuration adequate?7-14 Detection. Capability of

IA detection controls.E-2 6.2

59Elapsed time between when a penetration event occurred and when it was detected?

60 Number of events detected over events attempted.

61 Are TBMCS Intrusion Detection System(s) adequate?

TIEF for SUT Appendix C

6

Table C-1. JCOI to Navy Tasks and Measures

AFOTEC JCOI AFOTEC MOE# AFOTEC MOE Description COTF

COI COTF Subtask COTF M# COTF Measure Description

62 Is TBMCS audit logging adequate?63 Is TBMCS antivirus software adequate?

7-15 Response. Capability of IA response controls. E-2 6.3

64Number of successful reactions over the number of detected penetration events attempted.

65 Time taken by Test Asset personnel to react for each detectable event.

66 Course of action in response to event.

7-16Restoration. Capability of IA restoration controls.

E-2 6.467

Time to restore TBMCS support to operations after initiating restoration plan.

68 Is TBMCS system restoration adequate?69 Is TBMCS data restoration adequate?

8-1 Interoperability. E-1

1.1.2.1 45 Does TBMCS interface with AMDWS?1.1.2.2 46 Does TBMCS interface with AFATDS?1.1.2.3 47 Does TBMCS interface with TAIS?1.1.2.4 48 Does TBMCS interface with GCCS-M/J?1.1.2.5 49 Does TBMCS interface with DCGS-N?1.1.2.6 50 Does TBMCS interface with JMPS?1.1.2.7 51 Does TBMCS interface with GDSS?

8-2

Information Exchange. Capability of TBMCS s1.1.3 MR2 to send and receive critical information.

E-1 1.2.1.1 43

Percent of critical Information Exchange Requirements TBMCS can send and receive (# of critical IERs sent and received/# of critical IERs) (KPP)

9-1Impact of TBMCS s1.1.3 MR2 on current command and control TTP.

E-1 1.5.2 52 Does TBMCS capability affect existing TTPs?

7-2 Reliability. S-1 2 70 MTBOMF HW

7-3 Maintainability. S-2 3

71 MTBOMF SW72 MCMTOMF HW73 MCMTOMF SW

74Does TBMCS have adequate documentation and online help to support system maintenance?

75 Can TBMCS be operated and maintained safely with current manning levels?

7-4 Operational Availability (Ao). S-3 4 76 Ao

7-5 Logistic Supportability. Capability to support

S-4 5 78 Is there a Life Cycle Support Plan (LCSP) approved and implemented for

TIEF for SUT Appendix C

7

Table C-1. JCOI to Navy Tasks and Measures

AFOTEC JCOI AFOTEC MOE# AFOTEC MOE Description COTF

COI COTF Subtask COTF M# COTF Measure Description

TBMCS s1.1.3 MR2 logistics requirements as stated in the Product Support Management Plan (PSMP), 19 August 2002.

TBMCS?

7-6

Training. Overall sufficiency of TBMCS s1.1.3 MR2 difference training.

S-4 5 77 Is there an approved and implemented training solution for TBMCS?

7-7

Human Factors. Adequacy of the human factors associated with TBMCS s1.1.3 MR2.

S-2 3 75 Can TBMCS be operated and maintained safely with current manning levels?

7-8

Safety. Capability of TBMCS s1.1.3 MR2 to be safely operated and maintained in an operational environment.

S-2 3 75 Can TBMCS be operated and maintained safely with current manning levels?

ABP – Air Battle PlanACMREQ – Airspace Control Measures RequestACO – Airspace Control OrderAFATDS - Advanced Field Artillery Tactical Data SystemAMDWS - Air and Missile Defense WorkstationASR – Access Service RequestATO – Air Tasking OrderCAS – Close Air SupportDASC - Direct Air Support CenterDCGS-N - Distributed Common Ground System - NavyESTAT – Execution StatusFSTAT – Force StatusGCCS-M/J – Global Command and Control System – Maritime/JointGDSS - Global Decision Support SystemIA – Information Assurance

IIR – Imagery Interpretation ReportIPIR – Initial Phase Interpretation ReportJFACC – Joint Force Air Component CommanderJFC – Joint Force CommanderJIPTL - Joint Integrated Prioritized Target ListJMPS – Joint Mission Planning SystemJTASR – Joint Tasking RequestMAAP - Master Air Attack PlanMISREP – Mission ReportMOP – Measure of PerformanceMR2 – Maintenance Release 2SPINS – Special InstructionsTACC - Tactical Air Command CenterTAIS - Tactical Airspace Integration SystemUSMTF – U.S. Message Text Format

TIEF for SUT Appendix C

8

(U) BLANK PAGE

TIEF for SUT Appendix C

Blank pages are only added if the section (without the blank page) ends on an odd numbered page. Each section will begin with an odd numbered page.

1

APPENDIX D - ACRONYMS AND ABBREVIATIONSThere is a 2-column table below for entering acronyms and abbreviations. Insert rows as necessary. Leave a blank row between each alphabet grouping. In addition to acronyms in the body of the document, ensure that all acronyms used in the appendix tables (which may not have been previously defined in the document) are captured here.

MBTD Mission-Based Test Design

SoS System of SystemsSUT System Under Test

TEMP Test and Evaluation Master Plan

TIEF for SUT Appendix D

Use acronyms from the CAAL; if the correct acronym is not in the CAAL, request that it be added.

2

BLANK PAGE

TIEF for SUT Appendix D

Blank pages are only added if the section (without the blank page) ends on an odd numbered page. Each section will begin with an odd numbered page.

1

APPENDIX E - REFERENCESList all references used in construction of this IEF. Include all documents called out as sources of SUT and SoS measures. Also list anything used to create tasks and conditions including documentation on the kill chains. Include any IEFs used for comparison.EXAMPLE

(a) COMOPTEVFORINST 3980.2D, Operational Test Director’s Manual of 30 Sep 14

(b) COMOPTEVFOR PIN 10-01, Operational Reporting Guidance and Procedures of 2 Mar 10

(c) Previous IEF version of date (if applicable) (list all)

(d) SYSTEM ACRONYM Test and Evaluation Master Plan (TEMP) No. XXXX of date (U)

(e) SYSTEM ACRONYM ORD/CDD/CPD of date

(f) COMOPTEVFOR ltr 3980 Ser 54/S231 of 23 Aug 11

In this example, reference (d) is a classified TEMP, but the title is unclassified. Reference (f) is a classified letter, which is depicted by the "S" in the serial #. References (a) and (b) are examples of unclassified references.

TIEF for SUT Appendix E

If the document is classified but the title isn't, place a (U) after all references listed with the complete title. If the title is classified, use the appropriate classification for it.

2

BLANK PAGE

TIEF for SUT Appendix E

Blank pages are only added if the section (without the blank page) ends on an odd numbered page. Each section will begin with an odd numbered page.