How Do You Test an IT How Do You Test an IT Service?Service?

Pete GreenPete Green

Tom FreyTom Frey

PurposePurpose

Explore evolving influence of IT Explore evolving influence of IT systems and services on weapons systems and services on weapons systemssystems

Increasing complexity of testingIncreasing complexity of testing Our experience foundation is in NMCIOur experience foundation is in NMCI Foci are the test processesFoci are the test processes

How Did We Get Here?How Did We Get Here?

JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN

TESP InputDelivered

Veridian sole sourcedBy EDS to conduct

NMCI CT&E

BSA conductedat 4 test sites

Master TestPlan Development

CT&E 1 TestingConducted

CT&E 3Testing

CT&E 2Testing

Master Test PlanCompleted & Delivered

Interoperability Testing

& Monitoring Plan Delivery

Contract SLAs mapped to COIs

NMCI ContractAwarded to EDS

NMCI DesignatedDOT&E Oversight

Program

DASN DirectsTESP for NMCI

2001 20022000

Rolling InRolling In

Compressed ScheduleCompressed Schedule No Traditional Program DocumentationNo Traditional Program Documentation Our Response Based on 4-step Test ProcessOur Response Based on 4-step Test Process Build Up (Crawl, Walk, Run)Build Up (Crawl, Walk, Run) Concentric TestingConcentric Testing Familiar FundamentalsFamiliar Fundamentals

FacesFaces ProcessesProcesses

Test ApproachTest Approach

BSABSA Phased TestingPhased Testing

CTE1 - DT (lab, bench)CTE1 - DT (lab, bench) CTE2 - DT (networks)CTE2 - DT (networks) CTE3 - DT/OT (operational vignettes)CTE3 - DT/OT (operational vignettes) OPEVAL - OTOPEVAL - OT

SLA based - derived CTPs, COIs,SLA based - derived CTPs, COIs,

Test ApproachTest Approach

SECURITY

Electronic Survivability

Average Availability

Configuration Integrity

Data Integrity

Network Delays

Application Delays Interoperability Reconfigurability Customer

Satisfaction Help Desk

Support Technology

Insertion Latency

Recovery Latency

Security Report Latency

1 Desktop

Hardware and Operating System

2 3 1 3 2

3 E-mail Services 1 3 4 2

4 Directory Services 1 2,3 4,5,6

10 NMCI Intranet Performance 1 2 3 4

12 Internet Access 1 2 3

15 Moves, Adds, and

Changes 1,2,4 5

23 Basic Help Desk Services

1,2,3,4 ,5,6,7

25 BAN/LAN

Communications Services

1,3 2 6 4

33 NMCI Security

Operational Services General

4,5,6 9 7,8

36 NMCI Security

Planning Services 1,2 , 4

NMCI SERVICE LEVEL AGREEMENTS

USER UPGRADES

END USER SERVICES

MAINTENANCE & HELP DESK SERVICES

COMMUNICATIONS SERVIC ES

INFORMATION ASSURANCE

NMCI OPERATIONAL

REQUIREMENTS INTEROPERABILITY & RELIABILITY NETWORK OPERATIONS & MAINTENANCE

NMCI MEASURES OF

EFFECTIVENESS

Test ApproachTest Approach

IT TestingIT Testing

Similar to Early Aviation TestingSimilar to Early Aviation Testing How fast can you go?How fast can you go? How high can you go?How high can you go?

OSI layer 1-3 centricOSI layer 1-3 centric Parameter focusParameter focus

IT T&E and the OSI 7-Layer IT T&E and the OSI 7-Layer ModelModel

WhereWe

need to be

MissionBased

PresentCommercial

ITTesting

Parametric

7654321

“Is This SystemEffective andSuitable?”

“How Fast Can ThisCar Go?”

Physical

Data Link

Network

Transport

Session

Presentation

Application

Product Line MaturityProduct Line Maturity

1950 1960 1970 1980 1990 2000

Sys

tem

s M

atu

rity

Aircraft

Avionics

Software

Network

s

C4I TestingC4I Testing

Success of testing tied to maturity of program Success of testing tied to maturity of program executionexecution

OTRROTRR Schedule ConstraintsSchedule Constraints IP systems generate lots of dataIP systems generate lots of data Program Execution curve is fractalProgram Execution curve is fractal

Processes on critical path provide 1st order Processes on critical path provide 1st order approximationapproximation

Use of surveysUse of surveys AAoo

Testing and Program Testing and Program ExecutionExecution

% Complete

Time

Full Rate Production

OPEVAL

OTRR

Idealized Program Execution Curve

Test IssuesTest Issues

NMCI equal to or better than legacy NMCI equal to or better than legacy networksnetworks

Critical path processesCritical path processes Seat cutoverSeat cutover Legacy applicationsLegacy applications

SLAs to COIs not a reversible functionSLAs to COIs not a reversible function Operationalizing the metricsOperationalizing the metrics Early and combined CT/OT to minimize riskEarly and combined CT/OT to minimize risk

Baseline System Baseline System AssessmentAssessment

Innovative contract vehicleInnovative contract vehicle Lack of traditional documentationLack of traditional documentation Needed to measure the performance Needed to measure the performance

of the legacy systems for comparison of the legacy systems for comparison with NMCIwith NMCI

BSA ToolsBSA Tools AMPAMP NetIQNetIQ MIB scriptsMIB scripts Help Desk LogsHelp Desk Logs BelManageBelManage SurveysSurveys

End userEnd user CIO and IT professionalsCIO and IT professionals Information AssuranceInformation Assurance

Use same tools and processes for OPEVALUse same tools and processes for OPEVAL

BSA Lessons LearnedBSA Lessons Learned

Configuration management is Configuration management is historicalhistorical

Good management makes an Good management makes an immediate impactimmediate impact Firewalls and IDSFirewalls and IDS RoutersRouters ConnectivityConnectivity

Legacy applicationsLegacy applications

TESP ProcessTESP Process

Similar to a TEMPSimilar to a TEMP More responsiveMore responsive If it was not necessary, it was not includedIf it was not necessary, it was not included If it was necessary, it was tailored to NMCIIf it was necessary, it was tailored to NMCI

From blank sheet to approval in under From blank sheet to approval in under 11 months11 months

Represented a strategy for success for Represented a strategy for success for NMCINMCI

The State of IT Testing The State of IT Testing TodayToday

Operational assessments of interoperability Operational assessments of interoperability and information assurance in fielded systemsand information assurance in fielded systems

Inverse correlation of interoperability and Inverse correlation of interoperability and information assurance information assurance

Mission Thread AnalysisMission Thread Analysis Blue, Green, and Red TeamsBlue, Green, and Red Teams Common ToolkitsCommon Toolkits Information Operations Threat Capabilities Information Operations Threat Capabilities

SummariesSummaries Feedback and feed forwardFeedback and feed forward

Interoperability vs. Information Interoperability vs. Information Assurance:Assurance:

A Complex ChallengeA Complex Challenge

Interoperability (Iop)

InformationAssurance (IA)

Iop(ax) = by

IA

An Enterprise System Needs to Balance Iop and IA

Testing FrameworksTesting Frameworks

Device TestingDevice TestingSystem TestingSystem Testing

System of Systems TestingSystem of Systems Testing

Work Flow ThreadsWork Flow Threads

Mission-Focused TestingMission-Focused Testing

A

Z

Example Mission ThreadExample Mission Thread(ATO Distribution)(ATO Distribution)

BCDPLNS WS

GCC Prioritized

Target List

KCOIC

1

TWM &CTB-AMServer

KCOICCTAPSWS

HTACC

CTAPSAPSServer

CTAPSADSServer

HTACCCTAPSWS

RELROK WAN

Integrated

Tasking Order7

KCOICCTAPSWS

KCOIC

GCCS-KC2/COPWS

CTAPSITO DBServer

RELROK WAN

ROK AF (Suwon)

CTAPSTGTWS

5

KUNSAN

CTAPSTGTWS

6

Training Facility

GCCS-KWebServer

RELROK WAN

CP TANGO

GCCS-KC2/COPWS

8

RELROK WAN

RELROK WAN

RELROK WAN

2 3

4

2 ACCTargetNomination

ITOSent toTL # 1,6

Level of PerfSituation

Conditions

Joint Mission Areas UJTL

Assigned CoCom

MissionsCoCom JMETs

Lessons Learned CCTI

Strategies

Commanders Training

Guidance

Exercise Objectives

Joint DoctrineTT&Ps

Training ObjectivesTraining

Objectives

BLUFOR OPLAN

OPFOR Campaign

Control Plan

Scenario

MSEL

Collection Management

Plan

Integrated Architectures

C4ISP

Exercise Comm/Sys

Architecture

Mission Functional

Threads

Mission Functional

Threads

Execution

Assessment Plan

Assessment Plan

Performance Data

Performance Analysis

After Action Review

System SMEs

Mission SMEs

Operational Testers

MISSION ISSUESMISSION ISSUES

EXERCISE ISSUESEXERCISE ISSUES

SYSTEM ISSUESSYSTEM ISSUES

PE

RF

OR

MA

NC

E

AS

SE

SS

ME

NT

PE

RF

OR

MA

NC

E

AS

SE

SS

ME

NT

Standards

Assessment Methodology Assessment Methodology FlowFlow

JTS

QUESTIONS?QUESTIONS?