![Page 1: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/1.jpg)
11
Validation & Measurement Methods for the PHARE Demonstrations Validation & Measurement Methods for the PHARE Demonstrations
R A WhitakerR A Whitaker
Validation Project LeaderValidation Project Leader
![Page 2: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/2.jpg)
22
Analysis Is Not Easy!Analysis Is Not Easy!
![Page 3: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/3.jpg)
33
Unreasonable DemandsUnreasonable Demands
![Page 4: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/4.jpg)
44
Not UnderstoodNot Understood
![Page 5: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/5.jpg)
55
Does Anyone Listen?Does Anyone Listen?
![Page 6: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/6.jpg)
66
European Wide ValidationEuropean Wide Validation
NATSNATS
DLRDLR
CENACENA
NLRNLR
EECEEC
![Page 7: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/7.jpg)
77
European Wide Problems!European Wide Problems!
![Page 8: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/8.jpg)
88
European Wide SolutionEuropean Wide Solution
![Page 9: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/9.jpg)
99
What is Validation?What is Validation?
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
Validation builds confidence that Validation builds confidence that the system is fit for purposethe system is fit for purpose
![Page 10: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/10.jpg)
1010
Methodology Matureswith TimeMethodology Matureswith Time
PD1
PC/TC
PD1+
PD2
PD2+
PD3
PD1++
![Page 11: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/11.jpg)
1111
Experimental DesignExperimental Design
PD1 Validation PlanningPD1 Validation Planning
![Page 12: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/12.jpg)
1212
PHARE Experimental DesignPHARE Experimental Design
B A S E L IN E
O R G 1Too ls O n ly
O R G 2Too ls + 3 0 % D /L
O R G 3Too ls + 7 0 % D /L
Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be For Users to Understand Results - Baseline must be
Close to Current SystemClose to Current System
![Page 13: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/13.jpg)
1313
PHARE Experimental DesignPHARE Experimental Design
B A S E L IN E
O R G 1Too ls O n ly
O R G 2Too ls + 3 0 % D /L
O R G 3Too ls + 7 0 % D /L
Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be Close to For Users to Understand Results - Baseline must be Close to
Current SystemCurrent System Progressive Introduction of Experimental Variables in New Progressive Introduction of Experimental Variables in New
OrganisationsOrganisations
![Page 14: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/14.jpg)
1414
PHARE Experimental DesignPHARE Experimental Design
B A S E L IN E
O R G 1Too ls O n ly
O R G 2Too ls + 3 0 % D /L
O R G 3Too ls + 7 0 % D /L
Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be Close to For Users to Understand Results - Baseline must be Close to
Current SystemCurrent System Progressive Introduction of Experimental Variables in New Progressive Introduction of Experimental Variables in New
OrganisationsOrganisations
![Page 15: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/15.jpg)
1515
PHARE Experimental DesignPHARE Experimental Design
B A S E L IN E
O R G 1Too ls O n ly
O R G 2Too ls + 3 0 % D /L
O R G 3Too ls + 7 0 % D /L
Comparison To BaselineComparison To Baseline For Users to Understand Results - Baseline must be Close to For Users to Understand Results - Baseline must be Close to
Current SystemCurrent System Progressive Introduction of Experimental Variables in New Progressive Introduction of Experimental Variables in New
OrganisationsOrganisations
![Page 16: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/16.jpg)
1616
Experimental DesignExperimental Design Identifying Data CollectionIdentifying Data Collection
Objective DataObjective Data What; How; FormatWhat; How; Format
Subjective DataSubjective Data QuestionnairesQuestionnaires DebriefsDebriefs
Concentration on Controller Concentration on Controller WorkloadWorkload Instantaneous Self Assessment (ISA)Instantaneous Self Assessment (ISA) NASA TLX (paper)NASA TLX (paper)
PD1 Validation PlanningPD1 Validation Planning
![Page 17: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/17.jpg)
1717
Data Collected In One FileData Collected In One File Tedious data analysisTedious data analysis
No Recorded ObservationsNo Recorded Observations ““so-and-so said such-and-so-and-so said such-and-
such……I think”such……I think”
Limited Material For Debrief Limited Material For Debrief DiscussionsDiscussions
Video Data - but limited Video Data - but limited Controller Cognitive De-BriefController Cognitive De-Brief
PD1 Validation ExecutionPD1 Validation Execution
![Page 18: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/18.jpg)
1818
No Real Thoughts on Analysis No Real Thoughts on Analysis TechniquesTechniques Descriptive StatisticsDescriptive Statistics
Appropriate for Nominal Data?Appropriate for Nominal Data?
Inferential Statistics Employed as Inferential Statistics Employed as Analysis ProgressedAnalysis Progressed
Allowed Confidence IntervalsAllowed Confidence Intervals
Limited Considerations to Limited Considerations to Measures of MeritMeasures of Merit Capacity = fn(workload)?Capacity = fn(workload)? Quality of ServiceQuality of Service
PD1 Validation ExecutionPD1 Validation Execution
![Page 19: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/19.jpg)
1919
PC/TC, PD1+, PD2, PD2+ Validation PreparationPC/TC, PD1+, PD2, PD2+ Validation Preparation
Clearly Defined AimsClearly Defined Aims ObjectivesObjectives
HypothesesHypotheses
Measures of MeritMeasures of Merit Controller WorkloadController Workload Quality of ServiceQuality of Service CapacityCapacity UsabilityUsability
Data RequiredData Required Accessible StorageAccessible Storage
Formal ObservationFormal Observation PsychologistsPsychologists
![Page 20: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/20.jpg)
2020
Formal DocumentationFormal Documentation
Engineering PlanEngineering Plan Layout, tools, equipmentLayout, tools, equipment
Analysis PlanAnalysis Plan Aims, objectives, H0, H1Aims, objectives, H0, H1 DesignDesign
Scenario,Traffic samplesScenario,Traffic samples No. exercises, DurationNo. exercises, Duration Data recordingData recording OrganisationsOrganisations Ensures matched pairsEnsures matched pairs
MeasurementsMeasurements Analysis technique, Stat testsAnalysis technique, Stat tests
![Page 21: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/21.jpg)
2121
PC/TC, PD1+, PD2, PD2+ Validation ExecutionPC/TC, PD1+, PD2, PD2+ Validation Execution
Use of both Descriptive and Use of both Descriptive and Inferential StatisticsInferential Statistics Defined by Analysis PlanDefined by Analysis Plan
Supported with Subjective And Supported with Subjective And Formally Observed DataFormally Observed Data
Immediate Input to DebriefImmediate Input to Debrief Electronic TLXElectronic TLX Trajectory plotsTrajectory plots
Relatively Small and Well Relatively Small and Well Informed Core Group of Informed Core Group of “validators”“validators”
![Page 22: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/22.jpg)
2222
PD3 Validation PlanningPD3 Validation Planning
Validation Group expanded to Validation Group expanded to “non-analysts”“non-analysts”
Needed to Disseminate Needed to Disseminate Knowledge - DocumentedKnowledge - Documented Experimental Plan Experimental Plan
RequirementsRequirements Analysis Plan RequirementsAnalysis Plan Requirements
Aims, Objectives, H0, H1Aims, Objectives, H0, H1 …………..
Analytical MethodsAnalytical Methods
Acted as ConsultancyActed as Consultancy
![Page 23: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/23.jpg)
2323
PD3 Validation ExecutionPD3 Validation Execution
Planning Worked Well - Planning Worked Well - Execution of Analysis Execution of Analysis Limited to CENALimited to CENA
Documentation can Only be Documentation can Only be a Guide to Practitioners of a Guide to Practitioners of Best PracticeBest Practice Can’t Learn Analysis / Can’t Learn Analysis /
Validation from a BookValidation from a Book
![Page 24: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/24.jpg)
2424
PD1++ Validation Planning PD1++ Validation Planning
Planning - “routine”Planning - “routine” Fast-Time Simulation (TAAM) Fast-Time Simulation (TAAM)
IntroducedIntroduced Development of airspace, traffic Development of airspace, traffic
samplessamples Early examination of potential Early examination of potential
problem areasproblem areas
TrainingTraining Introduced Temporary Operating Introduced Temporary Operating
InstructionsInstructions
![Page 25: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/25.jpg)
2525
PD1++ Validation Execution PD1++ Validation Execution
Caught out by ComplacencyCaught out by Complacency Capacity measures used to date Capacity measures used to date
only applicable to non-changing only applicable to non-changing airspaceairspace
Cannot compare capacity between Cannot compare capacity between different airspace designs as used different airspace designs as used in PD1++in PD1++
Initiated research studyInitiated research study
![Page 26: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/26.jpg)
2626
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Core Validation Group with Representation from Core Validation Group with Representation from all Participating Sitesall Participating Sites ...those involved in the validation should have a thorough ...those involved in the validation should have a thorough
understanding of the operational concept under investigation understanding of the operational concept under investigation and of any restrictions/specifics of the simulation environment...and of any restrictions/specifics of the simulation environment...
Common Methodology CrucialCommon Methodology Crucial ……traceability of all input and output data is important for traceability of all input and output data is important for
correct analysis and can not be over-emphasised...correct analysis and can not be over-emphasised... ...the metrics applied in a research programme like PHARE ...the metrics applied in a research programme like PHARE
must be relevant to the performance of real life ATM...must be relevant to the performance of real life ATM...
![Page 27: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/27.jpg)
2727
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Methodology Improved with ExperienceMethodology Improved with Experience ……the Validation Methodology avoided disruptive data the Validation Methodology avoided disruptive data
collection. The use of intrusive measurements like eye collection. The use of intrusive measurements like eye tracking and heart-rate variability measurements could be tracking and heart-rate variability measurements could be valuable additions...valuable additions...
……the application of the appropriate technology to perform the application of the appropriate technology to perform the investigations is very important. The early choice of the investigations is very important. The early choice of PHARE to perform real time simulations has put a too strong PHARE to perform real time simulations has put a too strong focus on this type...focus on this type...
![Page 28: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/28.jpg)
2828
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Validation Should be Applied Throughout Validation Should be Applied Throughout Project - Not Just at the EndProject - Not Just at the End ...there has been too little input from the Validation project in ...there has been too little input from the Validation project in
the beginning of the PD/3 project. In hindsight it can be said the beginning of the PD/3 project. In hindsight it can be said that a stronger focus on the validation objectives of the that a stronger focus on the validation objectives of the project should have led to a better project definition...project should have led to a better project definition...
![Page 29: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/29.jpg)
2929
Validation is a Living SubjectValidation is a Living Subject
PHARE Validation Methodology developed PHARE Validation Methodology developed over time and through experienceover time and through experience
Methods are documentedMethods are documented
![Page 30: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/30.jpg)
3030
Validation is a Living SubjectValidation is a Living Subject
![Page 31: 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader](https://reader035.vdocument.in/reader035/viewer/2022062619/55154d9355034685568b5380/html5/thumbnails/31.jpg)
3131
Validation & Measurement Methods for the PHARE Demonstrations Validation & Measurement Methods for the PHARE Demonstrations
R A WhitakerR A Whitaker
Validation Project LeaderValidation Project Leadernextnext