EPA’s EPA’s Bioassessment Bioassessment

Performance and Performance and Comparability Comparability

“Guidance”“Guidance”

Why Develop this Why Develop this GuidanceGuidance

Previous NWQM Conferences expressed needPrevious NWQM Conferences expressed need Several states, tribes, and others want this Several states, tribes, and others want this

guidanceguidance EPA views this as a critical component to EPA views this as a critical component to

strengthen existing bioassessment programsstrengthen existing bioassessment programs Credible data laws; over-interpreting resultsCredible data laws; over-interpreting results Links with other bioassessment programs: Links with other bioassessment programs:

critical elements, reference condition criteriacritical elements, reference condition criteria

Document OutlineDocument Outline Background:Background: Bioassessments and DQOs; Bioassessments and DQOs;

Performance-based approach and comparability Performance-based approach and comparability defined; lit reviewdefined; lit review

Performance-based MethodsPerformance-based Methods: Performance : Performance characteristics defined; relationship to characteristics defined; relationship to comparability, data qualitycomparability, data quality

Documenting PerformanceDocumenting Performance: how to; challenges; : how to; challenges; examplesexamples

Determining ComparabilityDetermining Comparability: at data, metric and : at data, metric and assessment levels; rules; challenges; examplesassessment levels; rules; challenges; examples

Documentation and ReportingDocumentation and Reporting: metadata; forms: metadata; forms

EPA Bioassessment EPA Bioassessment Guidance: Guidance: PerformancePerformance

Includes:Includes: Recommended performance characteristics that Recommended performance characteristics that

should be documented for a bioassessment should be documented for a bioassessment protocol and its submethods (e.g., laboratory protocol and its submethods (e.g., laboratory procedures, taxonomy)procedures, taxonomy)

How to calculate performance characteristicsHow to calculate performance characteristics

Forms to help document performance and Forms to help document performance and recommended metadata (water quality data recommended metadata (water quality data elements)elements)

Examples of performance-based methods for Examples of performance-based methods for different DQOs different DQOs

Case study examples that calculated performance Case study examples that calculated performance for bioassessment protocols or submethodsfor bioassessment protocols or submethods

EPA Bioassessment EPA Bioassessment Guidance: Guidance: ComparabilityComparability

Includes:Includes: Recommended rules for combining Recommended rules for combining

bioassessment databioassessment data Recommended rules for combining Recommended rules for combining

assessmentsassessments Case study examples that examined Case study examples that examined

comparability of different protocols or comparability of different protocols or datasetsdatasets

Recommendations to enhance comparability Recommendations to enhance comparability evaluations and promote comparability of evaluations and promote comparability of information derived from different programsinformation derived from different programs

Identify objectives and Identify objectives and design of monitoring design of monitoring

projectproject

Study Objectives or Study Objectives or Monitoring QuestionMonitoring Question

Data-Quality Objectives Data-Quality Objectives (DQO)(DQO)

(includes sampling (includes sampling design)design)

Collect field samples and Collect field samples and on-site dataon-site data

Sampling PlanSampling Plan Field Certification and Field Certification and

TrainingTraining Quality-Assurance PlanQuality-Assurance Plan Field-activities Safety Field-activities Safety

PlanPlan

Produce data from Produce data from laboratory analyseslaboratory analyses

Quality-Assurance PlanQuality-Assurance Plan Specify Method Specify Method

Acceptance CriteriaAcceptance Criteria Laboratory Laboratory

AccreditationAccreditation Taxonomic certificationTaxonomic certification Method VerificationMethod Verification

Manage DataManage Data Required Metadata Required Metadata (data elements), (data elements), including data-quality including data-quality documentationdocumentation

Major Project Elements Comparability Requirements

Bioassessment Protocols Consist of Several Bioassessment Protocols Consist of Several Steps, Each Benefiting From a Performance Steps, Each Benefiting From a Performance

AssessmentAssessment

Generalized FlowchartGeneralized FlowchartState management objectives

Specify data quality objectives (DQO)

Specify measurement quality objectives (MQO) (= acceptance criteria)

Collect data

Document protocol performance

Evaluate performance relative to MQOs

Exclude/reject data not meeting MQOs

“Calculate” indicators

Address management objectives

Select indicators

Bioassessment ExampleBioassessment ExampleDetermine stream miles that are biologically impaired

Determine to a 90% degree of confidence whether or not a site is impaired or not

Precision of indicators for replicate samples 20% RSD or 80% similarity;

80% discrimination efficiency; stressor gradient MQOs? Sensitivity?

Collect field replicates; split lab samples; sort residue QC;

% taxonomic agreement

Using data meeting MQOs, calculate assessment values

Interpret findings in context of management objective

Metrics or index responsive to stressors and reliable, representative sampling methods

Performance CharacteristicsPerformance Characteristics

Tentatively: Tentatively: Precision, sensitivity, Precision, sensitivity, responsiveness and bias/accuracyresponsiveness and bias/accuracy

Could pertain to both field and Could pertain to both field and laboratory procedureslaboratory procedures

Sampling method also includes Sampling method also includes representativenessrepresentativeness

Need replication at sites along a Need replication at sites along a human disturbance gradient – HDG human disturbance gradient – HDG (from very best to very worst)(from very best to very worst)

Reference condition data are keyReference condition data are key Appropriate definition of the HDG is Appropriate definition of the HDG is

essentialessential

Performance; Performance; ComparabilityComparability

::

What is it? What is it? What do we What do we

mean?mean?

MethoMethodsds IndicatIndicat

orsors

Reference Reference ConditionCondition

SampliSampling ng DesignDesign

??

??

??

We Need Your We Need Your InputInput

Definitions: Performance CharacteristicsDefinitions: Performance Characteristics

Performance Characteristic Definition

Precision(Repeatability and reproducibility)

Repeatability variability in biological index or indicator measure using split samples. (lab error)Reproducibility variability in biological index or indicator measure using duplicate field samples. (measurement error – includes lab error)

Sensitivity

Level of generalized stressor gradient (or HDG) at which the biological index or indicator is statistically different from reference condition.

Bias/Accuracy(False positive or false negative rate)

Degree to which truly stressed sites are systematically classified as unimpaired (false negative) , or truly non-stressed sites are systematically classified as impaired (false positive).

ResponsivenessDegree to which a biological index or indicator measure decreases monotonically with increasing stress.

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6

HDG Classes

Bio

asse

ssm

ent

Ind

ex

Variance in Index Score - bioassessment precision

Variance in HDG Score – stressor precision

Two Sources of Error to AddressTwo Sources of Error to Address

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6

HDG Classes

Bio

asse

ssm

ent

Ind

ex

Sensitivity Method A distinguishes HDG 3 from HDG 2 Method B distinguishes HDG 4 from HDG 2

A

B

A

B

Reference variability

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6

HDG Classes

Bio

asse

ssm

ent

Ind

exPrecision

ResponsivenessResponsiveness

0

20

40

60

80

100

1 2 3 4 5 6

HDG Classes

Sim

ilar

ity

Ind

ex

Dissimilar

Identical

Similar to Reference

A

Each of these index values does not overlap with reference nor with each other – 5 distinct classes of stress identified.

0

20

40

60

80

100

1 2 3 4 5 6

HDG Classes

Sim

ilar

ity

Ind

ex

Similar to Reference

Three of these index values do not overlap with reference; the value for HDG 5 overlaps with HDG 4; therefore, only 3 different classes of stress distinguished (HDG 2, 4, and 6)

Responsiveness

Dissimilar

Identical

Bias / Assessment AccuracyBias / Assessment AccuracyNon-impaired mean and 95% CI based on bioassessment data from Non-impaired mean and 95% CI based on bioassessment data from

several reference sites; need several test sites in the HDG 1-3 several reference sites; need several test sites in the HDG 1-3 classes and in the HDG 4-6 classes to test for false positive and classes and in the HDG 4-6 classes to test for false positive and

negative rates.negative rates.

0

20

40

60

80

100

1 2 3 4 5 6

HDG Classes

Bio

asse

ssm

ent

Ind

ex

Non-impaired Impaired

False Negative

False Positive