integrated reliability index whitepaper draft

7/30/2019 Integrated Reliability Index WhitePaper DRAFT

1/11

1

Integrated Reliability Index Concepts

Thedevelopmentofanintegratedreliabilityindexaimstoinform,

increasetransparency,andquantifytheeffectivenessofriskreduction

and/ormitigationactions. Thegoalistoprovidetheindustrymeaningful

trendsofthebulksystemperformanceandguidanceonhowtheycanimprove

reliabilityandsupportriskinformeddecisionmaking.

Underthe

direction

of

the

Operating

Committee

(OC)

and

Planning

Committee

(PC),

the

Reliability Metrics Working Group (RMWG) has developed a portfolio of eighteen (18)

Adequate Level of Reliability (ALR) metrics.1 At its March 2011 meeting, the Operating

Committee (OC) and Planning Committee (PC) approved the Severity Risk Index (SRI)2

calculationandsupportedthedevelopmentofanIntegratedReliabilityIndex(IRI),whichcanbe

constructedbasedontheriskmodelillustratedinFigure1. Thismodelattemptstocapturethe

universeofriskandlinksthatriskcalculationtothereliabilityofthebulkpowersystem.

Figure1RiskModelforBulkPowerSystem

1 http://www.nerc.com/docs/pc/rmwg/RMWG_AnnualReport6.1.pdf2http://www.nerc.com/docs/pc/rmwg/Integrated_Bulk_Power_System_Risk_Assessment_Concepts_Final.pdf

EventDrivenIndex(EDI)IndicatesRiskfrom

MajorSystemEvents

Standards/StatuteDriven

Index(SDI)Indicates

Risks

fromSevereImpactStandardViolations

ConditionDrivenIndex(CDI)

IndicatesRisk

from

KeyReliabilityIndicators


2/11

2

EventDriven Index (EDI)TheEventDriven Index (EDI)providesabasis forprioritizationofevents based on bulk power system integrity, equipment performance, and/or engineering

judgment. The event severity indicators can serve as a high value risk assessment tool.

Stakeholderscanusethetooltomeasuretheseverityoftheseeventsandevaluatedisturbance

history.The

relative

severity

ranking

of

events

considers

both

the

occurrence

of

an

event

and

its impact, inordertoquantifytheeventrisktoreliability. The indexEDI increases iftherisk

valueassociatedwithsignificanteventsisreducedoveratrendingperiod. TheEDIcalculation

andmetricaggregationaredescribedinAppendixA.

ConditionDriven Index (CDI)ConditionDriven Index (CDI) focusesona setofmeasurablesystemconditionstoassessbulkpowersystemreliability. These reliability indicators identify

factors that positively or negatively impact reliability and are earlypredictors of the risk to

reliabilityfromconditionbasedALRmetrics. Acollectionoftheseindicatorsmeasureshowfar

reliabilityperformanceisfromdesiredoutcome,andiftheperformanceisheadingortrending

in the preferred direction. The index CDI increases if the risk value associated with key

reliabilitymetrics

covering

major

risk

factors

to

reliability

is

reduced

over

atrending

period.

TheCDIcalculationandmetricaggregationaredescribedinAppendixB.

Standards/Statute Driven Index (SDI) The Standards/Statute Driven Index (SDI) measuresimprovementincompliancewithReliabilityStandards.

3 TheviolationsincludedinSDIallhave

high Violation Risk Factors (VRFs) and were rated as potentially severe reliability impacts.

Basedon these twodimensions,knownunmitigatedviolationsareweightedandnormalized

overthenumberofapplicableregisteredentitiessubjecttoaparticularstandardrequirement.

Theindexincreasesifthecomplianceimprovementisachievedoveratrendingperiod. TheCDI

calculationandmetricaggregationaredescribedinAppendixC.

As

this

time,

the

IRI

is

intended

to

be

used

as

a

historical

measure,

neither

a

real

time,

nor

forwardlookingperformance score. As such, the index iscalculatedusingactual fielddata.4

The value of the IRI can be calculated based on the assessed risk of the above three

componentsandtheirrelativeweightings,asshowninEquation1below:

(1)

where:

weighting of event component;

normalized Event Driven Index;

weighting of metric component;

normalized Condition Driven Index;

weighting of standard compliance component; normalized Standards Driven Index;

Integrated Reliability Index for a speciic period.

3Detailedstandards/statutedrivenindicatorscanbeviewedathttp://www.nerc.com/filez/pmtf.html.4Fromarealtimeorforwardlookingpointofview,onecanalsocreateotherindiceslinkingrealtimeorforecast

variablesandfactoringinprobabilities. Astheindustrygainsexperiencewiththisbackwardlookingmetricitwill

continuetoconsiderwhethercertainrealtimeorforecastmetricsarevaluableindicatorsofthereliabilityofthe

bulkpowersystem.


3/11

3

Thevalueof IRIwill range from0 to100,and canbeaggregatedatvarious levelsofdetail,

includingattheNERC,InterconnectionorRegionallevels. Thethreeweightscanbeadjustedas

we learn more and gain experience after one to two years of trending. At this time it is

suggestedthataweightingof

(2)

isusedinitiallyastheEDIreflectsactualexperiencesofthepowersystemwhileCDIandSDIare

morereflectiveofriskstothepowersystem.

ThefollowingfactorsaretobeconsideredwhendevelopingeachoftheIRIcomponents:

Covermajorityofrisks

Showaconsistentlinktoreliability

Considerrandomnessandindependence

Shouldunderstandthedegreetowhichanyofthethreeindicesmayrelatetootherindices

Conceptually,thecomponentsmightbeconsideredassuccessratesforeventsexperienced

(asmeasuredbyEDI),conditionperformance(asmeasuredbyCDI),andcompliancehistory(as

measuredby SDI). Components shouldbemeasurable for specific timeperiodsof interest,

suchasquarterlyorannually.

Another important concept is the extent to which the IRImeasures the intersection or the

union of riskswithin the bulk powers system. At this time, there appears to be benefit in

shrinking the size of the populations (or weightings) for CDI and SDI. Also, currently the

equationreflectstheadditionofrisks,ratherthanthemeasurementofanyoftheunions(and

increasedweightings)ofeachofthecomponentsoftheIRI. Thisconceptisshowngraphically

inFigure

2.

Depending

upon

the

model

preferred,

the

equation

and

relevant

indices

should

be

modifiedappropriately.


4/11

4

Figure2UnionIRIorIntersectingIRI

IRIIndexComponentsAsdiscussedpreviouslyIRIisintendedtobeacompositemetricwhichintegratesseveralforms

of individual risks to the bulk power system. Upon their factoring andweighting a lagging

indicator will be developed on a quarterly or annual basis. At this time three component

indicesarebeingconsidered. Eachofthesecomponentsisdiscussedingreaterdetailbelow. It

is important to caution that since they range across many stakeholder organizations, these

concepts are developed as starting points for continued study and evaluation. Additional

supportingmaterialsarefoundintheAppendicesasmuchoftheindividualindicescalculations

andsupportingtrendinformationisinaformativestate.

SampleIntegratedReliabilityIndex(IRI)CalculationAt this time, the RMWG believes some form of blended weighting may serve to start to

populate IRI characteristic curves at a high and generic level. Based upon feedback from

stakeholders,thisapproachmaychange,orasdiscussedfurtherbelow,weightingfactorsmay

varybasedonperiodicreviewandriskmodelupdate.TheRMWGwillcontinuetherefinement

of the IRI calculation and consider other significant factors that impact reliability (e.g.,

intentionaland controlled loadshedding); further, itwillexploredevelopingmechanisms for

enablingongoingrefinementwhichshouldbeinfluencedbyawidesetofstakeholders.

RMWG

recommends

the

Event

Driven

Index

(EDI)

be

weighted

the

highest

(50%)

since

events

actuallyoccurred,indicatinghowsystemwasperforming.TheConditionDrivenIndex(CDI)and

Standards/Statute Driven Index (SDI) are weighted 25% each since they are indicators of

potentialriskstoBPSreliability. UsingEquation3,IRIcanbecalculatedasfollows:

0.50 0.25 0.25 (3)


5/11

5

AAppppeennddiixx AA EEvveenntt DDrriivveenn IInnddeexx ((EEDDII)) CCaallccuullaattiioonn

EventDrivenIndex(EDI)The

event

severity

risk

index

(SRI)

was

developed

to

measure

relative

severity

ranking

of

events

basedoneventoccurrence rateand their impact to thebulkpower system. Impact canbe

amongmultipledimensionssuchasload(asaproxyforcustomers)orlossoffacilities(suchas

generators, transmission lines, substations or communications facilities). These measures

provideaquantitativeapproachtodeterminewhicheventshavemore impactonbulkpower

systemreliability. Inotherwords,themetricsarean integratedmeasurementsystem,which

classifies an events impact on each of the components that are critical to the holistic

performanceofthebulkpowersystem.

Figure3plotsthe20082010NERCSRIperformance,includinghistoricbenchmarkeventsusing

theapprovedSRIformula.Theimpactconsidersloadlossandduration,ACtransmissioncircuit

andgeneration

losses

occurred

from

recorded

events,

including

weather

and

other

natural

events. The load lossanddurationdataweregathered fromEOP0045andOE417

6 reports.

ThedailygenerationandACtransmissioncircuitoutageswereobtainedfromGADS(Generating

AvailabilityDataSystem)andTADS (TransmissionAvailabilityDataSystem),respectively. The

eventcategory7anditsassociatedseverityriskrangearealsoshowninFigure3.

Figure3NERC20082010SeverityRiskIndexversusHistoricBenchmarkDays

5http://www.nerc.com/files/EOP0041.pdf6http://www.eia.doe.gov/cneaf/electricity/forms/instfor417.pdf7http://www.nerc.com/docs/eawg/Event_Analysis_Process_Field_test_DRAFT_102510Clean.pdf


6/11

6

The EDI component is derived from RMWGs Severity Risk Index (SRI) values8 and can be

computedasfollows:

(4)

wheretheSRIisadailySeverityRiskIndexandthedurationisaspecifictimeperiodofinterest,

suchasquarterlyorannually. ThevalueofEDIwillrangefrom0and100

BasedontheSRIvalues inFigure3andEquation4,quarterlyEDItrending forthepastthree

yearsispresentedinFigure4.

NosignificantEDItrendchangesareobservedfrom2008to2010.

Figure4NERC20082010EventDrivenIndexbyQuarter

8http://www.nerc.com/docs/pc/rmwg/Integrated_Bulk_Power_System_Risk_Assessment_Concepts_Final.pdf

99.70

99.75

99.80

99.85

99.90

99.95

100.00

EventDriven

Index(EDI)


7/11

7

AAppppeennddiixx BB CCoonnddiittiioonn DDrriivveenn IInnddeexx ((CCDDII)) CCaallccuullaattiioonnIn 2009, the RMWG developed the S.M.A.R.T criteria

9 (Specific, Measurable, Attainable,

Relevant and Tangible), shown in Table 1, as the ranking process providing a consistent

approachto

identify

ahigh

ranking

subset

of

metrics.

Table1SMARTMethodandRating(*SMARTScore=SumofRatingsforS,M,A,R,andT)

Table1SMARTMethodandRating*

Rating

S - Specific/Simple

Be easily understoodand not driven bycommercial factors (i.e.tariff)Identify factors thatpositively or negativelyimpact reliability

Address reliabilityproblems and solutions

M - Measurable

Be easily measuredwith regularly collectedinformation

Measure past andcurrent reliability

Measure progress in

ensuring reliability

Measure effectivenessof reliability standardsand enforcementprograms

A - Att ainable

The industry canprovide the rightresources (i.e. funding,time and ability) toimprove reliability

Reliability will bemeasurably improved

R - Relevant

Linked to reliabilitygoals

P rovide meaningfulinformation

P rovide feedback forimproving the Reliability

Standards

T - Tangible/Timely

Reflect current toppriority issues andpossess a sense ofurgency

Identify reliability gapsand point to existingstandards that need to

be modified or newstandards that need tobe developed

3

Defined in a NERCStandard

Not driven bycommercial factors

Addresses specificreliability issues

Easily measured andreported regularly

Historical data existsat REs and/or NERCand is currentlyrequired in NERCStandard

Directly measure

effectiveness ofstandard andenforcement programs

Compelling businesscase suggests goodchance of beingapproved throughbusiness planning andtariff approvalprocesses

Reliabilityimprovements will be

easily seen

Direct link to reliabilitygoalsFocus on failures andpossible solutions orimprovements

Provide directfeedback for improvingthe ReliabilityStandards

Directly links tocurrent top priorityreliability issues andpossess a sense ofurgency

Clearly identifiesreliability gaps andpoints to standardimprovement needs

2

Defined within theindustry

Not driven bycommercial factors

Addresses reliabilityissues

Easily measured andreported on occasionsSome historical dataexists at REs and/orNERC and is currentlyrequired in NERCstandard

Measure effectivenessof standard andenforcement programsin a long run

Additional resourceswill be required andhave a reasonablechance of beingapproved throughbusiness planning andtariff approvalprocesses

Reliabilityimprovements will beapparent within areasonable period oftime (months)

Some link to reliabilitygoals

Not directly focus onfailures and possiblesolutions orimprovements

May provide feedbackfor improving theReliability Standards ina long run

Some link to currenttop priority reliabilityissues

May reveal reliabilitygaps in a long run

1

Defined somewhere

May have somecommercial factors

May relate to reliabilityissues

Easily measured andnot reported

Some historical dataexists at REs and/orNERC

No link toeffectiveness ofstandard andenforcement programs

Significant resourceswill be required wellbeyond normalbusiness planning andtariff approval levels

Reliabilityimprovements will onlymarginal, or evidentover an extendedperiod of time (years)

No link to reliabilitygoals

Not Focus on failuresand possible solutionsor improvements

Not tied to a standardimprovement

No link to current toppriority reliability issuesand does not possess asense of urgency

Does not identifyreliability gaps.

9 http://www.nerc.com/docs/pc/rmwg/RMWG_Metric_Report090809.pdf


8/11

8

Figure 1: Metric Development Process

RMWG recommendsusingacombinationofRelevantandTangible ratingsofeachmetricas

weighting factors. Table 2 lists eachmetricsRelevant and Tangible scores10

and combined

metricweightingfactors.

Table2CDIMetricWeightingFactor(WF)andTrendRating(TR)

ALR Description Relevant Tangible WF2010

PerformanceTrend TrendRating

13 PlanningReserveMargin 2 3 5Slight

Improvement4

14BPSTransmissionRelatedEventsResultinginLoss

ofLoad3 3 6

Slight

Improvement4

15 SystemVoltagePerformance 3 2 5 NoData 0

112 FrequencyResponse 3 2 5 NoData 0

23 ActivationofUnderFrequencyLoadShedding 2 2 4 Inconclusive 3

24 AveragePercentNonRecoveryDCS 3 3 6 Inconclusive 3

25DisturbanceControlEventsGreaterThanMost

SevereSingleContingency2 2 4

Slight

Improvement4

35InterconnectionReliabilityOperatingLimit/

SystemOperatingLimit(IROL/SOL)Exceedance3 3 6 NewData 0

41AutomaticTransmissionOutagesCausedby

ProtectionSystemMisoperations3 3 6 Inconclusive 3

61 TransmissionConstraintMitigation 3 2 5 NewData 0

62 EnergyEmergencyAlert3(EEA3) 3 3 6Slight

Improvement4

63 EnergyEmergencyAlert2(EEA2) 3 3 6Slight

Improvement4


ProtectionSystemMisoperations3 2 5 Inconclusive 3


HumanError3 2 5 Inconclusive 3


FailedACSubstationEquipment3 2 5 Inconclusive 3


FailedACCircuitEquipment3 2 5 Inconclusive 3

615 ElementAvailabilityPercentage(APC) 2 2 4 NewData 0

616TransmissionSystemUnavailabilityon

Operational,PlannedandAutoSustained

Outages

2 2 4 NewData 0

10http://www.nerc.com/filez/Approved_Metrics.html


9/11

9

To integrate individualmetricswithdifferingunitsofmeasure,thefivecommontrendratings

(TR)were identifiedtoquantifyeachmetricperformance level,shown inTable2. Thesefive

ratingsrangefrom1to5. Ifnometricdatatrendisyetavailable,thetrendratingisassigned0.

Thesignificantimprovementtrendhasthehighestratingof5,andothertrendratings,aslisted

below:

TR 5 for Significant Improvement,

4 for Slight Improvement,

3 for Neutral or Inconclusive,

2 for Slight Deterioration, and

1 for Significant Deterioration.

0 for New Data or No Data

Theindividualmetriccontributioncanbecomputedas

(5)

Thevalue

of

metric

contribution

will

range

from

0to

100.

The

CDI

aggregates

each

metric

contributionasaweightedaveragebyrelativeimportance. Forexample,SMARTratingsofthe

abovefivemetricsservesastheirimportancevaluesintheCDIequationbelow:

(6)


10/11

10

AAppppeennddiixx CC SSttaannddaarrddss//SSttaattuuttee DDrriivveenn IInnddeexx ((SSDDII))

TheRMWGrecommendsusingtheReliabilityImpactStatement(RIS)andViolationRiskFactor11

(VRF)asselectioncriteriatoidentifythesubsetofstandardrequirementstobeincludedinthe

SDI. RISistheassessmentofrisktothebulkpowersystemwhenarequirementisviolated,as

determinedbytheRegionalEntity. ThethreeRISlevelsareminimalimpact,moderateimpact

andsevereimpact. ThefactorsincludedintheRISare

TimeHorizon

Relativesizeoftheentity

Relationshiptootherentities

Possiblesharingofresponsibilities

Voltagelevelsinvolved

Sizeofgeneratororequipmentinvolved

Abilityto

project

adverse

impacts

beyond

the

entitys

own

system

Failuretomaintainvegetationclearancecouldcauseorexacerbateacascadingoutageisan

example of severe impact; lack of regular maintenance and testing could result in

misoperations isanexampleofmoderate impact; theentity is conducting training,but its

training program did not include a plan for the initial operator training is an example of

minimalimpact.

Basedon theabovecriteria,thetwentysixstandard requirements inTable3were identified

fromNERCs3yearcompliancedatabaseashavingasevereRISandhighVRF.

Table3StandardRequirements*IncludedinSDI

*RequirementsareidentifiedfromtheNERCcompliancedatabasewheretheirviolationshave

severeRIS.

11http://www.nerc.com/page.php?cid=2|20|285

Standard Req. Standard Req. Standard Req. Standard Req. Standard Req.

EOP-001-0 R1. FAC-009-1 R1. PER-002-0 R3. PRC-005-1 R2. TOP-004-2 R1.

EOP-003-1 R7. IRO-005-2 R17. PER-002-0 R4. TOP-001-1 R3. TOP-004-2 R2.

EOP-005-1 R6. PER-001-0 R1. PRC-004-1 R1. TOP-001-1 R6. TOP-006-1 R6.

EOP-008-0 R1. PER-002-0 R1. PRC-004-1 R2. TOP-001-1 R7. TOP-008-1 R2.

FAC-003-1 R1. PER-002-0 R2. PRC-005-1 R1. TOP-002-2 R17. VAR-001-1 R1.

FAC-003-1 R2.


11/11

11

ByapplyingasimilaraggregationasEDI,theSDIcanbecalculatedas

100

(7)

where:

integrated standard compliance index for a speciic period;

weighting of a particular requirement violation; number of known unmitigated violations for the selected requirement;

number of registered entities who are required to comply

with the selected requirement.

Assuming an equal weighting (wV) for all requirements, Figure 5 provides the quarterly SDI

trendsforpastthreeyears.

Figure5NERC20082010SDITrendbyQuarter

Figure5providesthequarterlySDItrendsforpastthreeyears,indicatingtheriskduetoknown

severe impactviolationshasdecreased,andthe industryhasachievedahigherreliability level

throughstandardscompliancesincethirdquarter2009.

99.70

99.75

99.80

99.85

99.90

99.95

100.00

Stan

dardsDrivenIndex(%)

integrated reliability index whitepaper draft

Documents