successful nerc cip compliance - robert hoopes, ppl corporation
DESCRIPTION
Robert Hoopes - PPL Corporation, Speaker at the marcus evans Transmission & Distribution Summit Fall 2011, Wheeling, IL, delivered his presentation on Successful NERC CIP ComplianceTRANSCRIPT
11
SUCCESSFUL NERC SUCCESSFUL NERC CIP COMPLIANCECIP COMPLIANCE
Robert E. HoopesRobert E. Hoopes
PPL CorporationPPL Corporation
November 2, 2011November 2, 2011
22
OverviewOverview
Simple Compliance ModelSimple Compliance Model CIP CredentialsCIP Credentials Success RequirementsSuccess Requirements CIP ExperienceCIP Experience How Much?How Much? Audit Prep TimelineAudit Prep Timeline The PayoffThe Payoff
33
Reality CheckReality Check
““The issue is not whether your network is protected. The issue is not whether your network is protected. We know that your network is protected. The issue is We know that your network is protected. The issue is about strict compliance to the plain reading of the about strict compliance to the plain reading of the language in the CIP standards.” language in the CIP standards.”
Corporate Risk Solutions, Inc. (CRSI), December 2009Corporate Risk Solutions, Inc. (CRSI), December 2009
Translation…while the real objective is protecting Translation…while the real objective is protecting critical assets, the report card is based on compliance critical assets, the report card is based on compliance to the minutia in the CIP standards, as judged by the to the minutia in the CIP standards, as judged by the auditors.auditors.
44
Simple Compliance ModelSimple Compliance Model
Three ElementsThree Elements Clear requirementsClear requirements Clear accountabilityClear accountability Documented programDocumented program
All three elements are necessary for successAll three elements are necessary for success By far, clear accountability is the most importantBy far, clear accountability is the most important Accountable individuals make things happenAccountable individuals make things happen
This is the same for executives down to the Subject This is the same for executives down to the Subject Matter ExpertsMatter Experts
55
CIP CredentialsCIP Credentials Responsible for corporate NERC compliance since late 2006Responsible for corporate NERC compliance since late 2006
Assigned by the CEO as CIP “senior manager” Assigned by the CEO as CIP “senior manager”
Built CIP program for multiple Registered Entities from the Built CIP program for multiple Registered Entities from the ground up, as part of overall NERC compliance programground up, as part of overall NERC compliance program
External consultant supportExternal consultant support One failed gap analysis – Un-named consultantOne failed gap analysis – Un-named consultant One gap analysis and two mock audits - CRSIOne gap analysis and two mock audits - CRSI
One CIP audit (covered five GO/GOP Register Entities – 36 One CIP audit (covered five GO/GOP Register Entities – 36 requirements); verbal feedback:requirements); verbal feedback: Advanced documentation provided was “far superior than anything we Advanced documentation provided was “far superior than anything we
have seen”have seen” ““Best CIP compliance program we have seen”Best CIP compliance program we have seen” Audit completed in 3.5 daysAudit completed in 3.5 days Two minor issues identifiedTwo minor issues identified
66
Success Requirements (8)Success Requirements (8)
Successful NERC CIP Compliance requires:Successful NERC CIP Compliance requires: Leadership engagementLeadership engagement An organizational culture of complianceAn organizational culture of compliance An “effective” CIP Senior ManagerAn “effective” CIP Senior Manager A strong foundational programA strong foundational program Technically competent Subject Matter ExpertsTechnically competent Subject Matter Experts Sufficient resourcesSufficient resources Strong consulting supportStrong consulting support Extensive audit preparationExtensive audit preparation
77
Success Requirement #1:Success Requirement #1:Leadership EngagementLeadership Engagement
All compliance is localAll compliance is local
Executive management must communicate to senior Executive management must communicate to senior management that CIP compliance is importantmanagement that CIP compliance is important Senior management in turn must communicate this Senior management in turn must communicate this
message to line managementmessage to line management Line management makes it happenLine management makes it happen
Communicates importance of CIP complianceCommunicates importance of CIP compliance Provides sufficient resourcesProvides sufficient resources Sets the prioritiesSets the priorities Periodically checks on performancePeriodically checks on performance
Clear accountability is essentialClear accountability is essential
88
Success Requirement #2:Success Requirement #2:Org. Culture of ComplianceOrg. Culture of Compliance
Strict compliance is often counter-intuitive to Strict compliance is often counter-intuitive to individuals who have not been previously exposed to individuals who have not been previously exposed to itit Mountains of records are required…Why?? “Because…Mountains of records are required…Why?? “Because…
it’s the law.”it’s the law.”
Establishing a culture of compliance takes time. Establishing a culture of compliance takes time. People watch their leaders for cues…”Do as I do…” People watch their leaders for cues…”Do as I do…” will help, if leaders are engaged.will help, if leaders are engaged.
Always do the right thing. This sets the tone for the Always do the right thing. This sets the tone for the organization.organization.
99
Success Requirement #3:Success Requirement #3:Effective CIP Senior ManagerEffective CIP Senior Manager
CIP compliance does not just happen. While it is a function of CIP compliance does not just happen. While it is a function of smart people wanting to do the right things, absent sound smart people wanting to do the right things, absent sound leadership there will be gaps in compliance. Different parts of leadership there will be gaps in compliance. Different parts of the organization will do what they think is required but they the organization will do what they think is required but they may leave gaps in the “white space” between internal work may leave gaps in the “white space” between internal work groups.groups.
CIP-003 Requirement 2 calls for the assignment of a single CIP-003 Requirement 2 calls for the assignment of a single manager with overall responsibility and authority for leading manager with overall responsibility and authority for leading and managing adherence to the CIP standards.and managing adherence to the CIP standards.
The CIP founders got this one right.The CIP founders got this one right. However, the CIP senior manager does not relieve local line leadership However, the CIP senior manager does not relieve local line leadership
of CIP accountabilityof CIP accountability
1010
Effective CIP Senior Manager Effective CIP Senior Manager cont.cont.
Perfunctory assignment of a high level senior Perfunctory assignment of a high level senior leader as the required “senior manager” to leader as the required “senior manager” to meet the CIP-003 R2 requirement is meet the CIP-003 R2 requirement is problematic.problematic. If the assigned senior manager is too high in the If the assigned senior manager is too high in the
organization to be engaged in the ongoing issues organization to be engaged in the ongoing issues related to CIP compliance, problems will arise and related to CIP compliance, problems will arise and find you at a later time.find you at a later time.
1111
Success Requirement #4:Success Requirement #4:Strong Foundational ProgramStrong Foundational Program
A solid compliance program has three elementsA solid compliance program has three elements Clear requirements (CIP standards…)Clear requirements (CIP standards…) Clear accountability (engaged leadership)Clear accountability (engaged leadership) Documented programmatic controls (policies and Documented programmatic controls (policies and
procedures)procedures)
Programmatic controls must documentProgrammatic controls must document Who is responsible for what?Who is responsible for what? CIP-003 Requirement 1 calls for a cyber security policy CIP-003 Requirement 1 calls for a cyber security policy
that addresses the requirements in Standards CIP-002 thru -that addresses the requirements in Standards CIP-002 thru -009.009.
The cyber security policy should document the what and who is The cyber security policy should document the what and who is responsible (across the organization) for doing itresponsible (across the organization) for doing it
1212
Success Requirement #5:Success Requirement #5:Technically Competent SMEsTechnically Competent SMEs
Study and understand the CIP requirementsStudy and understand the CIP requirements
Identify how to comply and make it happenIdentify how to comply and make it happen
Accountable to their line leadershipAccountable to their line leadership
Should be responsible for producing and storing Should be responsible for producing and storing required evidence of compliancerequired evidence of compliance
Explain to the auditors how/why the entity is Explain to the auditors how/why the entity is compliant to the applicable CIP requirementcompliant to the applicable CIP requirement
1313
Success Requirement #6: Success Requirement #6: Sufficient ResourcesSufficient Resources
How much is enough?How much is enough? Line leadership must decide, based on competing Line leadership must decide, based on competing
objectives for available resourcesobjectives for available resources
Not enough can lead to painful shortfallsNot enough can lead to painful shortfalls Can result in expensive violationsCan result in expensive violations
CIP compliance must be part of individuals’ job CIP compliance must be part of individuals’ job functionsfunctions Full time CIP resources are the exceptionFull time CIP resources are the exception CIP audit preparation is labor intensive, beginning months CIP audit preparation is labor intensive, beginning months
before the audit and involving the various CIP SMEs in the before the audit and involving the various CIP SMEs in the business line, IT, security and other support groupsbusiness line, IT, security and other support groups
1414
Success Requirement #7:Success Requirement #7:Strong Consulting SupportStrong Consulting Support
Outside eyes on your CIP program and evidence is Outside eyes on your CIP program and evidence is absolutely essentialabsolutely essential They will see and interpret things differently than your They will see and interpret things differently than your
SMEsSMEs Based on their industry experience, they will be right most Based on their industry experience, they will be right most
of the timeof the time
Can help identify and help fix problem areasCan help identify and help fix problem areas
Choose good CIP consultantsChoose good CIP consultants Excellent audit support record (based on input from your Excellent audit support record (based on input from your
peers)peers) Those that perform CIP audits for Regions have a unique Those that perform CIP audits for Regions have a unique
perspective that is invaluableperspective that is invaluable
1515
Success Requirement #8:Success Requirement #8:Extensive Audit PreparationExtensive Audit Preparation
Begin immediately and do it annuallyBegin immediately and do it annually If you have not yet started, you are lateIf you have not yet started, you are late
Complete the CIP RSAWS and organize supporting evidence Complete the CIP RSAWS and organize supporting evidence of complianceof compliance
The audit package for some CIP standards can exceed 1,000 The audit package for some CIP standards can exceed 1,000 pagespages
Audit packages should be signed by a Preparer, Reviewer and Audit packages should be signed by a Preparer, Reviewer and Approver.Approver. Approver is the local VP or GM - responsible for CIP compliance in Approver is the local VP or GM - responsible for CIP compliance in
their organizationtheir organization Sometimes more than one preparer and reviewer sign the packages, Sometimes more than one preparer and reviewer sign the packages,
based on distribution of laborbased on distribution of labor
1616
Extensive Audit Preparation Extensive Audit Preparation cont.cont.
Recent GO/GOP CIP audit preparation and conduct Recent GO/GOP CIP audit preparation and conduct involved 33 CIP compliance personnel and SMEsinvolved 33 CIP compliance personnel and SMEs Evidence collectionEvidence collection Evidence reviewingEvidence reviewing Evidence packagingEvidence packaging SME reviewSME review Legal reviewLegal review SME audit presentation trainingSME audit presentation training SMEs standing by during audit to present and/or answer SMEs standing by during audit to present and/or answer
auditors’ questionsauditors’ questions
1717
CIP Experience:CIP Experience:Program Start-up to CIP AuditProgram Start-up to CIP Audit
JAN 2007 – Began development of the required CIP Cyber JAN 2007 – Began development of the required CIP Cyber Security Policy and other program documentsSecurity Policy and other program documents Laid out internal responsibilities for each CIP requirementLaid out internal responsibilities for each CIP requirement Identified key CIP compliance individual in each affected organizationIdentified key CIP compliance individual in each affected organization
MAY 2008 – Reviewed MAY 2008 – Reviewed CIP-002 and CIP-006 implementation CIP-002 and CIP-006 implementation with external consultants with external consultants
JUN 2008 – Turned on PSP securityJUN 2008 – Turned on PSP security
APR 2009 – Aborted CIP Gap AnalysisAPR 2009 – Aborted CIP Gap Analysis Consultants were the wrong fitConsultants were the wrong fit
JUL 2009 – Conducted JUL 2009 – Conducted CIP Gap AnalysisCIP Gap Analysis (CRSI) (CRSI) Numerous issues needed refinementNumerous issues needed refinement
1818
CIP Experience:CIP Experience:Start-up to Audit Start-up to Audit cont.cont.
SEP 2009 – Turned on security for the remaining ESPsSEP 2009 – Turned on security for the remaining ESPs
DEC 2009 – Conducted DEC 2009 – Conducted CIP Mock AuditCIP Mock Audit (CRSI) (CRSI) One major deficiency, self-reported prior to 1/1/2010 (mandatory One major deficiency, self-reported prior to 1/1/2010 (mandatory
enforcement date)enforcement date)
JAN 1, 2010 – CIP Standards mandatory and enforceable for JAN 1, 2010 – CIP Standards mandatory and enforceable for PPL Registered EntitiesPPL Registered Entities
MAY-OCT 2010 – Self-reported minor CIP violationsMAY-OCT 2010 – Self-reported minor CIP violations Several were residual issues from prior to January 1, 2010Several were residual issues from prior to January 1, 2010
MAR 2011 – Conducted MAR 2011 – Conducted CIP Mock AuditCIP Mock Audit (CRSI) (CRSI) Minor issues needed refinementMinor issues needed refinement
MAY 2011 – Conducted MAY 2011 – Conducted CIP AuditCIP Audit Two minor issues identifiedTwo minor issues identified
1919
How Much?How Much? CIP Gap Analysis and Mock AuditCIP Gap Analysis and Mock Audit
each around consulting 80 man-hours plus travel and each around consulting 80 man-hours plus travel and expensesexpenses
CIP Audit Prep – High volume of internal resources CIP Audit Prep – High volume of internal resources expendedexpended Audit included five Registered Entities (two with Critical Audit included five Registered Entities (two with Critical
Assets)Assets) Four compliance personnel and the various Subject Matter Four compliance personnel and the various Subject Matter
Experts put in many hoursExperts put in many hours Months of preparationMonths of preparation Post Audit – 33 individuals recognized for their Post Audit – 33 individuals recognized for their
contributions to the preparation and conduct of the auditcontributions to the preparation and conduct of the audit
2020
Audit Prep TimelineAudit Prep Timeline December 2010 December 2010
Began CIP Audit Package DevelopmentBegan CIP Audit Package Development
February 8, 2011 February 8, 2011 PPL received 90-day notification letterPPL received 90-day notification letter
February 28 – March 4February 28 – March 4 Conducted Third Party Mock AuditConducted Third Party Mock Audit
March 10March 10 Submitted Pre-Audit Survey and QuestionnaireSubmitted Pre-Audit Survey and Questionnaire
March 30 March 30 Submitted RSAW and EvidenceSubmitted RSAW and Evidence
May 9 -13May 9 -13 Conducted RFC Onsite AuditConducted RFC Onsite Audit
2121
Audit Package PreparationAudit Package Preparation One package for each of the eight CIP standardsOne package for each of the eight CIP standards
Most packages > 1,000 pagesMost packages > 1,000 pages
Work began in DecemberWork began in December Compliance Specialists led this effort with support from the SMEsCompliance Specialists led this effort with support from the SMEs
Compliance staff met weekly with SMEs to review RSAW Compliance staff met weekly with SMEs to review RSAW language and supporting evidence language and supporting evidence
Audit packages were reviewed by SMEs and OGCAudit packages were reviewed by SMEs and OGC
Two Day Offsite Meeting with all SMEs and Compliance staff Two Day Offsite Meeting with all SMEs and Compliance staff to review completed packagesto review completed packages
2222
The PayoffThe Payoff
Regional Entity Feedback:Regional Entity Feedback: CEO: Advanced documentation provided was “far CEO: Advanced documentation provided was “far
superior than anything we have seen”superior than anything we have seen” Audit Team: “Best CIP compliance program we Audit Team: “Best CIP compliance program we
have seen”have seen”
Audit completed in 3.5 daysAudit completed in 3.5 days Included the review of more than100 TFEsIncluded the review of more than100 TFEs
Two minor issues identifiedTwo minor issues identified
2323
Was It Worth It??Was It Worth It?? Enforcement space is very expensiveEnforcement space is very expensive
Even a minor violation receiving a minor penalty Even a minor violation receiving a minor penalty has many thousands of dollars in hidden has many thousands of dollars in hidden processing costsprocessing costs
While the real objective is protecting critical While the real objective is protecting critical assets, the report card is based on compliance assets, the report card is based on compliance to the details in the CIP standards, as judged to the details in the CIP standards, as judged by the auditorsby the auditors
You be the judge…You be the judge…
2424
Questions?Questions?