kevin behr: integrating controls and process improvement

38
Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46 © SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved © SANS Institute 2003, All Rights Reserved. 3 - 1A Kevin Behr - Integrating Controls and Process Improvement Integrating Controls and Process Improvement Integrating Controls and Process Improvement Kevin Behr CTO IP Services Kevin Behr CTO IP Services This space left intentionally blank

Upload: gene-kim

Post on 13-May-2015

882 views

Category:

Documents


5 download

TRANSCRIPT

Page 1: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 1 A

Kevin Behr - Integrating Controls and Process Improvement

Integrating Controls and Process Improvement

Integrating Controls and Process Improvement

Kevin BehrCTO IP Services

Kevin BehrCTO IP Services

This space left intentionally blank

Page 2: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 2 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 2

AgendaAgenda

The Problem : Are we smoking more and enjoying it less?

What we did about it. Control is possible!

How we did it. Blood, Sweat and VisibleOps

Measuring the results. The IMCA and other useful metrics

What we have built

The Problem : Are we smoking more and enjoying it less?

What we did about it. Control is possible!

How we did it. Blood, Sweat and VisibleOps

Measuring the results. The IMCA and other useful metrics

What we have built

We invest in redundancy and have smart engineers. Why is our infrastructure so unreliable?

I know there are best practices for security and audit but what about the ops guys?

These best practice volumes read like the tax code. How do I goabout implementing substantive change when all I have to go by is a picture of utopia?

Page 3: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 3 A

Kevin Behr - Integrating Controls and Process Improvement

The ProblemThe Problem

This space left intentionally blank

Page 4: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 4 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 4

The ProblemThe Problem

IDC, Meta etc say that security incidents cause less than 3 percent of down time.IDC Meta etc say that Hardware and environmental issues cause less than 6% of down time.Why aren’t our production systems more reliable?Why are our Ops people so busy and why are service levels getting worse? Our Data Center is always on fire!

IDC, Meta etc say that security incidents cause less than 3 percent of down time.IDC Meta etc say that Hardware and environmental issues cause less than 6% of down time.Why aren’t our production systems more reliable?Why are our Ops people so busy and why are service levels getting worse? Our Data Center is always on fire!

This space left intentionally blank

Page 5: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 5 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 5

The Problem - HumansThe Problem - Humans

Changes that authorized, tasked and directed IT people make cause 78%of all system outages!Our current way of working does nothing to address this.Many companies spend millions on change management systems – only to have them circumvented and never know it.

Changes that authorized, tasked and directed IT people make cause 78%of all system outages!Our current way of working does nothing to address this.Many companies spend millions on change management systems – only to have them circumvented and never know it.

IDC reports that authorized change by humans represents almost 80 percent of all IT outages.

Page 6: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 6 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 6

The Problem - HumansThe Problem - Humans

Many companies have developers maintaining production servers because of downsizing.In many companies Security and Operations have an adversarial relationship. Ops undoes what security puts in place. Security breaks what Ops provisions trying to minimize risk.Much of the critical knowledge on how things “Really work” lives in a few very busy minds.

Many companies have developers maintaining production servers because of downsizing.In many companies Security and Operations have an adversarial relationship. Ops undoes what security puts in place. Security breaks what Ops provisions trying to minimize risk.Much of the critical knowledge on how things “Really work” lives in a few very busy minds.

This space left intentionally blank

Page 7: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 7 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 7

The Problem- The way we work itThe Problem- The way we work it

Studies show that up to 80% of problem resolution time is spent determining the nature of the problem. The balance is spent actually correcting or bypassing the problem.Ops is so consumed with fighting fires that there is little or no accurate documentation of existing systems.There are no accurate golden builds – New servers are like snowflakes – No two are exactly the same.

Studies show that up to 80% of problem resolution time is spent determining the nature of the problem. The balance is spent actually correcting or bypassing the problem.Ops is so consumed with fighting fires that there is little or no accurate documentation of existing systems.There are no accurate golden builds – New servers are like snowflakes – No two are exactly the same.

This space left intentionally blank

Page 8: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 8 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 8

The Problem – Integrity DriftThe Problem – Integrity Drift

The purpose of deployed infrastructure “drifts” or changes over time. Suddenly a mail server is now also a DNS server, a DHCP server .Security is reduced to using detective controls to figure out what ops is deploying after the fact.New services deployed instantly become mission critical but there is no way to re-create the server that has evolved over time..

The purpose of deployed infrastructure “drifts” or changes over time. Suddenly a mail server is now also a DNS server, a DHCP server .Security is reduced to using detective controls to figure out what ops is deploying after the fact.New services deployed instantly become mission critical but there is no way to re-create the server that has evolved over time..

This space left intentionally blank

Page 9: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 9 A

Kevin Behr - Integrating Controls and Process Improvement

What we did about itWhat we did about it

This space left intentionally blank

Page 10: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 10 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 10

What we did about itWhat we did about it

Used a twelve step program to determine that we were powerless over our propensity to “light and fight” ops fires.We came to the conclusion that we needed a higher power (ITIL) and that if we worked the program we could find our way to Serenity and many nines of up time.We vowed to share our experience with others along the way.

Used a twelve step program to determine that we were powerless over our propensity to “light and fight” ops fires.We came to the conclusion that we needed a higher power (ITIL) and that if we worked the program we could find our way to Serenity and many nines of up time.We vowed to share our experience with others along the way.

This space left intentionally blank

Page 11: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 11 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 11

What we did – The Higher PowerWhat we did – The Higher Power

We needed a framework to put all of our activity into. So we could understand what it was we were supposed to be doing.The framework we chose was the Information Technology Infrastructure Library or ITIL (eye-til)Pros – Very Large and comprehensiveCons- Very Large and very descriptive (what it looks like) – we needed Prescriptive (what to do)

We needed a framework to put all of our activity into. So we could understand what it was we were supposed to be doing.The framework we chose was the Information Technology Infrastructure Library or ITIL (eye-til)Pros – Very Large and comprehensiveCons- Very Large and very descriptive (what it looks like) – we needed Prescriptive (what to do)

This space left intentionally blank

Page 12: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 12 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 12

What we did about it - What is ITIL?What we did about it - What is ITIL?

British Office of the Crown Government authors many well-known documents, including ISO17799 (BS7799) Created They realized Ops best practices have never been documented, and created ITIL (IT Infrastructure Library) and BS15000 to describe how world-class Ops processesExtremely widely used in Europe, but gaining acceptance in the U.S.HP OpenView, CA UniCenter, and IBM Tivoli are all basing their EMS

products on ITIL terminologyComputerWorld 10/7/2002: Proctor & Gamble reports saving $125

million per year on IT cost savings (10-15% of their annual IT budget)

British Office of the Crown Government authors many well-known documents, including ISO17799 (BS7799) Created They realized Ops best practices have never been documented, and created ITIL (IT Infrastructure Library) and BS15000 to describe how world-class Ops processesExtremely widely used in Europe, but gaining acceptance in the U.S.HP OpenView, CA UniCenter, and IBM Tivoli are all basing their EMS

products on ITIL terminologyComputerWorld 10/7/2002: Proctor & Gamble reports saving $125

million per year on IT cost savings (10-15% of their annual IT budget)

IT Infrastructure Library (ITIL) is the only consistent and comprehensive documentation of best practice for IT Service Management. Used by many hundreds of organizations around the world, a whole ITIL philosophy has grown up around the guidance contained within the ITIL books. ITIL consists of a series of books giving guidance on the provision of quality IT services, and on the accommodation and environmental facilities needed to support IT. ITIL has been developed in recognition of organizations' growing dependency on IT and embodies best practices for IT Service Management. The ITIL Online : http://www.ogc.gov.uk/itil/ The Office of Government and Commerce (owners of ITIL) http://www.ccta.gov.uk/

BS15000 / BS 15000 is the world's first standard for IT service management. The standard specifies a set of inter-related management processes, and is based heavily upon the ITIL (IT Infrastructure Library) framework. The BS15000 Site http://www.bs15000.org.uk/

Page 13: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 13 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 13

What Is “Visible Ops?”What Is “Visible Ops?”A closed-loop process methodology, aimed at increasing Operational efficiencies and increasing service levelsBased on studying “best in class” enterprise operationsVisible Ops goalsA small subset of ITIL and BS15000 frameworks, for terminology,

processes, and future improvementsIntended to 80% of the benefits at 20% of ITIL effort A “step by step” approach to three fundamental service management

disciplinesMethodology authors:Gene Kim, CTO, Tripwire, Inc.Kevin Behr, CTO, IP Services, Inc.

A closed-loop process methodology, aimed at increasing Operational efficiencies and increasing service levelsBased on studying “best in class” enterprise operationsVisible Ops goalsA small subset of ITIL and BS15000 frameworks, for terminology,

processes, and future improvementsIntended to 80% of the benefits at 20% of ITIL effort A “step by step” approach to three fundamental service management

disciplinesMethodology authors:Gene Kim, CTO, Tripwire, Inc.Kevin Behr, CTO, IP Services, Inc.

This space left intentionally blank

Page 14: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 14 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 14

What we did about it – VisibleOpsWhat we did about it – VisibleOps

Gene Kim and I studied many enterprise operations (A major trading company, The largest wireless carrier, a major stock exchange) and we began to note that these organizations had successfully implemented and benefited from preventive and detective control combinations.These controls were used to create audit points that made it easy to understand known good states.

Gene Kim and I studied many enterprise operations (A major trading company, The largest wireless carrier, a major stock exchange) and we began to note that these organizations had successfully implemented and benefited from preventive and detective control combinations.These controls were used to create audit points that made it easy to understand known good states.

This space left intentionally blank

Page 15: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 15 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 15

What we did about it What we did about it

We also began to see that if the infrastructure state was understood early on in the problem management cycle the time it took to accurately determine the nature of the problem could drastically be reduced.We would be able to stop many inappropriate and costly over-escalations if we could rule out change as early as possible.

We also began to see that if the infrastructure state was understood early on in the problem management cycle the time it took to accurately determine the nature of the problem could drastically be reduced.We would be able to stop many inappropriate and costly over-escalations if we could rule out change as early as possible.

When examining Problem resolution reports it was noticed that ifchange could be ruled out early the time it took to close the ticket was reduced.

Most every organization has a star quarterback in operations, and security. Many groups thought that everything wound up escalating to this person because the overall environment had grown so complex that only a few people could solve what used to be simple problems. This often results in a serious moral problem for thebrightest staff. We needed to put them in to an advisory role where they coach and consult instead of fighting fire full time on the front lines. The ultimate goal is to free up enough their time to turn them loose on creating additional operational efficiencies and process improvement.

Page 16: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 16 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 16

What we did about itWhat we did about it

Best in class operations had bounded remediation times for critical infrastructure.In order to have valid golden builds to accomplish this the change management process must have more teeth than just the “honor system”.These organizations also displayed the earliest integration of security in to the Ops lifecycle

Best in class operations had bounded remediation times for critical infrastructure.In order to have valid golden builds to accomplish this the change management process must have more teeth than just the “honor system”.These organizations also displayed the earliest integration of security in to the Ops lifecycle

We spoke to many large IT groups and heard them complain about the ineffective nature of their change management systems. One CTO even complained that his engineers were often so busy and backlogged in firefighting that they didn’t feel like they had enough time to even work through the Change Management processes. This meant that changes made during firefighting were never evendocumented!

Security would be completely on their own to detect and respond to these ad-hoc changes. They would certainly never know who made the changes let alone if they were made by friend or foe (although the odds are with “friend”)!

Page 17: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 17 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 17

Best In Class Ops and SecurityBest In Class Ops and SecurityBest in class Ops and Security organizations have:

•HighestServer/sysadmin ratios

•Lowest Mean Time To Repair (MTTR)

-Highest Mean Time Between Failures (MTBF)

•Earliest integration of Security into Ops lifecycle

This space left intentionally blank

Page 18: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 18 A

Kevin Behr - Integrating Controls and Process Improvement

How we did itHow we did it

This space left intentionally blank

Page 19: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 19 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 19

Where Is The Leverage?Where Is The Leverage?Ensure that I have predictability

around what goes into production

Ensure that I can control changes in my world in the

production environment

Equip me to deal with problems efficiently and feed the results

back into my environment

Help me learn to do this in an automated fashion.

Shift resources from fire fighting to implementing release management, controls and resolution processes.

Page 20: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 20 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 20

Process Area ObjectivesProcess Area ObjectivesRelease ManagementEnsure that provisioned systems match the “known, good build”Promote repeatable builds for all configurations

Control ProcessesEnsure that changes can be traced to a valid business reasonCreate a control point, where Ops, Dev, or Security can so stop a

change from occurringControl configuration drift and uncontrolled changes

Incident Management / ResolutionDecrease MTTR (mean time to resolve) outagesIncrease “culture of causality,” allowing better diagnosis and problem

management practices

Release ManagementEnsure that provisioned systems match the “known, good build”Promote repeatable builds for all configurations

Control ProcessesEnsure that changes can be traced to a valid business reasonCreate a control point, where Ops, Dev, or Security can so stop a

change from occurringControl configuration drift and uncontrolled changes

Incident Management / ResolutionDecrease MTTR (mean time to resolve) outagesIncrease “culture of causality,” allowing better diagnosis and problem

management practices

This space left intentionally blank

Page 21: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 21 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 21

How we did it – Stabilize the patientHow we did it – Stabilize the patient

Attack the 80%. Stop the bleeding caused by: change drive-bys ,integrity drift and changes made during firefighting.We used the combination of a preventive control (don’t touch that fence it’s electric!) and a detective control (why did you touch the fence at 2:11 am on March 3rd?) to get a handle on the state of every piece of critical infrastructure.

Attack the 80%. Stop the bleeding caused by: change drive-bys ,integrity drift and changes made during firefighting.We used the combination of a preventive control (don’t touch that fence it’s electric!) and a detective control (why did you touch the fence at 2:11 am on March 3rd?) to get a handle on the state of every piece of critical infrastructure.

Audit change and configuration controls

Tools: Tripwire, Tivoli auditing components, reports from change management toolsAudit configuration footprints to ensure complianceMap all changes to authorized work orderEnd-of-shift audit requires Ops managers to handover data center in the same state as they received it

Page 22: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 22 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 22

How we did it – Catch and ReleaseHow we did it – Catch and Release

We caught and foot-print audited all critical infrastructure configurations in the wild.We created golden builds for these devices.We tested and set bounded remediation times for all critical infrastructure.We determined audit frequency and methods necessary to support these times .

We caught and foot-print audited all critical infrastructure configurations in the wild.We created golden builds for these devices.We tested and set bounded remediation times for all critical infrastructure.We determined audit frequency and methods necessary to support these times .

Create repeatable buildsTools: Tivoli Configuration Manager, Tivoli Remote Control and others (Norton Ghost, InstallShield AdminStudio, Linux QuickStart, Sun Jumpstart)Automated provisioning of OS, configuration files, applications, and business rules

Create acceptance processTools: TripwireEnsure that provisioned servers matches “known, good build”

Page 23: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 23 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 23

How we did it – Manage the ChangeHow we did it – Manage the Change

Instituted a Change Advisory Board- Stake holders include: Security Lead ,Ops Systems Engineering Lead, VP of Operations , Service Desk Manager, Director of Network Operations, and Internal Audit.Made weekly change management meetings mandatory for all CAB members.Implemented a Change Transaction Process to make the correct path : Request For Change (RFC)

Instituted a Change Advisory Board- Stake holders include: Security Lead ,Ops Systems Engineering Lead, VP of Operations , Service Desk Manager, Director of Network Operations, and Internal Audit.Made weekly change management meetings mandatory for all CAB members.Implemented a Change Transaction Process to make the correct path : Request For Change (RFC)

Create change transaction workflow

Control points to document, authorize, schedule or deny, and audit change requests

Create change control meetings (include Security)

Tools: Tripwire, reports from change management tools (such as trouble ticketing system)

Page 24: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 24 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 24

How we did it – Managing ChangeHow we did it – Managing Change

All RFC are categorized based on a 1-4 severity system. Anything above a 2 goes to the CAB for review and comment.Changes can only be administered during maintenance windows and must be approved and scheduled by the CAB.Urgent changes trigger an emergency CAB meeting.

All RFC are categorized based on a 1-4 severity system. Anything above a 2 goes to the CAB for review and comment.Changes can only be administered during maintenance windows and must be approved and scheduled by the CAB.Urgent changes trigger an emergency CAB meeting.

Simple Change Management Meeting Agenda:

Discussion of:Failed Changes, backed-out Changes, or Changes that may have circumvented the CABRFCs to be assessed by CAB members Requests For Change that have been assessed by CAB members Change reviews The Change Management process, including any amendments made to it during the period under discussion, as well as proposed Changes Change Management wins/accomplishments for the period under discussion, i.e. a review of the business benefits accrued by way of the Change Management process. Review of Next Action assignments based on the above discussion.Dismiss.Meetings should have minutes taken and distributed to the CAB following the meeting.

Page 25: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 25 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 25

How we did it - First Response How we did it - First Response

Modified the problem management process to eliminate change as early as possible by identifying the assets directly involved in the ticket and auditing them against their configuration baseline for the last 72 hours. All changes found are attached to the ticket.If no changes are found the circle is widened to include changes made to infrastructure supporting the target systems.

Modified the problem management process to eliminate change as early as possible by identifying the assets directly involved in the ticket and auditing them against their configuration baseline for the last 72 hours. All changes found are attached to the ticket.If no changes are found the circle is widened to include changes made to infrastructure supporting the target systems.

Create inventory of all relevant evidence around issue or outageTools: Remedy/ CA Service Desk /Tivoli Configuration Manager and Tripwire; Configuration and asset management informationAll relevant scheduled and authorized changesActual changes on target system

Formalize post-incident assessment and reconciliation of changesTools: Tripwire, reports from Tivoli, reports from ticketing systemEnsure that changes are understoodEnsure that changes are incorporated into documentation and propagated to other systems, as appropriate

Page 26: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 26 A

Kevin Behr - Integrating Controls and Process Improvement

Measuring the resultsMeasuring the results

This space left intentionally blank

Page 27: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 27 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 27

Measuring the results - The IMCAMeasuring the results - The IMCABased on IT Infrastructure Library (ITIL) / BS 15000 standards and the Visible Ops methodologyAn interview-fueled process with a standardized scoring methodologyFocuses on high leverage areas:Release ProcessesControl Processes Resolution Processes

Based on IT Infrastructure Library (ITIL) / BS 15000 standards and the Visible Ops methodologyAn interview-fueled process with a standardized scoring methodologyFocuses on high leverage areas:Release ProcessesControl Processes Resolution Processes

This space left intentionally blank

Page 28: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 28 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 28

Measuring the results – IMCA Questions Measuring the results – IMCA Questions

All questions are answered with a number, “from zero to four” 0: Strongly disagree4: Strongly agree

Sample questions“Our IT department is understaffed to meet current workloads.”“Our Service levels are spiraling downwards.”“We can enforce a standard build across all our devices.”“We have a library of automated build systems for all our critical

devices.”“We have a clearly defined change control policy.”

All questions are answered with a number, “from zero to four” 0: Strongly disagree4: Strongly agree

Sample questions“Our IT department is understaffed to meet current workloads.”“Our Service levels are spiraling downwards.”“We can enforce a standard build across all our devices.”“We have a library of automated build systems for all our critical

devices.”“We have a clearly defined change control policy.”

This space left intentionally blank

Page 29: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 29 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 29

Measuring the results- IMCA reportMeasuring the results- IMCA report

This organization has no Request for Change process. Not having a correct path for changes to follow assures that they will go the path of least resistance and least documentation. Creating more gasoline to throw on the fire.

Page 30: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 30 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 30

Measuring the results- IMCA reportMeasuring the results- IMCA report

This represents a pretty tight shop with some room for improvement. They need to build on their strengths in audit and process to shore up their change transaction processes. Some detective control would certainly help their ailing rollback capabilities.

Page 31: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 31 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 31

Reliability and Validity of IMCAReliability and Validity of IMCA

Validity measuresBased on IT best practices frameworks of ITIL and

BS15000Questions are scored on the integrity of three key ITIL

processesReliability measuresAll answers are subjective, and can vary from day to dayAll answers do not have any quantitative significance (i.e.,

arithmetic operations cannot be done on the answers)

Validity measuresBased on IT best practices frameworks of ITIL and

BS15000Questions are scored on the integrity of three key ITIL

processesReliability measuresAll answers are subjective, and can vary from day to dayAll answers do not have any quantitative significance (i.e.,

arithmetic operations cannot be done on the answers)

This space left intentionally blank

Page 32: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 32 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 32

Measuring the results- Other MetricsMeasuring the results- Other Metrics

Number of changes made in data centerNumber of changes that map to authorized business reasonNumber of times change management system was circumventedPercent of outages caused by changeNumber of changes that obsolete repeatable buildsOps “clean shift handover” success rate

Number of changes made in data centerNumber of changes that map to authorized business reasonNumber of times change management system was circumventedPercent of outages caused by changeNumber of changes that obsolete repeatable buildsOps “clean shift handover” success rate

This space left intentionally blank

Page 33: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 33 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 33

Measuring the results- Other MetricsMeasuring the results- Other Metrics

Time to provision known, good buildNumber of fixes/turns to match known, good buildPercentage of deployed systems that match known, good buildPercentage of deployed systems that have security sign-off

Time to provision known, good buildNumber of fixes/turns to match known, good buildPercentage of deployed systems that match known, good buildPercentage of deployed systems that have security sign-off

This space left intentionally blank

Page 34: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 34 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 34

Measuring the results- Other MetricsMeasuring the results- Other Metrics

Outage and issue Mean Time To Repair (MTTR) Aggregate outage downtimeNumber of inappropriate escalationsIncreased change success rateIncreased systemic Mean Time Between FailureSmile to frown ration on Ops, Security and Audit staff

Outage and issue Mean Time To Repair (MTTR) Aggregate outage downtimeNumber of inappropriate escalationsIncreased change success rateIncreased systemic Mean Time Between FailureSmile to frown ration on Ops, Security and Audit staff

This space left intentionally blank

Page 35: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 35 A

Kevin Behr - Integrating Controls and Process Improvement

What you have builtWhat you have built

This space left intentionally blank

Page 36: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 36 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 36

What you have built - You Can Now:What you have built - You Can Now:Enforce change management process integrityDecreased firefighting and increase proactive controlsAvert revenue loss due to unplanned outagesDecrease Mean Time To Repair by efficient problem management processesCreate hard organizational change boundaries for accountability and responsibilityEstablish a beach head for operational best practices, allowing future process improvement

Enforce change management process integrityDecreased firefighting and increase proactive controlsAvert revenue loss due to unplanned outagesDecrease Mean Time To Repair by efficient problem management processesCreate hard organizational change boundaries for accountability and responsibilityEstablish a beach head for operational best practices, allowing future process improvement

This space left intentionally blank

Page 37: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 37 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 37

What you have builtWhat you have built

You now can measure and articulate the business benefit of process improvement effortsYou can target weak areas for quick winsRegain the confidence of the business by showing off your new and improving metricsFend off IT Budget Jenga with your CFO and CEO by showing where money needs to be invested and why.

You now can measure and articulate the business benefit of process improvement effortsYou can target weak areas for quick winsRegain the confidence of the business by showing off your new and improving metricsFend off IT Budget Jenga with your CFO and CEO by showing where money needs to be invested and why.

This space left intentionally blank

Page 38: Kevin Behr: Integrating Controls and Process Improvement

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46

© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

© S

ANS In

stitu

te 2

003,

All

Right

s Res

erve

d.

3 - 38 A

Kevin Behr - Integrating Controls and Process Improvement

© 2003 Tripwire, Inc. 38

Contact InformationContact Information

Gene Kim, CTO, Tripwire, [email protected]

Kevin Behr, CTO, IP Services, [email protected]

Gene Kim, CTO, Tripwire, [email protected]

Kevin Behr, CTO, IP Services, [email protected]

This space left intentionally blank