26.09.2012

Stefan Zahrer

HP SOFTWARE ANWENDERTREFFEN 2012

HP OO: FULL AUTOMATION STORY

Munich Re Group

Reinsurance | Munich Health | Group Functions

Asset Management

€ 32bn gross premium income

€ 200bn assets under management

14,000 employees

70+ office locations worldwide

ERGO primary insurance

€ 17bn gross premium income

33,000 employees

Key figures 2011 28.09.2012 2 Munich Re PowerPoint Vorlage / Beispielseiten

Datacenter Munich

1,000 Applications

1,400 Windows installations

250 Linux installations

4 PB storage capacity

Sourcing and offshore run

Agenda

History of Automation

Automation @ Munich Re

Automation Pilot: Managed Virtual Windows Server

Challenges and Achievements

28.09.2012 3

Automation Fascinated Mankind for Centuries

Windmill without automation

Source: Wikipedia

Automation Fascinated Mankind for Centuries

Since1745:

Windmill with automation

Source: Wikipedia

Run Book = Collection of

Operator tasks

Also known as Service

Automation, ITPA, etc.

250 Years Later: Automation in IT

In search for a funky moniker!

Features

Centralized automation platform

End-to-end automation

Encompassing all infrastructure components

9/28/2012 6

Run Book Automation

AUTOMATION @ MUNICH RE

Big Picture Run Book Automation

8

Self-Service Portal End-to-End Processes

Define Build Manage

Orchestration Platform

Technical Workflows

RB

A

RB

A

End-to-end Automation:

From Workshop to Industrial Manufacturing

General Standards

Infrastructure Architecture

System Management

Service Portfolio / Products

9/28/2012 9

A

u

t

o

m

a

t

i

o

n

Industrialized Fulfillment

Fixed service offering

Reliably short delivery times

Consistent high quality

Manual Fulfillment

All special requests are feasible

Manual fulfillment causes wait times

Limited workload

AUTOMATION PILOT

MANAGED VIRTUAL WINDOWS SERVER

End-to-end Automation:

Managed Virtual Windows Server

Deploying a virtual Windows installation with vmware is easy:

Now the server is online – but without integration into the infrastructure

A managed server needs more!

9/28/2012 11

Prepare

Vmware Inst.

Template

Deploy

Template

End-to-end Automation:

The Entire Process is Automated

Tasks 1. Create i3S change

2. Calculate server name / IP Address

3. Create i3S CMDB record

4. Customize VMWare installation template

5. Deploy server template

6. Configure server installation for standard

server roles

7. Register malware client in ePO

8. Configure Networker backup server and

client

9. Calculate backup timeslot and schedule

backup job in $Universe

10. Add standard monitoring pack according

to server role

11. Register server in SCOM and OVO

12. Add server to DTM tool *

13. Update i3S CMDB

14. Update i3S Change Task

Time-to-market

manual labor 5 days (+/- 2 days)

automated 4 hours

* Asynchronous manual task 12

Current Process Virt. Window Server Installation (Change 3967)

Malware ServiceNetwork ServiceWindows

ServiceBackup Service

Job Mgmt.

Service

Monitoring

ServiceRequester in i3S

Order

ClarificationManual Input

Fill Change

Servername

IP Address

Define

Servername

IP Address

Update CMDB

Prepare

Vmware Inst.

Template

Install Client

Software

Test BackupDefine $U Job

for Backup

Approve Job

Configuration

ePO Select

Groups

OVO Add

Object to

OMU

Relate

Downtime

Window

Manual

Quality Check

Add Server to

OpsMgr

No SOP

Apply XML

Config File

Check

Installation

Input Values

Project

Cost Center

Server Version

Server Type

DT Window

Pilot Server

No Backup

Location

AD Domain

Standalone

CPU

RAM

Size C-Drive

Size E-Drive

Size L-Drive

Size P-Drive

OU Group

Server Role

Add.

Permissions

Add. Service

Timezone

Desired Date

Add. Contact

i3S Email to

Requester

Manual Post

Processing

Configure

Networker DB

End

Input Values

Equipment

Domain

OS

Location

Input Values

<CAP> Required disk capacity. This parameter can occur more than once, depending

on

the number of virtual hard disks.

<CC> Cost center

<CPU> Number of CPUs

<DST> *) Destination data store

<ENV> Sever environment (e.g. Loc-Munich-MR, Loc-Munich-MEAG)

<ESXCL> *) Name of the ESX cluster (e.g. mr-munich-cl01)

<FQDN> Full qualified domain name of the destination domain (e.g. Standalone,

munichret02.t02)

<HOSTINGSITE> Hosting site (e.g. MUC, PCT)

<INST> *) Installation status (FIRST for a new installation, RE for a reinstallation)

<IPADDR> IP address of the server

<LOC> Location of the ESX host (e.g. RZ N2, RZ W5)

<NETENV> *) Destination network environment (e.g. MEAG_LAN_x86)

<NETLABEL> *) Network label (e.g. VM-MR-WINPRD)

<PATCH> *) Patch level

<RAM> Amount of main memory

<REQSVC> Requesting service (e.g. muc integration platform, muc dms)

<ROLE> Serer role (e.g. Terminal Server. Every role includes a specified set of

installation

scripts, which are different for different roles.)

<SRV> Name of the server to be installed (e.g. MUCV161749)

<STATE> State of the server (PRD for production, DEV or INT for test)

<VER> Version of the server to be installed (e.g. W2K3-SV4 32Bit (903-32))

<WINDOM> *) Windows domain (e.g. MR-WORKGROUP08, MUNICHRET02)

Update

Vmware Tools

WTS Add.

Manual Steps

Input Values

<SRV> Name of server

to maintain

<APPTYPE> Type of

application running on

this server

<Forest> Name of the

forest the server belongs

to

<Domain> Domain the

Server belongs to

<filer> Name of software

depot server. (see

Appendix 5.2)

<Loc> Location Code

Create XML

Config File

Deploy

Template

Update CMDB

Input Values

<SRV> Name of server

<DOMAIN> Name of the

domain the server belongs to

<APPTYPE> Type of

application the server runs

<CONTACT> Point of

contact for this server

Install SQL,

Exchange

Modules

Input Values

<SRV> Name of server

<DOMAIN>

No Backup

Input Values

<SRV> Name of server

<DOMAIN>

DT Window

Input Values

Object name (can be e.

g. a server, router,

switch, agent)

Objectgroup

Reference object

Machine type

Type of manage node

Reload

Console Mail

to 1st Level

OVO Reload

OMU Console

OVO Change

Object

Symbol

ePO Add

description

ePO Add

Server to

Group

Input Values

<SRV> Name of server

DT Window

Pilot Server

9/28/2012

Automation with HP Operations Orchestration:

Top Level Control Flow

Managed virtual Windows server

100 flows and sub flows

45 input variables

9000 steps per installation

Automated selection from

2 Virtual Centers

12 vmware Clusters

based on usage type

80 ESX-hosts

most free RAM, network label

15 LAN networks

>400 data stores

based on usage type and least

free space

And more…

Flow customizing experience

Flow are easy to understand – at second glance

Flow customizing more efficient than scripting

Significantly less bugs

Good maintainability

13

Initialize

Rollback

Install

Server

User Information

No rollback

after this point

Experience with Windows Pilot (1)

28.09.2012 14

Information gathering…

to design end-to-end automation

Infrastructure Architecture

Complexity of technology and processes is higher than expected

Future Architecture must be designed with automation in mind

Pic

ture

: a

uto

ma

tio

n.c

om

Experience with Windows Pilot (2)

28.09.2012 15

Organization & Processes

Major organizational change for IT

Service offering and customer interface hugely affected

Automation initially challenges peoples expectations

Architecture and process skills rise in importance

Collaboration across teams is important

Automation Project

End-to-end automation is a significant effort

Solid architecture framework for automation platform is important

Software development methods required

CHALLENGES AND ACHIEVEMENTS

Installed with HP OO

(06.09.12)

9/28/2012 17

246 Productive Windows Servers

68 Productive Linux Servers

500 IP Addresses and IP Names

120 Data Stores for VMware

35 Windows Server Deinstallations

40 Rollout Software Firewall Rule Sets to Servers

approx. 30 HP OO Users

Automation Pipeline

Storage Automation with (create/change SAN and NAS storage)

Create/change Oracle and SQL-Server database instances

Linux patching

Oracle patching

28.09.2012 18

Workflow request

Cache DB

Command ScriptObject

Filter & Finder

DFMWeb Service front end

Portal

Orchestrator

Job Log

Object Templates

Object Templates

Object Templates

Infrastructure

Powershell Toolkit

Automation Demystified

Automation isn’t easy – initially it is a very high effort

Automation is not about quick savings – it is a long term strategy with manifold

benefits

Automation is not a job killer – it provides higher skilled jobs

Automation implementation is not another infrastructure project – it is more of an

application development project

Automation is not just another tool – it hugely affects people and processes

28.09.2012 19

The Verdict on Automation

End-to-end Automation is a game changer for in-house IT!

Major organizational change for entire IT

Driver for higher standardization of infrastructure, processes and service

offering

Requires integrative architecture and forward planning

Technology: Finally it adds up!

Automation tools: Ready for large scale application

Infrastructure: Vendors offer versatile interfaces for automation

28.09.2012 20

End-to-end Automation Values

Customer satisfaction

Self-service portal

Reliable time-to-market, rapid provisioning, mass provisioning

IT satisfaction

Enabler for improved IT maturity level

Improved infrastructure standardization and fulfillment quality

Emotional satisfaction: Seeing automation at work is sheer pleasure!

28.09.2012 21

Questions…?

28.09.2012 22

Pic

ture

: a

uto

ma

tio

n.c

om

Stefan Zahrer

Email: szahrer@munichre.com

Phone: +49 89 3891-4625

© 2

01

0 M

ün

ch

en

er

Rückve

rsic

he

run

gs-G

ese

llsch

aft

© 2

01

0

Mu

nic

h R

ein

su

ran

ce

Co

mp

an

y

BACKUP

HP OO DESIGN FRAMEWORK

The Gordian Knot – How to Ensure Reusability and

Maintainability?

HP OO flows execute functions in infrastructure systems based on organizational processes

Process sequence and technical operations are closely interwoven

What happens if things were to change?

In the infrastructure: new software version, feature changes, new systems, …

In the organization: Changes to processes, new CMDB attributes, …

28.09.2012 25

Automation Flow

Automation Flow

Automation Flow

Automation Flow

Infrastructure

Are we Standardized?

YES!

Only x86_64 servers

Only Windows and Linux

Only Oracle and SQL Server database

Only rack server and blades

Only vmware and Citrix XenServer for virtualization

Only one storage vendor and two storage classes

NO!

Technical complexity

AD domains, networks, server names, Windows installations types, server names,

windows drives and sizes, software packages, etc.

Organizational complexity also very high

Interaction with HP Service Manager, support units, cost centers, CMDB maintenance

rules, etc.

28.09.2012 26

Major Topics of the HP OO Design Framework

28.09.2012 27

Flow Authoring Guidelines

Flow Architecture

Flow Call Relation-

ships

Naming Conventions

Authorization Model

ITIL Integration

Change Management

Incident Management

CMDB Updates

Processes

Change and Release Management

Staging and deployment

Manage HP OO repositories

Private Database

Advanced logging / reporting

Documentation of user input

Infrastructure Topology Model

Infrastructure Interface (II)

Action Plan (AP)

Optional: Subcontroller (SC)

Controller (CO)

User Interface (UI)

Utility

(U

T)

Transactional Data

Persistence Table

Key Value Store

Logical

Logging

Server Name

Table

Master Data Infrastructure Topology

Model

Select and Start User-

Interface-Flow in HP OO

Central

Input in User-Interface-

Flow of HP OO

System Call to I3S

(Create/Save Change)

System Call to I3S to

verify Change

Relevance

Change Relevant

Flag set?

Yes

Input Risk- and Impact-Parameters

Execution Time?

Start immediately after aproval

No

Submit Change

Input Parameters for Downtime

Request

Downtime Window

Input Parameters for

Time is agreed (Start

Time/Finish Time)

Fetch Downtimes via I3S for first

task

Time is agreed

Possibility that

downtime window

is not available

Writing of Variables in Private

Database

End of User Interface Flow

Aproval Process

First Automated Task (Status assigned)

I3S creates XML-File

Scheduled File Check picking up XML-File

Start of Scheduling Flow

- Acceptance of I3S Task

- Searching for Downtime of every CI in I3S

- Scheduling of Controller Flow for every CI at defined

Downtime or execute immediately

Start

Start of scheduled Controller Flow

Check if respective Change is canceled

No

Canceled? YesAbort

Report in Private Database

CI still exists in I3S

Change?

No

Yes

Check if running in

Downtime Window

Mode (Flow Input)

Check if Downtime is

still valid

Yes

No

Yes

Read Input Values from Private

Database

Do Work on CI

Exited with Error?

Update Data in Private DB

Check for last CI

Document in I3S

no

Create Incidentyes

Close Task

yes

Exit

no

Flow Architecture – Separates Process Control and

System Control

28.09.2012 28

Infrastructure Interface (II)

Action Plan (AP)

Optional: Subcontroller (SC)

Controller (CO)

User Interface (UI)

Utilit

y (

UT

)

Creates GUI for users

• Controlled by XML

• Populates selection lists

• Verifies user input

Controls processing order

of tasks

Executes a single function on

one infrastructure system

Combines functions to execute

a complex task on one system

Combines tasks on different

systems to reduce complexity

of Controllers

Characteristics of Flow Types

28.09.2012 29

Flow Type Complexity Reusability Modify and Test for

change to

Modify/Test

Effort

User Interface High High Test without affecting

infrastructure

Medium

Controller High Very Low Organizational Process High

Subcontroller Medium High Technical Process High

Action Plan Medium High Infrastructure systems and

applications

Low

Infrastructure Interface Low High Infrastructure systems and

applications

Low

Utility Low High HP OO versions Low

Rules for Flow Calls Limit Complexity

28.09.2012 30

Infrastructure Interface (II)

Optional: Subcontroller (SC)

Controller (CO)

User Interface (UI)

Utilit

y (

UT

)

Action Plan (AP)

Execute Configuration Changes

Infrastructure Interface (II)

Execute Configuration Changes

Action Plan (AP)

Infrastructure Systems Private DB

Logging, etc.

Scheduler Persistence

Table

• Gather information

• Create i3S Changes

• Logging

Configure

Infrastructure

Private Database Enhances HP OO Standard Features

Reporting of success/failure

Detailed reporting for services

Documentation of user input

Handover variables for scheduled runs

28.09.2012 31

Transactional Data

Persistence Table Flow run specific variables

Logical Logging Basis for result reporting

IP-NameTable Server Name Mgmt.

more to come…

Master Data

Infrastructure

Topology Model

Infrastructure Topology Model

Dependencies in infrastructure (Locations, Networks, AD Entities, SW Packages, order, preferences, etc.)

Supports guided user input

Reduced flow complexity

Infrastructure Topology Model Stores Information in a

Structured Easy to Maintain Form

Main purposes

Enhance User Input flows

Stores XML for user input

Content of list boxes, default selections

Limit selections to valid combinations

Dynamic selections in Controller flows based on

System architecture, location, usage, …

Preferences (e.g. prefer one ESX Cluster

whenever possible)

Organizational information (e.g. cost centers,

support units, templates)

References to HP OO System Accounts

Staging data from DEV to PRD

28.09.2012 32

RBA_Types

ID

Type

Description

int

varchar(512)

varchar(512)

<pk>

RBA_Entities

ID

Type_ID

Name

Description

int

int

varchar(512)

varchar(512)

<pk>

<fk>

RBA_Entities_RBA_Entities

ID

Parent_Entity_ID

Child_Entity_ID Description

int

int

int varchar(512)

<pk>

<fk2>

<fk1>

RBA_Entities_Extended_Attributes

ID

Entity_ID

Name Value

Description

int

int

varchar(512) varchar(512)

varchar(512)

<pk>

<fk>

RBA_Entities_RBA_Entities_Extended_Attributes

ID

Entitiy_Entity_ID Name

Value

Description

int

int varchar(512)

varchar(512)

varchar(512)

<pk>

<fk>

RBA_Entities_RBA_Entities_RBA_Entities

ID

Entity_ID

Entity_Entity_ID

Description

int

int

int

varchar(512)

<pk>

<fk1>

<fk2>

RBA_Entities_RBA_Entities_RBA_Entities_Extended_Attributes

ID

Entitiy_Entitiy_Entity_ID Name

Value

Description

int

int varchar(512)

varchar(512)

varchar(512)

<pk>

<fk>

No “hard encoded” values in flows

Reduce flow complexity – data sets defined by SQL queries

Reduced number of HP OO System Accounts to maintain

Less need to change to flows – edit database or SQL statement only

Database changes are immediately available in all affected flows

Infrastructure Entities and Relationships

28.09.2012 33

Infrastructure Entities and Attributes

28.09.2012 34

Execution Flows

User Interface Flows

Private DB

Interaction Between Flows and Private DB

28.09.2012 35

Dialog Step 1 Dialog Step 2 Input

Summary

SQL SQL SQL

HP OO

Scheduler

Controller Flow

Infrastructure Topology Model

Persistence

Table

Logical

Logging

Table

SQL

Create

Service

Request in

HP SM

XML File

Naming Conventions for Variables and Flows

Type Definition Example

Variables

Flow Inputs FI_purposeOfFlowInput FI_vmwarevmHostname

Step Inputs SI_purposeOfStepInput SI_vcenterHostname

Flow output fields FF_purposeOfFlowOutputField FF_vcenterTotalAvaiableStorage

Local variables LV_purposeOflocalVariable LV_vcenterHostname

Global Variables GV_purposeOfGlobalVariable GV_vcenterVmType

Singular for 1 object, plural for more than 1

object

FI_hostname for 1 value, FI_hostnames for a list of

hosts

Flows

User Interface (UI) Type.Component.[SubComponent].Function UI.linux.install

Controller (CO) Type.Component.[SubComponent].Function CO.windows.vm.create

Subcontroller (SC) Type.Component.[SubComponent].Function SC.vmware4.vm.deployLinuxHost

Action Plan (AP) Type.Component.[SubComponent].Function AP.windows.ad.findUserInGroup

Infrastructure Interface (II) Type.Component.[SubComponent].Function II.vmware4.datastore.getFreespace

Utilities (UT) Type.Component.[SubComponent].Function UT.privatedb.auxiliaryinfo.CompareUsedServernames

Private Flows MainFlowName.PR.prviateFlowFunction II.vmware4.datastore.getFreespace.PR.readFreespac

eFromDataStore

28.09.2012 36