Faster than a flash - behind the scenes of patching SharePoint Online

ALEKSANDAR DRAŠKOVIĆSERVICE ENGINEER, MICROSOFT IRELAND

SPONSORS

Aleksandar DraškovićSenior Service EngineerMicrosoft Ireland

AgendaIntroduction

Patching phasesNon-affecting work

Preparing environmentBits installation

Affecting workFarm upgradeContent databases upgrade

Automation

INTRODUCTION

„A service-level agreement (SLA) is a part of a standardized service contract

where a service is formally defined. Particular aspects of the service – scope,

quality, responsibilities – are agreed between the service provider and the

service user“source: Wikipedia

Introduction - SLA

Long bits extraction processTopology service endpoint discovery (round-robin)Long farm upgrade processTime and effort (requires engineer presence)Long downtimes

Limitations of the 2010/2013 mechanism

Challenges in SPO-D

AP Server FE Server0

2

4

6

8

10

12

14

16

Installation Time (hours)

Manual Automation

Server Farm0

100

200

300

400

500

600

700

800

900

Engineer interaction (min)

Manual Automation

Preparation workInstalling bitsFarm upgrade / WAC upgradeContent Database upgradePost-deployment work

Patching phases

NON-AFFECTING WORK

Usually 3 environment typesPre-production, Disaster Recovery and Production environment

Patch pre-production firstAnd perform basic functional tests

Patch disaster recovery as secondIn a case of any kind of disaster, it can’t be on the lower build level as production

Finally, patch the production environment

Diversity of environments

Preparation workRecession checkHealth check

Application poolsServicesweb.configDisk spaceLocal smoke test

Bits extractionDisable User Profile Incremental sync

Installing bitsCan be executed during the work week

HA setup required for no downtimeActivate maintenance mode for the server

In case you are using monitoring systems like SCOMTake the server out of rotation

So that the traffic won’t be directed to the affected server

Installing bits (contd.)Create services snapshot

Both SharePoint and Windows Services

Disable endpointsStop-SPServiceInstance

Stop and disable services on the serverIISAdmin, OSearch15, SPSearchHostController, SPAdminV4, SPTimerV4

Installing bits (contd.)Install bits

Copy app. bits and initialize resource securityInstall-SPApplicationContent, Initialize-SPResourceSecurity

Start servicesEnable endpointsRestart

Installing bits (contd.)Check server healthPerform a local smoke testBring the server back into rotationRemove server from the maintenance mode

Installing bits - tipsPatch multiple servers at once

Patch servers with different roles at the same time, eg. AP and FE

Watch the load on other frontends in the farmPerfmon: Web Service\Current Connections

Beware of July 2015 PUChanges in the search schema

AFFECTING WORK

Farm upgradeExecuted during agreed change window

There is a small period in time the services will be actually down

Place the whole farm in maintenance modePerform local smoke test on all serversCheck service health on all servers

Farm upgrade (contd.)Enumerate all content databases

Store all important parametersPrimary and mirror instanceDatabase nameWeb Application name

Dismount all content databasesPerform PSConfig on the first app serverPerform PSConfig on the rest of the servers

Farm upgrade (contd.)Mount all content databasesCheck service health on all serversPerform local smoke test on all frontendsPlace the farm out of the maintenance mode

WAC upgradeAffecting the whole WAC farmPlace the servers in the maintenance modeMake snapshot of the WAC configurationBreak WAC farmInstall bits on all WAC serversRebuild the WAC farmApply additional config (based on the snapshot)Remove servers from the maintenance mode

Content DB upgradePlace the farm in the maintenance modeUpgrade each content database

Upgrade-SPContentDatabaseCan be done in multiple change windows

Only sites in the currently upgraded database are affectedCan run multiple upgrade processes in parallelRemove the farm from the maintenance mode

AUTOMATION

AutomationChallenges

Effort: Human interaction with Servers Time: how long does it take to patch a SharePoint farmSecurity: limit the need to access the farm

GainsRepeated work pattern (on Monthly basis) Reduce the possibility of Human Error Co-op with increasing work load

ToolsSystem Center Orchestrator PowerShellAzure Automation

Automation

Stage TimeInstall bits 120 minutes

(depends of LPs installed)Farm update 90 - 120 minutes

(depends on farm size)WAC update (parallel to farm

update)90 minutes (depends of WAC farm size)

DB upgrade 180 minutes (depends of number of CDBs)

TOTAL (non-affecting work)TOTAL (affecting work)

120 minutes / server390 minutes

What are we gaining (SPO-D example)?

Stefan Gossner blogMost accurate information around PU / CU releaseshttps://blogs.technet.microsoft.com/stefan_gossner/

SharePoint patching demystifiedhttps://blogs.technet.microsoft.com/stefan_gossner/2014/08/18/sharepoint-patching-demystified/

Deploy software updates for SharePoint 2013https://technet.microsoft.com/en-us/library/cc263467.aspx

Additional info

thank youquestions?

live ratingsHTTP://WWW.ADRIT.DE/BLOG@ADRASKOVIC

spca.biz/NGYC