SharePoint Patching: Notes From the FieldSam Hassani, Premier Field Engineer, Microsoft UK

• The story of SharePoint patching• Answering the ‘what do I need

question’• Dismantle a released update• Update scenarios• Planning and upgrading• Monitoring and troubleshooting

Agenda

Terminology

• Hotfix• COD –Critical on Demand• Public Hotfix e.g. US DST Hotfix 941422• Post Service Pack rollup• Rollup of hotfixes build during SP-lock timeframe when

service pack is locked to not accept new fixes

• Cumulative Update• Since December *– rolling 2 month basis• Broadly affecting support issues

• Service Pack• Service Packs include all of the updates for MOSS plus deliver

important customer-requested stability and performance improvements, while incorporating further enhancements to user security.

Type of Updates Released

Build Numbers

1212 00 62196219 10001000

MajorMajor MinorMinor BuildBuild Revision

Revision

• Major = Office 12 : SharePoint 2007• Minor = RTM• Build = Internal build number

• 4518 = RTM• 6219 = SP1• 6320(WSS)/6322(MOSS) = Infrastructure Update• 6341 = February Cumulative Update

• Revision = Update type• 1000 = Service Pack• 30xx = Private build• 5000 = Cumulative Update• 500x = COD Build

Check Your Build – How?

• WSS – Check Version OWSSVR.DLL• WSS Localised – Check Installed KB Numbers• Content DB Versions Table

• MOSS – Check Version Microsoft.SharePoint.Portal.DLL

• MOSS Localised – Check Installed KB Numbers• Content DB Versions Table

• DLC – Check Version Microsoft.Office.Policy.DLL

• OSRV – Check Version of Microsoft.Office.Server.DLL

Check Your Build – How?

DEMO – Check Your Build

Patching Post IU

• http://support.microsoft.com/kb/953749/

• “Improvements to the time that is required to update and upgrade Windows SharePoint Services sites”

• PSConfig no longer updates Site Collection Version number unless a Schema update is required – Significant Performance Improvement

• February Cumulative Updates for WSS and MOSS• http://support.microsoft.com/kb/961756• http://support.microsoft.com/kb/961755

• Latest service pack + latest WSS Cumulative Update +latest MOSS Cumulative Update = completely up-to-date. 

How it used to be....

Build CoreServer GLB

CoreServer Loc

DLC-GLB

DLC-Loc

pjsrvwfe-glb

pjsrvapp-glb

Ifswfe -glb

Xlsrvapp -glb

6324.5000

July

955593 955586 955586

6327.5000

Aug

956056 953397

6331.5000

Oct

957693 958567 958569 957696

957694

How it was - MOSS

Build STS-GLB

STS-LOC

6324.5000

July

955594

6327.5000

Aug

957109

6332.5000

Oct

957691

How it was - WSS

The December CU and Beyond.....

Build CoreServer -glb

CoreServer -loc

DLC-GLB

DLC-loc

pjsrvwfe-glb

pjsrvapp-glb

Ifswfe

-glb

Xlsrvapp

-glb6324.5000

July

955593 955586 955586

6327.5000

Aug

956056 953397

6331.5000

Oct 957693 958567 958569 957696

957694

6335.5000

Dec

960011 960011 960011 960011

960011

960011

960011

960011

6341.5002

Feb

961755 961755 961755 961755

961755

961755

961755

961755

How it is- MOSS

Build STS-glb

STS-loc

6324.5000

July 955594

6327.5000

Aug

957109

6332.5000

Oct 957691

6335.5000

Dec

960010 960010

6341.5000

Feb 961756 961756

How it is- WSS

What’s In the MSP?

• Orca • Part of the Window’s development tools

portion of the SDK• http://msdn2.microsoft.com/en-us/libr

ary/aa370557.aspx

• Open MSI file• View MSP as transform

What’s In the MSP?

DEMO – Dismantling a Patch

Typical Scenarios

Updating to.........Minimum Build......Recommended

Build......Latest and Greatest Build

New Farm

New Server in Existing

Farm

Existing Server in Existing

Farm

• Minimum required build:• The oldest supported build• Currently this is SP1 for WSS 3.0/MOSS 2007• Once a Service Pack is released, the n-1 version is

supported for 1 year after Service Pack release. • After this time, patches for n-1 version will not be shipped

• Microsoft recommended build:• Product groups pubic recommendations:

http://blogs.msdn.com/sharepoint• The current recommended build is the Infrastructure

Update

• Latest and greatest build: • Running with maximum number of fixes

Updating To ....

• Which build should I be on?• Minimum required• Microsoft recommended• Latest and greatest

• "If it ain't broke don't fix it” attitude is becoming less prevalent (based on Case Histories)

• Software vendors are more diligent leading to increased consumer trust

Culture Shift

SP1 + IU

RTM SP1 + Dec CU

Improved Support from PSS

Version Distribution

• What is the best way to build it?• Slipstream it• Any package can be slipstreamed - extract the

MSPs into the update folder• Benefits of fixes in the provisioning and

configuration wizard code paths.• When you configure CA it is configured with the

fixes and updates already in place.• Long term is no different to manual updates but

is more efficient

New Farm

• Adding a new server to an existing farm:• Replacement • Scale-out process

• New server must be same version as existing servers

• How do we achieve this?• Slipstream installation• Manually install patches

• Do the above before running configuration wizard to join the farm

New Server in Existing Farm

• Patching an existing server in an existing farm:• New Microsoft recommendations• To correct an error that has been experienced (per

a KB article)• To be running the most recent code

• How do we achieve this?• Manually install patches

• Do the above before running configurationwizard

Existing Server in Existing Farm

• Requirement for deploying on Windows Server 2008

• Deployment guides• Windows SharePoint Services 3.0

• http://technet2.microsoft.com/windowsserver/WSS/en/library/91649a7e-6b5a-4e5a-9ee5-51951f4b857f1033.mspx

• Office SharePoint Server 2007• http://technet2.microsoft.com/Office/en-us/li

brary/f484f5f2-35bb-4d70-bf56-dd1c4c287c721033.mspx

Slipstream Creation

Slipstream Creation

Pre Upgrade Steps

• Take a complete farm backup and verify it restores• It is not supported to restore an earlier version onto a

newer version e.g. RTM to SP 1

• Backup any SharePoint out of box files you may have customized

• Run orphan repair tool, http://support.microsoft.com/kb/923904

• Defragment databases, http://support.microsoft.com/kb/943345

• Ensure adequate disk space exists on each server• Installation of the new binaries, and upgrade logging will take disk space

• Run a simulation on staging environment

• Load balance sites across databases if needed• Whitepaper released at SP 1, Performance

Recommendations for Storage Planning and Monitoring, http://go.microsoft.com/fwlink/?LinkID=105623&clcid=0x409

• Verify no upgrade Timer Jobs are currently running• In place upgrade needs to complete before installing an

update• Gradual upgrade needs to complete current site

upgrade before installing an update

• Take a full farm backup immediately after you know the update has applied successfully

Pre Upgrade Steps

Deployment Overview

• All updates follow the same steps and procedures

• Make sure the account you will be installing the update has adequate permissions

• Mechanics• Install the update on the SharePoint server(s)

• Installs the updated binaries

• Run SharePoint Configuration Wizard or psconfig.exe• Updates the web applications, databases, and versions

• The database upgrade is the longest running piece

• Until the database upgrade is complete access to content is blocked to prevent corruption

Deployment Options – In Place

• Single server Basic installation• Automatic updates

• In place installation for a farm• Install the update on each server until the following

dialog appears

• Click OK on the dialog on the first server • Wait until it has completed• Continue to each of the remaining servers

Deployment options – DB attach

• Document all databases assigned to each web application

• Remove all of the databases from the web applications

• Install the update on each server following the “in place” method

• Re-attach the content databases to respective web applications• Note: Attaching the content database has been reported to be faster and give more flexibility and control than normal “in place” upgrading of a farm

Minimising Downtime

• High performance db attach

• MSIT Scenario:• 4 Terabytes of data to upgrade• 80 content databases at 50GB per database

• Upgrade Procedure:1. Create the 4 farms (Identical target patch version)2. Backup databases3. Attach databases to upgrade farms (in parallel)4. Verify upgrade has completed successfully5. Detach databases from upgrade farms (in parallel)6. Attach upgraded databases to production farm

• Complete with quadrupled throughput!

Minimising Downtime

• Duplicate Read-Only Farm

• Internet Scenario:• 24x7 Internet Presence Required

• Upgrade Procedure:1. Create 2 farms (Identical original patch version)2. Backup databases3. Attach databases to target farm4. Change routing / DNS5. Verify access to ‘Read Only’ farm6. Upgrade production farm7. Switch back to Upgraded farm

• Complete with continued uptime!

Sequence of Installation

• WSSv3 Updates must be applied before the MOSS 2007 Updates

• FAQ – Can I install WSS and MOSS updates and run the configuration wizard once?

• FAQ – Can I uninstall any update if there are problems?

Upgrade Order

• Web01 = COMPLETE + Central Admin• Upgrade this server first

• Web01 = WFE + Central Admin• Upgrade index01 first if it is also a WFE

since it has the most roles/services• If index01 does not have WFE enabled

upgrade web01 first

• Multiple Central Admin• Upgrade “1st” Central Admin before

“2nd”

• Multiple Query/Index• If Search/Index contains Central Admin

upgrade it first Otherwise, whichever server has the most roles/services is before others

Verify Update was Successful

• The Upgrade Log file

• Finished upgrading SPFarm Name=<Configuration Database>

• In-place upgrade session finishes. Root object = SPFarm=<Configuration Database>, recursive = True. 0 errors and 0 warnings encountered.

• If you cannot find the entries look for Exception or Error, correct the problem and run the configuration wizard or psconfig.exe again

• Check build version

Monitoring and Troubleshooting

• Most support issues reported on deploying updates relates to failures running the Configuration wizard • SharePoint Products and Technologies Configuration wizard• psconfig.exe

• Log files• PSCDiagnostics log files, e.g.

PSCDiagnostics_1_21_2008_14_27_10_84_453374783.log

• Upgrade log files, e.g. Upgrade.log

• Event Logs

• PSCDiag and Upgrade log files are stored in: c:\Program Files\Common Files\Microsoft Shared\web server extensions\12\LOGS

Monitoring and Troubleshooting

Monitoring and Troubleshooting

psconfig -cmd upgrade -inplace b2b -wait -force

??

Upgrade Log• Refreshing the upgrade log• Sequences and Actions

• http://support.microsoft.com/kb/948780/en-us

• FAQ – What is ‘SyncUpgradeTimerJob: sleeping for 10 seconds’

Monitoring and Troubleshooting

• Sequences can repeat in certain scenarios• Multiple content databases• Multiple web applications

• If the SPTimerv3 account can not write to the default Upgrade.log, it may write to

\Documents and Settings\SPTimerv3 account\

Local Settings\Temp\Upgrade.log

• The Upgrade.log file is cumulative

Monitoring and Troubleshooting

• Issue: ‘Server Error’ when accessing SharePoint site

• Cause: WFEs are upgraded but Databases are not

• Resolution: Detach/reattach version mismatched database(s).

Common Failures

• Issue: PSConfig fails with inadequate install permissions

• Cause: Not logged in with adequate permissions to install the update• Overall #2 SP1 deployment failure

• Resolution: Ensure necessary permissions when deploying any update• Administrators on the WFEs• Administrators on the SQL Server or granted

the database role db_owner on all SharePointdatabases

Common Failures

• Issue: PSConfig fails during site collection upgrade sequence

• Cause:[NewWebParts] [12.1.6301.0] [DEBUG] [1/10/2009 8:15:40 AM]: Begin Upgrade()........[SiteSequence] [ERROR] [1/10/2009 8:15:40 AM]: Unknown server error number: d

• Resolution:. • Disable ForeFront Security for SharePoint• Run PSconfig, or detach/reattach content

databases

Common Failures

• Issue: PSConfig fails at Step 8

• Cause:[SPManager] [ERROR] [12/19/2008 10:46:02 AM]: Upgrade [SPSearchDatabase Name=OSST_SEARCH Parent=SPDatabaseServiceInstance Name=sql08,1433] failed.......[SPManager] [ERROR] [12/19/2008 10:46:02 AM]: Object reference not set to an instance of an object.

• Resolution : Corrupt dll in Dec CU [Build 3334]

Common Failures

Final Thoughts...

• Understand what you have...

• Understand where you want to be...

• While we cannot upgrade a live farm we do have processes that will move us closer to the nirvana of a zero downtime upgrade