clustering technology in windows nt server, enterprise edition jim gray microsoft research...

54
Clustering Clustering Technology In Technology In Windows NT Server, Windows NT Server, Enterprise Edition Enterprise Edition Jim Gray Jim Gray Microsoft Research Microsoft Research [email protected] [email protected]

Upload: ava-ward

Post on 27-Mar-2015

222 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Clustering Technology Clustering Technology In Windows NT Server, In Windows NT Server, Enterprise EditionEnterprise Edition

Jim GrayJim GrayMicrosoft ResearchMicrosoft [email protected]@Microsoft.com

research.Microsoft.com/~grayresearch.Microsoft.com/~gray

Page 2: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Today’s AgendaToday’s Agenda Windows NTWindows NT®® clustering clustering

MSCS (Microsoft Cluster Server) DemoMSCS (Microsoft Cluster Server) Demo MSCS backgroundMSCS background

Design goalsDesign goals TerminologyTerminology Architectural detailsArchitectural details

Setting up a MSCS clusterSetting up a MSCS cluster Hardware considerationsHardware considerations Cluster application issuesCluster application issues

Q&AQ&A

Page 3: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Extra CreditExtra Credit

Included in your presentation Included in your presentation materials but not covered materials but not covered in this session in this session Reference materialsReference materials SCSI primerSCSI primer

Speakers notes includedSpeakers notes included Hardware CertificationHardware Certification

Page 4: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

MSCS In ActionMSCS In Action

Page 5: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

High Availability Versus High Availability Versus Fault ToleranceFault Tolerance High Availability: mask outages High Availability: mask outages

through service restoration through service restoration Fault-Tolerance: mask local faultsFault-Tolerance: mask local faults

RAID disksRAID disks Uninterruptible Power SuppliesUninterruptible Power Supplies Cluster Failover Cluster Failover

Disaster Tolerance: masks Disaster Tolerance: masks site failuressite failures Protects against fire, flood, sabotage,..Protects against fire, flood, sabotage,.. Redundant system and service at Redundant system and service at

remote site remote site

Page 6: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Windows NT ClustersWindows NT ClustersWhat is clustering to Microsoft?What is clustering to Microsoft?

Group of independent systems that Group of independent systems that appear as a single systemappear as a single system

Managed as a single systemManaged as a single system Common namespaceCommon namespace Services are “cluster-wide”Services are “cluster-wide” Ability to tolerate component failuresAbility to tolerate component failures Components can be added Components can be added

transparently to userstransparently to users Existing client connectivity is not Existing client connectivity is not

effected by clustered applicationseffected by clustered applications

Page 7: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Microsoft Cluster ServerMicrosoft Cluster Server

2-node available 97Q3 2-node available 97Q3 Commoditize fault-tolerance Commoditize fault-tolerance

(high availability)(high availability) Commodity hardware Commodity hardware

(no special hardware)(no special hardware) Easy to set up and manageEasy to set up and manage Lots of applications work out of the box.Lots of applications work out of the box.

Multi-node Scalability in NT5 timeframeMulti-node Scalability in NT5 timeframe

Page 8: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

MSCA Initial GoalsMSCA Initial Goals ManageabilityManageability

Manage nodes as a single systemManage nodes as a single system Perform server maintenance without affecting usersPerform server maintenance without affecting users Mask faults, so repair is non-disruptiveMask faults, so repair is non-disruptive

AvailabilityAvailability Restart failed applications and serversRestart failed applications and servers

Un-availability ~ MTTR / MTBF , so quick repairUn-availability ~ MTTR / MTBF , so quick repair Detect/warn administrators of failuresDetect/warn administrators of failures

ReliabilityReliability Accommodate hardware and software failuresAccommodate hardware and software failures Redundant system without mandating a dedicated Redundant system without mandating a dedicated

“stand by” solution“stand by” solution

Page 9: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Client PCsClient PCs

Server AServer A Server BServer B

Disk cabinet ADisk cabinet A

Disk cabinet BDisk cabinet B

HeartbeatHeartbeat

Cluster managementCluster management

MSCS ClusterMSCS Cluster

Page 10: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Web Web sitesite

DatabaseDatabase

Web site filesWeb site files

Database filesDatabase files

Server 1Server 1 Server 2Server 2

BrowserBrowser

Web Web sitesite

DatabaseDatabase

Server 1Server 1 Server 2Server 2

Failover ExampleFailover Example

Page 11: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Basic MSCS TermsBasic MSCS Terms

Resource - basic unit of failoverResource - basic unit of failover Group - collection of resourcesGroup - collection of resources Node - Windows NTNode - Windows NT®® Server Server

running cluster softwarerunning cluster software Cluster - one or more closely-coupled Cluster - one or more closely-coupled

nodes, managed as a single entity nodes, managed as a single entity

Page 12: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Cluster nameCluster name

Node nameNode name Node nameNode name

VirtualVirtualserver nameserver name

VirtualVirtualserver nameserver name

VirtualVirtualserver nameserver name

VirtualVirtualserver nameserver name

MSCS NamespaceMSCS NamespaceCluster viewCluster view

Page 13: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

ClusterCluster Node 1Node 1 Node 2Node 2 VirtualVirtualserver 1server 1

VirtualVirtualserver 2server 2

VirtualVirtualserver 3server 3

IP address:IP address:1.1.1.11.1.1.1Network Network name:name:WHECCLUSWHECCLUS

IP address:IP address:1.1.1.21.1.1.2Network Network name:name:WHECNode1WHECNode1

IP address:IP address:1.1.1.31.1.1.3Network Network name:name:WHECNode2WHECNode2

IP address:IP address:1.1.1.41.1.1.4Network Network name:name:WHEC-VS1WHEC-VS1

IP address:IP address:1.1.1.51.1.1.5Network Network name:name:WHEC-VS2WHEC-VS2

IP address:IP address:1.1.1.61.1.1.6Network Network name:name:WHEC-VS3WHEC-VS3

Internet Internet Information Information ServerServerSQLSQL

MTSMTS““Falcon”Falcon”

Microsoft Microsoft ExchangeExchange

MSCS NamespaceMSCS NamespaceOutside world viewOutside world view

Page 14: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Windows NT ClustersWindows NT ClustersTarget applicationsTarget applications

Application & Database serversApplication & Database servers E-mail, groupware, E-mail, groupware,

productivity applications serverproductivity applications server Transaction processing serversTransaction processing servers Internet Web serversInternet Web servers File and print serversFile and print servers

Page 15: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

MSCS Design PhilosophyMSCS Design Philosophy Shared nothingShared nothing

Simplified hardware configurationSimplified hardware configuration

Remoteable toolsRemoteable tools Windows NT manageability enhancementsWindows NT manageability enhancements

Never take a “cluster” down: shell game Never take a “cluster” down: shell game rolling upgraderolling upgrade

MicrosoftMicrosoft®® BackOffice™ product support BackOffice™ product support Provide clustering solutions for all levels Provide clustering solutions for all levels

of customer requirementsof customer requirements Eliminate cost and complexity barriers Eliminate cost and complexity barriers

Page 16: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

MSCS Design PhilosophyMSCS Design Philosophy

Availability is core for all releasesAvailability is core for all releases Single server image for administration, Single server image for administration,

client interactionclient interaction Failover provided for unmodified server Failover provided for unmodified server

applications, unmodified clients applications, unmodified clients (cluster-aware server applications (cluster-aware server applications get richer features)get richer features)

Failover for file and print are default Failover for file and print are default Scalability is phase 2 focus Scalability is phase 2 focus

Page 17: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Non-Features Of MSCSNon-Features Of MSCS

NotNot lock-step/fault-tolerant lock-step/fault-tolerant Not able to “move” running applicationsNot able to “move” running applications

MSCS restarts applications that are failed over to MSCS restarts applications that are failed over to other cluster membersother cluster members

Not able to recover shared state between Not able to recover shared state between client and server (i.e., file position)client and server (i.e., file position) All client/server transactions should All client/server transactions should

be atomicbe atomic Standard client/server development Standard client/server development

rules still applyrules still apply ACID always winsACID always wins

Page 18: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Setting Up MSCS Setting Up MSCS ApplicationsApplications

Page 19: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Attributes Of Cluster- Aware Attributes Of Cluster- Aware ApplicationsApplications A persistence model that supports A persistence model that supports

orderly state transitionorderly state transition Database exampleDatabase example

ACID transactionsACID transactions Database log recoveryDatabase log recovery

Client application supportClient application support IP clients onlyIP clients only How are retries supported?How are retries supported?

No name service location dependenciesNo name service location dependencies Custom resource DLL is a good thing Custom resource DLL is a good thing

Page 20: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

MSCS Services For MSCS Services For Application SupportApplication Support Name service mapperName service mapper

GetComputerName resolves GetComputerName resolves to virtual server nameto virtual server name

Registry replicationRegistry replication Key and underlying keys and values Key and underlying keys and values

are replicated to the other nodeare replicated to the other node AtomicAtomic Logged to insure partitions Logged to insure partitions

in time are handled in time are handled

Page 21: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Application Deployment Application Deployment Planning Planning System configuration is crucialSystem configuration is crucial

Adequate hardware configurationAdequate hardware configuration You can’t run Microsoft BackOffice You can’t run Microsoft BackOffice

on a 32-MB 75mhz Pentiumon a 32-MB 75mhz Pentium Planning of preferred group ownersPlanning of preferred group owners

Good understanding of single-server Good understanding of single-server performance is criticalperformance is critical See Windows NT Resource Kit See Windows NT Resource Kit

performance planning sectionperformance planning section Understand working set sizeUnderstand working set size What is acceptable performance to the What is acceptable performance to the

business units? business units?

Page 22: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Evolution Of Cluster- Evolution Of Cluster- Aware ApplicationsAware Applications Active/passiveActive/passive - general out-of- the-box - general out-of- the-box

applicationsapplications Active/activeActive/active - applications that can run - applications that can run

simultaneously on multiple nodessimultaneously on multiple nodes Highly scalableHighly scalable - extending - extending

the active/active through I/O shipping, the active/active through I/O shipping, process groups, and other techniquesprocess groups, and other techniques

Page 23: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

ApplicationApplication Node 1Node 1 Node 2Node 2

Microsoft SQL ServerMicrosoft SQL Server Microsoft TransactionMicrosoft Transaction Server (MTS)Server (MTS)

Internet InformationInternet Information Server (IIS)Server (IIS)

Microsoft ExchangeMicrosoft Exchange ServerServer

Application EvolutionApplication Evolution

Page 24: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

ApplicationApplication Node 1Node 1 Node 2Node 2 Node 3Node 3 Node 4Node 4

Microsoft SQL ServerMicrosoft SQL Server Microsoft TransactionMicrosoft Transaction Server (MTS)Server (MTS)

Internet InformationInternet Information Server (IIS)Server (IIS)

Microsoft ExchangeMicrosoft Exchange ServerServer

Evolution Of Cluster- Evolution Of Cluster- Aware ApplicationsAware Applications

Page 25: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

ResourcesResourcesWhat are they?What are they?

Resources are basic system Resources are basic system components such as physical disks, components such as physical disks, processes, databases, IP addresses, processes, databases, IP addresses, etc., that provide a service to clients etc., that provide a service to clients in a client/server environmentin a client/server environment

They are online in only one place They are online in only one place in the cluster at a timein the cluster at a time

They can fail over from one system They can fail over from one system in the cluster to another system in the cluster to another system in the clusterin the cluster

Page 26: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

ResourcesResources MSCS includes resource DLL support for:MSCS includes resource DLL support for:

Physical and logical diskPhysical and logical disk IP address and network nameIP address and network name Generic service or applicationGeneric service or application File shareFile share Print queuePrint queue Internet Information Server virtual rootsInternet Information Server virtual roots Distributed Transaction Coordinator (DTC)Distributed Transaction Coordinator (DTC) Microsoft Message Queue (MSMQ)Microsoft Message Queue (MSMQ)

Supports resource dependenciesSupports resource dependencies Controlled via well-defined interfaceControlled via well-defined interface Group: offers a “virtual server”Group: offers a “virtual server”

Page 27: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Windows NTWindows NTcluster servicecluster service

ResourceResourcemonitormonitor

Physical diskPhysical diskresource DLLresource DLL

IP addressIP addressresource DLLresource DLL

Generic appGeneric appresource DLLresource DLL

DatabaseDatabaseresource DLLresource DLL

Resource eventsResource events

Initiate changesInitiate changes

DiskDisk NetworkNetwork AppApp DatabaseDatabase

Cluster Service To ResourceCluster Service To Resource

Page 28: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

ClusterCluster ResourceResourceGroupGroup ResourceResource

Resource: program or device managed by a clusterResource: program or device managed by a clustere.g., file service, print service, database servere.g., file service, print service, database servercan depend on other resources (startup ordering)can depend on other resources (startup ordering)can be online, offline, paused, failedcan be online, offline, paused, failed

Resource Group: a collection of related resourcesResource Group: a collection of related resourceshosts resources; belongs to a hosts resources; belongs to a clusterclusterunit of co-location; involved in naming resourcesunit of co-location; involved in naming resources

Cluster: a collection of nodes, resources, and groupsCluster: a collection of nodes, resources, and groupscooperation for authentication, administration, namingcooperation for authentication, administration, naming

Cluster AbstractionsCluster Abstractions

Page 29: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

ClusterCluster GroupGroup ResourceResource

ResourcesResources

Resources have...Resources have... Type: what it does (file, DB, print, Web…) Type: what it does (file, DB, print, Web…) An operational state (online/offline/failed)An operational state (online/offline/failed) Current and possible nodesCurrent and possible nodes Containing Resource GroupContaining Resource Group Dependencies on other resourcesDependencies on other resources Restart parameters (in case Restart parameters (in case

of resource failure)of resource failure)

Page 30: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

ResourceResource

Fails over (moves) from one Fails over (moves) from one machine to anothermachine to another Logical diskLogical disk IP addressIP address Server applicationServer application DatabaseDatabase

May depend on another resourceMay depend on another resource Well-defined properties Well-defined properties

controlling its behaviorcontrolling its behavior

Page 31: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Resource DependenciesResource Dependencies

A resource may depend A resource may depend on other resourceson other resources

A resource is brought online after A resource is brought online after any resources it depends onany resources it depends on

A resource is taken offline before A resource is taken offline before any resources it depends onany resources it depends on

All dependent resources must All dependent resources must fail over togetherfail over together

Page 32: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Drive E:Drive E:resource DLLresource DLL

IP addressIP addressresource DLLresource DLL

Generic Generic applicationapplication

resource DLLresource DLL

DatabaseDatabaseresource DLLresource DLL

Drive F:Drive F:resource DLLresource DLL

Dependency ExampleDependency Example

Page 33: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Payroll groupPayroll group

Drive E:Drive E:resource DLLresource DLL

IP addressIP addressresource DLLresource DLL

Generic Generic applicationapplication

resource DLLresource DLL

DatabaseDatabaseresource DLLresource DLL

Drive F:Drive F:resource DLLresource DLL

Group ExampleGroup Example

Page 34: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Cluster API stubCluster API stub

Cluster administratorCluster administrator

Database Database ManagerManager MembershipMembership

ManagerManager

Global Global Update Update ManagerManager

FailoverFailoverManagerManager

Event Event ProcessorProcessor

Node Node ManagerManager

ResourceResourceManagerManager

PhysicalPhysicalresource DLLresource DLL

LogicalLogicalresource DLLresource DLL

ApplicationApplicationresource DLLresource DLL

Resource Resource APIAPI

Reliable ClusterReliable ClusterTransport + HeartbeatTransport + Heartbeat

ApplicationApplicationresource DLLresource DLL

ResourceResourcemonitorsmonitors

ObjectObjectManagerManager

ClusterClusterAPIAPI

Cluster.ExeCluster.Exe

LogLogManagerManager

CheckpointCheckpointManagerManager

Cluster API DLLCluster API DLL

Cluster API DLLCluster API DLL

NetworkNetwork

MSCS ArchitectureMSCS Architecture

Page 35: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

MSCS ArchitectureMSCS Architecture Cluster service is comprised of the Cluster service is comprised of the

following objectsfollowing objects Failover Manager (FM)Failover Manager (FM) Resource Manager (RM)Resource Manager (RM) Node Manager (NM)Node Manager (NM) Membership Manager (MM)Membership Manager (MM) Event Processor (EP)Event Processor (EP) Database Manager (DM)Database Manager (DM) Object Manager (OM)Object Manager (OM) Global Update Manager (LM)Global Update Manager (LM) Checkpoint Manager (CM)Checkpoint Manager (CM)

More about these in the next sessionMore about these in the next session

Page 36: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Setting Up An Setting Up An MSCS ClusterMSCS Cluster

Page 37: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

MSCS Key ComponentsMSCS Key Components

Two serversTwo servers Multi versus uniprocessorMulti versus uniprocessor Heterogeneous serversHeterogeneous servers

Shared SCSI bus Shared SCSI bus SCSI HBAs, SCSI RAID HBAs, HW RAID boxesSCSI HBAs, SCSI RAID HBAs, HW RAID boxes

Interconnect Interconnect Many types can be supportedMany types can be supported

Remember, two NICs per node Remember, two NICs per node PCI for cluster interconnect PCI for cluster interconnect

Complete MSCS HCL configurationComplete MSCS HCL configuration

Page 38: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

MSCS SetupMSCS Setup

Most common problemsMost common problems Duplicate SCSI IDs on adaptersDuplicate SCSI IDs on adapters Incorrect SCSI cablingIncorrect SCSI cabling SCSI Card order on PCI busSCSI Card order on PCI bus Configuration of SCSI FirmwareConfiguration of SCSI Firmware

Let’s walk through getting Let’s walk through getting a cluster operationala cluster operational

Page 39: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Test Before You BuildTest Before You Build

Bring each system up independentlyBring each system up independently Network adaptersNetwork adapters

Cluster interconnectCluster interconnect Organization interconnectOrganization interconnect

SCSI and disk functionSCSI and disk function NTFS volume(s)NTFS volume(s)

Page 40: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Top Ten Setup “Concerns”Top Ten Setup “Concerns”

10.10. SCSI is not well known. Please use the MSCS and SCSI is not well known. Please use the MSCS and IHV setup documentation. Consider the SCSI book IHV setup documentation. Consider the SCSI book reference for this sessionreference for this session

9.9. Build a support model that will support clustering Build a support model that will support clustering requirements. For example, in clustering requirements. For example, in clustering components are paired exactly (i.e., SCSI bios components are paired exactly (i.e., SCSI bios revision levels. Include this in your plans)revision levels. Include this in your plans)

8.8. Build extra time into your deployment planning to Build extra time into your deployment planning to accommodate cluster setup, both for hardware and accommodate cluster setup, both for hardware and software. Hardware examples include SCSI setup. software. Hardware examples include SCSI setup. Software issues would include installation across Software issues would include installation across cluster nodescluster nodes

7.7. Know the certification processKnow the certification processand its support implicationsand its support implications

Page 41: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Top Ten Setup “Concerns”Top Ten Setup “Concerns”6.6. Applications will become more cluster-aware throughApplications will become more cluster-aware through

time. This will include better setup, diagnostics, and time. This will include better setup, diagnostics, and documentation. In the meantime, plan and test accordinglydocumentation. In the meantime, plan and test accordingly

5.5. Clustering will impact your server maintenanceClustering will impact your server maintenanceand upgrade methodologies. Plan accordinglyand upgrade methodologies. Plan accordingly

4.4. Use multiple network adapters and hubs to eliminateUse multiple network adapters and hubs to eliminatesingle points of failure (everywhere possible)single points of failure (everywhere possible)

3.3. Today’s clustering solutions are more complexToday’s clustering solutions are more complexto install and configure than single servers. Plan to install and configure than single servers. Plan your deployments accordinglyyour deployments accordingly

2.2. Make sure that your cabinet solutions and peripherals both Make sure that your cabinet solutions and peripherals both fit and function well. Consider the serviceability fit and function well. Consider the serviceability implicationsimplications

1.1. Cabling is a nightmare. Color coded, heavilyCabling is a nightmare. Color coded, heavilydocumented, Y cable inclusive, maintenance-designed documented, Y cable inclusive, maintenance-designed products are highly desirableproducts are highly desirable

Page 42: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Cluster Management ToolsCluster Management Tools Cluster administratorCluster administrator

Monitor and manage clusterMonitor and manage cluster Cluster CLI/COMCluster CLI/COM

Command line and COM interfaceCommand line and COM interface Minor modifications to existing toolsMinor modifications to existing tools

Performance monitorPerformance monitor Add ability to watch entire clusterAdd ability to watch entire cluster

Disk administratorDisk administrator Add understanding of shared disksAdd understanding of shared disks

Event loggerEvent logger Broadcast events to all nodesBroadcast events to all nodes

Page 43: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

In Search of Clusters; The Coming BattleIn Search of Clusters; The Coming BattleIn Lowly Parallel ComputingIn Lowly Parallel Computing Gregory F. PfisterGregory F. Pfister ISBN 0-13-437625-0ISBN 0-13-437625-0

The Book of SCSIThe Book of SCSI Peter M. RidgePeter M. Ridge ISBN 1-886411-02-6ISBN 1-886411-02-6

MSCSMSCSReference MaterialsReference Materials

Page 44: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

The Basics Of SCSIThe Basics Of SCSI

Why SCSI?Why SCSI? Types of Types of

interfaces?interfaces? Caching and Caching and

performance…performance… RAIDRAID The future…The future…

Page 45: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Why SCSI?Why SCSI? Faster then IDE - intelligent card/driveFaster then IDE - intelligent card/drive

Uses less processor time Uses less processor time Can transfer data up to 100 MB/sec.Can transfer data up to 100 MB/sec.

More devices on a single chain - More devices on a single chain - up to 15up to 15

Wider variety of devicesWider variety of devices DASDDASD ScannersScanners CD-ROM writers and optical drivesCD-ROM writers and optical drives Tape drivesTape drives

Page 46: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Types Of InterfacesTypes Of Interfaces SCSI and SCSI IISCSI and SCSI II

50-pin, 8-bit, max transfer = 10 MB/s 50-pin, 8-bit, max transfer = 10 MB/s (early 1.5 to 5 MB/s )(early 1.5 to 5 MB/s )

Internal transfer rate = 4 to 8 MB/sInternal transfer rate = 4 to 8 MB/s Wide SCSIWide SCSI

68-pin, 16-bit, max transfer = 20 MB/s68-pin, 16-bit, max transfer = 20 MB/s Internal transfer rate = 7 to 15.5 MB/sInternal transfer rate = 7 to 15.5 MB/s

Ultra SCSIUltra SCSI 50-pin, 8-bit, higher transfer rate, 50-pin, 8-bit, higher transfer rate,

max transfer = 20 MB/smax transfer = 20 MB/s Internal transfer rate = 7 to 15.5 MB/sInternal transfer rate = 7 to 15.5 MB/s

Ultra wideUltra wide 68-pin, 16-bit, max transfer rate = 40 MB/s68-pin, 16-bit, max transfer rate = 40 MB/s Internal transfer rate = 7 to 30 MB/sInternal transfer rate = 7 to 30 MB/s

Page 47: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Performance FactorsPerformance Factors Cache on the drive or controllerCache on the drive or controller Caching in the OSCaching in the OS Different variablesDifferent variables

Seek timeSeek time Transfer ratesTransfer rates

Page 48: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Raid Types A.K.A Description

RAI D 0 Striping the f astest RAI D - data is "stripped" across multiple volumes , no redundancy

RAI D 1 Mirroring a simple pair of drives with data replicated on both, writes are slower

RAI D 2 these sector stripe data across drives with some storing ECC inf o - done in HW now

RAI D 3 Sector striping but one drive dedicated to storing parity inf ormation f or the set

RAI D 4 identical to RAI D 3 but large stripes

RAI D 5 Best f or Multi-user environments, parity is spread across 3 or more drives

Redundant Array Of Redundant Array Of Inexpensive Disks (RAID)Inexpensive Disks (RAID) Developed from paper published in 1987Developed from paper published in 1987

at University of California Berkeleyat University of California Berkeley The idea is to combine multiple inexpensive drives The idea is to combine multiple inexpensive drives

(eliminate SLED - single large expensive drive)(eliminate SLED - single large expensive drive) Provided redundancy by storing parity informationProvided redundancy by storing parity information

Page 49: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

The Future For SCSIThe Future For SCSI

Faster interfaces - why?Faster interfaces - why? Fibre ChannelFibre Channel

Optical standard Optical standard Proposed as part of SCSI III (not final)Proposed as part of SCSI III (not final) Up to 100 MB/s transferUp to 100 MB/s transfer

Still using ultra-wide SCSI Still using ultra-wide SCSI inside enclosuresinside enclosures

Drives with optical interfaces not Drives with optical interfaces not available yet in quantity, higher cost available yet in quantity, higher cost than SCSIthan SCSI

Page 50: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

The Future Of SCISThe Future Of SCIS

Fibre Channel-arbitrated loopFibre Channel-arbitrated loop Ring instead of bus architectureRing instead of bus architecture Can support up to 126 devices/hostsCan support up to 126 devices/hosts Hot pluggable through the use Hot pluggable through the use

of a port bypass circuitof a port bypass circuit No disruption of the loop as devices No disruption of the loop as devices

are added/removedare added/removed Generally implemented using Generally implemented using

a backplane designa backplane design

Page 51: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

HCL List For MSCSHCL List For MSCS

Servers on normal Windows NT HCLServers on normal Windows NT HCL Self-test of MP machines soonSelf-test of MP machines soon

MSCS SCSI component HCLMSCS SCSI component HCL Tested by WHQLTested by WHQL Must pass Windows NT HCT as wellMust pass Windows NT HCT as well

MSCS interconnect HCLMSCS interconnect HCL Tested by WHQLTested by WHQL Not required to pass 100% of HCTNot required to pass 100% of HCT

I.e., point-to-point adaptersI.e., point-to-point adapters

Page 52: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Windows NT 4.0+Windows NT 4.0+

ServerServer

HCLHCL

Windows NT 4.0+Windows NT 4.0+

SCSISCSI

HCLHCL

Windows NT 4.0+Windows NT 4.0+

NetworkNetwork

HCLHCL

Windows NT 4.0+Windows NT 4.0+

MSCSMSCS

SCSISCSI

HCLHCL

Co

mp

lete

MS

CS

con

figu

ratio

n re

ad

y C

om

ple

te M

SC

S co

nfig

ura

tion

rea

dy

for se

lf-tes

tfo

r self-te

st

MSCS System Certification MSCS System Certification ProcessProcess

Page 53: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Testing PhasesTesting Phases

HW compatibility (24 hours)HW compatibility (24 hours) SCSI and interconnect testingSCSI and interconnect testing

One-node testing (24 hours)One-node testing (24 hours) Eight clients Eight clients

Two-node with failover (72 hours)Two-node with failover (72 hours) Eight-client with asynchronous failoversEight-client with asynchronous failovers

Stress testing (24 hours)Stress testing (24 hours) Dual initiator I/O, split-brain problemsDual initiator I/O, split-brain problems Simultaneous rebootsSimultaneous reboots

Page 54: Clustering Technology In Windows NT Server, Enterprise Edition Jim Gray Microsoft Research Gray@Microsoft.com research.Microsoft.com/~gray

Final MSCS HCLFinal MSCS HCL

Only complete configurationsOnly complete configurationsare supportedare supported

Self test results sent to MicrosoftSelf test results sent to Microsoft Logs checked and configuration reviewedLogs checked and configuration reviewed

HCL updated on Web and forHCL updated on Web and fornext major Windows NT releasenext major Windows NT release

For more details see the MSCS For more details see the MSCS Certification documentCertification document