MARFC Operational BackupMARFC Operational BackupA Case StudyA Case Study

January 26, 2006January 26, 2006

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 22

OutlineOutline

• ProblemProblem

• Proposed SolutionProposed Solution

• ResultsResults

• IssuesIssues

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 33

ProblemProblem• MARFC moving to new facilityMARFC moving to new facility• AWIPS unavailable for up to 6 days during AWIPS unavailable for up to 6 days during

movemove• How to conduct operations during AWIPS How to conduct operations during AWIPS

outageoutage– Maintain full operational outputMaintain full operational output

• Official ProductsOfficial Products• All web informationAll web information

– Worst case planning?Worst case planning?

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 44

Proposed SolutionProposed Solution

• Utilize in-house Linux serverUtilize in-house Linux server

• X-client LaptopsX-client Laptops

• ER WAN ConnectionER WAN Connection

• Data feed via LDMData feed via LDM

• Transmission via LDADTransmission via LDAD

• Additional seats for flooding and supportAdditional seats for flooding and support

• AWIPS OB4 BasisAWIPS OB4 Basis

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 55

Proposed Solution - ServerProposed Solution - Server

• Linux ServerLinux Server– Dell 4600 PowerEdge (late 2002)Dell 4600 PowerEdge (late 2002)– Dual 2 GHz Xeon CPUsDual 2 GHz Xeon CPUs– 1 GB Memory1 GB Memory– Raid 1 SCSI 73GB HDsRaid 1 SCSI 73GB HDs– Dual Power SuppliesDual Power Supplies– RH 7.3RH 7.3

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 66

Proposed Solution - AccessProposed Solution - Access• 4 Operational seats4 Operational seats• X access via 3 client laptopsX access via 3 client laptops

– Part of ER RFC Backup ProjectPart of ER RFC Backup Project– 1600x1050 resolution1600x1050 resolution

• 1 ½ screens1 ½ screens

– External mouse and keyboardExternal mouse and keyboard– Additional monitor for “non-AWIPS” displayAdditional monitor for “non-AWIPS” display

• Server console for HASServer console for HAS• 2 add’l clients for support and flooding2 add’l clients for support and flooding

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 77

Proposed Solution - CommsProposed Solution - Comms

• ER WAN ConnectionER WAN Connection

• Data feed via LDMData feed via LDM– Redundancy via ERH and SRHRedundancy via ERH and SRH

• Transmission via LDADTransmission via LDAD– Redundancy via PBZ with ERH backupRedundancy via PBZ with ERH backup

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 88

Proposed Solution - AppsProposed Solution - Apps• Based on AWIPS OB4Based on AWIPS OB4• Synchronize local apps changes Synchronize local apps changes

between AWIPS and serverbetween AWIPS and server– Crons as wellCrons as well

• Allow all auto processes to continueAllow all auto processes to continue– Shut off delivery via tokensShut off delivery via tokens

• Identify files to sync to go liveIdentify files to sync to go live– OFS, DB, controlOFS, DB, control

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 99

ResultsResults

• Difference in capabilitiesDifference in capabilities– No D2DNo D2D– No ArcView useNo ArcView use

• FOPFOP• Inundation mappingInundation mapping

– No 12Planet (within AWIPS only)No 12Planet (within AWIPS only)– 1 ½ vs. 3 screens1 ½ vs. 3 screens

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1010

ResultsResults• OperationsOperations

– Started with HAS AM shift 12/5/2005Started with HAS AM shift 12/5/2005– Product delivery easily shut off on AWIPS and Product delivery easily shut off on AWIPS and

initiated on backup via tokensinitiated on backup via tokens– Continued in backup mode until Thursday, 12/8, Continued in backup mode until Thursday, 12/8,

morning hydro shiftmorning hydro shift• AWIPS available for testing Wednesday night, 12/7AWIPS available for testing Wednesday night, 12/7

– Token reset shut off delivery from backup and Token reset shut off delivery from backup and initiated on AWIPSinitiated on AWIPS

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1111

ResultsResults

• PerformancePerformance– No forecaster perceived slowness in any No forecaster perceived slowness in any

operational applicationoperational application– No missed delivery of any text or graphical No missed delivery of any text or graphical

productproduct

• SUCCESS!SUCCESS!

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1212

IssuesIssues• Keeping in-step with AWIPSKeeping in-step with AWIPS

– Upgraded to OB5 on AWIPS prior to moveUpgraded to OB5 on AWIPS prior to move– Stayed on OB4 due to DB table changesStayed on OB4 due to DB table changes

• Not sure of best approach to incorporate changes of this Not sure of best approach to incorporate changes of this naturenature

• OB6?OB6?

• Help with problemsHelp with problems• Maintenance overhead – 2 systemsMaintenance overhead – 2 systems• Overall RFC solution?Overall RFC solution?

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1313

IssuesIssues

• Redundancy is critical!Redundancy is critical!

• 1 hour into backup operations one HD 1 hour into backup operations one HD on server failedon server failed– Continued with one HD with no problemsContinued with one HD with no problems

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1414

IssuesIssues

• Off-site computing capabilitiesOff-site computing capabilities

• If incapable of full operations support, If incapable of full operations support, what do you cut outwhat do you cut out– Customers expecting informationCustomers expecting information

• HD failure forced laptop server HD failure forced laptop server implementationimplementation

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1515

IssuesIssues

• Off-site computing capabilitiesOff-site computing capabilities– Part of ER RFC Backup projectPart of ER RFC Backup project– Dell 5150 “desktop replacement” laptopDell 5150 “desktop replacement” laptop– 1 year old1 year old– Pentium 4 3.0 GHzPentium 4 3.0 GHz– 1 GB Memory1 GB Memory– 1 100GB HD; 7200 RPM ATA 1 100GB HD; 7200 RPM ATA

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1616

IssuesIssues

• Off-site computing capabilitiesOff-site computing capabilities– Operations testOperations test

• 3 client laptops3 client laptops• Laptop displayLaptop display• Full cron loadFull cron load• Morning forecast operationsMorning forecast operations

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1717

IssuesIssues

• Off-site computing capabilitiesOff-site computing capabilities– Operations test – resultsOperations test – results

• SlowdownsSlowdowns– NMAP initiationNMAP initiation– Heavy disk access – e.g. Informix extractionsHeavy disk access – e.g. Informix extractions

• Slower than in-house serverSlower than in-house server• Acceptable by forecast staff knowing limitationsAcceptable by forecast staff knowing limitations

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1818

Performance ComparisonPerformance Comparison

• Identical cron instructionsIdentical cron instructions

• Handoff and execution of OFS jobs from Handoff and execution of OFS jobs from AWIPSAWIPS

• Laptop slowdown with heavy disk Laptop slowdown with heavy disk access activityaccess activity

• More analysis neededMore analysis needed

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 1919

updatedb,db_purge,sys_clean

db_purge

???

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 2020

Off-site CapabilitiesOff-site Capabilities• ““Shoebox” server, aka ShuttlePCShoebox” server, aka ShuttlePC• ““Server-like”; luggableServer-like”; luggable• Dual CPUsDual CPUs• Multi-GB memory capabilitiesMulti-GB memory capabilities• 7200 RPM SATA II7200 RPM SATA II

– SCSI-like performance?SCSI-like performance?– Larger, cheaper than SCSILarger, cheaper than SCSI– RAID 1RAID 1

1/26/20061/26/2006 MARFC Operational BackupMARFC Operational Backup 2121

Off-site CapabilitiesOff-site Capabilities

• Less than $3000Less than $3000– Dual 64-bit AMD Opteron, 2.4 GHzDual 64-bit AMD Opteron, 2.4 GHz– 2 GB memory2 GB memory– Dual 120 GB 7200 RPM SATA IIDual 120 GB 7200 RPM SATA II

• RAID 1RAID 1

– GB ethernetGB ethernet– 19” LCD19” LCD– External keyboard and mouseExternal keyboard and mouse

The EndThe End