Download - Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response
![Page 1: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/1.jpg)
Lessons Learned from Hurricane Katrina
Azim AshrafManager – Network Security & Incident Response
![Page 2: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/2.jpg)
Personal Naiveté
• Personal Preparations
• Some sense of excitement
• Estimation of what may occur
• Weather Channel – always on
• A bit of ‘Snow Day’ mentality
![Page 3: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/3.jpg)
04/20/23 LOUISIANA STATE UNIVERSITY 3
Hurricane Katrina
Thursday August 25
Sunday August 28
Tuesday August 23
Saturday August 27
Initial Projected Path
![Page 4: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/4.jpg)
Monday, August 29 - Landfall• Katrina’s Immediate Effects
– Makes landfall 6:10 a.m.– Lower LA Parishes swamped by storm surge; no real word out– Parts of New Orleans flooded, at least one levee over-topped, but
city seems to have survived– SE Louisiana devastated by winds/rain– Mississippi seems hardest hit
• Monday 5pm Meeting at LSUPD Station – LSU is OK– LSU Survived … just a little damage on campus– Data Center Lost power but fail-over to back-up worked perfectly– Everything Looks “Good to Go” for Tuesday clean-up, Wednesday
start-up, and Thursday-as-usual– Mood lightened– Power restored to campus ~6:15pm
![Page 5: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/5.jpg)
Tuesday 8/30 – Bad gets worse
• First confirmed reports of a levee failure in New Orleans occur at 1:30AM CDT
• By mid-day >80% of New Orleans is under water• Evacuees en route• LSU contacted about expanding routine special
evacuee facilities into a broader purpose– Medical Triage (Pete Maravich Assembly Center)– Special Needs Facility (Field House)– First IT needs – Phones, phones and more phones
![Page 6: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/6.jpg)
Called to assist
• IT personnel needed to respond
• It was not going to be anything like a ‘snow day’
![Page 7: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/7.jpg)
First Impressions
![Page 8: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/8.jpg)
First Impressions
![Page 9: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/9.jpg)
First Impressions
![Page 10: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/10.jpg)
LSU – A city within a city
Large H. Ed. institutions uniquely positioned to respond• Infrastructure, knowledge, manpower, affiliations
– PMAC/Field House – Became the largest acute care hospital to date in in U.S. history
• Over 40,000 (?) patients processed during Hurricanes Katrina and Rita
– Established a Hurricane command center• Coordinated information for students, and evacuees, as well as directing
resources to where they were needed
– Faculty, staff, and student volunteers– Housing for responders– Crowd control– Food and laundry services– Long distance charges– Managed volunteers– Received and distributed donations
![Page 11: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/11.jpg)
LSU – A city within a city (cont’d)– Tracked patients, volunteers, responders, supplies, etc..
– Provided Web page re-direction (and other IT services) for UNO
– Leveraged communications hardware and services to facilitate data or phone support for:
• Command centers
• Responders
• Govt. Agencies
• Affected Universities
• Evacuees
• Etc.
– LSU expended over $1M (not reimbursed)• Over $100K out of CIO’s budget
– LSU Became perhaps the most critical facility in support of disaster relief/response in the State of Louisiana
![Page 12: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/12.jpg)
Lessons Learned at LSU• Buildings can be rebuilt; hardware can be
replaced. Data is the basis of continuity.• Knowing what you’ll need to do and having
it organized is more important than knowing exactly ‘how’ you’ll do it
• IT enables everything in the 21st Century• IT Personnel = First Responders
• Disaster Recovery and Business Continuity Planning is not a luxury
• Be prepared to be flexible; adapt, improvise, overcome
![Page 13: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/13.jpg)
Lessons Learned at LSU (cont’d)
• Have a good stock of networking equipment, and mobile and desktop computing in the storeroom
• Have strong relationships with key vendors
• And most importantly…
![Page 14: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/14.jpg)
People are your most key asset• Know who does what and have them ‘on reserve’• Expect them to be burdened with other priorities• Be prepared to be amazed…
![Page 15: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/15.jpg)
Key changes in LSU’s Plan• Formal LSU EOC• Formal Memoranda
of Agreements (MOAs)– State agencies– Private sector
• diesel fuel from local refinery• water from local bottler, etc….
– Secondary suppliers backing up primaries• Chancellor requested written plans from all units
on campus• Full-time generator for PMAC• Logistics now pre-planned
![Page 16: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/16.jpg)
Traditional Disaster Recovery- You’re down, everything else is fine
• Do you have a workable DR plan?
• Do you know where on campus you’ll go?
• Did you take necessary back-ups and do you have them ready to re-produce production files?
• What vendors will you need to tap – and for what?
• How will you quickly re-establish network connectivity? Phone service? Web presence? E-mail? Mission critical information systems?
![Page 17: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/17.jpg)
Broader Disaster Recovery- You (and everyone around) you are down
• Are your off-sites conveniently (and perhaps tragically) close?
• Do you have arrangements to get key services restored at a distance– Web, E-mail, Financial/HR, Student Information, CMS
• Hot-sites may be too expensive – but can you find suitable raised floor/HVAC/power to ‘re-build’
• Can you support your administration “in exile?”– Internet access, computers, cell phones, e-mail, IM
• Is your ‘life-boat’ plan portable over larger distances?• Can you grab your key people? Can you care for them?
![Page 18: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/18.jpg)
One Possible Tool In The Arsenal:Data Center Lifeboat• Situation: What if we had very short notice
(4-8 hours) notice of the need to abandon our data center/campus and set-up elsewhere (>50miles away)
• Goal #1: Re-establish some critical subset of services
• Goal #2: Support the re-establishment of some subset of university administration
![Page 19: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/19.jpg)
Lifeboat• Key things to recover:
– Payroll/Financial Data– Web presence
• Splash/priority information screens
• As much content as possible
– E-mail service for faculty/staff/students
– Portal interface– Student Information Systems– HR, Procurement Systems– CMS– What else?
• Budgets ($25K, $50K, $100K)
• Key things to address– Off-site storage of critical back-ups– Ability to ‘grab and go’ key data
and hardware– List of key hardware needed later
from vendors– Disaster Supplies Crate
• What would we put into an 8x12 truck for rapid evac?
– Equipment for a mobile or relocated university command post
• Laptops, radios, phones, etc.– Identify Key IT personnel
• Who does what w/back-up• “Scoop ‘em up”
– Where might you go?
![Page 20: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/20.jpg)
Survivor Disaster RecoveryYou’re the last ones standing
• Dealing with unimaginable demands– Start imagining it
• Do you have a stock of equipment to set up a large support operation in short-order?– Networking gear, computers, cables, supplies, telephone service
• Value of a flexible and capable staff• Consider how you’ll do all this on top of your normal
jobs, as campus life resumes and student enrollment increases
• How ready is your campus administration to take on the role of disaster response center?– Facilities, public safety/police, communications, academic affairs– Is the CEO (Chancellor, or President) prepared?
![Page 21: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/21.jpg)
Final Thoughts
• Imagine the questions first so that you can find the answers
• Next time, you may not be watching it on CNN – you may be living it
• Do the right thing• Now is the time to think, plan, and take
action – later it will be too late
![Page 22: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/22.jpg)
Final Thoughts
• Data is the basis of continuity• Have a flexible plan• People are your most key asset• Do the right thing because in
the end its really all about…
![Page 23: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/23.jpg)
Service
![Page 24: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/24.jpg)
Credits• The staff of LSU ITS who helped make the relief
effort a success.
• Brian Voss (CIO) – ‘In the Wake of Katrina’
• Brian Nichols (CISO) – ‘At Katrina’s Edge’
• Frank O’Quinn (DR) – ‘Weathering the Storm’
• Sheri Thompson, Jim Zietz, and others- photographs
• John Borne – excerpts from Master’s Thesis
• Margo Jolet, LSU Office of Public Affairs - ‘LSU in the Eye of The Storm’
![Page 25: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/25.jpg)
![Page 26: Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response](https://reader031.vdocument.in/reader031/viewer/2022032206/56649eec5503460f94bfdf24/html5/thumbnails/26.jpg)
Lessons Learned from Hurricane Katrina
Azim AshrafManager – Network Security & Incident Response