the experiences of migrating a large scale, high performance healthcare network
Post on 21-May-2015
816 Views
Preview:
DESCRIPTION
TRANSCRIPT
April 2, 2008 SlipstreamUSA 1
The experiences of migrating a large scale, high performance healthcare network
Larry WilliamsCorporate Manager, Partners HealthCare
April 2, 2008 SlipstreamUSA 2
In the next half hour…
Partners Healthcare System overview Caché platform architecture & metrics The need to migrate Phased migration approach Benchmark testing and results Discoveries and production enhancements
April 2, 2008 SlipstreamUSA 3
Partners Healthcare System
Founded in 1994 Brigham & Women’s Hospital Massachusetts General Hospital
Now includes: Community physician network (1200 + 3500 MD’s) PCHi 3 community hospitals 2 rehab hospitals 3 specialty institutions
Enterprise-wide Information Systems 1100 employees Annual budget FY05 approximately $160 million
April 2, 2008 SlipstreamUSA 4
BWH
MGH
Logan Airport
10 km
6 km
Anchor Hospitals & Airport
April 2, 2008 SlipstreamUSA 5
MGH
BWH
Newton-Wellesley
Acute Care Hospitals
Community PhysicianPractices
April 2, 2008 SlipstreamUSA 6
Partners Domain Devices
Internet
12,000 Printers
1,450 Servers
32,000 Desktops
Firewall
Closely Managed
~30,000 other devices
Assumed Managed
April 2, 2008 SlipstreamUSA 7
Windows Production Architecture
3.5 TB
April 2, 2008 SlipstreamUSA 8
Enterprise Integration
# of
Interfaces
Est. Annual
Transactions
Daily
AverageChange from prior
year
2004 167 610,833,080 1,673,515
2005192 887,649,802 2,431,917 45%
2006170
1,240,712,044 3,399,211 40%
2007 196 1,330,962,017 4,659,035 37%
Over 30% are to and from Caché database
April 2, 2008 SlipstreamUSA 9
Integration Components
April 2, 2008 SlipstreamUSA 10
Gigabytes in Use
April 2, 2008 SlipstreamUSA 11
Annual Database Growth Rate
21%
37%
19%
24%
30%
37%
46%
39%
2000 2001 2002 2003 2004 2005 2006 2007
April 2, 2008 SlipstreamUSA 12
Database Utilization
Average Database References per day in Billions
April 2, 2008 SlipstreamUSA 13
The Need to Migrate - Availability
Uptime In Hours In Minutes 99 % 7.3
99.5 % 3.6
99.75% 1.8
99.9 % 44
99.99 % 4.4
99.999% 26 seconds
Monthly Downtime
Current State
Business need
April 2, 2008 SlipstreamUSA 14
Additional Business Requirements
Increase availability and reliability Decrease database risk from 5 single points of failure More robust hardware and OS Many less servers and OS instances to manage Clustering and automated failover Reduce monthly maintenance needs, updates once or twice per year
--------------------------------------------------------
Improve Performance 64 bit OS, more memory for cache
Caché 5.0.20 to Caché 2008.1, significantly improved ECP performance Increase Scalability
91 Terabytes available on EMC SAN DMX3
On-demand addition of processor cores
April 2, 2008 SlipstreamUSA 15
Caché Migration Decision Making Process
Only considered first tier vendors and support (IBM, HP) HP assumed much more risk with Professional Services Existing HP business yields more leverage & visibility with
regional office More headroom in HP configuration Price was not a distinguishing factor
April 2, 2008 SlipstreamUSA 16
Phased migration approach
Proof of Concept (benchmark testing) Completed 10/15/07
Phase 1 – Database tier 4 of 5 servers migrated, anticipated completion 4/14/08
Phase 2 – Application tier Big Bang migration 12/14/08
Phase 3 – Disaster Recovery January 2009
April 2, 2008 SlipstreamUSA 17
UNIX Benchmark Environment
April 2, 2008 SlipstreamUSA 18
Database Benchmark Load Testing Results
Goals Simulate current Production user counts &
transaction loads Verify support for load increases up to 300%
Benchmark Environment Isolated LAN, new DMX3 SAN 20 new Windows blade servers (10 app servers,
10 script ‘players’) Scripts for 8 apps (represent heaviest use,
Web/Telnet/VB apps) 2 batch jobs (screensaver simulation, NullGen
LMR functions)
Conclusions Able to simulate production load, 1.5x and 3x
load 2 HP rx8640 can handle growth projections
Metric Production peak (8/21, 11:20 am)
Benchmark “paced” script load
Benchmark full script load
Database Global Refs / sec.
35,000 30,000 135,000
LMR transactions (5 min. period)
11,806 40,000 40,000
LMR avg Caché app time (in sec.)
0.32 0.15 0.66
April 2, 2008 SlipstreamUSA 19
Design and Configuration Considerations
Database configuration simulation testing 1 to 5 Caché database instances were assessed 1 vs. 5 ECP channels per Caché instance were assessed Number of active cores were accessed (4 active, 2 reserved)
Results and unexpected discoveries Identify 5 Caché database instance as optimal design configuration
Journal synch bottleneck the biggest issue
o High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance)
Maintain same data distribution across 5 DB instances Determine 1 ECP channel per instance optimal
Additional channels did not improve throughput, still have only 1 Journal Deamon
April 2, 2008 SlipstreamUSA 20
Benchmark Discoveries led to Production Improvements
References to Undefined globals using $Data and $Get These commands require network round trip
Use of $increment Each call to $I requires network round trip
Excessive use of Cache locks Forces more than 1 round trip
Use of large strings Strings that require more than 3900–4000 bytes to represent the
string value are big strings and never cached on the ECP client.
Lesson Learned - Each trip to the database server results in overhead caused by a Journal Synch. Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions .
April 2, 2008 SlipstreamUSA 21
75% reduction in long running transaction
April 2, 2008 SlipstreamUSA 22
Phased Migration Approach
April 2, 2008 SlipstreamUSA 23
Monthly Average Caché Web Transaction Time
April 2, 2008 SlipstreamUSA 24
Application Models
Old NewBrowser
client
Web server
Cache
Cache
VB client
.Net server
CacheCache
.Net client
Browser client
Web server
Cache Web Services
Browser client.Net client
Scalability/Connection pooling, robustness/error handling,
Vism
Managed Obj.Vism.ocx
Managed Obj.
Cache Web Services
WebLink
April 2, 2008 SlipstreamUSA 25
The experiences of migrating a large scale, high performance healthcare network
Larry WilliamsCorporate Manager, Partners HealthCare
top related