one day version 10.3 upgrade - how a large biotech plant's deltav systems were successfully...
DESCRIPTION
Presented by Amgen's Auguste Dionne, Dan DiMatteo, and New England Controls' David Maglaya at the 2011 Emerson Exchange in Nashville, Tennessee.TRANSCRIPT
![Page 1: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/1.jpg)
DeltaV Version 10.3.1 Upgrade in 24hrs
DeltaV Version 10.3.1 Upgrade in 24hrs
Version 8.4.2 – Version 10.3.1
![Page 2: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/2.jpg)
PresentersPresenters
� Gus Dionne, Automation ARI
� Dan DiMatteo, IS ARI
� Josh Gangl, OIS Platform Lead
� David Maglaya, Emerson LBP
![Page 3: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/3.jpg)
IntroductionIntroduction
DeltaV Version Upgrade
� Planning the Upgrade
– Objectives, Deliverables, Schedule, Resources
� Executing the Plan
– Developing a Team, Direct and Manage the Execution
Monitoring and Controlling� Monitoring and Controlling
– Verify scope, Integrate Changes
� Closing the Project
– Lessons Learned
These are all requirements of a successful upgrade.
![Page 4: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/4.jpg)
ARI Mfg. Site 75 Acre FootprintARI Mfg. Site 75 Acre Footprint
At a site of this size, planning is key to success
![Page 5: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/5.jpg)
Planning and Defining Our ScopePlanning and Defining Our Scope
� Our goal was to upgrade two (2) DeltaV Systems with
minimum impact to users
� Our constraints were Time (24hrs) , Resources, and
Space
� We needed to upgrade not only the Software Application We needed to upgrade not only the Software Application
but we needed to upgrade most of the hardware that it
would run on
� And we also needed to successfully complete these goals
in the middle of a major project, which had extensive
software impact
![Page 6: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/6.jpg)
Additional ScopeAdditional Scope
� 8 of our 18 Servers would need to be updated from Dell
2850s to R710 (18 servers in the environment– 16 DV
servers plus 2 dedicated domain controllers which were
updated to PE2950 hardware)
� 70 out of 92 controllers would need to be upgraded from
MD’s to MD Plus and MX modelsMD’s to MD Plus and MX models
� All of our 65 OITs would need to be upgraded from 5 and
6 year old models to Optiplex 780s
� Add an additional 19th server to manage Critical Data
Backup Application (CDBA) as part of the upgrade
![Page 7: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/7.jpg)
Tools and ResourcesTools and Resources
� We would need high performance tools to complete the
upgrade in a 24hr window. The tool bag would have the
following +
– Imaging tool for OITs
– A Broadcast Server for deploying images
– Racks for Server buildup and testing– Racks for Server buildup and testing
– Spare OIT hardware for testing
– SI Dongles to turn all OITs to Development stations
– A team of Emerson Support Engineers
– Hard copy checklist for sequential steps
![Page 8: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/8.jpg)
Tools and Resources (additional)Tools and Resources (additional)
– A (Time smart) detailed DeltaV Excel Add-in spreadsheet for
saving and restoring critical controller online parameters
– A controller map depicting model, new Mac address of hardware
and installation location
– A team of hands on Tech’s and Engineers was created– A team of hands on Tech’s and Engineers was created
– A timed schedule was created with each task for each person on
the team
![Page 9: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/9.jpg)
The DeltaV Excel Add In Tool exampleThe DeltaV Excel Add In Tool example
Having a spreadsheet with macros simplifies work and
reduces risk.
![Page 10: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/10.jpg)
The Controller Tracking ID ToolThe Controller Tracking ID Tool
Controller Tracker
![Page 11: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/11.jpg)
The Simple Checklist goes a long wayThe Simple Checklist goes a long way
Organized tasks with priorities are a must for capturing
essentials and ensuring nothing is missed.
![Page 12: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/12.jpg)
The Early ExecutionThe Early Execution
� Offline Migration Testing from 8.4.2 to 10.3.1 at LBP office
� 60+ new OITs would be deployed with the 8.4.2 image
prior to the upgrade day
� 10.3.1 Images would be created and tested in plans for
broadcasting to the new OITs on the upgrade day
� Each new controller to be deployed was labeled and Mac � Each new controller to be deployed was labeled and Mac
address recorded for its intended new location. This saved
the light flashing process at each location for the new
controllers during commissioning.
![Page 13: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/13.jpg)
The Early Execution cont. The Early Execution cont.
� A disk was pulled from the Domain Controller and used to
copy the domain users onto the new DC box, (this was
done for offline code development in the Dev
environment, we used the live DC to migrate the Active
Directory during the outage)
� Critical parameters that needed to be restored would be � Critical parameters that needed to be restored would be
listed in Time Smart XL add in tool
� Server class boxes would be pre-built, racked, stacked
and configured as far as possible prior to execution and
left powered on until upgrade
� The Pro Plus Servers had Database imported at Version
8.4.2 and then the 10.3.1 upgrade wizard run
![Page 14: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/14.jpg)
The Execution prep(the night before) The Execution prep(the night before)
� Critical parameters that needed to be restored would be
uploaded to the XL add in tool with an individual time
record for each time-sensitive value
� Controllers with outputs that needed to be maintained
would have the manual field overrides enabled
� 70 of the 92 controllers would be decommissioned and � 70 of the 92 controllers would be decommissioned and
replaced with the new pre-labeled controllers
![Page 15: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/15.jpg)
The ExecutionThe Execution
� The execution started at 7:00am with the team
concurrently working the plan from a central automation
lab room, server room, and plant floor
� Servers previously racked and stacked are placed online
starting with the Pro Plus boxes
– Legacy DeltaV 8.4.2 PPS servers were “failed”
• Metadata cleanup, FSMO roles seized by dedicated DC• Metadata cleanup, FSMO roles seized by dedicated DC
– New PPS R710 hardware racked & stacked
• DCPROMO / DNS config, FSMO roles transferred to new PPS
![Page 16: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/16.jpg)
The ExecutionThe Execution
� 10.3.1 Images are concurrently broadcasted to multiple
OIT stations seven (7) at a time
� A team of IS engineers are running OIT configuration and
Sys Prep at pre-scheduled OIT locations
![Page 17: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/17.jpg)
The Execution continuedThe Execution continued
� As OITs became available controllers were commissioned
with no field assistance due to Mac address reservation,
then flashed, then downloaded
� Each OIT coming on line was used to RDP to field OITs to
multiply the Flashing capacity of the automation lab (SI)
� Critical online controller parameters were restored using
the DeltaV Time Smart Add-in tool. Online values that were
time sensitive were updated from within Excel, based on
the time the value was stored and the current time of
restoration.
![Page 18: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/18.jpg)
The ClosingThe Closing
� As Systems came back online they were turned over to
Validation then manufacturing
� A Lessons Learned session was held shortly after
![Page 19: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/19.jpg)
Lessons learnedLessons learned
Some Good Points
� Have a well defined plan and schedule
� Trained resources
� Pre building and Server Prep is key to saving time
� RAID 1 (mirroring) with hot spare disk setup to support DC
transfer (easily export Active Directory from dedicated transfer (easily export Active Directory from dedicated
domain controller – Pro Plus and other DeltaV servers use
standard RAID10 config)
� DeltaV Excel Add-in was key in restoring online values
quickly without manual calculations
� Broadcasting of OIT image saved time over upgrade
wizard
![Page 20: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/20.jpg)
Lessons learned cont.Lessons learned cont.
Some Bad Points
� We were overstaffed the morning of outage. Staffing
schedule could be improved, (Pro Plus delay due to Active
Directory DCPROMO issue added to this)
� VX dongles were needed and not known right awayVX dongles were needed and not known right away
� Some user accounts got corrupted due to Schema mis-
match
� Servers pulled from service should have been left in racks
for a week or two since some data configuration recovery
was required
![Page 21: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/21.jpg)
Lessons learned cont.Lessons learned cont.
Some more Bad Points
� Implementing an upgrade with concurrent complex
software changes is extremely difficult
� (Requirements were changed by people outside of core
project team – caused problems with keeping plan locked project team – caused problems with keeping plan locked
down) – point here is to get everyone to commit to
providing requirements up-front and early in the planning
![Page 22: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/22.jpg)
Business Results AchievedBusiness Results Achieved
1. There were no Safety related incidents due to any
upgrade associated actions
2. The DeltaV System went from running version 8.4.2 one
evening to running version 10.3.1 the following evening
3. 18 Servers, 65 OITs, and 70 Controllers were upgraded
(8 R710 replacements and 8 PE2950s were upgraded in
place using Upgrade Wizard, 2 DCs were upgraded to place using Upgrade Wizard, 2 DCs were upgraded to
PE2950 hardware)
4. There was no loss of production outside the planned
outage due to the upgrade
5. There was no loss of utility access due to the upgrade
![Page 23: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/23.jpg)
SummarySummary
� We have done our best to share how we planned,
managed, and executed a successful DeltaV software and
hardware upgrade within a large scale Bio-Tech
manufacturing Facility
� Our benefits are faster, better performing, and more
capable software and hardware to the end user as well as capable software and hardware to the end user as well as
a more robust system for the administrators
![Page 24: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/24.jpg)
QuestionsQuestions
I’d like to take this time to ask if anyone has any questions
that pertain to the project upgrade?
![Page 25: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/25.jpg)
FeedbackFeedback
If any of you would not mind sharing any feedback you might
have we would all like to know if you enjoyed the
presentation or not?
Thank you all so much for attending.
![Page 26: One Day Version 10.3 Upgrade - How a Large Biotech Plant's DeltaV Systems Were Successfully Upgraded in One Day](https://reader033.vdocument.in/reader033/viewer/2022051412/54c913044a79594f398b4625/html5/thumbnails/26.jpg)
Where To Get More InformationWhere To Get More Information
� Other training sessions
� List books, articles, electronic sources
� Consulting services, other sources