crashing the schedule in dcs validation projects - ipt · pdf filemembers and extensive...

Manufacturing

A ‘schedule crash’ represents an effort to reduce theoverall duration of a schedule by either addingresources (human or otherwise) or increasing workhours (overtime, weekend working). Crashing isgenerally done as a trade-off between shorter taskduration and higher task costs. It must bedetermined whether the total cost savings realisedfrom reducing the project duration are enough tojustify the higher costs associated with reducing theindividual tasks. If the cost savings on a delaypenalty are higher than the incremental cost ofreducing the project duration, then the crashing is justified.

In a pharmaceutical environment, enhancing projectperformance during a schedule crash is nostraightforward task. The approach is high-risk, as itusually leads to greater costs and diminished quality –unless a solid infrastructure is put in place to mitigate theimpact of changing the project plan. Project teams cancreate a viable infrastructure by using a combination oftried-and-tested project management practices, as well asLean Six Sigma techniques.

In this article, we examine how to apply projectmanagement practices and Lean Six Sigma successfullyto a schedule crash in a pharmaceutical automationenvironment.

CASE STUDYThe subject of this case study is a Distributed ControlSystem (DCS) validation project that formed part of theautomation of a new biotech pilot plant. In this project,schedule-crashing delivered a complex, compliant DCS tothe plant below cost and without compromising quality.

The project assumptions and baseline were aligned withthe client’s corporate policies regarding validation. Thevalidation effort benefited from leveraging a controlledsoftware library based on S88 standards, resulting in theelimination of a considerable amount of unnecessarytesting. Once tested, the library could be used multipletimes and by multiple sites.

The software was developed using a modular approach,allowing the re-use of the same code multiple times. Thisapproach helped build a testing strategy that minimisedredundancy and reduced risk. The project spannedapproximately 1.3 years and required a team ofapproximately 15 people, with an increase to about 30during the ‘crashing period.’

Project Management Institute methodologies, such asEarned Value reporting, were used. The project involveda high number of deliverables (approximately 7,000documents) with complex workflows (for remote andneighbouring locations). It involved multiple team

Crashing the Schedule in DCS Validation ProjectsSchedule-crashing is a high-risk strategy which can lead to greater costs and diminished quality – unless a solid infrastructure is put in place to mitigate the impact of changing the project plan

By Mark Cupryk and Doina Morusca at Invensys Validation Technologies, and Dean Takahata at Amgen

Mark Cupryk is Vice President and General Manager of Operations for Invensys Validation Technologies. He holds aChemical Engineering degree from McGill University and a Masters in Business Administration from Concordia University.He is also a certified Project Management Professional at the Pennsylvania Project Management Institute with a Master inProject Management. He has worked in automation and validation for over 15 years and is a Lean Six Sigma Black Belt.

Doina Morusca is a Senior Project Manager for Invensys Validation Technologies. She is specialised in both business andmanufacturing systems with Lean Six Sigma. She holds a Masters in Business Administration from Concordia Universityas well as a Masters in Education. She is also a certified Project Management Professional at the Pennsylvania ProjectManagement Institute.

Dean Takahata is a Project Manager for Amgen. He holds an Engineering degree from Miami University and has workedin automation and project management for over 22 years. He has held various positions in project operations andsoftware product development with ABB and Invensys.

76 Innovations in Pharmaceutical Technology

IPT 24 2007 3/1/08 13:51 Page 76

members and extensive documentation review andapproval. The project team exercised rigorous controlover the process flow and the progress of deliverables.Lean Six Sigma techniques for key metrics, such as cycle-time measurement, were implemented to ensure properfocus on success factors. Specifically, the projectmanagement team used the Variance and Earned Valueapproach to verify the budget and schedule. (EarnedValue is a forecasted variable used to predict whether theproject will finish as per initial estimates.)

The initial estimate of work was 11 months for 17people; however, due to the fact that the facility was anew pilot plant, the project team encountered a series ofchanges that expanded the scope of the project andcreated re-work. An updated schedule forecasted that theadditional work would require an additional six monthsat the existing planned pace.

THE CHALLENGEThe company was thus faced with two courses of action:either maintain current project delivery speed (hence

delaying product launch), or crash the schedule in aneffort to meet the set launch date. Given that the launchinvolved multiple products with high value to thebusiness, the decision was made to crash the projectschedule by doubling the team within two to three weeks.

Crashing a schedule comes with a series of problemsrelated to both cost and quality. The dangers ofexponential cost over-runs during a schedule crash arevery real due to the challenges of managing a larger team,and the potential for unforeseen inefficiencies. Qualitymay also be affected by the rapid ramp-up of the projectteam’s size with new members whose training orknowledge-set may not be sufficiently aligned with theproject’s specific dynamics.

As part of the transition planning, additional supervisionrequirements were identified to ensure a smoothtransition to the higher productivity model. The newwork model involved striking the proper balancebetween over-time and bringing in new talent, so as notto affect the team with attrition or burn-out. Finally,ideas to maintain morale were discussed and planned.The level of out-sourcing support was evaluated so as tohelp manage the rapid increase required for the project to be successful.

During the project, the team continuously analysed theprocess by using Lean Six Sigma tools. They kept focusingon critical path activities to ensure that all efforts werekept on schedule (the crashed schedule), while at the sametime preparing for the next critical path activity to ensuresuccess at each step. A process flow diagram was devisedwith key input and output variables, and value- and non-value-added activities were analysed (see Figure 1).

A cause-and-effect (fishbone) diagram was prepared tobreak out the root-cause categories of issues that had animpact on the process. Based on these results, the nextdrill-down exercise was to analyse why the scripts did notmatch the design specifications, repeating thecombination of cause-and-effect and Pareto tools tofurther isolate high-impact root causes. The followingroot causes were identified and analysed using a Paretochart (see Figure 2).

� Not match design – issues occurring when thesoftware script (that is, the documented test case)did not match the design specification. Thisresulted in corrections to either test scriptdocuments or the design specification.

� Test case outline – issues occurring when the testscript could not be executed as documented.


Figure 1: This process map was used to discussimprovement areas for existing activitiesbefore crashing theschedule. Durationand effort werereviewed and assessedfor each activity.Opportunities forbecoming leaner were identified andlater implemented

Input variables Output variables

Step completelyremoved. Did not

affect the outcomeof the testingsignificantly

Reduce the amount of re-work byusing new guidelinesand remove some of

the non-value added

Receive draftscripts

Technical review

Re-work

Quality review

(formatting)

Review script

Receive draft scriptsrevision 0.2

Receive draftscripts revision

0.2 ready for quality

Receive script for approval

Script reviewedtechnically

Script reviewedtechnically revision 0.2

Script reviewedquality revision

1.0

Script approved

Figure 2: The keycategories weremeasured using a Pareto chart todetermine the mostsignificant issues,providing immediatefocus areas forcorrective action bythe project team (note:actual numbers ofissues were factored)

PARETO – type of issues during review

3-no

t matc

h de

sign

10-te

st ca

se ou

tline

s

9-ins

uffic

ient i

nform

ation

4-co

verag

e

2-no

t use

d – t

empla

te

7-no

t use

d – g

uideli

nes

6-fil

e nam

e

1-de

sign

issue

s

1,800

1,600

1,400

1,200

1,000

800

600

400

200

0

1,547

997

215 174 128 113 78 76

IPT 24 2007 3/1/08 13:52 Page 78

� Insufficient information – issues occurring whenthe design specification did not have sufficientinformation to write the test script.

� Coverage – issues occurring when the structure ofthe test script did not cover all potential testingscenarios from the design specification.

� Not used template – issues occurring due toformatting errors such as footer/header, page breakand numbering.

� Not used guideline – issues occurring due to notrespecting guideline information such as phases or interlock strategy.

� File name – issues occurring specifically withrespect to improper file naming of scripts or design specifications.

� Design issues – issues occurring in designspecification such as missing information or not matching functional requirements.

The team determined that one of the main causes wasthat the scripts were written according to an olderversion of the design and had not been updated. Atighter control prior to review was implemented, and theimpact of these issues was significantly reduced.

Issues with the least impact on the process were removedfrom the reviewing activities, further reducing the timeneeded to review the scripts. (Fortunately, negligible-impact issues were few in number, which made theirremoval that much faster.) Some of the key process flowswere also further mapped according to Lean principlesfocusing on cycle times, value-added and non-value-added activities (see Figure 3).

As expected, crashing the schedule had a huge impact onproject management activities. Due to its high level ofrisk, crashing should only be attempted when the team isconvinced that it is fast-tracking the project, and hasplaced all activities in overlapping or parallel states toreduce the duration as much as possible.

A counter-measures matrix was built to ensure that allpotential risks associated with crashing were mitigated.Some of the key standard risks identified were lack ofcontrol over resources, unclear project processes,diminished quality in deliverables and unclear trainingguidelines for the ramped-up team.

DEALING WITH ISSUES IDENTIFIEDThe current Work Breakdown Structure (WBS) wasreviewed to identify the impact of crashing on eachactivity. The detailed project plan was revised for eachtask in light of the accelerated project time-line. Root-cause analysis guided continuous improvement, andLean techniques ensured improved workflow, removingdelays and non-value-added steps (see Figure 4).

Daily reporting replaced weekly reporting for furthercontrol on Earned Value (EV) enabling better monitoringand control of project progress. Key metrics were visuallycommunicated to the entire team, who became focusedon surpassing the daily goals. EV provides a tangiblemeasure of the actual progress of work in relation to theplanned value for the work. Too often, project teamsmeasure their progress against budgeted hours without


SCOPE BREAKDOWN TASKS

# Process cell

1 Water to injection 144 144 144 144 144

2 Water bulk 143 143 143 143 143

3 Glycol 42 42 42 42 42

5 Clean steam 49 49 49 49 49

6 CIP storage 32 32 32 32 32

9 Biowaste 40 40 40 40 40

10 Solvent waste 45 45 45 45 45

16 Harvest centrifuge 78 78 78 78 78

17 CIP 100 100 100 100 100

23 Fermentation tank 15 15 15 15 15

24 Column pack 23 23 23 23 23

26 Purification column 56 56 56 56 56

28 Fermentor 151 151 151 151 151

35 Make-up 76 76 76 76 76

36 Bioreactor 65 65 65 65 65

48 Clarification tank 67 67 67 67 67

53 Buffer 89 89 89 89 89

54 Buffer 45 45 45 45 45

Figure 3: A snapshotof the activitiesrequired to approvetest scripts forexecution, highlightingthe potential for leanimprovements in bothvalue- and non-value-added activities

Value-added activity = 5 hours – work time that adds value to the process

Non-value-added = 4 hours – work time that adds no value to the process

Figure 4: The Work Breakdown Structure (WBS) provides the number of test cases and associated tasks for each process cell – hence, the overall scope for

the project. (Note: actual numbers of units and test cases were factored.)

Tota

l tes

t cas

es

Test

cas

ede

velo

pmen

t

Test

cas

e re

view

Test

cas

eex

ecut

ed

Test

cas

e po

stre

view

and

clo

se

TC development testing

team (established

2 hours/TC)

TC technical review

team (established

<1 hour/TC)

TC technical review –

techwriters (established

15 minutes/TC)

Automation technical

review – auto team

(established 30 minutes/TC)

Issue resolution and

upgrades – testing team


Fix software fix team

(established

<30 minutes/TC)

TC re-execution reading

team (established

1 hour/TC)

A

Cycle time 2-3 days

Total cycle time 6-8 days

Cycle time 4-5 days

TC approval team


TC execution testing team

(established 3 hours/TC)

Failed PRN reviewers board


IPT 24 2007 3/1/08 13:54 Page 80

TECHNOLOGY AT YOUR FINGERTIPS

Trigger Sprayers for Pharmaceutical use

ISO 8317 Certified CRCIrradiation Resistant

••

Contact us today:4 Mallusk Road, Newtownabbey, County Antrim, BT36 4PR, Northern IrelandTel: +44 (0) 28 9084 1917 Fax:+44 (0) 28 9084 4528 Email: [email protected] Web: www.canyoneurope.com

any consideration for the deliverables. As a result, theproject team is not focused on completing the actual tasksin the required time; instead, it gets side-tracked bypreparing other deliverables that were not in the originalscope, or by enhancing the level of quality beyond theplanned scope (scope creep).

The required granularity and frequency are important forEV management. A simple, visual graph of EV reportedmonthly (see Figure 5), combined with detailed dailydeliverable progress in the form of a table, enabled theteam to truly focus on completing critical daily tasks. Theteam understood each task that needed to be completedat the test script level and executed it accordingly. A ‘RedFlag’ alarm system was implemented to rapidly assess thesituation and remove road-blocks to ensure success.

As part of the project crash, a revised detailedcommunication plan was presented to projectstakeholders – the procurement, quality, engineering,automation, and validation teams – delivering real-timeprogress of all critical-path activities.

LESSONS LEARNTApplying the principles of prototyping the project, the project team sat down and analysed the data

after 3-4 process cells delivered. A series of lessonswere drawn and action plans were developed andimplemented:

� Fix all problems prior to execution in all clonedprocess cells, so as not to propagate any issuesunnecessarily. The project benefited from replicationof similar process cells that enabled acceleratedcoding and testing. In order to further streamlinethe process, it was critical that the first prototypewas tested completely and that all generatedcorrective actions were considered prior to thereplication process.

Project budget hours

Crash forecasted hours

Earned value hours

Actual spent hours

Figure 5: The EarnedValue (EV) graph tracksproject progress aftercrashing the schedule.The initial budget wasmaintained – a keyattribute in definingproject success. (Note: actual hourswere factored.)

Testing effort

Effo

rt h

ours

(fa

ctor

ed)

80,000

70,000

60,000

50,000

40,000

30,000

20,000

10,000

-

Jun

04

Aug

04

Oct

04

Dec

04

Feb

05

Apr

05

Jun

05

Aug

05

Oct

05

Dec

05

Feb

06

Apr

06

Jun

06

IPT 24 2007 3/1/08 13:54 Page 81

� Make sure the Steam In Place (SIP) and Clean InPlace (CIP) charts are updated prior to testing. SIPand CIP are processes that impact the majority ofprocess cells. Due to their complexity, SIP and CIPare often subject to multiple changes (daily!).Communication of the modifications is critical tominimise re-work. Prior to executing the scripts,the team performed an informal check to ensurethat the latest versions of the charts were used.

� No additional technical review is needed for allcloned test scripts as the work is redundant. Sincean extensive assessment of the prototype wasperformed, including all associated documentation,the quality of the test scripts was assured and theteam decided that the risk of removing thetechnical review was minimal.

� Remove extensive comments in test summaries notrequired by the quality team. Quality decided thatall comments put in the comments section of thetest scripts needed to be transferred to the testsummary. A tremendous amount of effort wasrequired to transfer, review and track thecomments. The team decided to minimise the useof comments that provided little benefit to theparticular test. Such a reduction enabled savings in time and higher quality deliverables. The testsummary was less likely to contain errors and wasfocused on the value-added activities.

� Post-review of test scripts should be done just intime after the execution of the test cases so as tocorrect any items immediately and not losemomentum. Initially, the test scripts wereaccumulated, and momentum to complete wassomewhat diminished as the Quality team wereonly able to review them 3-4 weeks later. Theproject team decided that, once all test scripts wereexecuted, a review and approval session was heldwith Quality, which then augmented their capacity

to accommodate the accelerated review process. All key stakeholders were summoned to a ‘reviewand approve party.’ All issues were immediatelyidentified and rectified with high priority. Inaddition, any new quality requirements were added to the checklists of the team to ensure that other deliverables were appropriately prepared and executed.

� All documents must be prepared prior to execution to ensure focus on execution rather thandocuments or peripherals. The team realised thatby having all the tools and items ready prior to the start ensured easy tracking of all test scriptsand issues. Having to prepare binders or printdocuments during the testing sometimes causedminor distractions impacting schedule and quality.

CONCLUSIONCrashing should be considered only after fast-tracking(overlapping all tasks as much as possible) has beenoptimised. Crashing a project requires rigorousmovement towards effective control of the projectthrough instructions, statistical monitoring ofperformance (control of work processes) and – whenpossible – mistake-proofing (see Figure 6).

It should be recognised that crashing has become a wayof life for validation projects – and project teams areembracing accelerated timelines as a normal part of their projects. Solid project managementunderstanding and know-how ensure flexibility toadjust the course of the project efficiently andeffectively. The team must also be ready to cope with ahigh degree of stress due to heightened expectations.Hence, the leadership must motivate and communicatemuch more regularly. A positive work environment isparamount for the health of the talent engaged in theoverall project goal.

Due to the pharmaceutical industry’s continuous focuson cost, businesses are constantly trying new ways toderive maximum value from each dollar invested inprojects. Solid project management techniquescombined with Lean Six Sigma provide the requiredframework to align metrics with execution for success –whether crashing or simply executing a project. Thisscientific methodology translates into less effort andgreater control over the project during execution.

The authors can be contacted [email protected],[email protected] and [email protected]


Figure 6: Diminishing the effort for validation activities includes moving towards tools for more effective deployment of information and communication.The project was managed with the vision of moving towards mistake-proofing

and statistical control when possible

Amount of effort required to keep control

Effectiveness of control

Verbalinstructions

Writtenprocedures

Workinstructions SPC

Poka yokemistake proof

IPT 24 2007 3/1/08 13:54 Page 82

crashing the schedule in dcs validation projects - ipt · pdf filemembers and extensive...

Documents