single-use systems change-notification scorecard … · 2019. 12. 15. · biophorum operations...
TRANSCRIPT
1
CONNECT COLLABORATE
ACCELERATE TM
SINGLE-USE SYSTEMS CHANGE-NOTIFICATION
SCORECARD SURVEY
Scorecard survey 2©BioPhorum Operations Group Ltd
Contents
1.0 Executive summary........................................................................................................................................... 6
2.0 Introduction ........................................................................................................................................................ 7
2.1 Purpose ..............................................................................................................................................................................................7
3.0 Questions asked ................................................................................................................................................ 8
3.1 Respondentprofiling ......................................................................................................................................................................8
3.2 QuantityofSUSchangenotifications .......................................................................................................................................8
3.3 QualityofSUSchangenotifications ..........................................................................................................................................8
3.3.1 Categorization and timing ....................................................................................................................................................... 9
3.3.2 Change description effectiveness ........................................................................................................................................ 9
3.3.3 Data package quality ................................................................................................................................................................. 9
3.3.4 Workflow/Communicationeffectiveness .......................................................................................................................10
3.3.5 Overall satisfaction ..................................................................................................................................................................10
3.3.6 Ranking .........................................................................................................................................................................................10
4.0 Development of the survey .......................................................................................................................... 11
4.1 Guidanceoncompletingthesurvey ....................................................................................................................................... 11
5.0 Results ................................................................................................................................................................ 13
5.1 Respondentprofiles .................................................................................................................................................................... 13
5.1.1 Respondent company type ....................................................................................................................................................13
5.1.2 Respondent roles ......................................................................................................................................................................14
5.2 QuantityofSUSchangeshandled ........................................................................................................................................... 15
5.2.1 Mean number of SUS changes handled ............................................................................................................................15
5.2.2 Variability in number of SUS changes handled ..............................................................................................................16
5.3 QualityofSUSchangenotifications ....................................................................................................................................... 17
5.3.1 Outlier ..........................................................................................................................................................................................17
5.3.2 Mean rating for each aspect ................................................................................................................................................18
5.3.3 Mean scores for each aspect surveyed in different subgroups of the supply chain .........................................19
5.3.4 Inclusion of outlier data .........................................................................................................................................................20
5.3.5 Variabilitywithinrespondentgroups ...............................................................................................................................21
5.3.6 Ranking of perceived challenge and perceived impact ...............................................................................................23
Scorecard survey 3©BioPhorum Operations Group Ltd
List of figures
Figure1:16Companiesrespondedtothesurvey,eightoftheseself-identifiedasdrugsponsors,sixassuppliersofSUSassemblies andtwoassuppliersofSUScomponents................................................................................................................................................................................................................................................................. 13
Figure2:Cross-functionalteamswereusedbythreedrugsponsors.‘Other’functionsidentifiedbysuppliersincludedregulatorysupport andmarketing/productmanagementroles. ........................................................................................................................................................................................................................................................... 14
Figure3: Mean number of SUS changes handled by different respondent groups. A clear trend is seen, withanincreaseinthenumberofchangenotificationsissued/receivedasproductsmovefromcomponentmanufacture,throughassemblytouse. ......................................... 15
Figure4:Variabilityinthenumberofchangesrelatedtosingle-usesystemsreviewedbydrugsponsorsorissuedbysuppliersoverthelastyear. .............................................. 16
Figure5:Meanindustryscoreandtargetforeachaspectofchange-notificationpracticessurveyed ......................................................................................................................................... 18
Figure6:Comparisonofmeanscoresbetweenrespondentgroupsforeachaspectsurveyed ....................................................................................................................................................... 19
Figure7:Comparisonofmeanscoresbetweenrespondentgroupsincludingoutlier ......................................................................................................................................................................... 20
Figure8:Low,meanandhighscoresforresponsesreceivedfromdrugsponsors ................................................................................................................................................................................ 21
Figure9:Low,meanandhighscoresforresponsesreceivedfromsuppliersofSUSassemblies ..................................................................................................................................................... 22
Figure10:Low,meanandhighscoresforresponsesreceivedfromsuppliersofSUScomponents .............................................................................................................................................. 23
Figure11:Averageperceptionofchallengeofimprovingeachareaofthechange-notificationprocess ................................................................................................................................... 24
Figure12:Averageperceptionofimpactofimprovingeachareaofthechange-notificationprocess ......................................................................................................................................... 25
6.0 Discussion.......................................................................................................................................................... 26
6.1 Numberofchangeshandled ....................................................................................................................................................26
6.2 Estimationofqualitativeaspectsofsingle-usesystemchangenotifications ............................................................26
6.3 Categorizationandtiming ......................................................................................................................................................... 27
6.4 Changedescriptioneffectiveness ........................................................................................................................................... 27
6.5 Datapackagequality .................................................................................................................................................................. 27
6.6 Workflowandcommunicationeffectiveness ..................................................................................................................... 27
6.7 Rankingofthechallengeandimpactposedbyincreasingscoringineachaspect
ofthechange-notificationprocess ........................................................................................................................................ 28
7.0 Conclusion ......................................................................................................................................................... 29
Appendix ....................................................................................................................................................................... 30
References .................................................................................................................................................................... 31
List of tables
Table1: Industry mean score for each aspect and associated outcome from the scorecard model .............................................................................................................................................. 26
Scorecard survey 4©BioPhorum Operations Group Ltd
Authors
BioPhorum Sam Denby
Scorecard survey 5©BioPhorum Operations Group Ltd
About BioPhorum
TheBioPhorumOperationsGroup’s(BioPhorum’s)missionistocreateenvironmentswheretheglobalbiopharmaceutical industry can collaborate and accelerateitsrateofprogress,forthebenefitofall.Since its inception in 2004, BioPhorum has become theopenandtrustedenvironmentwhereseniorleaders of the biopharmaceutical industry come together to openly share and discuss the emerging trends and challenges facing their industry. Growingfromanend-usergroupin2008,BioPhorumnowcomprises53manufacturers and suppliers deploying their top 2,800 leaders and subject matterexpertstoworkinsevenfocusedPhorums,articulatingtheindustry’stechnologyroadmap,definingthesupplypartnerpracticesofthefuture,anddevelopingandadoptingbestpracticesindrugsubstance,fillfinish,processdevelopment and manufacturing IT. In each of these Phorums, BioPhorum facilitators bring leaders together to create future visions, mobilize teams ofexpertsontheopportunities,createpartnershipsthatenablechangeandprovide the quickest route to implementation, so that the industry shares, learns and builds the best solutions together.
Scorecard survey 6©BioPhorum Operations Group Ltd
1.0
Executive summaryThissurveyisakeytoolforcontinuousimprovementintheongoingmissionoftheBioPhorumdisposablesprogramtoachieveatleast90%right-first-timesupplierchangenotificationsforsingle-usesystems(SUS)usedinthebioprocessingindustry.
Itisanticipatedthatachievingthisgoalwillprovidesignificantcostandtimesavings,improve
qualityandprovideconfidencetomorerapidlyadoptsingle-usesystems,andtheflexibilitythey
provide as a strategic tool for biopharmaceutical manufacturers.
Thisisthefirsttimethissurveyhasbeenrunanditthereforeestablishesabaselineforthe
performanceofchangenotificationsforsingle-usesystemswithinthebioprocessindustry.
Resultsofthesurveyhighlightthatatpresenttheindustryisoperatingat50to75%right-first-
timesupplierchangenotificationsinfourkeyaspectsmeasuredinthesurvey(categorization
andtiming,changedescriptioneffectiveness,datapackagequalityandworkflow/
communicationeffectiveness).Thetargetamongbiomanufacturersandsupplypartnersisto
increase this to greater than 90%.
Thesurveyresultsrevealthatthereappearstobelimitedalignmentofwhatconstitutes
‘good’inrespectofdatapackages,whichpresentsakeychallenge.TheBioPhorumsupplier
changenotificationworkstreamisseekingtoaddressthisbyproducingcontenttostandardize
datapackagerecommendations.Ultimately,theteambelievesthataddressingthiswilldrive
significantimprovementinworkfloweffectivenessforchangenotification.
Movingforward,theintentionistousethistooltodemonstratethebenefitsthatalignmentof
change-notificationpracticesbringsandtheindustry’sprogressinthiscontinuousimprovement
initiative.Thenextstepwillbetorunthissurveywithabroaderaudience,andyouareencouraged
to invite your customers, your suppliers and their suppliers to participate in this survey.
Scorecard survey 7©BioPhorum Operations Group Ltd
2.0
IntroductionSince2015,theBioPhorum/BPSASingle-UseSystemsChangeNotificationteamhavebeenworkingtodevelopastandardized,industry-wideframeworkforthenotificationofchangestosingle-usesystemsusedinthebioprocessingindustry.In2015theteamidentifiedthechallengesposedbycurrentwaysofworkingandtheneedforanindustry-widesolution(WhiteandOtt,2015).Theteamdevelopedaproposedmethodologyandpublishedanindustryproposalforchange-notificationpracticesforsingle-usebiomanufacturingsystems(Carteretal.,2017a;Carteretal.,2017b).Asignificantnumberofmembercompaniesexpressedinterestinimplementingthesepracticesandderivingbenefitfromthem.
Followingpublicationofthepaperin2017,itwasclear
thattherewerechallengesinthechange-notification
processes for single-use biomanufacturing systems. The
industryproposalforbestpracticewaswrittentoimprove
change-notificationprocessesacrosstheindustry.The
teamevaluatedthesituationandacknowledgedthatwhile
therewasacollectiveunderstandingthattheprocess
couldbeimproved– andareasforimprovementhadbeen
identified– therewasnomechanismbywhichtosurvey
andmeasurethestateofchangenotificationsforsingle-
use biomanufacturing systems. Furthermore, the team
identifiedtheneedtomonitorwhetherthewaysofworking
outlinedinthe2017bestpracticeproposalwereeffective
inimprovingthestatusofchangenotificationsforthese
products. To this end, the team agreed to periodically run a
scorecardsurveytoestablishhoweffectivetheproposed
best practices are in bringing about improvements to
change-notificationprocessesintheindustry.
2.1 PurposeThe purpose of the survey is to understand the aggregated
perception of the industry from drug sponsors through the
supply chain. Currently, the survey is designed to capture
input from component manufacturers and single-use
systemintegrators,althoughitcouldbeextendedthrough
the supply chain.
Each time the survey is run, participating companies are
requested to complete the survey by providing one collated
response from their company. The purpose of the survey
hosted by BioPhorum is not for companies to complete the
scorecard survey separately for each supplier and customer
interaction.Companiesarewelcometousethescorecard
tomonitororimprovespecificrelationshipsaspartoftheir
quarterlybusinessrevieworotherprocessbutshould
not report the measurement of a single relationship to
BioPhorum.
A central tenet of the survey design is that it should be
easy to complete and should not provide an unnecessary
administrative burden. During the design of the scorecard
survey,itbecameapparentthatveryfewcompanieshave
readilyavailablehardmetricsaroundhowmuchrework
isnecessaryduringachangenotification,howeffectively
changes are categorized or the completeness of data
packagesprovided.Furthermore,whilemostcompanieswere
abletoidentifyhowmanychangestheyhadhandledintotal
duringagiventimeperiodandwhatpercentageofthose
wereattributabletosingle-usesystems,mostcompanies
werenotabletoprovideharddataabouthowlongithad
takenthemtohandlechanges,howmanyassociateshad
beeninvolvedorwhetherthisvariedby‘severityofchange’.
Thismaybesomethingtoreviewasbestpracticesbecome
embedded through the industry and there is increasing
standardizationonhowchangesshouldbecategorized.
Forthemoment,whatismostimportantisunderstandingthe
directionoftrendsrelatingtochangenotificationforsingle-
usesystemsandwhethertheproposedpracticesarehaving
thedesiredeffect.Tothatextent,theteamagreedthatasking
each company for consolidated feedback from the associates
handlingthechange-notificationprocessshouldprovide
a good measure of the current status of single-use change
notificationswithintheindustry.
Thisisthefirsttimethesurveyhasbeenrunandsothe
purpose is to establish a baseline of the current status of
changenotificationamongthecompaniesparticipatingwith
theworkstream.
Scorecard survey 8©BioPhorum Operations Group Ltd
3.0
Questions asked
3.1RespondentprofilingSomestandardquestionswereaskedtoenablefollowupandclarification(nameofrespondent,respondent’sorganization).Thisinformationwillnotbepublishedandalldatashownisaggregatedandanonymized.
Forthisroundofthesurvey,respondentswereprofiled
byrole(qualityassurance(QA),supplierquality,
regulatory affairs, process development, technical
services/manufacturingsciencesandtechnology(MSAT),
procurement,orother(specify)).Whereacross-functional
teamresponded,itwaspossibletoselectmultiple
roles.Respondentswerefurtherprofiledbyhowtheir
organization’sbusinessselfidentified(drugsponsor,
contractmanufacturingorganisation(CMO),supplierof
SUS assemblies, supplier of SUS components, supplier of
SUSrawmaterials).
3.2QuantityofSUSchangenotificationsRespondentswereaskedhowmanychangenotifications
theyhadreceived(drugsponsors)andissued(supply
partners)forsingle-usesystemsinthepreviousyear.They
werealsoaskedtoindicatewhetherthetrendinthese
numberswasincreasingordecreasing.
3.3QualityofSUSchangenotificationsQuestionswereaskedtoidentifyrespondentscurrentand
desiredexperienceofchangenotification.Ineachsection,
questionswereaskedinpairs–howwouldyourateyour
currentexperienceandwhatratingdoyouneedtoachieve?
Inallcases,companieswereaskedtoprovideascore
between0and10fortheirexperienceinrelationtoeach
questionwith0beingextremelynegativeand10being
extremelypositive.AframeworkisprovidedinPerforming
thissurveyhasbeenaveryusefulexercise,providing
some great insights into current areas of convergence
anddivergenceofcompanies’experienceswithchange-
notificationprocessthroughoutthebiopharmaceutical
supplychain.Importantly,sincethisisthefirsttime
the survey has been run it has established a baseline in
termsofcurrentperformanceofthechange-notification
processes employed by respondent companies. It has
highlightedsignificantdifferencesinthenumberofchange
notificationsdrugsponsorsarehandlingforsingle-use
systems. Establishing the current status of these practices
is important as the intention of the team is not to increase
thenumberofchangenotificationsreceived,butrather
toidentifyandfocusonthechangenotificationslikelyto
have the most impact.
The survey has also been very useful in uncovering a
‘revealedproblem’.End-usersconsiderdatapackagesto
behighlyimpactfultothechange-notificationprocess.A
gooddatapackage,deliveredwiththechangenotification,
providestheleanestworkflowandminimizestheneed
tofollowupwithadditionalquestionstothesupplier.
Since the drug sponsors are the ultimate customers in
thissupplychain,theywilltypicallydrivereworkand
requests for additional information from the supply
chain.Figure8showsthatthereisalackofagreement
betweendrugsponsorcompaniesonthequalityof
thedatapackagestheycurrentlyreceive.Reflecting
onthis,theteamacknowledgedthatatthetimeofthe
survey,littleworkhadbeencarriedouttodefinewhat
constitutes‘good’withrespecttosingle-usesystem
change-notificationdatapackages.Withoutalignment
betweendrugsponsorsitisverydifficultforsupplypartners
tomeetexpectationsasthesemaydifferbetweendrug
sponsor companies. The impact of this on timelines may be
exacerbatedwhenrequestsforadditionaldataneedtobe
passed on from tier 1 suppliers through the supply chain.
Team agreed to a subgroup composed of both drug sponsors
andsuppliersworkingtodevelopguidanceondatapackage
recommendations.
Scorecard survey 9©BioPhorum Operations Group Ltd
Movingforward,thesurveyhasallowedcompanies
participating in the BioPhorum initiative to measure the
currentstatusofkeyaspectsofthechange-notification
process that their best practice is intended to address. This
formsabaselineagainstwhichcontinuousimprovement
efforts can be measured. A key revealed problem has been
identifiedpromptingthegrouptodevelopcontentto
addressthis.Theintentionisnowtorunthissurveywitha
broader audience and continue to measure the impact that
ourbest-practiceproposalishavingonchange-notification
single-usesystemswithinthebioprocessingindustry.As
morecompaniesimplementthispractice,whenthissurvey
isrepeateditwillbepossibletomeasurewhetherthe
goalsofthecollaboration(>90%right-first-timechange
notifications)arebeingmetorwhetherthereisaneedto
correctthecoursebyadaptingwaysofworkingorrevising
published tools and guidance.
3.3.1Categorizationandtiming
The questions in this section are designed to understand
whetherthecomplexity/impactofthechangehasbeen
wellunderstoodandeffectivelyincorporatedintothe
changenotification.Inthecaseofdrugsponsorsthiswill
be directly based on the changes they have received. For
supplypartnerstheaimistounderstandhoweffectively
they believe they are categorizing changes, typically based
onfeedback(orlackoffeedback)fromtheircustomerbase.
Thetwoquestionsaskedinthissectionare:
How would you rate your experience of Categorization and Timing? (0–5)
What rating do you need to achieve for Categorization and Timing ? (0–5)
3.3.2Changedescriptioneffectiveness
The questions in this section are targeted at understanding
whetherthechangehasbeendescribedeffectivelyenough
toallowanend-usertomakearight-first-timeimpact
assessmentforthechange,withoutneedingtorevertto
the supplier for additional information.
Thetwoquestionsaskedinthissectionare:
How would you rate your experience of change description effectiveness (and anticipated end-user-impact assessment)? (0–5)
What rating do you need to achieve for change description effectiveness (and anticipated end-user-impact assessment)? (0–5)
3.3.3Datapackagequality
This section addresses the availability and quality of data
packagessupportingachangenotification.Questionsin
thissectionaredesignedtounderstandwhetherdata
packagesareprovidedwithoutrequestandwhetherit
ispossibletousetheavailabledatapackageswithout
requesting additional information.
Thetwoquestionsaskedinthissectionare:
How would you rate your experience in relation to data package quality? (0–5)
What rating do you need to achieve in relation to data package quality? (0–5)
Scorecard survey 10©BioPhorum Operations Group Ltd
3.3.4Workflow/Communicationeffectiveness
This section addresses the effectiveness of both the
overallworkflowandcommunicationaboutthechange.
Changenotificationsandpre-notificationsshouldbe
providedinlinewiththeguidancesetoutintheindustry
proposalforchange-notificationpracticesassociatedwith
single-usebiomanufacturingsystems(Carteretal.,2017b;
Carteretal.,2017a).Wheretheworkflowiseffective,
anyreworkshouldbeminimal,allowingnotificationand
implementation of changes in a timely manner and in line
withtheguidanceforeachlevelofchange.
Thetwoquestionsaskedinthissectionare:
How would you rate workflow/communication effectiveness? (0–5)
What rating do you need to achieve in relation to workflow/communication effectiveness? (0–5)
3.3.5Overallsatisfaction
Thisquestionaimstounderstandhoworganizations’
perceptionofthechange-notificationprocessforsingle-
usesystemsisevolvingovertime.Respondentswerenot
askedwhattheirdesiredsatisfactionlevelwouldbe.
How would you rate your overall satisfaction with the single-use change-notification process? (0–5)
3.3.6Ranking
Questionsinthissectionaredesignedtoidentifythemost
valuableaspectsofchangenotificationtodrugsponsors.
Impactful areas that are relatively easy to achieve
representquickwins.
Respondentswereaskedtorankthefouraspects
(categorizationandtiming,changedescription
effectiveness,datapackagequality,workflow
effectiveness)from1–4,asfollows:
Please rank categorization and timing, change description effectiveness, data package quality and workflow effectiveness from most (4) to least (1) challenging.
Please rank categorization and timing, change description effectiveness, data package quality and workflow effectiveness from most (4) to least (1) impactful.
Scorecard survey 11©BioPhorum Operations Group Ltd
4.0
Development of the surveyThesurveywasdevelopedbythechange-notificationworkstream.Theintentionistousethissurveyoverthecomingyearstoassesshoweffectivelythebestpracticeprinciplesarebeingembeddedandsustainedthroughouttheindustryandhowpositivetheimpactonchangenotificationsforsingle-usebiomanufacturingsystemshasbeen.
4.1GuidanceoncompletingthesurveyIdeallytheteamwantedtousekeyperformanceindicators
(KPIs)basedonharddatacollectedbyparticipating
companies.Duetothecomplexnatureofthisco-
implementation,itwasnotpossibletoagreehard,data-
basedkeyperformanceindicatorsthatwerereadily
available.Allcompanieswithintheworkstreamhave
rigorous and effective change management processes and
have a sense that they are frequently needing to request
additionaldata,thatreworkisoftenrequired,orthatthereis
opportunity to improve the process. Many companies handle
changes through cross-functional teams and do not typically
recordhowmanyfulltimeequivalent(FTE)hoursare
associatedwithhandlingagivenchange,howcompletethe
datapackagewas,howlongthechangeshouldhavetaken
to implement and the time it actually took to implement.
If companies have any of this data, there is no agreement
onhowitshouldbemeasuredsotryingtocomparehard
datafromdifferentcompanieswouldbechallenging.As
companies adopt these practices, development of hard,
data-drivenKPIsmaybeaconsiderationforthefuture.
WhilehardKPIsmaynotbeavailable,practitionersof
changenotificationsandleadersofthesefunctionswithin
companiesdohaveasenseofhowtheirsuppliersand
customers are performing in respect to each of the aspects
covered in this survey. By asking each organization to
provide an aggregated score, the organization has the
opportunity to survey a representative group of associates
from different functions. By surveying a cross functional
teamdifferingviewsandexperiencesrelatedtoparticular
changenotificationsarebalancedout.Tosupportreaching
aunifiedscoreforthecompanyandtoprovidesomelevelof
moderationbetweencompanyresponses(preventingoverly
optimisticorpessimisticviewsfrombiasingoutputs),the
scorecardframeworkwasdeveloped(seePerformingthis
surveyhasbeenaveryusefulexercise,providingsomegreat
insights into current areas of convergence and divergence
ofcompanies’experienceswithchange-notificationprocess
throughout the biopharmaceutical supply chain. Importantly,
sincethisisthefirsttimethesurveyhasbeenrunithas
established a baseline in terms of current performance of
thechange-notificationprocessesemployedbyrespondent
companies.Ithashighlightedsignificantdifferencesinthe
numberofchangenotificationsdrugsponsorsarehandling
for single-use systems. Establishing the current status of
these practices is important as the intention of the team is
nottoincreasethenumberofchangenotificationsreceived,
butrathertoidentifyandfocusonthechangenotifications
likely to have the most impact.
Scorecard survey 12©BioPhorum Operations Group Ltd
The survey has also been very useful in uncovering a
‘revealedproblem’.End-usersconsiderdatapackagesto
behighlyimpactfultothechange-notificationprocess.A
gooddatapackage,deliveredwiththechangenotification,
providestheleanestworkflowandminimizestheneed
tofollowupwithadditionalquestionstothesupplier.
Since the drug sponsors are the ultimate customers in this
supplychain,theywilltypicallydrivereworkandrequests
for additional information from the supply chain. Figure
8showsthatthereisalackofagreementbetweendrug
sponsor companies on the quality of the data packages they
currentlyreceive.Reflectingonthis,theteamacknowledged
thatatthetimeofthesurvey,littleworkhadbeencarried
outtodefinewhatconstitutes‘good’withrespecttosingle-
usesystemchange-notificationdatapackages.Without
alignmentbetweendrugsponsorsitisverydifficultfor
supplypartnerstomeetexpectationsasthesemaydiffer
betweendrugsponsorcompanies.Theimpactofthison
timelinesmaybeexacerbatedwhenrequestsforadditional
data need to be passed on from tier 1 suppliers through the
supply chain. Team agreed to a subgroup composed of both
drugsponsorsandsuppliersworkingtodevelopguidanceon
data package recommendations.
Movingforward,thesurveyhasallowedcompanies
participating in the BioPhorum initiative to measure the
currentstatusofkeyaspectsofthechange-notification
process that their best practice is intended to address. This
formsabaselineagainstwhichcontinuousimprovement
efforts can be measured. A key revealed problem has been
identifiedpromptingthegrouptodevelopcontentto
addressthis.Theintentionisnowtorunthissurveywitha
broader audience and continue to measure the impact that
ourbest-practiceproposalishavingonchange-notification
single-usesystemswithinthebioprocessingindustry.As
morecompaniesimplementthispractice,whenthissurvey
isrepeateditwillbepossibletomeasurewhetherthe
goalsofthecollaboration(>90%right-first-timechange
notifications)arebeingmetorwhetherthereisaneedto
correctthecoursebyadaptingwaysofworkingorrevising
published tools and guidance.
Appendix1
Itisimportanttohighlightthatthisscorecardframework
is intended to guide thinking and support a discussion
toreachaconsensusaroundwhatscoreanorganization
provides in response to the questions. It is not intended
that companies collect data to justify that they are meeting
specificpercentageslaidout.Thereisalsosomeroomfor
interpretation.Forexample,whereasupplierissues10
notificationsinayearto50customersandreceivesone
report for nine of those changes that the categorization of
thosechangeswasnotcorrectand30reportsforoneof
thosechangesthatthecategorizationwasnotcorrectthis
couldreasonablybeinterpretedasascoreof7–8,withthe
supplier considering that they are categorizing changes
correctly 75–90 % of the time.
Scorecard survey 13©BioPhorum Operations Group Ltd
5.0
Results
5.1RespondentprofilesThereareanumberofwaysinwhichrespondentscanbegrouped.Forthisfirstsurveyitwasconsideredthatnocompanieshaveyetimplementedthepractices.
Figure1:16Companiesrespondedtothesurvey,eightoftheseself-identifiedasdrugsponsors,sixassuppliersofSUSassembliesandtwoas
suppliers of SUS components
Figure 1: 16 Companies responded to the survey, eight of these self-identified as drug sponsors, six as suppliers of SUS assemblies and two as suppliers of SUS components
0
1
2
3
4
5
6
7
8
9
Drug Sponsor CMO Supplier of SUSAssemblies
Supplier of SUSComponents
Supplier of SUSRaw Materials
Nu
mb
er o
f res
po
nd
ents
Respondent company profile
Survey Mar 18
5.1.1Respondentcompanytype
Atotalof16responseswerecollected,includingtwo
responsesfromonedrugsponsorandtworesponsesfrom
one SUS assembly manufacturer. One SUS assembler
responsewasconsideredanoutlierandwillbediscounted
fromanalysisanddiscussedseparately(seesection5.3.1).
NoresponseswerecollectedfromcompaniesidentifyingprimarilyasCMOsorsuppliersofSUSrawmaterials.Therewere
onlytworesponsesfromcompaniesidentifyingassuppliersofSUScomponentsandconsequentlycautionmustbetakenin
drawingconclusions.ThechartinFigure1(orasimilarchart)willbeusedaspartofadashboardtomonitoruptakeofthe
proposedchange-notificationpractices.
Scorecard survey 14©BioPhorum Operations Group Ltd
5.1.2Respondentroles
In general, respondents tended to be from quality roles
(QA,supplierquality).Responsesfromtechnicalservices
andcross-functionalteamswerehigheramongdrug
sponsors than other groups.
Figure2:Cross-functionalteamswereusedbythreedrugsponsors.‘Other’functionsidentifiedbysuppliersincludedregulatorysupportand
marketing/productmanagementroles.
Figure 2: Cross-functional teams were used by three drug sponsors. ‘Other’ functions identified by suppliers included regulatory support and marketing/product management roles.
0
1
2
3
4
5
6
Drug Sponsor Supplier of SUS Assemblies Supplier of SUS Components
Nu
mb
er o
f res
po
nd
ents
Supplier quality Process development
Regulatory affairs
QA
Technical Services Procurement Other Cross functional team
Figure2showsthatprocurementandregulatoryfunctionsare
underrepresented from all respondent companies. Process
developmentfunctionswerenotsurveyedatall,whichmay
reflecttheriseofMSATfunctionsastechnicalcustodiansof
chemistryandmanufacturingcontrol(CMC)activities.
Severalrespondentsineachrespondentgroupidentified
as‘Other’:
Drugsponsor–onerespondentidentifiedas‘globaltechnical
operations’andwasincludedundertechnicalservices.
SupplierofSUSassemblies–onerespondentidentified
as‘regulatorysupport’,tworespondentsidentifiedas
‘marketingoperations’.
SupplierofSUScomponents–onerespondentidentifiedas
‘productmanagement’.
This probably represents a cross-section of individuals
withinacompanywhoareengagedwithchangenotification
for single-use systems and demonstrates that a number of
stakeholderswithdiverseviewsarelikelytobeengaged
withnotifyingandreviewingchanges.
Scorecard survey 15©BioPhorum Operations Group Ltd
5.2QuantityofSUSchangeshandled
5.2.1MeannumberofSUSchangeshandled
Respondentswereaskedtoindicatehowmanychange
notificationsrelatingtosingle-usesystemstheyhandleon
an annual basis. The simplest snapshot of this is the mean
numberofSUSchangeshandled(reviewedornotified)by
each respondent group.
Figure3:MeannumberofSUSchangeshandledbydifferentrespondentgroups.Acleartrendisseen,withanincreaseinthenumberof
changenotificationsissued/receivedasproductsmovefromcomponentmanufacture,throughassemblytouse.
Figure 3: Mean number of SUS changes handled by different respondent groups. A clear trend is seen, with an increase in the number of change notifications issued/received as products move from component manufacture, through assembly to use.
Drug Sponsor Supplier of SUSAssemblies
Supplier of SUSComponents
Mean Changes Mar 18 182 66 4
0
20
40
60
80
100
120
140
160
180
200
Mea
n n
um
ber
of S
US
chan
ges
han
dle
d
Comparison of mean number of changes handled bydifferent respondent groups
Figure3showshowthenumberofchange
notificationsforsingle-usesystemsappearsto
increaseorpropagatethroughthesupplychainwith
significantlygreaternumbersofchangesbeinghandled
by drug sponsors than any other group surveyed.
Toanextentthismaybeexpected,sincethereisan
approximatelythree-folddifferencebetweenthe
number of changes handled by drug sponsors versus
suppliers of SUS assemblies. If a drug sponsor is using
multiplesourcing(twoorthreevendors)thenthisratio
wouldmakesense.ThehighlevelviewshowninFigure
3willbereportedaspartofthedashboardonthe
BioPhorumwebsite.
Furtherbreakdownofthesenumbersisprovidedin
section5.2.2andwillbemonitoredinthefullreport
provided for each iteration of the survey.
Scorecard survey 16©BioPhorum Operations Group Ltd
5.2.2Variabilityinnumberof SUSchangeshandled
Onestrikingaspectofthesurveywasthedifferencein
thenumberofchangesreviewedorissuedbycompanies.
There are several factors that should be taken into
consideration. For drug sponsors, respondents from global
functionsarelikelytoseemorechangesthanthosewith
morelocalresponsibility.Largercompanieswithabroader
product portfolio are likely to have more changes to
review.Similarly,forsuppliers,thosewithlargerormore
complexportfoliosarelikelytoissuemorechangesthan
thosewithsmallerorlesscomplexportfolios.
Figure4:Variabilityinthenumberofchangesrelatedtosingle-usesystemsreviewedbydrugsponsorsorissuedbysuppliersoverthelastyear.
Figure 4: Variability in the number of changes related to single-use systems reviewed by drug sponsors or issued by suppliers over the last year.
Drug Sponsor
Lowest 50
Mean 182
Highest 426
Supplier of SUSAssemblies
5
66
150
Supplier of SUSComponents
3
4
5
0
50
100
150
200
250
300
350
400
450
Nu
mb
er o
f SU
S ch
ange
sm
anag
ed in
last
yea
r
Variation in number of SUS changes reviewed orissued (March 2018)
Scorecard survey 17©BioPhorum Operations Group Ltd
Despite these factors, it appears that there is an eight-fold
differencebetweenthelowestandthehighestnumber
ofchangesreviewedbydrugsponsorsanda30-fold
differenceinthenumberofchanges(lowesttohighest)
initiated by suppliers of single-use assemblies. This is likely,
atleastinpart,tobecausedbydifferencesinwhatisand
isnotnotified/reviewedbetweencompanies.Publication
of a standardized decision tree to support categorization
ofchangesshouldgosomewaytowardsharmonizingthe
approach taken by different companies.
A very small number of component suppliers responding
to this survey and it may be that the average number of
changesissuebycomponentsuppliersissomewhathigher.
Based on the numbers reported in the survey it does seem
that a very small number of changes at the component
manufacturer level propagates to a very large number at
the drug sponsor level. This is something the team may
wishtorevisit,alongwiththenotionofacommonchange
identifiertoallowdrugsponsorstoknow/understand
whenmultiplechangespresentedfromSUSassemblers
havethesamerootatthecomponentorrawmateriallevel.
5.3QualityofSUSchangenotifications
5.3.1Outlier
One response from suppliers of SUS assemblies appeared
tobeanoutlierwithscoressignificantlylowerthanthose
provided by their peers. The objective of the survey
wastounderstandhowsuppliersscoredthemselveson
thenotificationstheyprovidetotheircustomers.Upon
discussion, it emerged that the outlier had provided
scoresbasedonthenotificationstheyreceivefromtheir
suppliersandconsequently,theoutlier’sscoreswere
removedfromanalysiswiththeSUSassemblysuppliers.In
someinstances,theoutlierresultsareshownseparately
as the scoring does reveal potential challenges that may
need to be addressed for these best practices to become
embedded and sustainable throughout the value chain.
Notably,workingwithtier1ortier2suppliersmaynotbe
sufficienttoimprovetheoverallperformance.
Scorecard survey 18©BioPhorum Operations Group Ltd
Figure5:Meanindustryscoreandtargetforeachaspectofchange-notificationpracticessurveyed
Figure 5: Mean industry score and target for each aspect of change-notification practices surveyed
Categorization& timing
Changedescription
effectivenessData packages
Workflow &communication
Overall score
Mar 18 3.4 3.5 3.2 3.2 3.5
Target 4.5 4.5 4.5 4.4
0
1
2
3
4
5
Change notification scorecard surveyscores: all respondents
Mar 18 Target
5.3.2Meanratingforeachaspect
Figure5showsthemeanscoreacrossallrespondentsfor
each aspect surveyed. Generally, there is a reasonable
levelofsatisfactionwiththecurrentstatus,althoughwith
a desire to improve further in all areas.
The team has focused most closely on categorization
and change description effectiveness and this is perhaps
reflectedintheresults.Analysisofvariabilitybetween
andwithinsubgroupsisshowninsubsequentsectionsand
reveals areas of focus for the team.
Scorecard survey 19©BioPhorum Operations Group Ltd
5.3.3Meanscoresforeachaspectsurveyedindifferentsubgroupsofthesupplychain
Although the mean scores for each aspect of change
notificationsurveyedarerelativelyhigh,thereissome
differenceinscoringbetweendrugsponsorsandthe
supply base. Generally, drug sponsors score the current
state less positively than supply partners. In part, this may
be related to drug sponsors being asked to score based
onwhattheyreceiveandsupplypartnersbeingaskedto
scorebasedonwhattheyprovide.
Figure6:Comparisonofmeanscoresbetweenrespondentgroupsforeachaspectsurveyed
Figure 6: Comparison of mean scores between respondent groups for each aspect surveyed
3.4 3.1 2.6 2.9 3.1
3.3 4.0 4.0 2.6 3.9
Drug sponsor
Supplier of SUS
Assemblies Supplierof SUS components
4.0 4.0 3.5 3.0 4.5
0
1
2
3
4
5
Mea
n s
core
Comparison of mean scores between drug sponsorand supply partner subgroups
Categorization& timing
Changedescription
effectivenessData packages
Workflow &communication
effectiveness
Overallsatisfaction
Scorecard survey 20©BioPhorum Operations Group Ltd
What is particularly notable is a difference in scoring
betweendrugsponsorsandsupplypartnersinrelation
todatapackagequality.Thisdiscrepancywilldrive
challengesinrelationtoworkfloweffectiveness.Whatis
clearisthatinvestigatingthediscrepancyinexpectations
fordatapackageswillbeimportantinimprovingthe
generalperformanceofchangenotificationsforsingle-
use systems. Perhaps reassuringly, this is an area the
teamhadpreviouslyidentifiedbutnotyetfocusedon.
Conversely, areas such as categorisation and timing,
whichhavereceivedsignificantattention,scoremuch
more highly and much more consistently. While these
are still benchmarking data, it is an early indication that
improvementshavebeenmadeinareasonwhichtheteam
has already focused.
5.3.4Inclusionofoutlierdata
Inclusionoftheoutliermentionedpreviously,whereone
supply partner responded based on the quality of change
notificationstheyhavereceivedratherthanissued,
provides some valuable insights.
When scoring their supply base, this supplier of SUS
assemblies indicated very poor performance across all
aspects. This is based on only one data point and may not
beatruereflectionofthesituationseenbyotherSUS
assemblysuppliers.However,itprovidesanindicationthat
drugsponsorswillnotbeabletoimprovetheperformance
ofthechange-notificationsystemontheirown,anditis
likely to be necessary to engage as broad a cross-section
of the SUS supply chain as possible. It highlights the
importanceofcontinuingeffortstoengagelower-tier
suppliersinthiswayofworkingandinadoptingthetools
andpracticesdevelopedbytheworkstream.
Figure7:Comparisonofmeanscoresbetweenrespondentgroupsincludingoutlier
Figure 7: Comparison of mean scores between respondent groups including outlier
3.4 3.1 2.6 2.9 3.1
3.3 4.0 4.0 2.6 3.9
4.0 4.0 3.5 3.0 4.5
Drug sponsor
Supplier of SUS assemblies
Supplier of SUS
Components outlier 2.0 2.0 3.0 3.0 3.0
0
1
2
3
4
5
Mea
n s
core
Inclusion of outlier
Categorization& timing
Changedescription
effectivenessData packages
Workflow &communication
effectiveness
Overallsatisfaction
Scorecard survey 21©BioPhorum Operations Group Ltd
5.3.5Variabilitywithinrespondentgroups
Aswellasdifferencesinresponsesbetweengroups,
significantdifferenceswereobservedinexperience
withingroups.Thiswasmostobviousinthedrug
sponsor grouping.
Figure8showsvariabilitywithinthedrugsponsorgroup.
Although current scoring is likely to vary due to differing
experiencesandexpectations,thereisatleastathree-
pointdifferencebetweenthelowestandhighestscorefor
eachaspectwithaseven-pointdifferencefordatapackage
qualityandworkflow/communicationeffectiveness.
Based on comments received data package quality is,
inmanycases,linkedtoworkflowandcommunication
effectivenesswithseveralrespondentscommentingthat
datapackagesmustberequestedmultipletimesand/or
are incomplete.
Itisclearthatworkremainstoharmonizefurtherdrug
sponsors’expectations,particularlyinrelationtodata
packagequality.Thoseareaswhichhavehadmorefocus
(e.g.categorizationandtiming)havemorealignmentin
termsofscoring.Itwillbeimportantfordrugsponsor
companiestomaintainengagementwiththeprocess
ofsustainingandembeddingthesechangenotification
practices.Ifexpectationsarealignedanddefinedbetween
drug sponsor companies, the likelihood of the supply chain
beingabletomeettheexpectationsincreases.
Figure8:Low,meanandhighscoresforresponsesreceivedfromdrugsponsors
Figure 8: Low, mean and high scores for responses received from drug sponsors
Figure 9: Low, mean and high scores for responses received from suppliers of SUS assemblies
0
1
2
3
4
5
Drug sponsors: current status range
Low Mean High
0
1
2
3
4
5
Suppliers of SUS assemblies: current status range
Categorization& timing
Changedescription
effectivenessData packages quality
Workflow &communication
effectiveness
Overall satisfaction
Categorization& timing
Changedescription
effectivenessData packages quality
Workflow &communication
effectiveness
Overall satisfaction
Low Mean High
Scorecard survey 22©BioPhorum Operations Group Ltd
Figure9:Low,meanandhighscoresforresponsesreceivedfromsuppliersofSUSassemblies
Figure 8: Low, mean and high scores for responses received from drug sponsors
Figure 9: Low, mean and high scores for responses received from suppliers of SUS assemblies
0
1
2
3
4
5
Drug sponsors: current status range
Low Mean High
0
1
2
3
4
5
Suppliers of SUS assemblies: current status range
Categorization& timing
Changedescription
effectivenessData packages quality
Workflow &communication
effectiveness
Overall satisfaction
Categorization& timing
Changedescription
effectivenessData packages quality
Workflow &communication
effectiveness
Overall satisfaction
Low Mean High
Figure9showsalignedscoringbetweensuppliersofSUS
assembliesonthequalityofthenotificationstheysupply.
Thisestablishesagoodbenchmark,butitwillbeimportant
to continue monitoring this to ensure there is adequate
feedback from drug sponsors to supply partners so they
cancontinueimprovingandaligningexpectationswith
whatisprovided.
Scorecard survey 23©BioPhorum Operations Group Ltd
Figure10:Low,meanandhighscoresforresponsesreceivedfromsuppliersofSUScomponents
Figure 10: Low, mean and high scores for responses received from suppliers of SUS components
0
1
2
3
4
5
Low Mean High
Suppliers of SUS components: current status range
Categorization& timing
Changedescription
effectivenessData packages quality
Workflow &communication
effectiveness
Overall satisfaction
Figure10showsgoodalignmentbetweensuppliersofSUS
components,however,itmustberememberedthatthisis
basedonsmallsamplesize(n=2).
5.3.6Rankingofperceivedchallengeandperceivedimpact
Inthefinalaspect,surveyrespondentswereaskedto
rankhowchallengingandimpactfultheybelievedeach
aspectofthechange-notificationprocesstobe.When
interpreting data, it is important to remember that
the number respondents is relatively small, especially
forsuppliersofSUScomponents.Figure11shows
howchallengingcompaniesperceivedittobetomake
improvements in each area.
Scorecard survey 24©BioPhorum Operations Group Ltd
Figure11:Averageperceptionofchallengeofimprovingeachareaofthechange-notificationprocess
Figure 11: Average perception of challenge of improving each area of the change-notification process
Figure 12: Average perception of impact of improving each area of the change-notification process
0.0
1.0
2.0
3.0
4.0
Val
ue
Perceived challenge to improve
Drug Sponsor
Difficulty
Supplier of SUS assemblies Supplier of SUS Components
0.0
1.0
2.0
3.0
4.0
Val
ue
Perceived impact of improving
Categorization & timing Change
descriptioneffectiveness
Data packages qualityWorkflow &
communicationeffectiveness
Difficulty
Categorization & timing Change
descriptioneffectiveness
Data packages qualityWorkflow &
communicationeffectiveness
Drug Sponsor Supplier of SUS assemblies Supplier of SUS Components
Scorecard survey 25©BioPhorum Operations Group Ltd
Figure12:Averageperceptionofimpactofimprovingeachareaofthechange-notificationprocess
Figure 11: Average perception of challenge of improving each area of the change-notification process
Figure 12: Average perception of impact of improving each area of the change-notification process
0.0
1.0
2.0
3.0
4.0
Val
ue
Perceived challenge to improve
Drug Sponsor
Difficulty
Supplier of SUS assemblies Supplier of SUS Components
0.0
1.0
2.0
3.0
4.0
Val
ue
Perceived impact of improving
Categorization & timing Change
descriptioneffectiveness
Data packages qualityWorkflow &
communicationeffectiveness
Difficulty
Categorization & timing Change
descriptioneffectiveness
Data packages qualityWorkflow &
communicationeffectiveness
Drug Sponsor Supplier of SUS assemblies Supplier of SUS Components
Figure12showsresponsestothequestion‘Howimpactful
do you believe improvements to each aspect of change-
notificationpracticewouldbe?’Effectively,theaimof
thesequestionswastoidentifyareaswhichmightbeboth
easy and impactful to address.
Scorecard survey 26©BioPhorum Operations Group Ltd
6.0
DiscussionThesurveyprovedtobeveryusefulinrevealingdifferencesinhowchangenotificationsarehandledbetweencompaniesthroughoutthebiopharmaceuticalsupplychain.
6.1NumberofchangeshandledThere are clear differences in the number of changes
handledbydrugsponsors.Thismayreflectthedifferent
sizes of the organizations surveyed, or their current level
ofinvestmentinsingle-usetechnologies,howeverall
respondentsweremajorbiopharmaceuticalcompanies.
Similarly, there are differences in the number of changes
reported by single-use equipment manufacturers. Size of
company probably plays a role in the numbers reported
by supply partners, but there is still a three-fold difference
betweenthelargersuppliers(Figure4).
It is possible that the timing of the survey accounts for these
differences, but it seems unlikely that a three-fold difference
inthenumberofchangesreportedwouldbecausedby
externaleventsalone.Suchdifferencesinthenumbers
of changes handled indicate a need to harmonize on the
appropriate level of change reporting to avoid over- or
under-notificationofchangestosingle-usesystems.
6.2Estimationofqualitativeaspectsofsingle-usesystemchangenotificationsTheteamwasverykeentounderstandwhatwasworkingwell
andwhichareasofferedmostopportunityforimprovement
and asked companies to rate different aspects of the change-
notificationprocess,astheircompanywasexperiencingit
overall across the industry.
Figure5showstheglobalresponses,whichshowthesystemis
performingmoderatelywellacrosstheindustry,withatypical
scoreof3.Referringtothescorecardframeworkinappendix
1(asrespondentswereaskedto),thistranslatestochange
categorizationandtiming,changedescriptionandworkflow
andcommunicationeffectivenessbeingcorrect/effective
around50–75%ofthetime,asshowninTable1.Meanscores
for data package quality indicate that across the industry data
packagesareprovidedwithminorgaps.Thisglobalviewdoes
nothowevertellthewholestorywithend-userstypicallyless
satisfiedinmostareasthansupplypartners.Perhapsmost
notably, the average score for data package quality declines
from3.2to2.6whenonlydrugsponsorsareconsidered.As
thefinalcustomerinthischain,thisiscriticallyimportant,
especiallywhenthescorecard-modeloutcomeisconsidered
whichatscoresof2indicatesthattherearemajorgapsindata
packages that frequently impact implementation.
Table1: Industry mean score for each aspect and associated outcome from the scorecard model
Industrymeanscore Level3:maturing
Categorization and timing 3.4 50–75% of changes are categorized correctly according to the industry standard.
Change description
effectiveness
3.5 AchangedescriptionisprovidedthatwillallowtheEUtocompletetheRFTIA50–75%ofthetime.
Theremaybeaminornumberoffollow-uprequiredtoobtainproperdescriptions.
Data package quality 3.2 Datapackageisprovidedwithminorgapsthatareeasilyaddressedwithoutimpactingon
timelines for implementation.
Workflowand
communication
effectiveness
3.2 Notification(andwhenapplicablepre-notification)areprovidedwithsufficienttimetoproperlyimplement
50–75% of the time.
Implementation timelines are acceptable 50% of the time or more.
SPOCs are used and there is opportunity for timely and effective bidirectional feedback 50–75% of the time.
Scorecard survey 27©BioPhorum Operations Group Ltd
Inthesubsequentanalysiswewilllookateachaspectfor
whichaqualitativemeasurewasrequestedanddiscuss
differencesbetweendrugsponsor,supplierofSUS
assemblies,andsupplierofSUScomponentgroups.Wewill
alsolookatdifferenceswithinthedrugsponsorandsupplier
ofSUSassemblies.Wewillnotdothisforcomponent
manufacturersasthenumberofrespondentswaslow(n=2).
6.3CategorizationandtimingCategorization and timing received an average score above
3 from drug sponsors, suppliers of SUS assemblies, and
SUScomponentmanufacturers(Figure8,Figure9,Figure
10).Scoringwasalsorelativelyconsistentwithingroups.
Thisisoneoftheareasinwhichtheteamhasfocused
most, so it is good to see good consistency of scoring in
relation to categorization. It is surprising that the overall
averagescoreisslightlylowerthanforchangedescription
effectiveness,butthismayreflectthelackofprevious
guidance/systemsfortheclassificationofchanges.
6.4ChangedescriptioneffectivenessMeanscoresforchangedescriptioneffectivenesswere
alsoreasonablyconsistentbetweendifferenttiersofthe
supplychain,withaveragescoresabove3inallcases.
TherewasgoodagreementwithinthesupplierofSUS
assemblies group on the current position, but slightly more
variabilitywithinthedrugsponsorgroup.Thisprobably
reflectsthefactthatthedrugsponsorgroupwereasked
torespondonthebasisofthechangestheyreceivewhile
thesuppliersofSUSassemblieswererespondingonthe
basis of the changes they issue. Effectively, it is almost
certainly easier for the authoring party to understand the
description of a change than it is for the receiving party
as the change issuer is likely to have more background
information about the process and the change itself. There
is an opportunity to improve change communication by
diggingintohoweffectivethechangedescriptionisin
theabsenceofcontextabouttheprocessordetailthat
the supply partner holds about the change. At present,
this aspect is relatively aligned, but it may need to be
considered in future as an improvement target.
6.5DatapackagequalityDatapackagequalitywasidentifiedasthemajorarea
of focus for the team through 2018. As highlighted
previously,thiswas,inpart,duetothemeanscoreofthe
drug sponsor group indicating that typically there are
major gaps in the data packages received leading to delays
in implementation and the need to request additional
information(Figure6).Theteamhadidentifiedhigh-
quality data packages as an area that needed focus and
wouldultimatelydrivebenefitsinworkfloweffectiveness.
Whatwasinterestinghowever,wasthediscrepancyin
scoringwithintheend-usergroup(Figure8).Thisled
theteamtoaskthequestion,havewe(asdrugsponsors)
definedwhat‘good’or‘high-quality’lookslikewithregard
todatapackages?Inturn,thishasledtoasignificanteffort
betweensupplypartnersandend-userstodefinewhat
elements should typically be considered for inclusion
inadatapackageforeachtypeofchangeidentified
previouslybytheworkstreaminthe2018paperA guide
to the classification of changes to single-use biomanufacturing
systemsthecategorizationpaper.Itisexpectedthat
outputsfromthisworkwillbemadeavailableinQ2
2019. It is encouraging that this is an area the team had
previouslyidentifiedassignificant,butforwhichasolution
hadnotyetbeendeveloped.Thesignificantlylowerscores
(particularlyamongdrugsponsors)comparedtochange
categorisationdoindicatethatinareaswheretheteam
hasfocusedsignificantenergy(changecategorisation)it
has been possible to improve the overall performance of
theindustry.Oncepublisheditwillbeexcitingtoseewhat
impactdefining‘good’datapackageshasonthismetric.
6.6WorkflowandcommunicationeffectivenessPerceptionofworkflowandcommunicationeffectiveness
acrosstheindustryisrelativelyconsistentbetween
allgroups,withanaveragescorecloseto3inall
cases.Thisisreflectiveofthefactthatroughly50%
of the time communication is considered effective,
withcorrecttimelinesattributedtobothnotification
and implementation of the change. The ambition of
theindustryistomovetowardsachievingeffective
communication 90% of the time. There is some
discrepancyinscoringwithinthesupplierofSUS
assembliesgroup(Figure9)andevenmoresowithinthe
drugsponsorgroup(Figure10).Thediscrepancywithin
the drug sponsor group appears to be of roughly the
same magnitude as for data packages, and it is possible
thatthisdifferenceinviewofwhatconstitutesahigh-
qualitydatapackagemayalsobeimpactingworkflow
effectiveness. Current thinking is that improving data
Scorecard survey 28©BioPhorum Operations Group Ltd
packagequalitywillmakeasubstantialcontributionto
improvingworkfloweffectiveness.Twootheraspectsof
workfloweffectivenessthatneedtobeconsideredare
the use of a single point of contact and particularly the
feedbackmechanismbywhichsuppliersarenotifiedof
acceptance of the change. It is important to remember that
ineffectiveworkflowscanadverselyimpactsuppliersof
SUSequipmentassignificantlyorpossiblyevenmoreso
thandrugsponsors.AsdemandforSUSequipmentgrows,
capacity and supply chain planning becomes important. If
drugsponsorsdon’tclosetheloopandindicateacceptance
of a change it can pose challenges to SUS suppliers, in
extremecasesmeaningthatbringinganewfacilityonline
maybedelayed.Inturnthismayputpressureonexisting
facilities, potentially creating backorder situations and
ultimately posing drug sponsors challenges in supplying
therapeutics to market. Effectively these are challenges
tobusinesscontinuityplanningposedbybringingnew
products into a regulated environment, This may be one
reasonwhySUSsuppliersonaveragescoreworkflow
effectiveness less highly than drug sponsors.
6.7Rankingofthechallengeandimpactposedbyincreasingscoringineachaspectofthechange-notificationprocessFirstRespondentswerefirstaskedtorankhow
challenging they perceived improving each aspect of
thechange-notificationprocesstobe(Figure11).The
firstobservationisthatevenwithinrespondentgroups
thereislimitedagreementastowhichaspectitwould
bemostchallengingtoimprove.Ifallscoreswere
distributedequallyeachaspectwouldberankedat2.5.
More deviation is seen from this starting point among the
suppliers of SUS components, primarily driven by the small
samplesize(n=2).Thereissomeindicationthatingeneral,
improving data package quality and categorisation and
timing are seen as slightly more challenging aspects.
Change description effectiveness is generally seen as less
challenging to improve by drug sponsors and suppliers
of SUS assemblies. Wheras, although the sample size is
small, there is strong agreement among suppliers of SUS
components that the change description effectiveness
aspect is particularly challenging. It is essential to collect
datafromalargersamplesizebeforedrawinganyfirm
conclusions, but there are some early indications that
different challenges may be faced by different tiers of the
supply chain. It underlines the criticality of continuing to
build understanding and rapport through the supply chain
sothatsolutionsimplementedbytier2supplierslower
in the supply chain, ultimately address the challenges
faced by the drug sponsors. Conversely, drug sponsors
and SUS assemblers have a role to play in understanding
thechallengesfacedbylowertiersofthesupplychain,
enteringintodialoguewiththem,anddevelopingsolutions
thatworkforalltiersofthesupplychain.
Consideringhowimpactfulimprovingeachaspectof
thechange-notificationprocessmightbe(Figure12)
some trends begin to emerge. Generally, SUS assembly
manufacturers consider all aspects to be equally impactful.
This may be indicative of the different challenges faced
bydifferentrespondentsanditwillbeinterestingto
seehowthisevolves,asfurthertoolsandguidanceare
published and implemented. Interestingly, there is an
inversetrendinhowdifferenttiersofthesupplychain
rank particular aspects. Data package quality is ranked as
mostimpactfulbydrugsponsorswiththeimpactranking
decreasing through the supply chain. Conversely change
description effectiveness is seen as least impactful by
the drug sponsors and the impact ranking rises through
the supply chain. This may indicate that drug sponsors
rely heavily on critical analysis of the data provided to
support a change and less on the description of the change
provided by the supplier. This data-driven approach taken
bydrugsponsorsiswhollyconsistentwithoperationina
regulated environment. Change description effectiveness
mayhoweverhaveakeyroletoplayinsettingthecontext
of the change or in providing better understanding of the
technicalaspectsofsuppliermanufacturingprocesseswith
whichdrugsponsorsmaynotbefamiliar.Thismaybewhy
SUS component manufacturers have scored this highly.
Scorecard survey 29©BioPhorum Operations Group Ltd
7.0
ConclusionPerformingthissurveyhasbeenaveryusefulexercise,providingsomegreatinsightsintocurrentareasofconvergenceanddivergenceofcompanies’experienceswithchange-notificationprocessthroughoutthebiopharmaceuticalsupplychain.Importantly,sincethisisthefirsttimethesurveyhasbeenrunithasestablishedabaselineintermsofcurrentperformanceofthechange-notificationprocessesemployedbyrespondentcompanies.Ithashighlightedsignificantdifferencesinthenumberofchangenotificationsdrugsponsorsarehandlingforsingle-usesystems.Establishingthecurrentstatusofthesepracticesisimportantastheintentionoftheteamisnottoincreasethenumberofchangenotificationsreceived,butrathertoidentifyandfocusonthechangenotificationslikelytohavethemostimpact.
Thesurveyhasalsobeenveryusefulinuncoveringa‘revealedproblem’.End-usersconsiderdatapackages
tobehighlyimpactfultothechange-notificationprocess.Agooddatapackage,deliveredwiththechange
notification,providestheleanestworkflowandminimizestheneedtofollowupwithadditionalquestions
tothesupplier.Sincethedrugsponsorsaretheultimatecustomersinthissupplychain,theywilltypically
drivereworkandrequestsforadditionalinformationfromthesupplychain.Figure8showsthatthereis
alackofagreementbetweendrugsponsorcompaniesonthequalityofthedatapackagestheycurrently
receive.Reflectingonthis,theteamacknowledgedthatatthetimeofthesurvey,littleworkhadbeen
carriedouttodefinewhatconstitutes‘good’withrespecttosingle-usesystemchange-notificationdata
packages.Withoutalignmentbetweendrugsponsorsitisverydifficultforsupplypartnerstomeet
expectationsasthesemaydifferbetweendrugsponsorcompanies.Theimpactofthisontimelinesmay
beexacerbatedwhenrequestsforadditionaldataneedtobepassedonfromtier1suppliersthrough
thesupplychain.Teamagreedtoasubgroupcomposedofbothdrugsponsorsandsuppliersworking
todevelop guidance on data package recommendations.
Movingforward,thesurveyhasallowedcompaniesparticipatingintheBioPhoruminitiativetomeasure
thecurrentstatusofkeyaspectsofthechange-notificationprocessthattheirbestpracticeisintended
toaddress.Thisformsabaselineagainstwhichcontinuousimprovementeffortscanbemeasured.A
keyrevealedproblemhasbeenidentifiedpromptingthegrouptodevelopcontenttoaddressthis.The
intentionisnowtorunthissurveywithabroaderaudienceandcontinuetomeasuretheimpactthat
ourbest-practiceproposalishavingonchange-notificationsingle-usesystemswithinthebioprocessing
industry.Asmorecompaniesimplementthispractice,whenthissurveyisrepeateditwillbepossibleto
measurewhetherthegoalsofthecollaboration(>90%right-first-timechangenotifications)arebeing
metorwhetherthereisaneedtocorrectthecoursebyadaptingwaysofworkingorrevisingpublished
tools and guidance.
Scorecard survey ©BioPhorum Operations Group Ltd 30
Appendix
BioPhorumBPSAChangeNotificationScorecardFramework
Attribute 0 1 2 3 4 5
Categorizationand
timing
BioPhorum or
other industry
standards(ASTM,
USP,etc.)toassign
change categories
are not used.
Less than 25%
of changes are
categorized
correctly
according to the
industry standard.
25–50% of
changes are
categorized
correctly
according to the
industry standard.
50–75% of
changes are
categorized
correctly
according to the
industry standard.
75–90% are
categorized
correctly
according to the
industry standard.
At least 90%
are categorized
correctly
according to the
industry standard.
Changedescription
effectiveness(and
anticipatedenduser(EU)
impactassessment(IA))
A description
of the change is
not provided or
the description
provided does
notallowanEU
IA. There may
beasignificant
amountoffollow-
up required to
obtain proper
descriptions.
A change
description is
provided that
willallowtheEU
to complete the
right-first-time
(RFT)IAless
than 25% of the
time. There may
beasignificant
amountoffollow-
up required to
obtain proper
descriptions.
A change
description is
providedthatwill
allowtheEUto
complete the RFT
IA 25–50% of the
time. There may
beasignificant
amountoffollow-
up required to
obtain proper
descriptions.
A change
description is
providedthatwill
allowtheEUto
complete the RFT
IA 50–75% of the
time. There may be
a minor number of
follow-uprequired
to obtain proper
descriptions.
A change
description is
providedthatwill
allowtheEUto
complete the RFT
IA at least 75–90%
of the time.
The change
description
providessufficient
information to
allowtheEUto
complete the RFT
IA at least 90% of
the time.
Datapackagequality Data packages
are not sent
withthechange
notification,
and data is not
available.
Data package is
providedbutwith
major gaps and
reworkisrequired,
impacting
significantlyon
timelines for
implementation.
Data package is
providedwith
significantgaps
andrework
is required,
impacting on
timelines for
implementation.
Data package is
providedwith
minor gaps that are
easily addressed
withoutimpacting
on timelines for
implementation.
Data is available
withnogapsbut
itisnotalways
sentwiththe
notificationand
must be requested.
Dataisalways
available,withno
gapsandsentwith
thenotification
(rightfirsttime).
Workflow/
communication
effectiveness
Pre-notification
is not provided
(whenneeded).
The quality and
timing of the
notificationisnot
acceptable.
Timelines
for change
implementation
are not acceptable
most of the time.
SPOCs are not
used. There is
no opportunity
for bidirectional
feedback.
Notification(and
whenapplicable
pre-notification)
isprovidedwith
sufficienttime
to properly
implement less
than 25% of the
time.
Implementation
timelines are
acceptable less
than 25% of the
time.
SPOCs are not
used. There is
opportunity
for timely
and effective
bidirectional
feedback less than
25% of the time.
Notification(and
whenapplicable
pre-notification)
are provided
withsufficient
time to properly
implement less
than 25–50% of
the time.
Implementation
timelines are
acceptable 25% of
the time or more.
SPOCs are
used and there
is opportunity
for effective
bidirectional
feedback 25– 50%
of the time.
Notification(and
whenapplicable
pre-notification)
are provided
withsufficient
time to properly
implement
50–75% of the
time.
Implementation
timelines are
acceptable 50% of
the time or more.
SPOCs are
used and there
is opportunity
for timely
and effective
bidirectional
feedback 50–75%
of the time.
Notification(and
whenapplicable
pre-notification)
are provided,
withsufficient
time to properly
implement,
75–95% of the
time.
Implementation
timelines are
acceptable 75% of
the time or more.
SPOCs are used
and have timely
and effective
bidirectional
feedback 75–90%
of the time.
Pre-notification
(whenapplicable)
andnotification
are provided,
withsufficient
time to properly
implement, at least
90% of the time.
Implementation
timelines are
acceptable 90% of
the time or more.
SPOCs are used
and have timely
and effective
bidirectional
feedback at least
90% of the time.
Scorecard survey ©BioPhorum Operations Group Ltd 31
ReferencesCarterJ,IsbergE,VogelJ,RestrepoS.2017a.Anindustryproposalforchangenotificationpracticesforsingle-use
biomanufacturing systems. Bioprocess Int. 15.
CarterJ,RestrepoS,VogelJ,IsbergE.2017b.AnIndustryProposalforChangeNotificationPracticesforSingle-Use
BiomanufacturingSystems.BioPhorum.http://bpsalliance.org/wp-content/uploads/2017/06/BPOG-BPSA-Change-
Notification-June-2017-FINAL.pdf.
WhiteT,OttK.2015.Management,notification,anddocumentationofsingle-usesystemschangeorders:Challengesand
opportunities. Bioprocess Int. 13.
Restreppo et al 2018 Aguidetotheclassificationofchangestosingle-usebiomanufacturingsystems BioPhorum.com
Scorecard survey ©BioPhorum Operations Group Ltd 32
Permission to useThe contents of this report may be used unaltered as longasthecopyrightisacknowledgedappropriatelywithcorrectsourcecitation,asfollows“Entity,Author(s),Editor,Title,Location:Year”
DisclaimerThisdocumentrepresentsaconsensusview,andassuch it does not represent fully the internal policies of the contributing companies.
Neither BioPhorum nor any of the contributing companies accept any liability to any person arising from their use of this document.
Theviewsandopinionscontainedhereinarethatofthe individual authors and should not be attributed to theauthors’employers.