otncologist he european perspectives - alphamed...

Critical Appraisal of Translational Research Models for Suitability inPerformance Assessment of Cancer Centers

ABINAYA RAJAN,a,b RICHARD SULLIVAN,c SUZANNE BAKKER,a WIM H. VAN HARTENa,b

aDepartment of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute-Antoni vanLeeuwenhoek Hospital, Amsterdam; bDepartment of Health Technology and Services Research, University ofTwente, The Netherlands; cInstitute of Cancer Policy, Kings Health Partners Integrated Cancer Center, King’s

College London, United Kingdom

Key Words. Translational medical research • Models • Performance assessment • Cancer centers •Performance improvement

Disclosures: Richard Sullivan: Boehringer (C/A); Wim H. van Harten: Organisation of European Cancer Institutes (E). The other authorsindicated no financial relationships.

C/A: Consulting/advisory relationship; RF: Research funding; E: Employment; H: Honoraria received; OI: Ownership interests;IP: Intellectual property rights/inventor/patent holder; SAB: scientific advisory board

ABSTRACT

Background. Translational research is a complex cumula-tive process that takes time. However, the operating envi-ronment for cancer centers engaged in translationalresearch is now financially insecure. Centers are chal-lenged to improve results and reduce time from discoveryto practice innovations. Performance assessment can iden-tify improvement areas that will help reduce translational de-lays. Currently, no standard method exists to identify modelsfor use in performance assessment. This study aimed to crit-ically appraise translational research models for suitability inperformance assessment of cancer centers.

Methods. We conducted a systematic review to identifymodels and developed a set of criteria based on scientomet-rics, complex adaptive systems, research and developmentprocesses, and strategic evaluation. Models were assessedfor linkage between research and care components, new

knowledge, systems integration, performance assessment,and review of other models.

Results. Twelve models were identified; six described phas-es/components for translational research in different blocks(T models) and six described the process of translational re-search (process models). Both models view translational re-search as an accumulation of new knowledge. However,process models more clearly address systems integration, linkresearch and care components, and were developed for eval-uating and improving the performance of translational re-search. T models are more likely to review other models.

Conclusion. Process models seem to be more suitable forperformance assessment of cancer centers than T models.The most suitable process models (the Process MarkerModel and Lean and Six Sigma applications) must be thor-oughly tested in practice. The Oncologist 2012;17:e48–e57

INTRODUCTIONTranslational research is a complex, cumulative, and often unpre-dictable process focused on moving a single or combination ofbasic research findings into clinical practice. The recent identifi-cation of and attention to this field is not just meant to raise aware-ness, but also to improve performance in terms of efficiency andeffectiveness. A particular challenge to translational research in

oncology, as in other clinical fields, are perceptions about unnec-essary delays in or complete blockage of translation.

In the fiscal year 2004–2005, global spending on cancerresearch reached approximately €14 billion ($17.64 billion).The U.S. (dominated by the National Cancer Institute) ac-counted for most of the spending, with per capita spending al-most three times greater than Europe. However, in terms of

Correspondence: Wim H. van Harten, M.D., Ph.D., The Netherlands Cancer Institute-Antoni van Leeuwenhoek Hospital (NKI-AVL),Division of Psychosocial Research and Epidemiology, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands. Telephone: �31 (0) 20512 2860; fax: �31 (20) 66 91 449; E-mail: [email protected] Received May 21, 2012; accepted for publication November 2, 2012.©AlphaMed Press 1083-7159/2012/$20.00/0 http://dx.doi.org/10.1634/theoncologist.2012-0216

TheOncologist® European Perspectives

The Oncologist 2012;17:e48–e57 www.TheOncologist.com

by guest on May 25, 2018

http://theoncologist.alphamedpress.org/

Dow

nloaded from

http://dx.doi.org/10.1634/theoncologist.2012-0216


publications and an increasing trend towards more appliedclinical outputs, relative research productivity was better inEurope [1]. Apart from effectiveness issues, translation of re-search into practice still takes a lot of time. There are claimsthat translation of only 14% of new health-related scientificdiscoveries to clinical practice [2] takes an average of 17 years[3]. Ioannidis et al. examined 101 promising claims of new dis-coveries with clear clinical potential that were reported in ma-jor basic science journals between 1979 and 1983; only fiveresulted in interventions with licensed clinical use by 2003 andonly one had extensive clinical use [4].

Imatinib is an example of successful translation from on-cology. It shows the time it took for an intervention to reachlicensed clinical use based on knowledge that emerged slowlyover many decades. The drug focuses on disrupting one spe-cific protein that seems to fuel the cancer while sparing otherenzymes. The initial knowledge appeared in the 1960s whenscientists first noticed chromosomal abnormalities in the bloodof patients with chronic myeloid leukemia. However, it wasnot until the 1980s that genetic mapping helped determine thatchromosomal abnormalityproduces a cancer-causingkinase enzyme. It took 2years to create and test 400molecules to find one thatwould target this enzymewithout disrupting any of thehundreds of other similar en-zymes in a healthy cell. An-other 8 years of safetytesting and development wasneeded before the drug couldbe tested with patients, fi-nally giving remarkable re-sults. While clinical trialswere being expanded, the U.S. Food and Drug Administrationput the drug on fast track for approval in 2001 [5].

Translational research is cumulative. To improve its per-formance and reduce unnecessary delays, acquiring insightinto the process and performance assessment can add value.This means assessing performance in cancer centers against aset of predetermined criteria of the economy, efficiency, andeffectiveness of that organization in conducting translationalresearch (adapted from the Organisation for Economic Co-operation and Development definition) [6] with the purpose ofsupporting continuous improvement and transparent account-ability at multiple organizational levels. This would help ad-dress delays by identifying areas for improvement, includinginnovation transfer management, organizational administra-tion of research projects, incentive mechanisms to motivate re-searchers, and communication strategies between researchersand other key stakeholder groups. These areas can promotemultidisciplinary collaboration that in turn can speed the rate atwhich basic research discoveries eventually become clinicallyviable health technologies.

For performance assessment, it is essential to know what isbeing translated and how it is being translated. Initially, mod-

els need to be systematically identified and critically appraisedbefore they can be tested in practice. To a large extent, the pro-cess of translational research seems to be generic, and it is notclear if a specific model should be preferred for oncology. Atpresent, it is unknown how many models exist and which of thoseare suitable for performance assessment. Most recent referencesare based on two studies. Trochim et al. reviewed and synthesizedfour models to illuminate important issues to evaluate transla-tional research [7]. Morris et al. looked at quantification of trans-lational time lags; in that context, they offered a tentative modelbased on synthesis of a few models [8]. However, the studies donot specify if they conducted a systematic identification of mod-els, nor did they use systematic criteria to appraise the identifiedmodels. Moreover, in the study by Morris et al., it is not clear howmany models were used to synthesize their model.

The current study aims to identify models of translationalresearch using a systematic literature review and critically ap-praise them by using common criteria that were specificallydeveloped for this purpose. The rationale is to identify themodels that are most suitable for assessing the performance of

cancer centers in transla-tional research.

METHODSA systematic literature re-view was carried out to iden-tify translational researchmodels using a combinationof search terms in four data-bases: PubMed, Embase,Trip Database, and Scopus(supplemental online data).The first search included sci-entific terms and commonexpressions for translational

research and terms associated with models and performanceassessment, whereas the second search included scientific termsand common expressions for translational research and differentphases of translation (Fig. 1). In addition, we tracked the refer-ences and citations for a few papers that were identified throughthe previous search method, which either proposed a modeland/or identified other models. We did not limit our search tomodels specific to oncology nor to the year of their publication.

Criteria Development to Appraise ModelsAt present, there is no standard methodology to assess the suit-ability of translational research models for performance as-sessment purposes. We developed a set of criteria (CR; Table1). The models were awarded a yes or no answer for each ques-tion, in which yes meant that the model seemed suitable forperformance assessment. Model appraisal focused on howtranslational research was presented in terms of its main pur-pose, component(s) that can be evaluated, strategies to evalu-ate the identified components, and testing of the chosenstrategies in practical settings. To validate our focus, we re-ferred to a range of literature from both medical and nonmed-ical disciplines, such as organizational management.

Apart from effectiveness issues, translation of researchinto practice still takes a lot of time. There are claims thattranslation of only 14% of new health-related scientificdiscoveries to clinical practice takes an average of 17years. Ioannidis et al. examined 101 promising claims ofnew discoveries with clear clinical potential that werereported in major basic science journals between 1979and 1983; only five resulted in interventions with licensedclinical use by 2003 and only one had extensive clinicaluse.

e49Rajan, Sullivan, Bakker et al.

www.TheOncologist.com



Dow

nloaded from


With reference to the scientometric analysis conducted byJones et al., we deduced that translational research emerged tolink the research and care components (CR1) [29]. Cancer re-search is a complex adaptive system in which the componentsmust be regularly assessed to improve their performance (CR2and CR3) [30]. Fifth-generation research and developmentsuggests that performance assessment strategies should inte-grate organizational systems to link the process of translationthat occurs through cross-boundary learning and knowledgeflow (CR4) [31]. Using the theory of the evaluation of strategicoptions by Johnson and Scholes, we framed criteria for evalu-ating the strategies of the models for suitability and feasibility(CR5 and CR6) [32]. A seventh criterion based on acceptabil-ity (CR7) was meant to check if models have been tested orapplied in practice. This last criterion is not been presented inTable 1 because we were able to assess only one model.

RESULTS

Identified Translational Research ModelsA total of 2,397 studies were identified after removing the dupli-cates (Fig. 2). Title screening showed that the majority of studieswere related to specific biomedical discoveries focusing on basicand translational issues. Many studies referred to animal models,not conceptual models. Only 385 papers contained a descriptionof translational research. Abstract screening led to 182 papers thatcontained bench-to-bedside issues; 89 studies used descriptivestatements to define translational research. Only 12 studies thatcontained and described a model were included in the resultingappraisal. Of these, 6 studies described the main phases/compo-nents for translational research within different translationalblocks (T models) [2, 9, 12–14, 19]. The remaining 6 papersmapped the steps/processes for translational research (processmodels) [7, 22–26]. Both type of models start at basic discovery;the following phases extend to clinical trials or even beyond towidespread diffusion or population impact (Fig. 3).

Overview of T ModelsThe terminologies and position of the types of translations areinconsistent in all T models. Overall, the T blocks identify thespecific translational areas that are also barriers for translation,but steps to overcome these barriers and improve performanceare not clearly addressed.

Type 1 TranslationIn the six models, descriptions of type 1 translation (T1) havesimilar starting points but are phrased differently. T1 encom-passed “basic research to patient based research” [9], “basicscience research to human clinical research” [2], “basic sci-ence research (phase 0) to early human trials (phase 1) andearly clinical trials (phase 2)” [14], “basic biomedical scienceto clinical efficacy knowledge” [13], “basic biomedical to clin-ical science knowledge” [12], and “gene discovery to healthapplications” [19]. Because of these variations, it is hard to es-tablish where T1 ends.

Type 2 TranslationThe description of type 2 translation (T2) is also inconsistentover all models. T2 encompassed “patient oriented to popula-tion oriented research” [9], “human clinical research to prac-tice based research” [2], “early clinical trials (phase 2) to lateclinical trials (phase 3)” [14], “clinical efficacy knowledge toclinical effectiveness knowledge” [13], “clinical scienceknowledge to improved health” [12], and “health applicationsto evidence-based guidelines” [19].

Type 3 TranslationThe location and extent of type 3 translation (T3) also varies inall models. T3 encompassed “population-based research to ba-sic research” [9], “practice-based research to clinical practice”[2], “late clinical trials (phase 3) to implementation phase(phase 4)” [14], and “clinical effectiveness knowledge to im-proved population health” [13]. In Sung et al.’s model, therewas no T3 [12]. In Khoury et al.’s model, T3 was the transla-tion of guidelines to health practice [19].

Type 4 TranslationOnly Khoury et al.’s model contained type 4 translation (T4),which was the translation of practice to population health impact[19].

Overview of Process ModelsThree process models used T terminologies. The early transla-tional pathways by Ernest et al. [23] used the T1-T2 model, butthe pathways were mapped only for T1. They were developed toaid the transformation of scientific discoveries into new clinical

Figure 1. Search terms used to identify models of translational research.

e50 Translational Research Models for Performance Assessment



Dow

nloaded from


Table 1. Critical appraisal of translational research models

Model

Criterion

1. Does the modelpresent translationalresearch as a continuumwith bidirectional flowbetween research andpractice?

2. Was the purpose ofthe model performanceassessment oftranslational research?

3. Is translationalresearch aboutgeneration of newknowledge?

4. Does the modeladdress systemsintegration?

5. Does the modelexplain any strategiesfor performanceassessment oftranslational research?

6. Has the modelreviewed othertranslational researchmodels?

T models

Rubio et al.(T1-T3) [9]

Yes: Bidirectional arrowsbetween T1, T2, and T3

No: Defines translationalresearch as a basis fordeveloping appropriatetraining programs

Yes: Recognizes theintegration of basic,patient-oriented, andpopulation-based researchto move multidisciplinaryknowledge fromdiscovery to theimplementation phase

No: Focused on trainingprograms intranslational research,although it suggestscollaboration amongscientists from multipledisciplines

No: Does not explain howthe translational researchcontinuum can be assessed,only provides a logicmodel for performanceassessment training andeducation programs fortranslational research

Yes: NIH roadmap [10];IOM roundtable [11];Westfall et al. [2]; Sung etal. [12]; Dougherty andConway [13]

Sung et al.(T1-T2)[12]

No: Unidirectional arrowsbetween T1 and T2

No: Describes the majorphases of translationalresearch

Yes: T2 is called newknowledge into clinicalpractice and health decisionmaking

Yes: Translation is seenfrom a systemsperspective thataddresses incompatibledatabases, fragmentedinfrastructure, andpractice limitations forknowledge to flow better

No: Does not give anystrategies for performanceassessment but doesrecognize the need forstrong informationsystems as they affectresearch and clinicaldecisions

Yes: IOM roundtable [11]

Thornicroftet al. (T1-T3) [14]

No: Unidirectional arrowsbetween 5 phases and 3blocks


Yes: Need to look at thefactors that promote ordelay knowledge flowacross the threecommunication blocksthat they identified (T1-T3)

No: Primarily focusedon points wherecommunication blockscan occur, but does notfocus on how toovercome these withbetter systemsintegration

No: Examines factors thatpromote or delayknowledge flow but doesnot give any strategies toassess them

Yes: MRC framework[15]; Craig et al. [16];Sung et al. [12]; Crowleyet al. [17]; NIH roadmap[10]; Presidents’ CancerPanel [18]; Westfall et al.[2]

DoughertyandConway(T1-T3)[13]



Yes: Clinical efficacyknowledge between T1and T2 and clinicaleffectiveness knowledgebetween T2 and T3

Yes: Uses cohesivehealth informationtechnology andtransdisciplinary researchteams; important to carryactivities in eachtranslational step thatenable translationalmovement to the nextstep

No: Identifies keyfacilitators of translation:shared leadership,transdisciplinary teams,tools that help improvequality and value, andbetter financial resources;does not give anystrategies for performanceassessment

Yes: NIH roadmap [10]

Khoury etal. (T1-T4)[19]

Yes: Connects the fourphases T1-T4, althoughno bidirectional arrowsare shown


Yes: It looks at the typesof knowledge that areimportant for each phase

No: Refers to multipledisciplines beinginvolved but notsystems integrationdirectly

No: Presents a frameworkwith questions related toperformance assessmentof genomics; unclear ifthese strategies that canbe used to assess theperformance along theentire continuum oftranslational research

Yes: NIH roadmap [10];Human GenomeEpidemiology Network[20]; ACCE framework[21]

Westfall etal. (T1-T3)[2]



Yes: It refers to practicebased research as alaboratory to generatenew knowledge

Yes: Rethinks theinterface between basicscience and clinicalpractice; practice-basedresearch is the commonpathway on whichdifferent stakeholdersand interests can beengaged to improvepatient care andoutcomes

No: Advocates forpractice-based research asa crucial scientific step inthe continuum, but doesnot give any strategies toassess translationalresearch performance

Yes: NIH roadmap [10]

Processmodels

Trochim etal. (ProcessMarkerModel) [7]

Yes: Views translationalresearch as bidirectional;shows that translationalresearch can be evaluatedat any level by assessinglength of any segment orsubsegment of processesalong the continuum

Yes: Assessestranslational efforts thatseek to reduce the time ittakes to move researchinto practice and healthimpacts

Yes: Provides a commonframework that can linkmany studies and types ofknowledge together togive a shared basis forassessing and reducingtranslational time

Yes: Shows thatprocesses can be trackedacross three systems:basic research, clinicaltrials, and practiceresearch

Yes: Identifies processand subprocess markersthat can trackperformance at differentpoints in translationalresearch continuum

Yes: Sung et al. [12];Westfall et al. [2];Dougherty and Conway[13]; Khoury et al. [19]

Drolet andLorenzi(BiomedicalResearchTranslationcontinuum)[22]

Yes: Describes the zoneof translation with threetranslational chasms;findings at any stage inthe continuum feed backto previous researchstages for moreexamination and action

No: Reviews,synthesizes, and clarifiescurrent models andterminology andproposes a new modelcalled the biomedicaltranslational continuum;does not proposestrategies forperformance assessment

Yes: Translationalresearch occurs along theentire continuum asknowledge progresses topublic health gains

No: Attempts to mapthe zone of translation,particularly translationalchasms where activitiesremain vague; does notaddress systemsintegration

No: Presents translationalresearch in way thatmakes sense to physiciansbut does not look atperformance assessment

Yes: NIH roadmap [10],IOM roundtable [11],Sung et al. [12], Westfallet al. [2], Dougherty andConway [13]

Ernest et al.(early-stagedevelopmentalpathways)[23]

Yes: Views translationalresearch as bench tobedside and vice versa,but the pathwaysthemselves are confinedto the early translationalresearch phase

No: Pathways areengineering flowchartsthat schematize theprocess of earlytranslational research;however, the directpurpose was not toevaluate the performanceof translational research

Yes: Development andapplication of pathway-based tools to enhance theproductivity oftranslational research; canbe adapted based on thelevel of knowledge

No: Addresses systemsintegration only for theearly translationalphase; recognizes thatpathways are idealizedrepresentations that donot capture real-worldcomplexity

No: Pathways identifyopportunities forcollaboration acrossresearch disciplines, butwere not directlydeveloped forperformance assessment

No: Recognizes T1-T2 bythe Presidents’ CancerPanel [18] but did notreview any models.

(continued)





Dow

nloaded from


modalities for oncology—specifically risk assessment modalities(biospecimen-based risk assessment devices and image-basedrisk assessment) and interventive modalities (agents, immune re-sponse modifiers, interventive devices, lifestyle alterations).

The biomedical research continuum by Drolet and Lorenzi

[22] consisted of a zone of translation with three translationalchasms (T1-T3): T1 was laboratory to clinical research be-tween basic science discovery to proposed human application;T2 was safety and efficacy research between proposed humanapplication and proven clinical application; T3 was implemen-

Table 1. (Continued)

Model

Criterion

1. Does the modelpresent translationalresearch as a continuumwith bidirectional flowbetween research andpractice?

2. Was the purpose ofthe model performanceassessment oftranslational research?

3. Is translationalresearch aboutgeneration of newknowledge?

4. Does the modeladdress systemsintegration?

5. Does the modelexplain any strategiesfor performanceassessment oftranslational research?

6. Has the modelreviewed othertranslational researchmodels?

Schweikhartand Dembe(Lean andSix Sigmaapplicationsfor clinicalandtranslationalresearch)[24]

Yes: Continuum oftranslational research isthe context in which thetechniques are beingapplied across thecontinuum

Yes: Focuses onimproving the processesinvolved in clinical andtranslational researchthrough performanceassessment using theprinciples, practices, andmethods from Lean andSix Sigma strategies

Yes: Knowledge transferneeds stakeholders towork outsideorganizational boundariesin transdisciplinary teams;this shapes the type ofknowledge produced

Yes: Business strategiesare applied to all phasesof translational researchto show how to makethe process moreefficient and costeffective, thusimproving the researchquality and translation

Yes: Details themanagement strategiesassociated with Lean andSix Sigma, showing howapplication of the twoapproaches is relevant toall translational researchphases

Yes: Westfall et al. [2],Dougherty and Conway[13]; Khoury et al. [19];IOM roundtable [11]

Lane andFlagg(Need toKnowledgeModel)[25]

Yes: Represents thecomplete continuum ofactivities from problemstatement to solutiondelivery, which needcollective actions bystakeholders and thosemay be recursive oriterative; in short, looks atthe flow of knowledge

Yes: Gives an operationalframework in which anapplication needsknowledgetransformations to reachthe marketplace as adevice or service; actioncycle for each phaseshows the performanceassessment focus

Yes: Based on discovery,intervention, andinnovation, there is a needto ensure that productknowledge is effectivelycommunicated to therelevant stakeholdergroups

Yes: Integrates threephases: discoverycreation, interventioncreation, and innovationcreation; examines howto accommodate thecommercializationaspect of newknowledge

Yes: Uses a seven-stagemodel: discovery creationin stages 1–3 (discoveryoutputs), intervention instages 4–6 (inventionoutput), innovation instage 7 (innovationoutput)

No: It adapts theKnowledge to Actionmodel [27] but did notreview any models.

Ogilvie etal.(TranslationalFrameworkfor PublicHealthResearch)[26]

Yes: Reviews the criticalpathway for translation ofhealth research in theU.K., which isbidirectional; refers to theCooksey report [28],which describes atranslation pathway forhealth research into healthcare development

No: Poses a researchagenda to advancetranslational research;does not provide clearstrategies to assesstranslational researchperformance

Yes: Acknowledges thatknowledge flows alongthe elements of thepathway and that manytypes of researchcontribute to shapingpolicy practice and newresearch

Yes: Translation shouldmove frominstitutionalizingeffective interventionsto improving populationhealth by influencingthe individual andcollective determinantsof health

No: Only outlines atranslational frameworkfor public health research;does not give any specificstrategies for performanceassessment

No: Examinestranslational pathwayspresented in Cookseyreport [28], but it does notreview other models.

Abbreviations: ACCE, Analytical validity, clinical validity, clinical utility, and ethical, legal, and social implications ofgenetic testing; IOM, Institute of Medicine; NIH, National Institutes of Health; T1, type 1 translation; T2, type 2 translation;T3, type 3 translation; T4, type 4 translation.

Figure 2. A systematic review to identify models of translational research.




Dow

nloaded from


tation and adoption research between proven clinical applica-tions and clinical practice. A pathway, inquiry, and action foreach chasm were given.

The Lean and Six Sigma applications to clinical and trans-lational research by Schweikhart and Dembe [24] used theT1-T4 phases by Khoury et al. [19] to improve the efficiency oftranslational research. Each phase consisted of business man-agement strategies for process assessment. The ProcessMarker Model by Trochim et al. identified key steps of trans-lational research, which were not represented by Ts but de-scribed as three integrated systems: basic research system,clinical trials system, and practice-based system. The modelaims to evaluate the process of translational research in orderto reduce the time lag [7].

The Need to Knowledge model by Lane and Flagg [25]identified unmet needs that lead to the generation of knowl-edge through the outputs of three activities: research discov-

ery, prototype intervention, and product innovation. Itrecognized that knowledge implementation and beneficial so-cietal impacts involve effective communication of each suc-cessive knowledge state to the relevant stakeholders. Finally,Ogilvie et al.’s model is a framework to advance translationalresearch that identifies a pivotal role for evidence synthesisthat translates knowledge of nonlinear and intersectoral inter-faces to the public realm [26].

Oncology-Specific ModelsIt was difficult to confirm which of the appraised models arecurrently being used to inform translational research in cancercenters in Europe and/or the U.S. However, only one modelwas specifically developed for oncology: the early-stage trans-lational pathways by Ernest et al. [23]. They used the T1-T2model proposed by the President’s Cancer Panel [18]. This wasone of the first models in translational research to emerge and

Figure 3. An overview of translational models and process models of translational research.





Dow

nloaded from


is also known as bench-to-bedside-to-practice. The pathwayswere developed in T1 phase to facilitate the process of basicdiscoveries in cancer to be developed into clinical modalities,but they have not been adopted in practice.

Evidence From Appraisal of T Models andProcess ModelsThe process models were more favorable when appraised againstour criteria than T models (Table 1), suggesting that they may bebetter suited for performance assessment in cancer centers. Thereis only one similarity between the two types of models: they viewtranslational research as accumulation of new knowledge. Thedifferences are that process models more clearly address systemsintegration, link research and care components, and were devel-oped for evaluating and improving the performance of transla-tional research. In contrast, T models tend to review other models;their purpose is to present the phases of translational research butnot to assess and improve its performance.

Three process models (Lean and Six Sigma applications,the Process Maker Model, and the Need to Knowledge Model)seem to have been developed to evaluate translational re-search. In particular, the first two models scored highest in theappraisal (Figs. 4 and 5). They track the time between varioussteps of the different translational phases in order to improvetranslational process efficiency. Lean and Six Sigma is the

only model that clearly gave evidence that it had been tested inpractice in a process improvement project focused on redesignof the scheduling system at the clinical trials unit of Ohio StateUniversity [24].

Possible Implementation of Lean and Six SigmaTechniques in Performance Assessment ofTranslational ResearchA research process improvement project involving redesign of thescheduling system in the clinical trials unit of the Ohio State Uni-versity (Fig. 5) used a five-stage intervention. The aim was to im-prove the efficiency of the patient scheduling process by replacingpaper-based calendar system with a more coherent data-drivencomputerized scheduling system. It is a practical example of theapplicability of Lean and Six Sigma techniques in assessing andimproving the performance of translational research.

In stage 1, an environmental scan was undertaken by a re-search team to determine stakeholder needs, as well as to suf-ficiently identify and understand various steps that areinvolved in the patient accrual and scheduling process, includ-ing protocol requirements, the total number of trials being con-ducted, software requirements, inpatient bed capacity, numberof available nurses and other staff per shift, examination andtreatment room availability, number of expected visits and spe-cific visit number in the sequence of protocol. The improve-

Figure 4. Examples of process marker models at three levels of scale. Reprinted from [7] with permission from Wiley.Abbreviations: FDA, U.S. Food and Drug Administration; IRB, institutional review board.




Dow

nloaded from


ment strategy was to develop acuity measures to gaugeresource intensity in each step.

Next, in stage 2, the team identified and mapped each processstep and relationship between those steps using value streammaps or process flow maps. As an improvement strategy, they de-veloped different scheduling algorithms based on acuity mea-sures and other factors. In stage 3, the team identified obstacles forand inefficiencies between patient scheduling and planning of theavailable resources. The improvement strategy led to the devel-opment of standardized scheduling instructions for physiciansand patients to improve resource utilization.

In stage 4, the team performed repeated field testing of var-ious scheduling algorithms. As an improvement strategy, anacuity table with estimates for each activity was calculated. Forexample, the activity of “simple specimen collection” wasgiven an acuity score of 5. A scheduling algorithm matched thescores with key internal and external factors (e.g., availabilityof a specific number of research staff per shift, room availabil-ity, protocol-related requirements) to optimize patient and staffscheduling on a given day.

Finally, in stage 5, an assessment of organizational struc-ture and culture was done in the research unit to evaluate read-iness for change. The improvement strategy led to cross-disciplinary training of research staff to make them understandand use the new patient scheduling system. The concerns andsuggestions by staff regarding the practical use of the systemwere addressed during the training. The above stages led to theadoption of the system in daily practice [24].

Drawing on this example in more generic terms, a five-stageintervention for applying performance assessment models intranslational research in cancer centers should address the follow-ing:

1. Environmental scanning to understand key activities intranslational research

2. Elaborating different algorithms in which the identified keyactivities will be efficiently performed

3. Evaluation of these algorithms by performing continu-ous improvement cycles to check which algorithm ismost suitable

4. Using estimates (metrics such as frequency/duration) to mapthe key activities identified and correlating that to key internaland external factors that may affect those estimates

5. Training of research staff on the new system and ensuringthat its implementation within the cancer center is accept-able to key stakeholders

Based on these stages, qualitative and quantitative indica-tors can be derived.

DISCUSSIONThis study aimed to identify models of translational research andappraise their suitability for performance assessment of cancercenters. We managed to identify 12 models of translational re-search: six T models and six process models.

T models contribute to our understanding of translational re-search by mapping its key components. However, these compo-nents vary from model to model, confirming the statement ofAustralia’s chief scientist, Professor Ian Chubb: “If you were toask ten people what translational research means, you’re likely toget ten different answers” [33]. It is not clear whether the varia-tions in T models reflect actual variations in practice or are relatedto specific objectives or circumstances of various stakeholders.These variations may also reflect models being developed for spe-cific research and/or clinical domains. In contrast, process modelsidentify methods to facilitate, track, and assess knowledge flowsand interfaces along the continuum, including multiple startingpoints for innovation, pathway mapping, process markers, usingstrategies and tools from business management, and inclusive ev-idence synthesis.

Figure 5. Example of a process improvement project at a clinical trial unit using Lean techniques. Reprinted from [24] with permissionfrom Wolters Kluwer Health.





Dow

nloaded from


Based on our appraisal, two process models seem to bemost suitable for performance assessment of cancer centers:the Process Marker Model and Lean and Six Sigma appli-cations. Process markers can help cancer centers assess theperformance of translational research by tracking the timetaken between markers, such as prepiloting of studies, sub-mission of research proposals, funding of studies, the startand end of data collection for studies, and inclusion of thestudy in research synthesis (e.g., publications or main-streaming of research activities) that leads to subsequentstages of translational research. Process markers can in-clude both process steps as well as reflect the transfer pro-cess per step (known as subprocess markers). Processmarkers can be defined for phases in clinical trials, proposalsubmission, Institutional Review Board approval, fundingof proposal, accrual of first subject, closed to accrual, andpresentation and publishing of results etc [7]. Process mark-ers might help to identify and possibly reduce the time be-tween different phases of clinical trials in cancer.

Lean and Six Sigma applications are complementary tothe Process Marker Model and might help cancer centersdefine markers more clearly.For example, in basic re-search, process makerscould include turnaroundtime of toxicology results,transfer of samples in labo-ratory, and response to regu-latory requests. In clinicaltrials, cancer centers couldtrack the unnecessary timeand/or added value perprocess step for biostatis-tical consultations, minimiz-ing protocol amendments,checking if placebos areneeded, patient recruitment campaigns, patient monitoringprocess, and elimination of early-phase design errors [24].

However, the models still have some limitations. Lean andSix Sigma applications are derived from a nonmedical field.Although their pilot results are positive, they need to be testedin other phases of translational research to fully validate theiruse along the continuum. The Process Marker Model lacks pre-cisely stated operational definitions of markers and an inferen-tial statistical analysis framework [7]. In addition, althoughmarkers primarily measure time lags, qualitative value relatedcriteria are still lacking.

The five-stage intervention for the possible implementa-tion of Lean and Six Sigma techniques can be adapted to dif-ferent phases of the translational research continuum. It can aidperformance improvement from basic science along the con-tinuum to population impact. However, defining activities ormarkers for the earlier phases is relatively easier than for laterphases, such as population impact. These later phases tend tobe beyond the primary scope of some comprehensive cancercenters. Hence, inclusive evidence synthesis is needed to un-derstand the later phases from a broader public health perspec-

tive [26] before performance assessment models can beimplemented.

Translational research is not a simple linear process. Somemay argue that its complex and unpredictable nature prohibitsthe use of models for performance assessment. The fear re-garding such assessments among some stakeholder groups isthat it might jeopardize serendipity that is characteristic formany research processes, fail to capture research excellencethat might exist partially or completely outside the scope of as-sessment criteria, and enable bureaucrats to take control offields they do not really comprehend. A cautious and stepwiseapproach is therefore advisable if cancer centers intend to usethese models for performance assessment. As a first step, ac-quiring structured insight into the various aspects of the trans-lational process and comparing these results between cancercenters might help centers identify improvement opportuni-ties. For that purpose, more precise operating definitions areneeded at three levels: performance dimensions, performanceindicators, and sufficiently detailed metrics [34]. It is hard tosay whether the cancer field has specific needs, but all stake-holders, including clinicians, should be open to the idea that

models from other medicaland/or nonmedical fields canalso be used to assess cancercenters. These modelsshould be thoroughly testedin practice to know their po-tential for actual perfor-mance assessment.

The strengths of thisstudy are that, to our knowl-edge, this is the first time thata systematic review has beenundertaken to identify mod-els of translational researchthat were appraised using a

set of criteria. These criteria were based on a range of issues fortranslational research identified from relevant literature. Un-doubtedly, the criteria that we used can be critiqued. However, itis necessary for cancer centers to carefully select models for per-formance assessment and our framework provides a basis for that.The criteria can be refined with views from key stakeholdergroups (e.g., basic researchers, clinical researchers, clinicians,funding agencies, senior executives, and patients).

There are two possible limitations to our study. First, wecould not check if all the models had been tested and imple-mented in practice. One could argue that the elements of thesemodels are supported by “findings” or evidence from aca-demic or experiential literature. The second limitation is that,because of a lack of consensus on terminologies in transla-tional research, it was hard to identify models. Therefore, therecould be models that we did not consider in this appraisal. Toincrease the possibility of identifying models in future, the title,abstract, and keywords of studies should clearly use a commonterm and/or commonly associated terms of translational research.Substitutions such as “bench-to-bedside,” “implementation sci-ence,” and “biomedical research” should be restricted to the main

The fear regarding [performance] assessments amongsome stakeholder groups is that it might jeopardizeserendipity that is characteristic for many researchprocesses, fail to capture research excellence that mightexist partially or completely outside the scope ofassessment criteria, and enable bureaucrats to takecontrol of fields they do not really comprehend. Acautious and stepwise approach is therefore advisable ifcancer centers intend to use these models for performanceassessment.




Dow

nloaded from


content of the papers, with clear explanation of these terms thatcan help the reader understand the model. Addition of a specificMeSH term for models in databases (e.g., conceptual models oftranslational research) may be useful to ensure that models areeasily listed and identified.

CONCLUSIONPerformance assessment can help improve the process of transla-tional research by identifying areas for improvement in its manage-ment, knowledge exchange, and engagement of multidisciplinaryteams to deliver efficient and effective translational research,which would help reduce unnecessary time lag. Two models oftranslational research appear to be more suitable for performanceassessment: the Process Marker model and the Lean and SixSigma applications to clinical and translational science. It will benecessary to thoroughly test them in practice. Finally, cancer cen-ters need to come to a consensus on terminologies in translationalresearch, which will help to identify and select models for perfor-

mance assessment that can improve the performance of transla-tional research for the benefit of patients.

ACKNOWLEDGMENTSThe study was supported by the FP7 program of the EuropeanCommission as part of a Eurocan platform project (An Excel-lence Designation System for Comprehensive Cancer Centersin Europe). The project consists of 28 cancer institutions acrossEurope that currently collaborate on different work packagesto help advance translational cancer research.

AUTHOR CONTRIBUTIONSConception/Design: Abinaya Rajan, Richard Sullivan, Wim van HartenCollection and/or assembly of data: Abinaya Rajan, Suzanne BakkerData analysis and interpretation: Abinaya Rajan, Wim van HartenManuscript writing: Abinaya Rajan, Wim van HartenFinal approval of manuscript: Abinaya Rajan, Richard Sullivan, Suzanne

Bakker, Wim van Harten

REFERENCES

1. Eckhouse S, Lewison G, Sullivan R. Trends in theglobal funding and activity of cancer research. Mol On-col 2008;2:20–32.

2. Westfall JM, Mold J, Fagnan L. Practice-based re-search: “Blue highways” on the NIH roadmap. JAMA2007;297:403–406.

3. Balas EA, Boren SA. Managing Clinical Knowledgefor Health Care Improvement. Yearbook of Medical In-formatics. Stuttgart, Germany: Schattauer Verlagsge-sellschaft mbH, 2000.

4. Contopoulos-Ioannidis D, Alexiou G, Gouvias T etal. Life cycle of translational research for medical inter-ventions. Science 2008;321:1298–1299.

5. Pray L. Gleevec: The breakthrough in cancer treat-ment. Nat Educ 2008;1:1.

6. Organisation for Economic Co-operation and De-velopment. Effects of European Union Accession, Part1: Budgeting and Financial Control. Paris, France: Or-ganisation for Economic Co-operation and Develop-ment, 1998.

7. Trochim W, Kane C, Graham M et al. Evaluatingtranslation research: A process marker model. ClinTransl Sci 2011;4:153–162.

8. Morris ZS, Wooding S, Grant J. The answer is 17years, what is the question: Understanding time lags intranslational research. J R Soc Med 2011;104:510–520.

9. Rubio DM, Schoenbaum EE, Lee LS et al. Definingtranslational research: Implications for training. AcadMed 2010;85:470–475.

10. Zehrouni E. The NIH roadmap. Science 2003;302:63–72.

11. Institute of Medicine. Crossing the quality chasm:A new health system for the 21st century. WashingtonDC: Institute of Medicine, 2001.

12. Sung NS, Crowley WFJ, Genel M et al. Central

challenges facing the national clinical research enter-prise. JAMA 2003;289:1278–1287.

13. Dougherty D, Conway PH. The “3T’s” road map totransform US health care. JAMA 2008;299:2319–2321.

14. Thornicroft G, Lempp H, Tansella M. The place ofimplementation science in the translational medicinecontinuum. Psychol Med 2011;41:2015–2021.

15. National Institute for Health Research. Better Re-search for Better Health. London, UK: Department ofHealth, 2006.

16. Craig P, Dieppe P, Macintyre S et al. Developingand evaluating complex interventions: the new MedicalResearch Council guidance. BMJ 2008;337:a1655.

17. Crowley WF Jr., Sherwood L, Salber P et al. Clin-ical research in the United States at a crossroads: pro-posal for a novel public-private partnership to establish anational clinical research enterprise. JAMA 2004;291:1120–1126.

18. The President’s Cancer Panel. Translating Re-search into Cancer Care: Delivering on the Promise.Washington DC: Department of Health and Human Ser-vices, 2005.

19. Khoury MJ, Gwinn M, Yoon PW et al. The contin-uum of translation research in genomic medicine: howcan we accelerate the appropriate integration of humangenome discoveries into health care and disease preven-tion. Genet Med 2007;9:665–674.

20. Khoury MJ, Little J, Burke W, eds. Human Ge-nome Epidemiology: Scope and Strategies. New York:Oxford University Press, 2004.

21. Haddow JE, Palomaki GE. A model process for theevaluating data on emerging genetic tests. In: KhouryMJ, Little J, Burke W, editors. Human Genome Epide-miology: Scope and Strategies. New York: Oxford Uni-versity Press, 2004:217–233.

22. Drolet BC, Lorenzi NM. Translational research:

Understanding the continuum from bench to bedside.Transl Res 2011;157:1–5.

23. Ernest TH, Matrisian LM, Nelson WG et al. TheTranslational Research Working Group developmentalpathways: Introduction and overview. Clin Cancer Res2008;14:5664–5671.

24. Schweikhart SA, Dembe AE. The applicability ofLean and Six Sigma techniques to clinical and transla-tional research. J Investig Med 2009;57:748–755.

25. Lane JP, Flagg JL. Translating three states ofknowledge: Discovery, invention, and innovation. Im-plement Sci 2010;5:1–14.

26. Ogilvie D, Craig P, Griffin S et al. A translationalframework for public health research. BMC PublicHealth 2009;9:116.

27. Graham ID, Logan J, Harrison MB et al. Lost inknowledge translation: Time for a map? J Contin EducHealth Prof 2006;26:13–24.

28. Cooksey D. A review of UK health research fund-ing. London: HM Treasury; 2006.

29. Jones DS, Cambrioso A, Mogoutov A. Detectionand characterization of translational research in cancerand cardiovascular medicine. J Transl Med 2011;9:57.

30. Schwab ED, Pienta KJ. Cancer as a complex adap-tive system. Med Hypotheses 1996;47:235–241.

31. Amidon DM. The challenge of fifth generationR&D. Res Tech Manage 1996;39:33–41.

32. Johnson G, Scholes K, Whittington R. ExploringCorporate Strategy, 8th Ed. Essex, UK: Prentice Hall,2008.

33. Chubb I. Can Australia afford to fund translationalresearch? Presented at: Australian Government Bio-breakfast; April 3, 2012; Melbourne, Australia.

34. Pozen R, Heather K. Defining success for trans-lational research organisations. Sci Transl Med 2011;3:94cm20.





Dow

nloaded from


otncologist he european perspectives - alphamed...

Documents