SYSTEMATIC REVIEW Open Access

Effectiveness of implementation strategiesfor clinical guidelines to communitypharmacy: a systematic reviewKim Watkins1*, Helen Wood1, Carl R. Schneider2 and Rhonda Clifford1

Abstract

Background: The clinical role of community pharmacists is expanding, as is the use of clinical guidelines in thissetting. However, it is unclear which strategies are successful in implementing clinical guidelines and what outcomescan be achieved. The aim of this systematic review is to synthesise the literature on the implementation of clinicalguidelines to community pharmacy. The objectives are to describe the implementation strategies used, describe theresulting outcomes and to assess the effectiveness of the strategies.

Methods: A systematic search was performed in six electronic databases (Medline, EMBASE, CINAHL, Web of Science,Informit, Cochrane Library) for relevant articles. Studies were included if they reported on clinical guidelinesimplementation strategies in the community pharmacy setting. Two researchers completed the full-search strategy,data abstraction and quality assessments, independently. A third researcher acted as a moderator. Quality assessmentswere completed with three validated tools. A narrative synthesis was performed to analyse results.

Results: A total of 1937 articles were retrieved and the titles and abstracts were screened. Full-text screening wascompleted for 36 articles resulting in 19 articles (reporting on 22 studies) included for review. Implementationstrategies were categorised according to a modified version of the EPOC taxonomy. Educational interventionswere the most commonly utilised strategy (n = 20), and computerised decision support systems demonstratedthe greatest effect (n = 4). Most studies were multifaceted and used more than one implementation strategy(n = 18). Overall outcomes were moderately positive (n = 17) but focused on process (n = 22) rather than patient (n = 3)or economic outcomes (n = 3). Most studies (n = 20) were rated as being of low methodological quality and havinglow or very low quality of evidence for outcomes.

Conclusions: Studies in this review did not generally have a well thought-out rationale for the choice of implementationstrategy. Most utilised educational strategies, but the greatest effect on outcomes was demonstrated using computerisedclinical decision support systems. Poor methodology, in the majority of the research, provided insufficient evidence to beconclusive about the best implementation strategies or the benefit of clinical guidelines in this setting. However, thegenerally positive outcomes across studies and strategies indicate that implementing clinical guidelines to communitypharmacy might be beneficial. Improved methodological rigour in future research is required to strengthen the evidencefor this hypothesis.

Protocol registration: PROSPERO 2012:CRD42012003019.

Keywords: Community pharmacy, Pharmacy, Pharmacists, Implementation, Information dissemination, Guideline,Evidence-based medicine

* Correspondence: kim.watkins@uwa.edu.au1School of Medicine and Pharmacology, University of Western Australia,Perth, AustraliaFull list of author information is available at the end of the article

ImplementationScience

© 2015 Watkins et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Watkins et al. Implementation Science (2015) 10:151 DOI 10.1186/s13012-015-0337-7

BackgroundIn the last 30 years, there has been a major shift in health-care towards evidence-based medicine and the use of clin-ical guidelines to facilitate evidence-based practice [1–4].Clinical guidelines are defined as “systematically devel-oped statements to assist practitioner and patient deci-sions about appropriate healthcare for specific clinicalcircumstances” [5]. Their primary aim is to improve pa-tient care and, ultimately, patient health outcomes [4].They also have many potential benefits for health profes-sionals including improved clinical decision-making andconsistency of care [4].While research has demonstrated that clinical guidelines

can achieve improvements in health professional practiceand patient outcomes, there is substantial variability inobserved effectiveness [2, 6]. A recent study in Australiareported that patients receive appropriate, evidence-basedcare, on average, only 57 % of the time [7]. Despite theproliferation of clinical guidelines, there are still barriersin translating the evidence in clinical guidelines into prac-tice across all healthcare settings [8, 9]. This has resultedin an increase in research attempting to identify thefactors that influence successful clinical guideline imple-mentation [10].Development and dissemination of clinical guidelines

does not necessarily translate to health professional uptakeand adherence [1, 11]. The impact of clinical guidelinesdepends on the characteristics of the guidelines them-selves as well as the implementation strategies employed[12]. Characteristics such as the complexity of guidelinesand the evidence base used in their development areimportant factors influencing uptake into clinical practice[1]. However, the evidence regarding effective implemen-tation strategies is less clear, and in some instances contra-dictory, as seen in the conflicting conclusions of researchin this field [1, 2, 8, 11, 13]. For example, it is still unclearwhether multifaceted implementation strategies, thattarget multiple barriers to implementation or use multiplestrategies, are more effective than single strategies, despitethe commonly held assumption that a multifaceted ap-proach is superior [1, 2, 14, 15].These contradictions have led to the recommendation

that implementation strategies should be based around aclear rationale, such as overcoming barriers to change(tailored implementation interventions) [14, 16]. However,methodology on how to determine barriers and how todesign strategies to address them, is less clear [17, 18].Recently, the literature regarding guideline implementa-tion has also advocated the use of theoretical frameworksto inform implementation strategies [19–22]. While the-ory has not been routinely used in implementationresearch to date, this approach is gaining momentum [23].Most of the research into guideline implementation

has been conducted with medical practitioners and in

the hospital setting [1, 2, 24–26]. In the systematic reviewof guideline dissemination and implementation by Grim-shaw, medical practitioners alone were the target of 74 % ofinterventions [2]. There have been a small number ofreviews looking at guideline implementation for allied-health practitioners, but only a few studies relate to phar-macists, and many were in the hospital setting [13, 16, 22].Understanding the impact of clinical guidelines in com-

munity pharmacy is important, given the expanding roleof community pharmacists in primary healthcare. It isacknowledged that community pharmacists are often apatient’s first point of contact with the health system, andin some instances, the only health professional to see a pa-tient [27]. Worldwide, community pharmacy practice isgradually evolving to incorporate the provision of clinicalservices and a greater focus on patient care [28]. Inresponse to these practice changes, there has been anincrease in the development of clinical guidelines for usein this setting. However, little is known about the imple-mentation of clinical guidelines in community pharmacy.The aim of this systematic review is to synthesise the

literature on implementation of clinical guidelines tocommunity pharmacy. The objectives are to:

� Describe the types of implementation strategiesutilised.

� Describe the outcomes resulting from guidelineimplementation at a:– Practitioner level: process outcomes measuring

community pharmacy practice– Patient level: patient outcomes measuring clinical

and/or humanistic outcomes– Health system level: economic outcomes

� Assess the effectiveness of the implementationstrategies.

MethodsThe methodology used in conducting this systematicreview was in accordance with the Preferred ReportingItems for Systematic Reviews and Meta-Analyses(PRISMA) statement and checklist, using the accom-panying explanation and elaboration document [29, 30].The review protocol was registered with the Inter-national Prospective Register of Systematic Reviews(PROSPERO) on 27 September 2012 and updated inMay 2015 (registration number CRD42012003019) [31].

Search strategy and study selectionAfter consulting with a librarian from the Medical andDental Library at the University of Western Australia, acomprehensive search strategy was devised. Search termsand medical subject headings (MeSH headings) chosenwere those relevant to “community pharmacy” AND “clin-ical guidelines” AND “implementation” (Additional file

Watkins et al. Implementation Science (2015) 10:151 Page 2 of 23

1—Search terms). The search was conducted in six elec-tronic databases (Medline, EMBASE, CINAHL, Web ofScience, Informit and Cochrane Library) up to and includ-ing the 9 November 2014. No restrictions were placed onthe search.The search results from each of the databases were

collated. Articles retrieved that were not printed inEnglish were removed along with duplicate results.The decision to remove articles not written in Englishwas based on resource limitations. There is evidenceto suggest this was unlikely to introduce systematicbias into this review[32]. Articles were then screenedfor eligibility based on the previously devised PICO(participants, interventions, comparators, outcomes)framework [33]. Initially titles and abstracts werescreened. Full-text screening was undertaken in arti-cles meeting the inclusion criteria or where the ab-stract provided insufficient information. A final list ofarticles was identified for inclusion in the review.Two authors (KW and HW) independently conductedthe full-search strategy and eligibility assessment. Dis-crepancies in study selection were resolved throughdiscussion, and where consensus could not beachieved, by mediation with a third author (CS).

Eligibility criteriaStudies were not excluded based on research design.Experimental, quasi-experimental, intervention and obser-vational studies were all eligible for inclusion. Studies wereselected based on the PICO (participants, interventions,comparators, outcomes) framework [33] outlined below.

ParticipantsStudies were included if the intervention was in thecommunity pharmacy setting. This included interven-tions directed to professional staff (pharmacists, interns(unregistered pharmacists) and pharmacy students) andpharmacy support staff (pharmacy assistants, pharmacytechnicians). The definition of a community pharmacyused was “a retail business registered for the provision ofpharmaceutical services and from which goods and ser-vices relating to the provision of pharmaceutical servicesmay be available to the public” [34]. Not included in thereview were studies set in hospital pharmacies, multi-disciplinary medical clinics run by health funds orother private organisations, outpatient clinics, consult-ant pharmacist services and Accredited Drug Dispens-ing Outlets (ADDOs).

InterventionsOf interest were interventions involving the implementa-tion of clinical guidelines. Studies were included if theyreported dissemination and implementation strategies

directed to community pharmacy which aimed to influ-ence behaviour of pharmacists, and/or other staff, to-wards uptake and adherence of guideline-based practice.Studies were excluded if they were “patient programmes”and not aimed at influencing practice behaviours in thecommunity pharmacy setting. This included interventionswhere pharmacists were instructed to provide services forthe course of the project, but the aim was not to make this“usual practice” for the pharmacist. For example, thepaper by Armour and colleagues in 2007, that described aguideline-based asthma management programme [35].The description of the intervention strategies was a

modified version of the Cochrane Effective Practice andOrganisation Care Review Group (EPOC) taxonomy’ssection on implementation strategies [36]. This includededucational materials, educational meetings, educationaloutreach visits, mass media campaign, audit and feed-back, reminders, practice support and fee for service [36].Also investigated was the basis for the interventions. In-terventions were considered “tailored” if barriers to imple-mentation were initially identified and then addressed inthe development of the implementation strategy [36, 37].“Based on theory” was the term for interventions thatused a documented theory, model or framework for thedesign of the intervention [20]. Interventions addressing“organisational culture” represent those interventions thatfocus on changing some aspect of the community phar-macy as an organisation [36, 38]. These include many con-structs such as leadership effectiveness, workflows withinthe organisation, staffing and time management issues.The operational definition of clinical guidelines was

“systematically developed statements to assist practitionerand patient decisions about appropriate healthcare forspecific clinical circumstances” [5, 36]. This definition waschosen on the basis that it has been included in theupdated EPOC taxonomy [36] and also been used in twosimilar reviews with a similar scope, but in different prac-tice settings [1, 16]. In the application of this definition, itwas acknowledged that some studies would be excluded,for example, studies that looked at the implementation ofpractice standards that did not include guidance on clin-ical activities [39, 40] and articles that looked at policydocuments and frameworks [41, 42]. One of the key deter-minations from the definition was that the guidelinesneeded to relate to “specific clinical circumstances.” Theuse of two reviewers and a moderator provided methodo-logical rigour in application of the definition.

ComparatorsFor most of the studies in this review, the primary com-parator was “usual practice”. Usual practice is indicatedby no implementation strategy being directed to thecommunity pharmacy to change current practice.

Watkins et al. Implementation Science (2015) 10:151 Page 3 of 23

OutcomesOutcomes were based on the EPOC classification schemeand included patient outcomes, practitioner outcomesand economic outcomes [43]. Patient health outcomemeasures of interest included, physical health and treat-ment outcomes (e.g. mortality, morbidity and surrogatephysiological health measures) and psychosocial outcomes(e.g. quality of life) [43]. Practitioner/process outcomemeasures related to quality of care provided by commu-nity pharmacy. This included adherence to recommendedpractice or guidelines (e.g. extent to which health careproviders gave appropriate advice, delivered clinical inter-ventions and followed referral guidelines) [43]. Economicoutcomes calculated resource use and measured costs andcost savings associated with guideline implementation(e.g. human resources, consumables and equipment) [43].Secondary (surrogate) outcomes are those that may indir-ectly reflect important outcomes, including measures suchas knowledge, attitudes and satisfaction of both patientsand practitioners [43]. Also of interest was how outcomeswere determined. Objective measures involve an impartialmeasurement and are usually considered more reliablethan self-reported measures, which are subjective [44].Examples of objective data include information frommedical or health records, while subjective data may beobtained from a self-report questionnaire.

Data abstraction, synthesis and quality assessmentA comprehensive data abstraction table was developedbased on two standard checklists: The Cochrane datacollection form (for intervention reviews of randomisedcontrolled trials and non-randomised controlled trials)and the transparent reporting of evaluations with non-randomised designs (TREND) statement checklist (fornon-randomised evaluations of behavioural and publichealth interventions) [45, 46]. The data abstraction tablewas piloted initially using two articles and revised byconsensus (KW and HW). Using the revised table, dataabstraction was completed independently by two authors(KW and HW). Discrepancies in data abstraction wereresolved through discussion, and where consensus couldnot be achieved, by mediation with a third author (CS).A narrative synthesis was undertaken because the meth-

odological and clinical heterogeneity of the studies in thisreview determined that meta-analysis was not appropriate.A narrative synthesis satisfied the aim by considering dif-ferent implementation strategies and determining theireffectiveness in achieving outcomes reflecting the benefitof clinical guidelines in the community pharmacy setting.Risk of bias was assessed for all studies included in the

review. Studies that were either randomised controlledtrials (RCTs), non-randomised controlled trials (NRCTs)or controlled before and after studies (CBAs) were

evaluated for bias using the EPOC risk of bias tool [47].Other studies in this review were cohort or quasi-experimental studies and were evaluated for qualityusing the Newcastle-Ottawa quality assessment scale forcohort studies [48]. These tools were chosen based onthe Cochrane recommendation to use a domain-basedevaluation for risk of bias assessments, rather than a toolwith a summary score [49]. The Newcastle-Ottawa toolwas one of only two tools recommended by Deeks in areview of 182 instruments for measuring risk of bias incohort studies [50].The Grading of Recommendations Assessment, Develop-

ment and Evaluation working group (GRADE) approach[51] was used, in conjunction with the risk of bias tools, toevaluate quality of evidence for outcomes. This is becauserisk of bias can vary across outcomes, a considerationwhich is often ignored in systematic reviews [52]. Tocomplete the GRADE assessment, worksheets were used toprepare a summary of findings table [53].This involved athree-step process of assessing the relative importance of allstudy outcomes, assessing the certainty of evidence acrossstudies for an outcome and summarising the findings [53].All quality evaluations including risk of bias and quality

of evidence for outcomes were completed independentlyand then by consensus of two authors, KW and HW, withmediation by CS when required.A critical appraisal of economic outcomes and eco-

nomic modelling was not undertaken as it was beyondthe scope of this review.

ResultsStudy selectionSearching the databases resulted in a total of 1937 arti-cles. After adjusting for duplicates and non-English lan-guage articles, 1446 remained. Title and abstractscreening excluded all but 36 articles. Examining thefull-text records and reference lists of the remaining 36articles resulted in 19 articles reporting on 22 studies,meeting the inclusion criteria. Five articles were dis-cussed with the moderator to achieve consensus (Kappascore 0.86) Fig. 1.

Study characteristics including risk of bias assessmentsThe 22 studies comprised 10 RCTs [54–63], 3 NRCTs[64–66], 1 CBA trial [67] and 8 quasi-experimental orobservational studies [68–75] (Tables 1 and 2). Thestudies were conducted in Australia (n = 8) [54–56, 60,64, 65, 68, 69], the USA (n = 4) [57–59, 73], the UK(n = 3) [62, 63, 75], the Netherlands (n = 2) [61, 72],Belgium (n = 1) [67], Canada, (n = 1) [71], Finland(n = 1) [74], Germany (n = 1) [70] and Switzerland(n = 1) [66]. The time frame for intervention-durationranged from 1 day to 2 years. The included studies hadsample sizes ranging from a group of 8 pharmacists to an

Watkins et al. Implementation Science (2015) 10:151 Page 4 of 23

intervention group of 1222 community pharmacies. Powercalculations to determine an appropriate sample size wereperformed in three studies [62, 66, 70].The 22 studies included in the review were assessed for

risk of bias using separate tools depending on the studydesign. Consensus was achieved through discussion.Moderation was required for one domain (“comparabilitydomain” in the Newcastle-Ottawa tool), which was themain area of disagreement (Tables 3 and 4).

Fourteen studies were assessed using the EPOC risk ofbias tool for RCTs, NRCTs and CBA studies. Of these,11 studies were evaluated as having a high risk of bias[55–59, 61, 63–67] and three studies assessed as havingan unclear risk of bias [54, 60, 62]. Several domains con-tributed to risk of bias in multiple studies. These in-cluded, inadequately addressing incomplete data, lack ofsimilarity in baseline measurements and protection fromcontamination in the study. It was not clear if these

Fig. 1 Flow diagram of study selection

Watkins et al. Implementation Science (2015) 10:151 Page 5 of 23

Table 1 Summary of included studies evaluating implementation of clinical guidelines in community pharmacy

First author, year Country Sample size Time frame Clinical area (guidelines) Methodology for collectingoutcome data

Key findings for mainoutcome (++, +, minimaleffect, variable results, noevidence of effect)

Curtain, C. et al,2011 [54]

Australia Community pharmacies, Total(n = 185), Intervention (n = 73),Control (n = 112)

12 weeks Proton pump inhibitors (NationalPrescribing Service proton pumpinhibitor dosage recommendation)

Clinical interventionsoftware data, Prescriptiondata, Patient survey,Economic data

++

De Almeida Neto,A.C. et al, 2000[55]

Australia Pharmacists, Total (n = 43) fromcommunity pharmacies, Total(n = 30), Intervention, (n = 15pharmacies), Control (n = 15pharmacies)

Three-week baseline datacollection followed by a6-week interventionperiod, immediately afterthe workshop

Non-prescriptiondrugs—analgesics

Simulated patientmethodology, Pharmacistsurvey

+

(Protocol for non-prescriptionmedicines with a focus onidentifying inappropriate “off-label”use of compound analgesics)

De Almeida Neto,A.C. et al, 2001Study 1 [56]

Australia Community pharmacies, Total(n = 24), Intervention group(n = 16), Control group (n = 8)

Three 4-week periods: beforeand immediately

Non-prescriptiondrugs—analgesics

Simulated patientmethodology

++

after a 3-h training workshop,and after a further interval of14 weeks

(Protocol for non-prescriptionmedicines with a focus onidentifying inappropriate “off-label”use of compound analgesics)

De Almeida Neto,A.C. et al, 2001Study 3 [64]

Australia Not stated Three 2-week periods(baseline, post workshop1and post workshop2)

Non-prescription drugs—coughand cold medicines

Simulated patientmethodology

++

(Protocol for non-prescriptioncough and cold medicines)

De Almeida Neto,A.C. et al, 2001Study 5 [68]

Australia Pharmacists and pharmacyassistants from communitypharmacies, total (n = 99)

12 weeks of pseudo-patronand feedback visits, post atraining visit

Non-prescription drugs—heartburnand indigestion treatments(protocol for heartburnmanagement)

Simulated patientmethodology

++

De Almeida Neto,A.C. et al, 2001Study 5A [69]

Australia Not stated Not mentioned in paper Non-prescriptiondrugs—analgesics

(Protocols specific for analgesics)

Egen, V. et al,2003 [70]

Germany Gynaecologists, total (n = 311),interviewed baseline (n = 24),post-intervention (n = 27),pharmacists, total (n = 418),baseline (n = 21), post (n = 21),women in childbed: baseline(n = 131), post (n = 118)

16 months intervention withinterviews pre and post

folic acid (The Societies ofNutrition, Gynaecology andObstetrics, Human Genetics,Paediatrics, and Neuropaediatricsjointly issued correspondingrecommendations)

Simulated patientmethodology, Patientinterview, Gynaecologisttelephone interview

No evidence of effect

Watkins

etal.Im

plementation

Science (2015) 10:151

Page6of

23

Table 1 Summary of included studies evaluating implementation of clinical guidelines in community pharmacy (Continued)

Guirguis, L.M. et al,2007 [71]

Canada Practicing pharmacists, Total(n = 8) tested the diabetes tool

Participants were introducedto the tools, and theirexperience was evaluatedafter 2 weeks. One year latera survey was faxed toinvestigate any sustaineduse/change in practice

Diabetes Pharmacist self-reportforms. Pharmacist survey,Focus group discussion

+

(Canadian Diabetic Guidelines)

Koster, E.S, et al,2014 [72]

TheNetherlands

Community pharmacies, Total(n = 78), Pharmacists, (n = 95)and technicians (n = 337) wereinterviewed, dispensingdata—only available for(n = 52) pharmacies

Dispensing data wascollected for the periodbetween 1 Jan. 2008 to 10May. 2011.

Methotrexate (Safe MethotrexateDispensing Recommendationspublished by the Royal DutchPharmaceutical Society inaccordance with the Dutch HealthCare Inspectorate)

Pharmacist-structuredinterviews, Electronicdispensing records

+

Kradjan, W.A. et al,1999 [57]

USA Community pharmacies, Total(n = 90)

Intervention period 4th Asthma Patient survey No evidence of effect

Intervention, (n = 44) March 1996 to 30th June1996

(Current asthma treatmentguidelines)

Control, (n = 46)

Legrand, S.A. et al,2012 [67]

Belgium Pharmacists, Total (n = 100) Intervention pharmaciescompleted a baselinequestionnaire, and after a 6-month intervention periodparticipants (includingcontrols) were asked tocomplete a post-questionnaire

Medicines and driving Pharmacist survey +

(IS) intervention group (n = 68),(SA) intervention group(n = 12), Control group (n = 20)

(DRUID (driving under theinfluence of drugs, alcohol andmedicines project) dispensingguidelines)

Martin, B.A. et al,2010 [73]

USA Pharmacists, Total (n = 25) The study was conductedduring 2002–2003

Smoking cessation Pharmacist survey +

(National tobacco cessationguidelines (Treating Tobacco Useand Dependence: Clinical PracticeGuideline), which incorporates the5A’s counselling process)

Pharmacist telephoneinterview

Invoicessubmitted—(remunerationclaims)

Naunton, M. et al,2004 [65]

Australia GPs, total (n = 200)—74 %visited

Baseline data collection Osteoporosis Pharmacist survey +

Watkins

etal.Im

plementation

Science (2015) 10:151

Page7of

23

Table 1 Summary of included studies evaluating implementation of clinical guidelines in community pharmacy (Continued)

Community pharmacies, total(n = 69)—100 % visitedpharmacists, total (n = 81) tocomplete surveys

(Mar–Sept 2001) Interventionmail out (Oct 2001) Detailingvisits (Jan–May 2002), postintervention data collection(Mar–Sept 2002)

(Locally produced guidelinesadapted from American College ofRheumatology, UK ConsensusGroup and Osteoporosis Australiaguidelines on the management ofglucocorticoid inducedosteoporosis)

GP survey

Hospital admission data

Prescriptiondata—(remunerationclaims)

Patwardhan, P.D.et al, 2012 [58]

USA Intervention group: The research was carried outfrom July 2008 until March2009 with a 1-month studyperiod in November 2008

Smoking cessation Pharmacist self-report forms +

Community pharmacies(n = 8), Pharmacists (n = 16),Technicians (n = 24)

(Treating tobacco use anddependence: Clinical practiceguideline (2008 update)

Quit-line referral reports(from an external agency)

Control group The specific recommendation touse AAR in situations in which the5A’s approach may not be feasible)

Community pharmacies (n = 8),

Pharmacists (n = 16),Technicians (n = 24),

Puumalainen, I. etal, 2005 [74]

Finland Pharmacists, Total (n = 734) TIPPA implementation,4 years (2000–2003).

Guideline-based counselling Pharmacist survey Minimal effect

Data collection for thisresearch, 1 month—June2002

(The United States Pharmacopeia(USP) Medication CounsellingBehaviour Guidelines disseminatedthrough a 4-year project (TIPPA))

Raisch, D.W. 1998[59]

USA (NewMexico)

Community pharmacies, Total(n = 301)

Ketorolac claims records werereviewed for 3 monthsbefore intervention (Aug–Oct1995) and for 3 months afterintervention (Dec–Feb 1996)

Ketorolac Dispensing data +

Intervention (n = 150), Control(n = 151)

(Manufacturers prescribingguidelines for ketorolac)

Economic data

Data obtained from:

Community pharmacies(n = 167), Intervention (n = 90)

Control (n = 77)

Australia Diabetes ++

Watkins

etal.Im

plementation

Science (2015) 10:151

Page8of

23

Table 1 Summary of included studies evaluating implementation of clinical guidelines in community pharmacy (Continued)

Reeve, J.F. et al2008 [60]

Community pharmacies, Total(n = 52), Pharmacists, Total(n = 150) recruited to attendtraining

6-week study period wherethe computer-generatedprompt was active plusanother 2-week period whereinterventions were recordedbut the prompt wasdeactivated

Clinical interventionsoftware data, Prescriptiondata, Pharmacist survey

Intervention (n = 31)pharmacies

(American Diabetes Association-Clinical practice recommendations.Aspirin therapy in diabetes.Recommendation for the additionof low-dose aspirin therapy tomedication regimen of high-riskpatients with diabetes)

Control (n = 21) pharmacies

Sigrist, T. et al,2002 [66]

Switzerland Community pharmacies, Total(n = 27)

2 months Non-prescription drugs Simulated patientmethodology

Variable results

Intervention (n = 14) (Personalised advice protocolbased on change and health beliefmodels and used in assessment ofappropriate use of non-prescription medications)

Control (n = 13),

intervention participants toattend workshops,

Pharmacists (n = 20),

Pharmacy assistants (n = 65),

Thorley, T. et al,2006 [75]

UK Community pharmacies, Total(n = 1222)

March 2003 (initialimplementationcommunication), mysteryshopping data collected over4 months (May–Aug 2003)

Asthma Simulated patientmethodology

+

(Evidence-based questions (×3)from Royal College of Physicians(RCP) to determine patient asthmacontrol and to direct responsebased on answers)

Van de Steeg-vanGompel, C. et al2011 [61]

TheNetherlands

Community pharmacies, Total(n = 71) grouped into 36clusters

Sept 2006–Feb 2008 Statins drugs Prescription data No evidence of effect

Intervention (n = 37) (18clusters), Control (n = 34)(18 clusters)

(Protocol for Education at FirstDispensing of a Statin (EAFD) and

Pharmacist self-report forms

Watkins

etal.Im

plementation

Science (2015) 10:151

Page9of

23

Table 1 Summary of included studies evaluating implementation of clinical guidelines in community pharmacy (Continued)

Protocol for Education at SecondDispensing of a Statin (EASD))

Pharmacist telephoneinterview,

Watson, M.C. et al,2002 [62]

UK Community pharmacies, Total(n = 60)

Mar–Apr 2000 baseline dataJuly–Nov 2000 postintervention data collection

Non-prescription Simulated patientmethodology, Pharmacistsurvey, Economic data

No evidence of effect

EO intervention (n = 15), CPEintervention (n = 15), EO andCPE intervention (n = 15),Control (n = 15)

Drugs—vulvovaginal candidiasis

(Evidence-based guidelines for OTCtreatment of vulvovaginalcandidiasis)

Watson, M.C. et al,2007 [63]

UK Community pharmacies, Total(n = 20)

The intervention comprisedtwo training sessions1 month apart (Sept and Oct2005)

Good pharmacy practice Simulated patientmethodology

No evidence of effect

Medication care assistants(n = 30)

(Royal Pharmaceutical Society ofGreat Britain (RPSGB) guidelinesand WWHAM guideline.Professional and good practiceguidelines for the supply ofnon-prescription medicines)

Pharmacist survey

Intervention (n = 20 MCAs),Control (n = 10 MCAs)

Key:Simulated patient methodology: this involves data collection using covert patients (mystery shoppers) to assess pharmacy practice [95]

Watkins

etal.Im

plementation

Science (2015) 10:151

Page10

of23

Table 2 Comparison of studies

Curtain,C. 2011

De AlmeidaNeto, A.C.2000

De AlmeidaNeto, A.C.2001 Study 1

De AlmeidaNeto, A.C.2001 Study 3

De AlmeidaNeto, A.C.2001 Study 5

De AlmeidaNeto, A.C. 2001Study 5A

Egen, V.2003

Guirguis,L.M. 2007

Koster,E.S. 2014

Kradjan,W.A. 1999

Legrand,S.A. 2012

Study design Randomised controlledtrial (RCT)

✓ ✓ ✓ ✓

Non-randomisedcontrolled trial

✓

Controlled beforeand after studies

✓

Other ✓ ✓ ✓ ✓ ✓

Participantsself-selected

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Participantsincluded in theintervention

Pharmacists ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Pharmacy supportstaff (techniciansand assistants)

✓ ✓ ✓ ✓ ✓

Other healthprofessionals

✓

Patients/Public ✓

Intervention type(EPOC taxonomy)

Educational materials ✓ ✓ ✓ ✓ ✓

Educational meetings ✓ ✓ ✓ ✓ ✓ ✓

Educational outreachvisits

✓

Mass media Campaign ✓

Audit and feedback ✓ ✓ ✓ ✓ ✓

Reminders ✓ ✓ ✓

Practice support ✓ ✓ ✓

Fee for service ✓

Basis forintervention

Improve clinicalknowledge

✓ ✓ ✓ ✓ ✓ ✓ ✓

Improvecommunication skills

✓ ✓ ✓ ✓

Based on addressingorganisational culture

✓

Based on a specifiedtheory of behaviourchange

✓ ✓

Based on addressingidentified barriers(tailored)

Watkins

etal.Im

plementation

Science (2015) 10:151

Page11

of23

Table 2 Comparison of studies (Continued)

Outcomemeasures

Practitioneroutcomes—subjectivemeasures

✓ ✓ ✓ ✓ ✓

Practitioneroutcomes—objectivemeasures

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Secondary measures(practitioner orpatient)

✓ ✓ ✓ ✓

Patient healthoutcomes

✓ ✓ ✓

Economic outcomes ✓

Effectiveness Overall effective inachieving mainoutcomes

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Martin,B.A. 2010

Naunton,M. 2004

Patwardhan, P.D. 2012 Puumalainen,I 2005

Raisch,D.W. 1998

Reeve,J.F. 2008

Sigrist,T. 2002

Thornley,T. 2006

Van de Steeg-vanGompel, C.H. 2011

Watson,M.C. 2002

Study design Randomised controlledtrial (RCT)

✓ ✓ ✓ ✓ ✓

Non-randomisedcontrolled trial

✓ ✓

Controlled beforeand after studies

Other ✓ ✓ ✓

Participantsself-selected

✓ ✓ ✓ ✓ ✓ ✓ ✓

Participantsincluded in theintervention

Pharmacists ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Pharmacy supportstaff (techniciansand assistants)

✓ ✓ ✓ ✓ ✓

Other healthprofessionals

✓

Patients/Public ✓

Intervention type(EPOC taxonomy)

Educational materials ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Educational meetings ✓ ✓ ✓ ✓ ✓

Educational outreachvisits

✓ ✓

Mass media Campaign

Audit and feedback ✓

Watkins

etal.Im

plementation

Science (2015) 10:151

Page12

of23

Table 2 Comparison of studies (Continued)

Reminders ✓

Practice support ✓ ✓ ✓ ✓

Fee for service

Basis forintervention

Improve clinicalknowledge

✓ ✓ ✓ ✓ ✓ ✓

Improvecommunication skills

✓ ✓ ✓ ✓ ✓

Based on addressingorganisational culture

✓

Based on a specifiedtheory of behaviourchange

✓ ✓ ✓

Based on addressingidentified barriers(tailored)

✓

Outcomemeasures

Practitioneroutcomes—subjectivemeasures

✓ ✓ ✓ ✓ ✓ ✓ ✓

Practitioneroutcomes—objectivemeasures

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Secondary measures(practitioner orpatient)

✓ ✓ ✓ ✓ ✓

Patient healthoutcomes

Economic outcomes ✓ ✓

Effectiveness Overall effective inachieving mainoutcomes

✓ ✓ ✓ ✓ ✓ ✓

Key:Educational materials: distribution of educational materials to support guideline-based practice (paper-based, electronic, patient focused, practice tools), disseminated by mail, email or in person. Educational meetings:conferences, lectures or workshopsEducational outreach visits: use of a trained person to meet with health professionals to give information with the intent of changing the professional’s practice, includes academic detailingMass media: varied use of communication that reaches a large number of people including television, radio, newspapers etc. targeted at a population level. Audit and feedback: any summary of clinical performance,which may also include recommendations for clinical actionReminders: interventions involving computer prompts to support practicePractice support: follow-up contact (e.g. visits or phone calls) to provide motivation and support to practitioners post education

Watkins

etal.Im

plementation

Science (2015) 10:151

Page13

of23

inadequacies were due to lack of reporting or flaws inresearch design. Two of the studies evaluated as havingan overall unclear risk of bias used implementationstrategies involving a computer prompt [54, 60].The Newcastle-Ottawa tool for cohort studies was

used to evaluate eight studies [68–75]. The study byEgen showed the least risk of bias with the domainsaround selection of study groups and outcomes deemedlow risk [70]. However, like all of the cohort studies, inthis review, the study design or analysis meant the com-parability of cohorts could be a source of bias. Therewas little to suggest that studies had used measures to

correct for confounding variables in any of the eightstudies evaluated.Use of the two risk of bias tools lead to the observa-

tion that many of the studies in this review were subjectto selection bias and performance bias. Selection biaswas inherent in the self-selection of study participants.Self-selection of participants occurred in 18 of the 22studies, even if randomisation methods were subsequentlyused [54–58, 60–64, 66–69, 71–74]. Performance bias re-lates to the propensity for the studies to be threatened bythe Hawthorne effect, which is the tendency to modifybehaviour when being observed [76, 77]. Simulated patient

Table 3 Results of risk of bias assessment using the EPOC risk of bias tool for RCTs, NRCTs, CBA studies

Watkins et al. Implementation Science (2015) 10:151 Page 14 of 23

methodology is one method that has been used inpharmacy-practice research to avoid the Hawthorne effect[78]. In this review, simulated patient methodology wasused in ten studies [55, 56, 62–64, 66, 68–70, 75].However, in most instances, the studies used the simu-lated patients overtly rather than covertly, which whileethically sound, does not eliminate the risk of bias.

Implementation strategies and their effectivenessThe areas of clinical practice that guideline implementa-tion were designed to impact were varied. They involvedchronic disease states [57, 60, 65, 71, 75], guidelines forthe supply of particular classes of medications [55, 56, 59,61, 64, 66, 68–70, 72], community pharmacy’s role in pre-ventative health [58, 67, 73] and guidelines for appropriatepharmacy practice [63, 74]. The variation did not allow forany conclusions to be made about the effectiveness ofimplementation based on guideline characteristics.Implementation activities were targeted to community

pharmacy, and in particular to pharmacists in 20 studies[54–62, 65–75] and pharmacy support staff in 11 studies[56, 58, 61–64, 66, 68, 69, 72, 75]. In two studies, imple-mentation activities were directed exclusively to pharmacysupport staff, while three studies involved implementationactivities to other health professionals and/or patients as

well as community pharmacy staff [58, 65, 70]. Both thestudies in this review involving comparisons between pro-fessional and support staff demonstrated that better out-comes were achieved when the patient encountered apharmacist [62, 75].Eighteen studies involved multifaceted interventions

[54–58, 60, 62–70, 73–75], and single intervention strat-egies were used in four studies [59, 63, 71, 72]. The multi-faceted nature of the majority of implementationstrategies did not allow for a clear understanding of the ef-fectiveness of individual strategies. Overall, eight differentimplementation strategies were reported in studies in thereview including, use of educational materials, educationalmeetings, educational outreach visits, mass media cam-paigns, audit and feedback, reminders (including CDSS),practice support and fee for service.The most commonly used implementation strategies

were educational interventions. All but two of the stud-ies in the review had an educational component in theirintervention including provision of educational mate-rials, educational meetings or educational outreach visits[54–59, 61–70, 72–75]. Seven studies used educationalstrategies exclusively [59, 63, 65, 72–75]. Of these, twostudies used behaviour change theory to inform theintervention [63, 73] resulting in variable outcomes. The

Table 4 Results of risk of bias assessment using the Newcastle-Ottawa risk of bias tool for cohort studies

De AlmeidaNeto, A.C. 2001Study 5

De AlmeidaNeto, A.C. 2001Study 5A

Egen, V.2003

Guirguis,L.M. 2007

Koster,E.S. 2014

Martin,B.A. 2010

Puumalainen,I. 2005

Thornley,T. 2006

Selection Representativenessof the exposed cohort

– – * – – – * *

Selection of thenon-exposed cohort

* * * * * * * *

Ascertainment ofthe exposure

* * * – * – – *

Demonstration thatoutcome of interestwas not present atstart of the study

* * * – * * – –

Comparability Comparability ofcohorts on thebasis of the designor analysis

Outcome Assessment of outcome * * * – * – – *

Was follow-up longenough for outcomesto occur

– – * * * * – –

Adequacy of follow-up cohorts – – * – – – – –

Reference: Newcastle-Ottowa Quality Assessment ScaleNB, where interventions were directed to more than one group; analysis was only for the component that related to community pharmacyKeyBased on a star system (*) with a range of 0 to 9 stars possible. Three domains are tested:1. Selection of study groups (up to one star allowed for each item)2. Comparability of the groups (up to two stars allowed)3. Outcomes (up to one star allowed for each item)

Watkins et al. Implementation Science (2015) 10:151 Page 15 of 23

remaining five studies achieved modest positive outcomes.Educational outreach visits (sometimes called academic de-tailing) were undertaken in three studies [62, 65, 70]. Oneof the studies by Watson compared educational outreachand education meetings, individually and in combination,against a control [62]. This study did not demonstrateeffectiveness for either strategy.In six studies, educational interventions were combined

with audit and feedback [55, 56, 64, 66, 68, 69]. Benrimojand colleagues authored all of these studies. Five of thestudies (from two papers) were in collaboration with DeAlmeida Neto [55, 56, 64, 68, 69] as the primary author,and one study was in collaboration with Sigrist [66] as theprimary author. The studies using audit and feedbackwere generally effective. All reported positive outcomesexcept one, which had variable outcomes [66].Practice support, in the form of follow-up visits, phone

calls or via provision of practice tools (e.g. checklists,patient handouts, documentation forms), was used inseven studies [55, 56, 58, 60–62]. Only one study usedpractice support as a single intervention [71]. The studywas a pilot study that provided pharmacists with practicetools for diabetes management. The proposed methodwas to implement the tools in conjunction with an educa-tion programme; however, the education programme wasnot available for the pilot study. The use of the practicesupport tools alone resulted in a modest positive outcome.Computer prompts, as reminders to support practice

change, were the used in four studies [54, 57, 60, 67, 71].The studies using this implementation strategy in thecommunity pharmacy setting produced variable out-comes. The sub-studies by Curtain and Reeve [54, 60],which were both part of the larger Pharmacy Recording ofMedication and Services (PROMISe) project [79], demon-strated very strong evidence of effect, whereas the studyby Kradjan [57] demonstrated no effect. The stronglypositive outcomes and comparatively robust methodology,seen in the Curtain [54] and Reeve [60] studies, indicatethe best evidence for effectiveness of an implementationstrategy in the community pharmacy setting.The study by Egen used an intensive mass media cam-

paign along with specific educational interventions directedto pharmacists and gynaecologists [70]. This multifacetedapproach did not demonstrate evidence of effect inpharmacist- or patient-outcome measures.A monetary incentive (fee for service) was only used

in one study as an implementation strategy [57]. How-ever, it was part of a multifaceted implementation, whichalso involved educational interventions and reminders.There was no evidence of effectiveness demonstrated.

Basis for implementation strategiesMost implementation strategies were educational anddesigned to improve knowledge and/or communication

skills. However, this choice of implementation strategywas, in almost all instances, not based on an identifieddeficit in these areas. Tailored implementation strategies,based on addressing identified barriers to successfulguideline implementation, were only evident in twostudies [61, 63]. The barriers identified included, organ-isational barriers, knowledge deficits, social factors suchas patient indifference and suboptimal communicationby non-professional pharmacy staff. Both the studies,which used tailored strategies, were ineffective in produ-cing positive primary outcomes. Two other studies alsoaddressed organisational factors (including workflow andtime constraints), but without mentioning that thechoice of strategy was tailored [58, 71]. Both proved tobe moderately effective. Studies were based on specificbehavioural theorems or frameworks in six instances[55, 56, 58, 63, 66, 73]. The behavioural theories,frameworks and strategies used included “MotivationalInterviewing” techniques [55]; “Stages of Change Model”[55, 56, 66]; “Trans‐theoretical Model for Change” [73];“Social Cognitive Theory” [58]; “Health Beliefs Model”[66]; “Theory of Planned Behaviour” [63]; “Calgary-Cambridge Model of Communication Skills” [63] and“Cognitive Behavioural Therapy” [63] techniques. Despitemost of the studies in this review reporting positive out-comes, two out of the six studies based on behaviourchange models did not [63, 66].

Characteristics of outcome measures including quality ofevidence assessmentsFifteen studies [54–56, 58–60, 64, 65, 67–69, 71–73, 75]reported positive outcomes as a result of clinical guide-lines implementation. Five studies indicated no significantimprovement in primary outcomes measures as a result ofthe implementation strategy [57, 61–63, 70] and two stud-ies reported minimal or variable results [66, 74]. Almost allstudies reported multiple outcomes making assessment ofprimary outcomes difficult. Objective measures were usedto report outcomes in 18 studies [54–56, 59–69, 71, 73–75].Self-reported outcomes, both patient and practitioneroutcomes, were included in 15 studies [54, 55, 57, 58,60–63, 65, 67, 70–74]. In many instances, these out-comes were assessed via novel un-validated tools, modi-fied validated tools or the reporting was inadequate toascertain validity and reliability of the tool. Such mea-sures are of limited benefit in evaluating successfulguideline implementation. Surrogate measures wereused in nine studies and included assessments ofattitudes, self-efficacy, and patient and practitioner sat-isfaction/acceptability of interventions [55, 57, 60, 61,65, 67, 71, 73, 74]. Four studies did not use any objectivemeasures and relied upon self-reported (subjective)measures [57, 67, 71, 74].

Watkins et al. Implementation Science (2015) 10:151 Page 16 of 23

All studies in the review measured process outcomesrelated to changes in the practice of community pharmacystaff. Patient outcomes were only measured in three stud-ies and all relied upon self-reported outcomes usingpatient surveys [54, 57, 70]. Only one of the patientoutcomes measured was based on a health outcome, butit was a surrogate measure for perceived asthma control[57]. The lack of measurement of robust (objective)patient outcomes determines that few conclusions can bemade about the evidence of effectiveness for implementa-tion of guidelines to community pharmacy.Most studies did not comment on sustainability of

outcomes or mentioned it as a limitation of the research.Sustainable practice change was noted in two studies,but the sample sizes were small [71, 73]. Four studiesindicated a decline in outcomes over the course of datacollection [54, 60, 61, 71]. One study indicated thatongoing communication was required for sustainability[75]. Thus there is little evidence that the positiveoutcomes generated by implementation of clinical guide-lines in most of the studies, are sustainable effects.Economic outcomes were determined in three studies

[54, 59, 62]. All the economic evaluations looked atdifferent measurements. These included the cost ofimplementing guidelines [62], cost savings to the healthsystem due to improved guideline adherence [54] andan attempt at a more thorough cost-benefit analysis[59]. No analysis was undertaken to assess the quality ofthe economic modelling undertaken.Six primary outcome measures were agreed upon and

assessed using the GRADE approach [51]. Five outcomeswere assessed as having very low quality of evidence. Oneoutcome was assessed as having low quality of evidence.The main reasons for this were the heterogeneity of thestudies, variability of outcomes and the potential for biasin studies. None of the outcomes were considered to havemoderate or high quality of evidence. Thus, even thoughthe studies demonstrated positive outcomes from clinicalguideline implementation, these outcomes are generallynot a reliable indication of the likely effect. The likelihoodthat the effect will be substantially different is very high(Table 5).

DiscussionThis systematic review expands on the limited researchinto guideline implementation to allied health practi-tioners [13, 16] and the extensive research on guidelineimplementation to medical practitioners in hospitals andthe primary healthcare setting [1, 2, 25, 26]. The findingsof this review indicate that there is a growing body ofevidence on clinical guideline implementation to com-munity pharmacy, but conclusions that can be drawnfrom this evidence are limited. Many studies lack rigourin their methodology and are at risk of substantial bias.

There is also a great deal of variability in the studies inthis field making analysis challenging. The studies todate generally do not provide evidence of a grounded ap-proach to the development of their implementation strat-egies. The strategies employed are mostly multifaceted,with an over-reliance on educational interventions. Report-ing of multiple outcomes complicates assessment of theeffectiveness of clinical guidelines implementation. Thefocus for outcomes is on process and surrogate outcomes,rather than patient outcomes, and the quality of evidencefor outcomes is low. However, despite the limitations inthe research to date, there are indications that clinicalguideline implementation may be of benefit and that CDSSmay be an effective implementation strategy in this setting.In this review, variability was seen in the types of

guidelines and areas of clinical practice, the interventionstrategies utilised and the resultant outcomes, despiteremaining focused in the community pharmacy setting.Other reviewers in implementation science have foundsimilar variability. A recent systematic review, whichonly looked at the implementation of asthma protocols,noted inconsistent results in practitioner and patientoutcomes within the one disease state [80].The lack of rationale in intervention design and over-

reliance on educational implementation strategies are alsonot unique to research in community pharmacy. There is asimilar over-reliance on educational interventions in the lit-erature [17, 18, 22]. This is surprising because educationalinterventions have been demonstrated to be minimallyeffective, particularly if they simply involve passive dissem-ination of information [2, 18, 22, 81]. This review demon-strated a similar minimal effectiveness of educationalstrategies, although it was hard to determine due to themultifaceted nature of most interventions. As the evidencein the literature for a multifaceted approach remains incon-clusive, it would seem sensible, wherever possible, to useless complex interventions [14]. Potentially, less compli-cated interventions would be more cost effective, easier tosustain and better able to inform future practice [14].A clear rationale involves using a tailored approach or the-

oretical framework to inform the implementation strategy.Current consensus in the literature is that implementa-

tion strategies should consider the barriers and facilitators(determinants) of change [2, 16–18, 25, 82]. Althoughresearchers agree on the use of a tailored approach, thereare currently no recommended, reliable ways to identifyand overcome barriers to successful implementation[22, 82]. Krause et al. demonstrated the complexity oftailoring implementation strategies in the area ofchronic disease management [82]. The complexity ofdevising a tailored strategy is what may have resulted inthe small number of studies assessing barriers to imple-mentation in this review. It may also explain the lack ofeffectiveness of tailored implementation strategies

Watkins et al. Implementation Science (2015) 10:151 Page 17 of 23

Table 5 Summary of outcomes and quality of evidence using GRADE

Watkins et al. Implementation Science (2015) 10:151 Page 18 of 23

observed, which seems to challenge the recommenda-tions in the literature. Community pharmacy is a com-plex, retail environment with many staff and possiblymany barriers to clinical guideline implementation. Agreater understanding of the barriers unique to thecommunity pharmacy setting may improve outcomes.The current literature also promotes the use of theory to

underpin study design, but with little knowledge of how tochoose between the many behaviour change theories andhow to successfully translate their constructs into an inter-vention [20, 21, 23, 83]. It is these challenges that may haveresulted in very few studies in this review using a theory-based approach to implementation. Furthermore, the stud-ies in this review that did use theory to guide implementa-tion produced variable results. This is also consistent withthe literature. While there are strong advocates for the useof theory in implementation science, interventions basedon theory are not necessarily more effective [84]. However,there are potential advantages to the generalisability andreplicability of theory-based implementation interventions[20], which would be useful to improving the evidence forguideline implementation in community pharmacy.The implementation strategy to show the most promise

in the community pharmacy setting was CDSS. Two stud-ies using this strategy measured objective outcomes anddemonstrated a strong effect, as well as being compara-tively rigorous in their methodology. This observationholds logical appeal because CDSS has the potential to in-tegrate with the existing workflows of pharmacists [85].As pharmacists routinely use computers in the dispensingprocess, CDSS reminders possibly have fewer barriers toovercome in terms of integration into practice. Legrand etal. demonstrated the superiority of computer-integratedreminders compared to non-integrated information intheir study on medicines and driving [67]. However,further research is required to truly understand the poten-tial of computer prompts in community pharmacy: whattypes of prompts are effective, what variables influenceeffectiveness, the sustainability of such interventions andultimately their effect on patient health outcomes. Sup-porting the evidence of benefit for CDSS in the commu-nity pharmacy setting is the fact that reminders haveimproved outcomes in other settings [2, 25, 86].While the majority of studies in this review reported posi-

tive outcomes, there was variability in the types of outcomemeasures used and the magnitude of the effect, both be-tween and within studies. These findings are consistentwith the current literature for both allied health and themedical profession [2, 6, 13, 16, 20]. Multiple outcomemeasures in studies were common, and this added to thechallenge of interpreting the results. Interpretation of theseresults also requires consideration of the quality of evidenceand whether the research outcomes are valid measures ofclinically significant changes in practice and patient care.

Detection of patient health outcomes can be difficult inthe time frames seen for most studies in this review. Also,many of the studies implemented guidelines for the treat-ment of minor ailments using non-prescription medica-tions. The expected clinical benefit in such instanceswould be small and challenging to measure. Worrall andcolleagues support this notion with their review, whichfound little evidence to suggest that clinical practiceguidelines produce significant changes in clinical out-comes in primary care [26].Understandably, very few of the studies in this review

looked at patient outcomes, and only one study measureda patient health outcome, which was a surrogate measureabout perceptions of asthma control using a non-validatedtool [57]. Consequently, no objective health outcomeswere measured in this review. Given that improvingpatient health outcomes is the main objective in the devel-opment and implementation of clinical guidelines, thecurrent research in community pharmacy is failing to pro-vide substantive evidence of effect. Researchers have noteda similar lack of patient outcome data regarding clinicalguideline implementation in other settings and professions[2, 16, 26, 87]. The main outcome measures seen in theliterature [87] and in this review were process measures,at the level of the practitioner.All the studies in this review assessed practitioner

behaviour change, and most used objective measures toassess practice. While such measures may indicate be-haviour change on the part of the practitioner, they donot necessarily indicate successful guideline implementa-tion, translation of evidence into practice and may notcorrespond with patient outcomes.One of the more interesting observations from this

review was the importance of considering the role of sup-port staff in the community pharmacy setting when imple-menting guidelines. Two studies in this review measuredoutcomes achieved when strategies were directed at pro-fessional pharmacy compared to support staff. Both stud-ies demonstrated that better outcomes were achievedwhen the patient encountered a pharmacist. [62, 75]. Thisobservation has also been substantiated in other pharmacypractice research [88, 89]. The implications of this surro-gate outcome measure lie more in the consideration ofstudy design in the community pharmacy setting. It ispossible that lack of acknowledgement of the influence ofpharmacy support staff, in designing an implementationintervention, could be responsible for some of the variabil-ity in outcomes observed.Another major problem that limits the ability of re-

searchers to make firm conclusions about guideline imple-mentation is the poor methodological quality of studiesand the poor reporting of interventions [2, 10, 13, 22, 26,76, 90]. Unfortunately, this review is no different. Manystudies used self-selection in the recruitment process.

Watkins et al. Implementation Science (2015) 10:151 Page 19 of 23

Self-selection results in samples that are unlikely to berepresentative of the wider population [91]. This selectionbias leads to recruitment of participants who are moti-vated and more likely to adopt practice changes [91].Performance bias was also inherent in many studies in thisreview. Unfortunately, by its very nature, practice-basedresearch is vulnerable to participant-behaviour change dueto the consent and awareness created by the research [13].For controlled trials, selection and performance bias occurin both intervention and control groups. This can lead tosmall differences in effect between groups, which perhapsunderestimates the value of the implementation strategy,but it may also overestimate the absolute effect of anintervention [91]. The concern is that findings do notnecessarily translate to the “real world” [9, 87, 90].

StrengthsThe strength of this review is that it focuses on the singleorganisational setting of community pharmacy, which isunique. A criticism of implementation science has beenthat it can be hard to decipher the components of anintervention and the basis for using a strategy [92]. Thisreview has managed to detail these characteristics for eachstudy, providing a greater understanding of the gaps in re-search. A comprehensive assessment of the quality of allincluded studies and primary outcomes, using three vali-dated tools, also allowed for insight into the strength ofevidence from the studies.

LimitationsA limitation of this narrative synthesis is that of the 22studies synthesised, half were from Australia or the UK,and there were multiple studies by only three researchgroups (Benrimoj, The PROMISe researchers andWatson) [54–56, 60, 62–64, 66, 68, 69]. When theweighting of evidence is clustered in such a way, there isa reduced ability to make generalised conclusions.Conclusions were also limited by the determination thatmost of the studies were at a high risk of bias. However,the authors of the papers in this review were not con-tacted for further information. Such information mayhave helped to better discern methodological quality andprovided stronger evidence. Another concern is that themajority of positive outcomes were the result of publica-tion bias. There are also limitations in a narrative syn-thesis compared to meta-analysis.

Practice implicationsThe findings of this review make an important contribu-tion to the evidence base for the role of clinical guidelinesin the community pharmacy setting. The review has thepotential to guide pharmacy practice research and informsuccessful implementation, of clinical guidelines, in com-munity pharmacy in the future. As highly accessible

primary healthcare practitioners, the scope of practice ofcommunity pharmacists is increasingly important in influ-encing patient health outcomes. In recent years, there hasbeen a focus by governments and non-government orga-nisations on primary healthcare due to the recognisedbenefits to patients and society that a strong primaryhealthcare system generates [28, 93, 94]. Improving theeffectiveness of pharmacists to provide evidence-basedcare through the use of clinical guidelines can strengthenprimary healthcare.

ConclusionWhile this review points to the potential of clinicalguidelines to influence practice in community pharmacy,at present, there is little to suggest that they positivelyaffect patient outcomes. There is also little evidence onthe best strategies for implementation. This lack of evi-dence is not surprising due to the complexity of imple-mentation science, the heterogeneity of studies and thepoor methodological quality of research in this setting.In the future, study design should focus on using a moresystematic approach. More attention should be given tothe rationale of an implementation intervention and thechoice of outcome measures. The community pharmacysetting is unique in the influence that pharmacy supportstaff can have on outcomes achieved by guideline imple-mentation. As a result, careful consideration should begiven to their role in any study design. Improvedmethodological rigour in future research will strengthenthe evidence of the benefits of clinical guidelines andthe best strategies to implement them in communitypharmacy.

Additional file

Additional file 1: Search terms.

AbbreviationsADDOs: Accredited Drug Dispensing Outlets; CBA: controlled before-and-afterTrial; CDSS: computerised clinical decision support system; EPOC: CochraneEffective Practice and Organisation Care Review Group; GRADE: Grading ofRecommendations Assessment, Development and Evaluation; MCAs: medicinecounter assistant; NRCT: non-randomised controlled trial; PICO: participants,interventions, comparators, outcome framework; PRISMA: preferred reportingfor systematic reviews and meta-analyses; RCT: randomised controlled trial;TREND: transparent reporting of evaluations with non-randomised designs.

Competing interestsKW is the proprietor of a community pharmacy in Perth, Western Australiaand thus has a financial interest in community pharmacy.

Authors’ contributionsKW conceptualised and designed this review as part of her PhD studies,guided and supervised by CS, and RC. KW and RC secured funding for theresearch from the Health Department of Western Australia. HW was thesecond reviewer in the team and CS acted as moderator. KW prepared aninitial draft paper with all authors contributing to subsequent drafts andapproving the final manuscript.

Watkins et al. Implementation Science (2015) 10:151 Page 20 of 23

AcknowledgementsWe would like to acknowledge that KW is the recipient of an unrestricted PhDscholarship from AstraZeneca and a grant from the Health Department ofWestern Australia. There are no further funding arrangements for this research.

Author details1School of Medicine and Pharmacology, University of Western Australia,Perth, Australia. 2Faculty of Pharmacy, The University of Sydney, Sydney,Australia.

Received: 17 June 2015 Accepted: 13 October 2015

References1. Francke AL, Smit MC, de Veer AJ, Mistiaen P. Factors influencing the

implementation of clinical guidelines for health care professionals: a systematicmeta-review. BMC Medical Informatics & Decision Making. 2008;8(38) doi:10.1186/1472-6947-8-38.

2. Grimshaw J, Thomas R, MacLennan G, Fraser C, Ramsay C, Vale L, et al.Effectiveness and efficiency of guideline dissemination and implementationstrategies. Health Technol Assess. 2004;8(6):iii–iv.

3. Pilling S. History, context, process, and rationale for the development ofclinical guidelines. Psychol Psychotherapy. 2008;81(Pt 4):331–50. doi:10.1348/147608308x324923.

4. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines:potential benefits, limitations, and harms of clinical guidelines. BMJ (Clinicalresearch ed). 1999;318(7182):527–30.

5. Field MJ, Lohr KN. Clinical practice guidelines: directions for a new program.vol IOM publication; 90–08. Washington D.C: National Academy Press; 1990.

6. Grimshaw J, Russell I. Effect of clinical guidelines on medical practice: asystematic review of rigorous evaluations. Lancet. 1993;342(88883):1317–22.

7. Runciman WB, Hunt TD, Hannaford NA, Hibbert PD, Westbrook JI, CoieraEW, et al. CareTrack: assessing the appropriateness of health care delivery inAustralia. Med J Aust. 2012;197(2):100–5.

8. Grol R, Buchan H. Clinical guidelines: what can we do to increase their use?Med J Aust. 2006;185(6):301–2.

9. Grol R, Grimshaw J. From best evidence to best practice: effectiveimplementation of change in patients’ care. Lancet. 2003;362(9391):1225–30.

10. Colagiuri R. Implementing evidence based guidelines: unlocking the secrets.Diabetes Res Clin Pract. 2009;85(2):117–8. http://dx.doi.org/10.1016/j.diabres.2009.06.011.

11. Fretheim A, Schunemann HJ, Oxman AD. Improving the use of researchevidence in guideline development: 15. Disseminating and implementingguidelines. Health Res Policy Syst. 2006;4:27.

12. National Health and Medical Research Council. In: A guide to thedevelopment, implementation and evaluation of clinical practice guidelines.1999. Canberra: AusInfo Government Info Bookshops.

13. Thomas L, Cullum N, McColl E, Rousseau N, Soutter J, Steen N. Guidelines inprofessions allied to medicine. Cochrane Database Syst Rev. 2009;1:CD000349.

14. Squires JE, Sullivan K, Eccles MP, Worswick J, Grimshaw JM. Are multifacetedinterventions more effective than single-component interventions in changinghealth-care professionals’ behaviours? An overview of systematic reviews.Implement Sci. 2014;9:152. http://www.implementationscience.com/content/9/1/152.

15. Wensing M, van der Weijden T, Grol R. Implementing guidelines andinnovations in general practice: which interventions are effective? Br J GenPract. 1998;48(427):991–7.

16. Hakkennes S, Dodd K. Guideline implementation in allied health professions: asystematic review of the literature. Qual Saf Health Care. 2008;17(4):296–300.

17. Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al.Tailored interventions to address determinants of practice. Cochrane DatabaseSyst Rev. 2015;4:CD005470. http://dx.doi.org/10.1002/14651858.CD005470.pub3.

18. Galgliardi AR, Alhabib S. Members of the Guidelines International NetworkImplementation Working Group. Trends in guideline implementation: ascoping systematic review. Implement Sci. 2015;10:54. doi:10.1186/s13012-015-0247-8.

19. Eccles MP, Grimshaw JM, MacLennan G, Bonetti D, Glidewell L, Pitts NB,et al. Explaining clinical behaviors using multiple theoretical models.Implement Sci. 2012;7:99. doi:10.1186/1748-5908-7-99.

20. ICEBeRG Research Group. Designing theoretically-informed implementationinterventions. Implement Sci. 2006;1:4.

21. Michie S, van Stralen MM, West R. The behaviour change wheel: a newmethod for characterising and designing behaviour change interventions.Implement Sci. 2011;6:42. http://dx.doi.org/10.1186/1748-5908-6-42.

22. Scott SD, Albrecht L, O’Leary K, Ball GD, Hartling L, Hofmeyer A, et al. Systematicreview of knowledge translation strategies in the allied health professions.Implement Sci. 2012;7:70. http://dx.doi.org/10.1186/1748-5908-7-70.

23. Davies P, Walker AE, Grimshaw JM. A systematic review of the use of theoryin the design of guideline dissemination and implementation strategies andinterpretation of the results of rigorous evaluations. Implement Sci.2010;5:14. http://dx.doi.org/10.1186/1748-5908-5-14.

24. Freemantle N. Implementation strategies. Fam Pract. 2000;17 Suppl 1:S7–10.25. Grol R. Implementing guidelines in general practice care. Qual Health Care.

1992;1(3):184–91.26. Worrall G, Chaulk P, Freake D. The effects of clinical practice guidelines on patient

outcomes in primary care: a systematic review. CMAJ. 1997;156(12):1705–12.27. Habeeb Ibrahim AR, Deepak J, Jegan RS. Pharmacists in the wider public

health workforce. Arch Pharm Pract. 2012;3(2):166–9.28. Mossialos E, Naci H, Courtin E. Expanding the role of community pharmacists:

policymaking in the absence of policy-relevant evidence? Health Policy.2013;111(2):135–48. http://dx.doi.org/10.1016/j.healthpol.2013.04.003.

29. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al.The PRISMA statement for reporting systematic reviews and meta-analysesof studies that evaluate health care interventions: explanation andelaboration. J Clin Epidemiol. 2009;62(10):e1–34. http://dx.doi.org/10.1016/j.jclinepi.2009.06.006.

30. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reportingitems for systematic reviews and meta-analyses: the PRISMA statement.Journal of Clinical Epidemiology. 2009;62(10):1006–12. http://dx.doi.org/10.1016/j.jclinepi.2009.06.005.

31. Watkins K, Wood H, Schneider C, Clifford R. Clinical guideline implementationin community pharmacy. (PROSPERO) International prospective register ofsystematic reviews, 2012 CRD42012003019.

32. Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M,et al. The effect of English-language restriction on systematic review-basedmeta-analyses: a systematic review of empirical studies. Int J Technol AssessHealth Care. 2012;28(2):138–44. http://dx.doi.org/10.1017/S0266462312000086.

33. Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a knowledgerepresentation for clinical questions. AMIA. [Evaluation Studies]. 2006;AnnualSymposium Proceedings/AMIA Symposium 2006 359-63. Available from:MEDLINE.

34. Moullin JC, Sabater-Hernández D, Fernandez-Llimos F, Benrimoj SI.Defining professional pharmacy services in community pharmacy.Res Soc Adm Pharm. 2013;9(6):989–95. http://dx.doi.org/10.1016/j.sapharm.2013.02.005.

35. Armour C, Bosnic-Anticevich S, Brillant M, Burton D, Emmerton L, Krass I, et al.Pharmacy Asthma Care Program (PACP) improves outcomes for patients in thecommunity. Thorax. 2007;62(6):496–502.

36. Cochrane Effective Practice and Organisation of Care Group (EPOC). EPOCtaxonomy. Oslo: Norwegian Knowledge Centre for the Health Services; 2015.http://epoc.cochrane.org/epoc-taxonomy.

37. Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S,et al. Tailored interventions to overcome identified barriers to change:effects on professional practice and health care outcomes. CochraneDatabase Syst Rev. 2010;3:CD005470. http://dx.doi.org/10.1002/14651858.CD005470.pub2.

38. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affectingimplementation of health innovations: a systematic review of structural,organizational, provider, patient, and innovation level measures. ImplementSci. 2013;8:22. http://dx.doi.org/10.1186/1748-5908-8-22.

39. Benrimoj SC, Gilbert A, Quintrell N, Neto AC. Non-prescription medicines: aprocess for standards development and testing in community pharmacy.Pharm World Sci. 2007;29(4):386–94.

40. Benrimoj SI, Gilbert AL, de Almeida Neto AC, Kelly F. National implementationof standards of practice for non-prescription medicines in Australia. PharmWorld Sci. 2009;31(2):230–7. http://dx.doi.org/10.1007/s11096-008-9266-8.

41. Mestrovic A, Stanicic Z, Hadziabdic MO, Mucalo I, Bates I, Duggan C, et al.Individualized education and competency development of Croatiancommunity pharmacists using the general level framework. Am J PharmEduc. 2012;76(2):23.

Watkins et al. Implementation Science (2015) 10:151 Page 21 of 23

42. Paphassarang C, Wahlstrom R, Phoummalaysith B, Boupha B, Tomson G.Building the national drug policy on evidence—a cross sectional study onassessing implementation in Lao PDR. Southeast Asian J Trop Med PublicHealth. 2002;33(3):647–53.

43. Cochrane Effective P, Organisation of Care Group (EPOC). What outcomesshould be reported in EPOC reviews? EPOC resources for review authors.Oslo: Norwegian Knowledge Centre for the Health Services; 2013. http://epoc.cochrane.org/sites/epoc.cochrane.org/files/uploads/06%20What%20outcomes%20should%20be%20reported%20in%20EPOC%20reviews%202013%2008%2012_1.pdf.

44. Rothstein JM. On defining subjective and objective measurements. PhysTher. 1989;69(7):577–9.

45. Des Jarlais DC, Lyles C, Crepaz N, Group T. Improving the reporting qualityof nonrandomized evaluations of behavioral and public healthinterventions: the TREND statement. Am J Public Health. 2004;94(3):361–6.

46. The Cochrane Collaboration. Data collection form: Intervention review RCTsand non-RCTs. 2014. http://community.cochrane.org/sites/default/files/uploads/forums/u389/ERC%20data%20collection%20form%20for%20intervention%20reviews%20for%20RCTs%20only.doc.

47. Cochrane Effective Practice and Organisation of Care Group (EPOC).Suggested risk of bias criteria for EPOC reviews. Oslo: NorwegianKnowledge Centre for the Health Services 2013. http://epoc.cochrane.org/sites/epoc.cochrane.org/files/uploads/14%20Suggested%20risk%20of%20bias%20criteria%20for%20EPOC%20reviews%2009%2002%2015.pdf.

48. Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M et al. TheNewcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomisedstudies in meta-analyses. 2011 http://www.ohri.ca/programs/clinical_epidemiology/nosgen.pdf.

49. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviewsof Interventions Version 5.1.0 [updated March 2011] The CochraneCollaboration,. http://handbook.cochrane.org/chapter_9/9_1_4_when_not_to_use_meta_analysis_in_a_review.htm.

50. Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al.Evaluating non-randomised intervention studies. Health Technol Assess.2003;7(27):iii–x. 1-173.

51. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al.GRADE: an emerging consensus on rating quality of evidence and strengthof recommendations. BMJ. 2008;336(7650):924–6. http://dx.doi.org/10.1136/bmj.39489.470347.AD.

52. Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, et al.GRADE guidelines: 4. Rating the quality of evidence—study limitations (riskof bias). J Clin Epidemiol. 2011;64(4):407–15. http://dx.doi.org/10.1016/j.jclinepi.2010.07.017.

53. Cochrane Effective Practice and Organisation of Care Group (EPOC). EPOCworksheets for preparing a summary of findings (SoF) table using GRADE.Oslo: Norwegian Knowledge Centre for the Health Services; 2013.http://epoc.cochrane.org/epoc-specific-resources-review-authors.

54. Curtain C, Peterson GM, Tenni P, Bindoff IK, Williams M. Outcomes of adecision support prompt in community pharmacy-dispensing software topromote step-down of proton pump inhibitor therapy. Br J Clin Pharmacol.2011. doi:10.1111/j.1365-2125.2010.03890.x.

55. De Almeida Neto AC, Benrimoj SI, Kavanagh DJ, Boakes RA. Novel educationaltraining programme for community pharmacists. Am J Pharmaceut Educ.2000;64(Fall 2000):302–7.

56. De Almeida Neto AC, Kelly F, Benrimoj SI. Shaping practice behaviour: noveltraining methodology (study 1). Int J Pharm Pract. 2001;9(3):203–10.

57. Kradjan WA. Patients’ perceived benefit from and satisfaction with asthma-related pharmacy services. J Am Pharm Assoc. 1999;39(5):658.

58. Patwardhan PD, Chewning BA. Effectiveness of intervention to implementtobacco cessation counseling in community chain pharmacies. J Am PharmAssoc. 2012. doi:10.1331/JAPhA.2012.10117.

59. Raisch DW. Effect of a pharmacist-education initiative on ketorolac use andcosts in a Medicaid program. Am J Health Syst Pharm. 1998;55(12):1274.

60. Reeve JF, Tenni PC, Peterson GM. An electronic prompt in dispensingsoftware to promote clinical interventions by community pharmacists: arandomized controlled trial. Br J Clin Pharmacol. 2008. doi:10.1111/j.1365-2125.2007.03012.x.

61. Van de Steeg-van Gompel CH, Wensing M, De Smet PA. Implementation ofpatient education at first and second dispensing of statins in Dutchcommunity pharmacies: the sequel of a cluster randomized trial. BMCHealth Serv Res. 2011;11:313. http://dx.doi.org/10.1186/1472-6963-11-313.

62. Watson MC, Bond CM, Grimshaw JM, Mollison J, Ludbrook A, Walker AE.Educational strategies to promote evidence-based community pharmacypractice: a cluster randomized controlled trial (RCT). Fam Pract. 2002;19(5):529–36.

63. Watson MC, Cleland J, Inch J, Bond CM, Francis J. Theory-basedcommunication skills training for medicine counter assistants to improveconsultations for non-prescription medicines. Med Educ. 2007;41(5):450–9.

64. De Almeida Neto AC, Kelly F, Benrimoj SI. Shaping practice behaviour: noveltraining methodology (study 3). Int J Pharm Pract. 2001;9(3):203–10.

65. Naunton M, Peterson GM, Jones G, Griffin GM, Bleasel MD. Multifacetededucational program increases prescribing of preventive medication forcorticosteroid induced osteoporosis. J Rheumatol. 2004;31(3):550–6.

66. Sigrist T, Benrimoj SI, Hersberger K, Langford J. Changing pharmacists’ andpharmacist assistants' practice in dealing with direct requests for non-prescription analgesics. 2002. doi:10.1111/j.2042-7174.2002.tb00584.x

67. Legrand SA, Boets S, Meesmann U, Verstraete AG. Medicines and driving:evaluation of training and software support for patient counselling bypharmacists. Int J Clin Pharm. 2012;34(4):633–43. http://dx.doi.org/10.1007/s11096-012-9658-7.

68. De Almeida Neto AC, Kelly F, Benrimoj SI. Shaping practice behaviour: noveltraining methodology (study 5). Int J Pharm Pract. 2001;9(3):203–10.

69. De Almeida Neto AC, Kelly F, Benrimoj SI. Shaping practice behaviour: noveltraining methodology (study 5A). Int J Pharm Pract. 2001;9(3):203–10.

70. Egen V, Hasford J. Prevention of neural tube defects: effect of anintervention aimed at implementing the official recommendations. SozPraventivmed. 2003;48(1):24–32.

71. Guirguis LM, Schindel TJ, Johnson JA, Tsuyuki RT. Evaluating pharmacists’experience using diabetes practice tools: a pilot study. Can Pharm J.2007;140(5):312–9. http://cph.sagepub.com/content/140/5/312.full.

72. Koster ES, Walgers JCD, van Grinsven MCJ, Winters NA, Bouvy ML.Adherence to national recommendations for safe Methotrexate dispensingin community pharmacies. J Manag Care Pharm. 2014;20(2):194–200.

73. Martin BA, Bruskiewitz RH, Chewning BA. Effect of a tobacco cessationcontinuing professional education program on pharmacists’ confidence,skills, and practice-change behaviors. J Am Pharm Assoc (2003).2010;50(1):9–16. http://dx.doi.org/10.1331/JAPhA.2010.09034.

74. Puumalainen I, Kansanaho H, Varunki M, Ahonen R, Airaksinen M. Usefulnessof the USP medication counselling behavior guidelines. Pharm World Sci.2005;27(6):465–8.

75. Thornley T, Gray N, Anderson C, Eastham S. A study to investigate the extent ofdelivery of an intervention in asthma, in a UK national community pharmacychain, using mystery customers. Patient Educ Couns. 2006;60(2):246–52.

76. Grimshaw J, Campbell M, Eccles M, Steen N. Experimental and quasi-experimental designs for evaluating guideline implementation strategies.Fam Pract. 2000;17(Supplement):s11–6.

77. McCambridge J, Witton J, Elbourne DR. Systematic review of the Hawthorneeffect: new concepts are needed to study research participation effects. J ClinEpidemiol. 2014;67(3):267–77. http://dx.doi.org/10.1016/j.jclinepi.2013.08.015.

78. Watson M, Skelton J, Bond C, Croft P, Wiskin C, Grimshaw J, et al. Simulatedpatients in the community pharmacy setting. Pharm World Sci. 2004;26:32–7.

79. Peterson G, Tenni P, Jackson S, Bereznicki L, Hughes J, Kong D, et al.Documenting Clinical Interventions in Community Pharmacy:PROMISe III 2010.

80. Dexheimer JW, Borycki EM, Chiu KW, Johnson KB, Aronsky D. A systematicreview of the implementation and impact of asthma protocols. BMC MedInformDecision Making. 2014;14:82. http://dx.doi.org/10.1186/1472-6947-14-82.

81. Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F et al.Continuing education meetings and workshops: effects on professionalpractice and health care outcomes. Cochrane Database of Systematic Reviews.2009(2):CD003030. http://dx.doi.org/10.1002/14651858.CD003030.pub2.

82. Krause J, Van Lieshout J, Klomp R, Huntink E, Aakhus E, Flottorp S et al.Identifying determinants of care for tailoring implementation in chronicdiseases: an evaluation of different methods. Implement Sci. 2014;9:102.http://dx.doi.org/10.1186/s13012-014-0102-3.

83. French SD, Green SE, O’Connor DA, McKenzie JE, Francis JJ, Michie S et al.Developing theory-informed behaviour change interventions to implementevidence into practice: a systematic approach using the Theoretical DomainsFramework. Implement Sci. 2012;7:38. http://dx.doi.org/10.1186/1748-5908-7-38.

84. Nilsen P. Making sense of implementation theories, models and frameworks.Implement Sci. 2015;10:53. http://dx.doi.org/10.1186/s13012-015-0242-0.

85. Mandt I, Horn AM, Ekedahl A, Granas AG. Community pharmacists’prescription intervention practices—Exploring variations in practice in

Watkins et al. Implementation Science (2015) 10:151 Page 22 of 23

Norwegian pharmacies. Research in Social and Administrative Pharmacy.2010;6(1):6–17. http://dx.doi.org/10.1016/j.sapharm.2009.03.003.

86. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ,Beyene J, et al. Effects of computerized clinical decision support systems onpractitioner performance and patient outcomes: a systematic review. JAMA.2005;293(10):1223–38.

87. Hakkennes S, Green S. Measures for assessing practice change in medicalpractitioners. Implement Sci. 2006;1:29.

88. Schneider CR, Everett AW, Geelhoed E, Kendall PA, Murray K, Garnett P, et al.Provision of primary care to patients with chronic cough in the communitypharmacy setting. Ann Pharmacother. 2011;45(3):402–8.

89. Watson M, Bond C, Grimshaw J, Johnston M. Factors predicting theguideline compliant supply (or nonsupply) of non-prescription medicines inthe community pharmacy setting. Quality Safety Health Care. 2006;15:53–7.doi:10.1136/qshc.2005.014720.

90. Foy R, Eccles M, Grimshaw J. Why does primary care need more implementationresearch? Fam Pract. 2001;18(4):353–5.

91. Grol R, Jones R. Twenty years of implementation research. Fam Pract.2000;17 Suppl 1:S32–5.

92. Tavender EJ, Bosch M, Gruen RL, Green SE, Michie S, Brennan SE et al.Developing a targeted, theory-informed implementation intervention usingtwo theoretical frameworks to address health professional andorganisational factors: a case study to improve the management of mildtraumatic brain injury in the emergency department. Implement Sci.2015;10:74. http://dx.doi.org/10.1186/s13012-015-0264-7.

93. Australian Government Department of Health and Ageing. Primary HealthCare Reform in Australia: Report to Support Australia’s First National PrimaryHealth Care Strategy. Commonwealth of Australia 2009. http://apo.org.au/files/Resource/NPHC-supp.pdf.

94. World Health Organization. The World Health Report 2008: Primary HealthCare: Now More than Ever. 2008. http://www.who.int/whr/2008/whr08_en.pdf.

95. Watson MC, Norris P, Granas AG. A systematic review of the use ofsimulated patients and pharmacy practice research. International Journal ofPharmacy Practice. 2006;14(2):83–93. http://dx.doi.org/10.1211/ijpp.14.2.0002.

Submit your next manuscript to BioMed Centraland take full advantage of:

• Convenient online submission

• Thorough peer review

• No space constraints or color figure charges

• Immediate publication on acceptance

• Inclusion in PubMed, CAS, Scopus and Google Scholar

• Research which is freely available for redistribution

Submit your manuscript at www.biomedcentral.com/submit

Watkins et al. Implementation Science (2015) 10:151 Page 23 of 23