Vaccine 27 (2009) 792–802

Contents lists available at ScienceDirect

Vaccine

journa l homepage: www.e lsev ier .com/ locate /vacc ine

The comparative evaluation of expanded national immunization policiesin Korea using an analytic hierarchy process

Taeksoo Shina,1, Chun-Bae Kimb,∗, Yang-Heui Ahnc,2, Hyo-Youl Kimd,3, Byung Ho Chae,4,Young Uhf,5, Joo-Heon Leeg,6, Sook-Jung Hyunb,7, Dong-Han Leeh,8, Un-Yeong Goh,8

a Department of Management Information Systems, College of Government & Business, Yonsei University, 234 Maeji, Heungup, Wonju City, Gangwon-do 220-710,Republic of Koreab Department of Preventive Medicine and Institute of Lifelong Health, Yonsei University Wonju College of Medicine, 162 Ilsan-dong, Wonju-city, Gangwon-do 220-701,Republic of Koreac Department of Nursing, Yonsei University Wonju College of Medicine, 162 Ilsan-dong, Wonju-city, Gangwon-do 220-701, Republic of Koread Department of Internal Medicine (Infectious Disease), Yonsei University Wonju College of Medicine, 162 Ilsan-dong, Wonju-city, Gangwon-do 220-701, Republic of Koreae Department of Pediatrics, Yonsei University Wonju College of Medicine, 162 Ilsan-dong, Wonju-city, Gangwon-do 220-701, Republic of Koreaf Department of Laboratory Medicine, Yonsei University Wonju College of Medicine, 162 Ilsan-dong, Wonju-city, Gangwon-do 220-701, Republic of Koreag Department of Business Administration, College of Government & Business, Yonsei University, 234 Maeji, Heungup, Wonju City, Gangwon-do 220-710, Republic of Koreah Division of Vaccine Preventable Disease Control and National Immunization Program, Korea Centers for Disease Control and Prevention, 194 Tongil-lo, Eunpyung-gu,Seoul 122-701, Republic of Korea

a r t i c l e i n f o

Article history:Received 1 September 2008Received in revised form 15 October 2008Accepted 20 October 2008Available online 17 November 2008

Keywords:

a b s t r a c t

The purpose of this paper is to propose new evaluation criteria and an analytic hierarchy process (AHP)model to assess the expanded national immunization programs (ENIPs) and to evaluate two alternativehealth care policies. One of the alternative policies is that private clinics and hospitals would offer freevaccination services to children and the other of them is that public health centers would offer thesefree vaccination services. Our model to evaluate the ENIPs was developed using brainstorming, Delphitechniques, and the AHP model. We first used the brainstorming and Delphi techniques, as well as literature

National immunization programAnalytic hierarchy process modelDelphi methodB

reviews, to determine 25 criteria with which to evaluate the national immunization policy; we thenproposed a hierarchical structure of the AHP model to assess ENIPs. By applying the proposed AHP modelto the assessment of ENIPs for Korean immunization policies, we show that free vaccination services

vate c

1

hd

acdl

T

0d

rain storming should be provided by pri

. Introduction

National immunization programs have an important role in theealth status of the population, because there are many infectiousiseases that are prevented through compulsory immunization.

∗ Corresponding author. Tel.: +82 33 741 0344; fax: +82 33 747 0409.E-mail addresses: tsshin@yonsei.ac.kr (T. Shin), kimcb@yonsei.ac.kr (C.-B. Kim),

hn57@yonsei.ac.kr (Y.-H. Ahn), hyksos@yonsei.ac.kr (H.-Y. Kim),ha12bho@yonsei.ac.kr (B.H. Cha), u931018@yonsei.ac.kr (Y. Uh),ardenlee@yonsei.ac.kr (J.-H. Lee), hsj119@yonsei.ac.kr (S.-J. Hyun),

dhmd@cdc.go.kr (D.-H. Lee), unyng@cdc.go.kr (U.-Y. Go).1 Tel.: +82 33 760 2335; fax: +82 33 763 4324.2 Tel.: +82 33 741 0383; fax: +82 33 742 1434.3 Tel.: +82 33 741 1206; fax: +82 33 745 6461.4 Tel.: +82 33 741 0561; fax: +82 33 748 2709.5 Tel.: +82 33 741 1592; fax: +82 33 731 6953.6 Tel.: +82 33 760 2317; fax: +82 33 763 4324.7 Tel.: +82 33 741 0347; fax: +82 33 747 0409.8 Tel.: +82 2 380 1446; fax: +82 2 352 8235.

nameimm

aapaapp

gd

264-410X/$ – see front matter. Crown Copyright © 2008 Published by Elsevier Ltd. All rioi:10.1016/j.vaccine.2008.10.075

linics and hospitals rather than public health centers.Crown Copyright © 2008 Published by Elsevier Ltd. All rights reserved.

here are many available strategies that work to increase immu-ization coverage, but there are some that do not work as wells others. The value of a strategy depends upon how it is imple-ented, its potential effectiveness, and how well it is matched to

xisting problems. Some strategies do not work because they arell-conceived from the beginning or because they are poorly imple-

ented. Strategies that address the wrong problems are simplyisdirected [1].Effective program evaluation is a systematic method to improve

nd account for public health actions by involving procedures thatre useful, feasible, ethical, and accurate. The framework guidesublic health professionals in their evaluation of programs. It ispractical, non-prescriptive tool that is designed to summarize

nd organize the essential elements of the program evaluation

rocedure. The framework comprises steps in program evaluationractice and standards for effective program evaluation [2].

A well-designed strategy to evaluate health interventionenerally involves a formative evaluation during the project’sevelopmental phase, process evaluations throughout the project

ghts reserved.

ine 27

ieobcprlh

1(natKrfri

Mhnturnfc

aeAhov

2

aaaipaeagud

pim

raRt

n

pimqtbpla

iasaPsitab

2

epeptdct

ganeho

efttap

2

isfo

ip

T. Shin et al. / Vacc

mplementation phase, and an impact evaluation to assess the netffects of the intervention [3]. In particular, estimating the costsf health interventions9 is important to policy-makers for a num-er of reasons including the fact that the results can be used as aomponent in the assessment and improvement of health systemerformance. For example, the costs can be used to assess if scarceesources are being used efficiently or whether they can be real-ocated in a way that would lead to improvements in populationealth [4].

Since the vaccination was first introduced in Korea in the960s, the most vaccines including the measles–mumps–rubellaMMR) vaccine have been incorporated into the national immu-ization program (NIP) in 1985 [5,6]. To date, the development ofnational immunization strategy in Korea has been conducted by

he Korea Centers for Disease Control and Prevention. The NIP inorea covers diseases including tuberculosis, hepatitis B, diphthe-ia, pertussis, tetanus, polio, MMR, Japanese encephalitis, typhoidever, influenza, chickenpox (varicella) and hemorrhagic fever withenal syndrome. In the private sector, vaccinations for Haemophilusnfluenza type b (Hib) are also provided.

Vaccines available from the NIP include BCG, HBV, DTaP, OPV,MR, JEV, and oral or parenteral vaccines against typhoid fever and

emorrhagic fever with renal syndrome. The standards for immu-ization practices are in accordance with the recommendations ofhe Korea Advisory Committee on Immunization Practices (KACIP)nder the Ministry for Health, Welfare and Family Affairs. Someegions in Korea have recently executed expanded national immu-ization programs (ENIPs) that were oriented to private medical

acilities in 2005 [7,8] and ENIPs that were oriented to public healthenters in 2006 [9].

The purpose of this paper is to propose new evaluation criteriand an analytic hierarchy process (AHP) model to assess ENIPs andvaluate two alternative health care policies of the ENIPs using theHP model. One of the alternative policies is that private clinics andospitals would offer free vaccination services to children and thether of them is that public health centers would offer these freeaccination services.

. Evaluation models of health care policies

Several methodologies need to be utilized selectively to developnd convey the information that is most pertinent to the goalsnd objectives of the particular health care project being evalu-ted. The challenge here is to develop an evaluative strategy thats most suitable to (1) the intervention’s scope, design, and pur-ose; (2) its potential for being scaled up or replicated elsewhere;nd (3) the potential value of the information that will be gen-rated from the evaluation. When assessing the health outcomesnd system impacts of health interventions, the challenge is thereatest when the assumed benefits of many interventions remainnproven, and when methodological issues, evaluative capacity,ata, and resource constraints make impact evaluations difficult [3].

Immunization program planning is a key component of the pro-osed national immunization strategy. The goal of this component

s to use common criteria to support collaborative, national assess-ent, and prioritization of new vaccines [10].Albrecht and Perry [11] conducted a study to identify priority

esearch questions regarding home healthcare nursing by usingbrainstorming approach. Another recent study initiated by the

esearch Committee of the Home Healthcare Nurses Association inhe United States used the Delphi technique to determine research

9 In this study, we used such factors as “adequacy of repayment process of vacci-ation costs” and “economic satisfaction (immunization costs, indirect costs, etc.)”.

faDg

fieo

(2009) 792–802 793

riorities based on the perspectives of home healthcare nurse clin-cians [12,13]. Chung et al. [14] developed a strategic evaluation

odel for public informatization projects that aim to improve theuality of life and national competitiveness of people working inhe healthcare sector. Their evaluation model was developed as aalanced scorecard (BSC) model that was based on performanceerspective, user perspective, information system perspective, and

earning and growth perspective; the last three perspectives canlso be used to monitor informatization projects.

Many researchers in the health care field, including those work-ng on national immunization policies, have developed a systematicpproach to planning and evaluation of intervention [15,16]. Thistudy reviews the following three different evaluation models thatre used to assess national immunization policies. The logic andRECEDE–PROCEED models are basically a process-oriented andystematic (input-process-output) model for evaluating nationalmmunization policies. The main structure of our proposed evalua-ion models was based on this systematic approach. This structure islso organized hierarchically using AHP method and then evaluatedy expert groups.

.1. Logic model

The logic model is a beneficial evaluation tool that facilitatesffective program planning, implementation, and evaluation ofublic health programs. A logic model describes the sequence ofvents required to bring about change by synthesizing the mainrogram elements into a picture of how the program is supposedo work. The most basic logic model uses words and/or pictures toescribe the sequence of activities that are required to bring abouthange and illustrates how these activities are linked to the resultshat the program is expected to achieve [17].

The assumptions and expectations about how and why a pro-ram will solve a particular problem, generate new possibilities,nd make the best use of valuable assets are fundamentals thateed to be taken into consideration by the stakeholder to achieve anffective evaluation and program success. The logic model approachelps create a shared understanding of the goals and methodologyf the program; it also relates activities to projected outcomes.

Multiple indicators are needed to track the implementation andffects of a program, but defining too many indicators may detractrom the goals of the evaluation. One approach to developing mul-iple indicators is based on the program logic model (developed inhe second step of the evaluation). The logic model can be used astemplate to define a spectrum of indicators that lead from the

rogram activities to the expected effects [2,18,19].

.2. PRECEDE–PROCEED model

One of the most extensively used models in the health care fields the PRECEDE–PROCEED model, because it provides a comprehen-ive structure for assessing health and quality-of-life needs and alsoor designing, implementing, and evaluating health promotion andther public health programs to meet these needs.

PRECEDE (Predisposing, Reinforcing, and Enabling Constructsn Educational Diagnosis and Evaluation) outlines a diagnosticlanning process to assist in the development of targeted andocused public health programs. PROCEED (Policy, Regulatory,nd Organizational Constructs in Educational and Environmentalevelopment) guides the implementation and evaluation of pro-

rams designed using PRECEDE [20].

The PRECEDE–PROCEED model consists of nine phases. Therst five phases are diagnostic. These phases assess the social,pidemiological, behavioral and environmental, educational andrganizational, and administrative and policy issues that are

7 ine 27

rpeao

2

ciadhaiadmt

adsctcst0

mre[aa

stt

3

3

tcwtpE

rwphA

sc

Uf

iffp

94 T. Shin et al. / Vacc

elevant to the health education program. The remaining fourhases address the implementation and evaluation of the healthducation program. The PRECEDE–PROCEED model has encour-ged and facilitated more systematic and comprehensive planningf public health programs.

.3. Analytic hierarchy process (AHP) method

The AHP method [21] is a widely used method for analyzingomplex discrete alternative decision problems with multiple qual-tative criteria. In an AHP, the decision problem is decomposed intotree-like hierarchical structure, with the overall goal at the top andiscrete alternatives at the bottom. The intermediate levels of theierarchy represent lower level criteria that contribute to the over-ll goal. The AHP is designed to cope with both the rational and thentuitive when selecting the best of a number of alternatives thatre evaluated according to several criteria. During this process, theecision-maker carries out only simple, pairwise, comparison judg-ents that are then used to develop overall priorities for ranking

he alternatives.The benefit of AHP is in the formalization of the structuring and

ssessment of all the factors and their interactions in a decisionomain. The great strength of AHP is also its ability to incorporateystematic checks on the consistency of judgments in the pairwiseomparison matrices; other approaches to multi-attribute decisionheory fail to incorporate consistency checks [22]. The ratio of theonsistency index (C.I.) to the average random index (R.I.) of theame order matrix is called the consistency ratio (C.R.). A consis-ency ratio of 0.10 or less is considered acceptable, and a ratio of.20 or less may be tolerated [23].

The application of the AHP to support group decisions in theedical and health care fields has proven to be useful in several

esearch studies [24–28]. There has been a recent increased inter-st in applying the AHP to the evaluation of health care facilities27,29,30]. A total of 50 articles have applied the AHP to medicalnd health care decision making between 1981 and 2006. Liber-tore and Nydick [28] have illustrated that AHP is a promising

stitw

Fig. 1. Development process of the evaluation model

(2009) 792–802

upport tool for shared decision making between patients and doc-ors, the evaluation and selection of therapies and treatments, andhe evaluation of health care technologies and policies.

. Research methodology

.1. Developing an AHP model to assess ENIPs

This study developed new evaluation criteria and an AHP modelo assess ENIPs and evaluated two alternative health care poli-ies of ENIPs using the AHP model. The alternative policies werehether private clinics and hospitals should provide free vaccina-

ion services to children or whether this should be provided byublic health centers. The AHP development process to evaluateNIP models is summarized in Fig. 1.

The decision group formulated the AHP decision hierarchyequired for the selection structure of the model. The participantsere advised to thoroughly evaluate all of the principal criteriarior to their application because the development of a decisionierarchy is very important for the successful application of theHP.

To identify both the criteria and subcriteria for the policyelection of national health programs using the AHP model, weonducted a survey to enumerate the critical success factors.

The AHP modeling process involves six phases as shown in Fig. 1.sing this six-phase approach, we first formulated an AHP model

or the policy selection of national health programs.First, the theoretical and prior research related to this study

s outlined and an empirical investigation was conducted with aew expert groups (Phases 1 through 4 in Fig. 1). Then, the factorsor evaluating the alternatives were generated with healthcarerofessionals using multiple qualitative methods including brain-

torming and Delphi methods. The idea that interacting groupsend to produce a larger numbers of ideas when brainstormings used is well accepted. Brainstorming is based on the premisehat within groups, members hear the ideas of others that theyould not have otherwise thought of themselves. In principle,

for expanded national immunization programs.

T. Shin et al. / Vaccine 27 (2009) 792–802 795

h inst

bastf

ptAsmibw

3

efp

camalopnsceba

ocn

etlithgtt

dgfinetiiitupiie

Fig. 2. Hierarchical structure of the AHP model for choosing whic

rainstorming requires members to initially state as many ideass possible and this includes improving on or merging previouslytated ideas [31,32]. The Delphi method was also used in this studyo design AHP questionnaires completed by a few expert groupsrom widely distributed geographic areas of Korea.

All of the criteria (Phases 5–6 in Fig. 1) are compared in aairwise manner when the AHP technique is used. The basic func-ions of the AHP include structuring, measuring, and synthesis. TheHP aims to structure a hierarchy that contains a goal or missiontatement, objectives or criteria, and alternatives. The AHP alsoeasures the ratio scale properties (i.e. weights) for elements (or

nputs) in the model, which is referred to as prioritization. The otherasic function of AHP is synthesis, which involves summarizing theeights of all of the alternatives according to their criteria.

.2. Decision criteria selection for decision making

Monitoring of inputs, outputs, and immediate outcomes is anvaluative strategy expected of nearly all projects and is neededor program accountability, continuing investment decisions, androgram improvement [3].

A system that collects data on the structure, process, and out-omes of an immunization program can identify groups of children,dolescents, and adults at risk of vaccine-preventable diseases,onitor racial and ethnic disparities in vaccine coverage, evalu-

te the effectiveness of programs designed to increase coverageevels, track uptake of new vaccines, assess differential impactsf vaccine shortages, measure performance by various types ofroviders, and provide insight on sociodemographic and attitudi-

al factors associated with vaccination. Such information that theystem for an immunization program provides is used by healthare providers, program designers, policymakers, and parents tonhance current strategies and develop new strategies to provideetter protection against vaccine-preventable diseases in people ofll ages [33].

Ttipt

itute should offer an expanded national immunization program.

However, within the current system, even though expert rec-mmendations regarding immunizations are made, there is nooordinated, national mechanism for assessing and prioritizingew service system of immunization program.

Therefore, this study suggests a hierarchical evaluation model ofxpanded national immunization policies using AHP. In this study,he decision hierarchy consists of five levels, which are a goal, threeevels of criteria, and alternatives. These elements are representedn a tree-like structure. The hierarchy represents the structure ofhe decision problem and forms the basis of the comparisons thatave to be made in the following phases. In this paper, the primaryoal is to choose an alternative for ENIPs. The hierarchy includeshree levels of criteria (criteria, first subcriteria, and second subcri-eria).

In our study, the three major evaluation criteria of ENIP areefined as program infrastructure, process, and outcome. The pro-ram infrastructure stage evaluates the personal, physical, andnancial resources needed to implement a program. A strongational immunization infrastructure is critical to assure the deliv-ry of the vaccine, to conduct outreach and education programs,o maintain coverage rates, and to achieve outbreak control, butt also requires adequate funding. Increased support for nationalmmunization programs and targeted educational campaigns aremportant to increase the demand for vaccines and to ensure thathere is an adequate supply [34]. The program process stage eval-ates the feasibility and adequacy of the program. In this programrocess, we identified issues related to project management and

mplementation that can be used to improve program quality bothmmediately and in the future. Finally, the program outcome stagevaluation determines the efficiency of disseminating the program.

his includes an assessment of the vaccination coverage levels,he adequacy of the immunization reports, the immunization reg-stration rates, and the timeliness and up-to-date status of therogram. In our AHP model the critical factors of the first subcri-eria are classified into six major groups: (1) investment resources

796 T. Shin et al. / Vaccine 27 (2009) 792–802

Table 1The distribution and consistency ratios of AHP questionnaire responders accordingto the expert group.

Expert groups Responders (%) Consistency ratio

Consultation professor group (pediatrics,infectious diseases division) forimmunization (PD/INF)

10 (11.4) 0.08

Participating institutes (private medicalfacilities) of ENIP (PMF)

40 (45.4) 0.09525

Implementing institutes (public healthcenter, etc.) of ENIP (PHC)

18 (20.5) 0.12778

Research group (2005/2006) of ENIP 15 (17.0) 0.12067Decision-maker/execution group (National

Assembly/KCDC) of ENIP5 (5.7) 0.056

T

Pp

fmagthbg

Table 2The importance (relative weights) of group opinions.

Expert groups Group importance Weighting (%)

Consultation professor group(pediatrics, infectious diseasesdivision) for immunization(PD/INF)

5.71 21.0

Participating institute (privatemedical facilities; PMF)

5.66 21.0

Implementing institute (publichealth center; PHC)

5.47 20.0

Research group (2005/2006) 5.41 20.0Decision-maker/execution group 4.86 18.0

T

Pa

4

4

The expert groups included experts in vaccine-preventable

TA

L

P

P

P

otal 88 (100.0)

D: pediatrics; INF: infection medicine; ENIP: expanded national immunizationrogram; KCDC: Korea Centers for Disease Control and Prevention.

or building infrastructure, (2) participation systems within the com-unity, (3) adequacy of the service provided, (4) adequacy of the

dministrative procedure, (5) immunization output, and (6) pro-ram performance. The second subcriteria in the fourth level of

he hierarchy were then divided into 25 critical factors. Given theierarchy illustrated in Fig. 2, the priority setting procedure beginsy pairwise comparison of the criteria with respect to the overalloal.

domh

able 3nalyzing the relative importance of evaluation criteria using an AHP model.

evel 1 criteria Level 2 criteria (first subcriteria)

rogram infrastructure (Globalweight: 0.3670; ImportanceRank: 2)

Investment resources for buildinginfrastructure (Global weight: 0.2299;Importance Rank: 1)

Participation system within community(Global weight: 0.1371; Importance Rank: 5)

rogram process (Global weight:0.2544; Importance Rank: 3)

Adequacy of service provided (Global weight:0.1773; Importance Rank: 3)

Adequacy of administrative procedures (Globalweight: 0.0770; Importance Rank: 6)

rogram outcome (Global weight:0.3786; Importance Rank: 1)

Immunization output (Global weight: 0.1673;Importance Rank: 4)

Program performance (Global weight: 0.2113;Importance Rank: 2)

Total

(National Assembly/KCDC)otal 27.11 100.0

D: pediatrics; INF: infection medicine; KCDC: Korea Centers for Disease Controlnd Prevention.

. Empirical results

.1. The validity of a hierarchical structure for assessing ENIPs

iseases and vaccination programs from academic and researchrganizations, federal agencies, and local public health depart-ents. They investigated the validity of the criteria and the

ierarchical structure of the AHP model.

Level 3 criteria (second subcriteria) Priorities (globalweights)

Importancerank

Governmental budget for program 0.0533 5Specialty and adequacy of manpower 0.0523 6Construction of stable demand andsupply system of immunization vaccine

0.0458 9

Feasibility of plan of demonstrationprogram

0.0373 14

Construction of service system forimmunization program

0.0412 11

Networks and partnership betweenpublic health centers and privatemedical facilities

0.0704 2

Methods leading to participation fromcommunity residents

0.0353 17

Methods and strategies of program’s PR 0.0314 20

Vaccine adverse event reporting 0.0860 1Time required for immunization 0.0358 15Recall and remind services 0.0555 4Adequacy of selection ofunder-privileged classes

0.0152 23

Adequacy of problem-solving method 0.0138 25Application of immunizationmanagement database

0.0174 21

Adequacy of repayment process ofvaccination costs

0.0164 22

Compliance with immunizationguidelines

0.0142 24

Immunization coverage, etc. 0.0356 16Adequacy of immunization reports 0.0347 18Immunization registration rate 0.0378 13Improvement of timeliness andup-to-date immunization coverage

0.0593 3

Qualitative satisfaction with service 0.0454 10Geographical accessibility 0.0323 19Economic satisfaction (immunizationcosts, indirect costs, etc.)

0.0478 7

Participation rates of communityresidents and private medical facilities,etc.

0.0469 8

Efficiency of program (input versusoutput)

0.0390 12

1.000

T. Shin et al. / Vaccine 27

Fig. 3. Analyzing the relative importance of evaluation criteria using an AHP model(level 1).

arsbaqard

eOct1t

Fig. 4. Analyzing the relative importance of evalu

Fig. 5. Analyzing the relative importance between e

(2009) 792–802 797

The validity of the criteria and the hierarchical structure wasnalyzed through six rounds of questionnaires. In the first threeounds, expert panel members were summoned to brainstormingessions. In the other three rounds, another panel of expert mem-ers were provided with questionnaires and requested to reviewnd return questionnaires using the Delphi technique. After theuestionnaires were returned, we built the decision hierarchy byssociating a subset of attributes with each criterion identified. Theesulting hierarchy, which is depicted in Fig. 2, classified all of theecision criteria into five levels.

The AHP survey was conducted in five expert groups over sev-ral independent sessions within a span of 12 weeks starting from 9

ctober 2006 to 12 January 2007. The numerical scale used to makeomparisons ranges from 1 to 9, where a 1 denotes equal impor-ance and 9 denotes the highest degree of importance. In our study,46 participants selected randomly from different areas respondedo this survey but our study only used the 88 of the survey results

ation criteria using an AHP model (level 2).

valuation criteria using AHP model (level 3).

798 T. Shin et al. / Vaccine 27

Table 4Priorities by alternative policy (alternative preference).

A

PP

tat

ci

4

a

io[

maaaag

ccalculated by aggregating them according to the following two-step

TG

L

P

P

P

P

lternative Priorities (alternative preference)

rivate medical facilities (PMF) 0.5590ublic health centers (PHC) 0.4410

o meet the consistency requirements of the AHP. The distributionnd consistency ratios of AHP questionnaire responders accordingo their expert group is shown in Table 1.

The overall priority of alternative national health programs isalculated by multiplying the local priorities with their correspond-ng weights along the hierarchy.

.2. Aggregating AHP weights for group decision making

In a group setting, there are several ways to include the viewsnd judgments of each person in the priority setting process, which

a

ga

able 5ap analysis of alternative priorities (alternative preferences) based on evaluation criteria

evel 1 criteria Level 2 criteria (first subcriteria) Levelsubcr

rogram infrastructure (PMF:0.2023; PHC: 0.1545; Gap:0.0478; Gap Rank: 2)

Investment resources for buildinginfrastructure (PMF: 0.1150; PHC:0.0997; Gap: 0.0152; Gap Rank: 4)

GoveprogrSpecmanpConsand simmFeasidemoConsfor im

Participation systems within thecommunity (PMF: 0.0873; PHC:0.0547; Gap: 0.0326; Gap Rank: 2)

Netwbetwand pMethfromMethprogr

rogram process (PMF: 0.1225;PHC: 0.1430; Gap: 0.0205; GapRank: 3)

Adequacy of service providing(PMF: 0.0892; PHC: 0.0948; Gap:0.0056; Gap Rank: 6)

VaccTimeReca

Adequacy of administrativeprocedure (PMF: 0.0333; PHC:0.0481; Gap: 0.0149; Gap Rank: 5)

AdequndeAdeqmethApplmanaAdeqvacciCompguide

rogram outcome (PMF: 0.2342;PHC: 0.1436; Gap: 0.0906; GapRank: 1)

Immunization output (PMF:0.0967; PHC: 0.0705; Gap: 0.0262;Gap Rank: 3)

ImmAdeqreporImmImprup-tocover

Program performance (PMF:0.1374; PHC: 0.0730; Gap: 0.0644;Gap Rank: 1)

QualserviGeogEcon(immcostsPartiresidfaciliEfficiversu

Total

MF: private medical facilities; PHC: public health centers; Gap: the absolute value of dif

(2009) 792–802

nclude (1) consensus, (2) vote or compromise, (3) geometric meanf the individuals’ judgments, and (4) separate models or players35].

The participants were categorized into five groups in the AHPodel, which consisted of a consultation professor group (pedi-

trics, infectious diseases division) for immunization (PD/INF),participating institute (private medical facilities; PMF) group,

n implementing institute (public health center; PHC) group,research group (2005/2006), and a decision-maker/execution

roup (National Assembly/KCDC).The evaluation score for ENIPs is calculated by combining the

riteria weightings and the ratings of the alternatives, which are

ggregating method of expert groups.The geometric mean was first calculated by averaging the

roups’ individual responses at each point of comparison to formcomposite matrix, which was then used to obtain the relative

.

3 criteria (seconditeria)

PMF PHC Gap Gap Rank

rnmental budget foram

0.027 0.0219 0.0051 12

ialty and adequacy ofower

0.032 0.0166 0.0154 5

truction of stable demandupply system forunization vaccines

0.0198 0.0231 0.0033 18

bility of plan ofnstration program

0.0178 0.0156 0.0022 22

truction of service systemmunization program

0.0183 0.0225 0.0041 14

orks and partnershipseen public health centersrivate medical facilities

0.0486 0.0285 0.0201 3

ods leading to participationcommunity residents

0.0207 0.0129 0.0078 8

ods and strategies ofam’s PR

0.018 0.0133 0.0047 13

ine adverse event reporting 0.0437 0.0456 0.0019 23required for immunization 0.0211 0.018 0.0031 19

ll and remind services 0.0244 0.0312 0.0068 10uacy of selection ofr-privileged classes

0.0044 0.0113 0.007 9

uacy of problem-solvingod in the process

0.0063 0.0081 0.0017 24

ication of immunizationgement database

0.0077 0.01 0.0023 21

uacy of repayment ofnation costs

0.0084 0.01 0.0016 25

liance with immunizationlines

0.0064 0.0088 0.0023 20

unization coverage, etc. 0.0219 0.0125 0.0095 6uacy of immunizationts

0.02 0.0164 0.0036 16

unization registration rate 0.0205 0.0166 0.0039 15ovement of timeliness and-date immunizationage

0.0343 0.0251 0.0092 7

itative satisfaction withce

0.036 0.0131 0.0229 2

raphical accessibility 0.0238 0.007 0.0168 4omic satisfactionunization costs, indirect, etc.)

0.0192 0.0252 0.006 11

cipation rates of communityents and private medicalties

0.0369 0.0097 0.0271 1

ency of program (inputs output)

0.0215 0.0179 0.0036 17

0.5587 0.4409 – –

ference between the priorities of alternatives; Gap rank: the rank of gap values.

T. Shin et al. / Vaccine 27 (2009) 792–802 799

lterna

amisistsegtiiw

aweae

cip

Fig. 6. Comparison of the priorities of the a

nd composite weights in the usual manner. After the geometricean of the individual weights within each group was calculated,

t was then multiplied by the group weight for each group to synthe-ize the different group judgments. The importance of the nationalmmunization program was determined in each group using theurvey results to calculate the group weight. The arithmetic mean ofhe group weights of each group is used as a group weight to synthe-ize the different group judgments. In other words, to aggregate thexpert groups weightings, the respondents were asked to rate each

roup weight using the seven-point scale from “not very impor-ant” to “very important”. The results of the survey are summarizedn Table 2. From these weight results, the geometric mean of thendividual weights within each group is multiplied by the group

eight of each group and the calculated results are summed up for

wmipa

Fig. 7. Comparison of the priorities of the alterna

tives in terms of level 1 evaluation criteria.

ll groups. The two-step aggregating method for group decisionsas used in this study because it eliminates unequally averaged

ffects due to different sample sizes of participants in each groupnd reflects the relative importance of a group decision amongxpert groups.

The AHP provides an infrastructure for the decomposition of aomplex decision into a hierarchy of many factors. By establish-ng the hierarchy, a difficult decision making problem becomesairwise comparisons between factors sharing common criterion,

hich are much easier for people to process than an overall judg-ent call. The pairwise judgments are then translated into weights

n the AHP. Using the same hierarchy, the AHP further assists peo-le to synthesize ‘local’ weights into an overall rating of all thelternatives.

tives in terms of level 2 evaluation criteria.

800 T. Shin et al. / Vaccine 27 (2009) 792–802

lterna

tafi

oripmitipoTfltecnsaltipta(aapc

tE

aCPimiwm

lsrt

4ap

tpPjm

vnc

aotS

Fig. 8. Comparison of the priorities of the a

In this study, the results of the AHP modeling are divided intohe criteria evaluation and the relative priorities of the decisionlternatives of the ENIPs. The criteria evaluations for ENIPs wererst divided into three stages according to the hierarchical criteria.

First, Table 3 and Fig. 3 show that the evaluation of programutcome is the most important of the level 1 evaluation crite-ia. However, the program infrastructure factor has similar relativemportance to the program outcome factor. This indicates that therogram outcome and infrastructure should be more carefullyonitored and invested in than the program process to successfully

mplement an ENIP. Second, the three most important factors ofhe level 2 evaluation criteria are (1) investment resources for build-ng infrastructure as one of the program infrastructures, (2) programerformance as one of the program outcomes, and (3) adequacyf service provided as one of the program processes, shown inable 3 and Fig. 4. The results also shows that investment resourcesor building infrastructure is the most important factor of theevel 2 evaluation criteria. Third, the five most important fac-ors of the level 3 evaluation criteria are (1) vaccine adversevent reporting, (2) networks and partnership between public healthenters and private medical facilities, (3) improvement of timeli-ess and up-to-date immunization coverage, (4) recall and remindervices, and (5) governmental budget for the program (Table 3nd Fig. 5). These results also indicate that the most importantevel 3 evaluation factors in terms of each of the level 2 evalua-ion criterion are (1) governmental budget for program as part ofnvestment resources for building infrastructure, (2) networks andartnerships between public health centers and private medical facili-ies as part of participation system within the community, (3) vaccinedverse event reporting as part of adequacy of service providing,4) application of immunization management database as part ofdequacy of administrative procedure, (5) improvement of timelinessnd up-to-date immunization coverage as part of immunization out-

ut, and (6) economical satisfaction (immunization costs, indirectosts, etc.) as part of program performance.

In our study, the vaccine adverse event reporting factor was foundo be the most important criterion of all the factors for assessingNIP. The networks and partnership between public health centers

ct(la

tives in terms of level 3 evaluation criteria.

nd private medical facilities were the second most important factor.ommunity participation and networks and partnerships betweenHC and PMF are required effective to improve access to healthnformation resources. This is of particular interest to healthcare

anagers, researchers, and policymakers who currently have lim-ted access to evidence-based health information resources and

ould like to learn how to implement and understand virtual infor-ation systems within their organizations.These results illustrate that the high ranking indicators from the

evel 1 to level 3 evaluation criteria should be the first priority touccessfully introduce and manage an ENIP. Based on the final AHPesults of our study, we conclude that PMF are a better alternativeo PHC with respect to providing free vaccine services (Table 4).

.3. The comparative analysis of national health policylternatives: comparison of the priorities of the alternativeolicies according to the evaluation criteria

Next, this study performs a preference gap analysis betweenhe PMF and the PHC according to the evaluation criteria. Theurpose of this analysis is to specify how much more preferable theMF is compared to the PHC with respect to each criterion. That is,udgments regarding the relative preference of the alternatives are

ade relative to each criterion.Table 5 shows the priorities of the alternatives, the absolute

alues of the differences (gap) between the priorities of the alter-atives, and the rank of gap values in terms of the evaluationriteria.

First, the PMF alternative is preferred to the PHC alternativet both the program infrastructure and the outcome of the level 1f the evaluation criteria, but the PHC alternative is preferred tohe PMF alternative only at the program process (Fig. 6, Table 5).econd, when evaluating alternatives using the level 2 evaluation

riteria, the PMF is preferable to the PHC particularly at the par-icipation system within community and the program performanceFig. 7, Table 5). Finally, when evaluating alternatives using theevel 3 evaluation criteria, the PMF is preferable to the PHC. Thislso means that the PMF makes a larger contribution than PHC

ine 27

wtfi(fafqaf

5

ficiiu

ntmis

uftmatvtgteaftct

atpprtm3ua(va

tac

A

e

pAfR

R

[

[

[

[

[

[

[

[[

[

[

[

[

[

[

[

[

[

T. Shin et al. / Vacc

ith respect to achieving program goals. The most contributive fac-or in this study is defined as the factor with the largest gap. Theve most contributive factors of level 3 evaluation criteria were1) participation rates of community residents and private medicalacilities, etc. (2) qualitative satisfaction with service, (3) networksnd partnerships between public health centers and private medicalacilities, (4) geographical accessibility, and (5) specialty and ade-uacy of manpower (Fig. 8, Table 5). This implies that the PMFlternative is preferred over the PHC alternative because of theseactors.

. Concluding remarks

When developing a national immunization strategy, one of therst initiatives should be to create a reliable, systematic, logi-al evaluation mechanism that can effectively assess the nationalmmunization strategy. Thus, optimal implementation of themmunization strategy depends on the presence of such an eval-ation mechanism.

This study developed an evaluation model to find the best alter-ative for implementing a national immunization policy by usinghe AHP as the most promising method for prioritizing require-

ents. In this study, the results of the AHP modeling were dividednto the evaluation of criteria and the relative priorities of the deci-ion alternatives of the ENIPs.

We first analyzed the relative importance of evaluation criteriasing our proposed AHP model. The vaccine adverse event reporting

actor and networks and partnerships between public health cen-ers and private medical facilities factor were found to be the two

ost important criteria of all the factors tested in our study forssessing an ENIP. This result may be due to the fact that whenhere is an increase in public concern regarding vaccine risks andaccine safety it is increasingly difficult to convince the populaceo accept immunizations, thereby raising the spectre of resur-ence of vaccine-preventable diseases [36]. Thus, it is importanto take steps to ensure the safety of vaccinations by vaccine adversevent reporting as an active surveillance system for ENIPs. Networksnd partnerships between public health centers and private medicalacilities was the second most important factor; community par-icipation, and networks and partnerships between public healthenters and private medical facilities are required to improve accesso health information resources.

We then performed a gap analysis of the PMF and the PHCccording to the evaluation criteria. By using the level 1 evalua-ion criteria to evaluate the alternatives, we found that the PMF isreferable to the PHC with respect to the program infrastructure androgram outcome groups. When using the level 2 evaluation crite-ia, the PMF is again preferable to the PHC particularly with regardo the participation system within the community and program perfor-ance. Finally, when alternatives are evaluated based on the levelevaluation criteria, the PMF is also preferable to the PHC partic-larly with respect to (1) participation rates of community residentsnd private medical facilities; (2) qualitative satisfaction with service;3) networks and partnerships between public health centers and pri-ate medical facilities; (4) geographical accessibility; and (5) specialtynd adequacy of manpower.

Application of the proposed AHP model to Korean immuniza-ion policies showed that private clinics and hospitals are a betterlternative to public health centers in terms of providing free vac-ination services.

cknowledgements

We are grateful to our colleagues in the Korea Centers for Dis-ase Control and Prevention for this research project. We also thank

[

[

(2009) 792–802 801

rofessional experts of the participating institutes for surveying theHP questionnaire. This study was supported by the health policy

unding from the Korea Centers for Disease Control and Prevention,epublic of Korea in 2006.

eferences

[1] Department of Health and Human Services, Centers for Disease Control and Pre-vention. Epidemiology and prevention of vaccine-preventable diseases. 10thed. Atlanta: Safer Healthier PeopleTM; 2007.

[2] The Centers for Disease Control and Prevention. Framework for program eval-uation in public health. MMWR 1999;48(RR-11):1–57.

[3] Wynn BO, Dutta A, Nelson MI. Challenges in program evaluation of healthinterventions in developing countries. Santa Monica: Rand Corporation; 2005.

[4] Johns B, Baltussen R, Hutubessy R. Programme costs in the economic evaluationof health interventions. Cost Eff Resour Alloc 2003;1(1):1–10.

[5] Ministry of Health and Welfare/National Institute of Health, Ministry of Edu-cational and Human Resources Development (Korea). Final report of nationalserological survey for measles. Seoul: NIH; 2001.

[6] Kim SS, Han HW, Go U, Chung HW. Sero-epidemiology of measles and mumpsin Korea: impact of the catch-up campaign on measles immunity. Vaccine2004;23(3):290–7.

[7] Lee JJ, Yang JH, Hwang IS, Chun BY, Gam S, Lee KS, et al. The BCG, DTaP andIPV vaccination rate and the proportion of vaccination in the public healthcenters: demonstration project for expansion of National Immunization Pro-gram coverage in Daegu metropolitan city. J Korean Soc Matern Child Health2007;11(1):33–43.

[8] Lee SG, Jeon SY, Kim YT, Lee JH, Kim YG, Lim MR. Experience on demonstrationproject for expansion of National Immunization Program coverage in Gunphocity, Korea. J Korean Soc Matern Child Health 2008;12(2):166–80.

[9] Park HS, Kim KH, Kim OS, Baik SH, Yoo WS, Choi YJ, et al. Final report of 2006demonstration project for expansion of National Immunization Program in the3 public health centers. Seoul: Korean Centers for Disease Control and Preven-tion; 2007.

10] The Federal/Provincial/Territorial (F/P/T) Advisory Committee on PopulationHealth and Health Security (ACPHHS). National immunization strategy: Finalreport 2003. A report in the Conference of F/P/T Deputy Ministers of Health.Public Health Agency of Canada; 2003.

11] Albrecht MN, Perry KM. Home health care: delineation of research prioritiesand formation of a national network group. Clin Nurs Res 1992;1(3):305–11.

12] Madigan E. HHNA Research Committee update. Home Healthc Nurse2002;20(4):274.

13] Annells M, DeRoche M, Koch T, Lewin G, Lucke J. A Delphi study of districtnursing research priorities in Australia. Appl Nurs Res 2005;18(1):36–43.

14] Chung YC, Song HJ, Lee K. A strategic evaluation model for public informati-zation projects in the healthcare sector: Research project report. Seoul: KoreaInstitute for Health and Social Affairs; 2003.

15] McKenzie JF, Smeltzer JL. Planning, implementing, and evaluating health pro-motion programs: a primer. Bonston: Allyn and Bacon; 2001.

16] Spencer B, Broesskamp-Stone U, Ruckstuhl B, Ackermann G, Spoerri A, CloettaB. Modeling the results of health promotion activities in Switzerland: develop-ment of the Swiss Model for outcome classification in Health Promotion andPrevention. Health Promot Int 2008;23(1):86–97.

17] W.K. Kellogg Foundation. Logic Model Development Guide. 2004.18] Moyer A, Verhovsek H, Wilson VL. Facilitating the shift to population-based

public health programs: innovation through the use of framework and logicmodel tools. Can J Public Health 1997;88(2):95–8.

19] McEwan KL, Bigelow DA. Using a logic model to focus health services on pop-ulation health goals. Can J Program Eval 1997;12(1):167–74.

20] Green LW, Kreuter MW. Health promotion planning: an educational and eco-logical approach. 3rd ed. Mayfield: Mountain View; 1999.

21] Saaty TL. Multicriteria decision making: The analytic hierarchy process. reviseded. Pittsburgh: University of Pittsburgh; 1988.

22] Boucher TO, MacStravic EL. Multiattribute evaluation within a present valueframework and its relation to the analytic hierarchy process. Eng Economist1991;37(1):1–32.

23] Saaty TL. Priority setting in complex problems. IEEE Trans Eng Manage1983;30(3):140–55.

24] Hannan EL, O’Donnell J, Freedland T. A priority assignment model for stan-dards and conditions in a long term care survey. Socioecon Plann Sci1981;15(6):277–89.

25] Odynocki B. The forward and backward processes in health policy planning.Math Comput Simul 1983;XXV:146–55.

26] Longo M, Masella C. Organisation of operating theatres: an Italian benchmark-ing study. Int J Oper Product Manage 2002;22(4):425–44.

27] Hariharan S, Dey PK, Chen DR, Moseley HSL, Kumar AY. Application of ana-lytic hierarchy process for measuring and comparing the global performance

of intensive care units. J Crit Care 2005;20:117–25.

28] Liberatore MJ, Nydick RL. The analytic hierarchy process in medical and healthcare decision making: a literature review. Eur J Oper Res 2008;189(1):194–207.

29] Hariharan S, Dey PK, Moseley HSL, Kumar AY, Gora J. A new tool for measure-ment of process-based performance of multispecialty tertiary care hospitals.Int J Health Care Qual Assur 2004;17(6):302–12.

8 ine 27

[

[[

[

[

2007.

02 T. Shin et al. / Vacc

30] Dey PK, Hariharan S, Kumar AY, Moseley HSL. Performance measurement ofintensive care services in hospitals: the case of Barbados. Int J Serv TechnolManage 2004;5(5/6):579–94.

31] Osborn AF. Applied imagination. revised ed. New York: Scribner; 1957.

32] Kunene KN, Petkov D. Task structuring a brainstorming group activity with

an AHP-based group support system: Proceedings of the 35th Annual HawaiiInternational Conference on System Sciences. 2002. p. 2889–98.

33] The Centers for Disease Control and Prevention. Program in brief: Immuniza-tion assessment, National Center for Immunization and Respiratory Diseases,2007.

[

[

(2009) 792–802

34] The Association of State and Territorial Health Officials (ASTHO). Vaccinesupply, immunization infrastructure and emergency vaccine shortages: Arepository for all of ASTHO’s Policy Statements (source: www.astho.org),

35] Dyer R, Forman E. Group decision support with the analytic hierarchy process.Decision Support Syst 1992;8(2):99–124.

36] Chen RT, DeStefano F, Pless R, Mootrey G, Kramarz P, Hibbs B. Chal-lenges and controversies in immunization safety. Infect Dis Clin North Am2001;15(1):21–39.