primary health care · 2014. 5. 12. · primary health care (phc) is the first and continuing point...

PPrriimmaarryy HHeeaalltthh CCaarree EEvvaalluuaattiioonn RReeffeerreennccee MMaannuuaall

March 31, 2008

Prepared By:

Research Power Inc. and Pyra Management Consulting Services Inc.

Department of Health

.

TTAABBLLEE OOFF CCOONNTTEENNTTSS

IINNTTRROODDUUCCTTIIOONN && BBAACCKKGGRROOUUNNDD 1

PRIMARY HEALTH CARE 1

AN OVERVIEW OF THE PHC EVALUATION FRAMEWORK 1

WHY AN EVALUATION REFERENCE MANUAL? 3

TTHHEE BBAASSIICCSS OOFF EEVVAALLUUAATTIIOONN 4

WHAT DOES THIS CHAPTER CONTAIN? 4

PROGRAM EVALUATION: WHAT IS IT? 4

WHY EVALUATE? 4

HOW EVALUATION LINKS TO PLANNING 5

TYPES OF EVALUATION 6

STEPS IN EVALUATION 7

EVALUATION CONSIDERATIONS 7

FOUR STANDARDS OF PROGRAM EVALUATION 8

EENNGGAAGGIINNGG SSTTAAKKEEHHOOLLDDEERRSS 9


APPLYING THE STANDARDS 11

DDEEVVEELLOOPPIINNGG YYOOUURR EEVVAALLUUAATTIIOONN FFRRAAMMEEWWOORRKK 12


DESCRIBE THE PROGRAM – DEVELOPING YOUR LOGIC MODEL 12

FOCUS THE EVALUATION – DEVELOPING THE EVALUATION MATRIX 19

DDEEVVEELLOOPP YYOOUURR IINNSSTTRRUUMMEENNTTSS AANNDD GGAATTHHEERR TTHHEE DDAATTAA 25


WHY IS IT IMPORTANT TO GATHER CREDIBLE DATA? 25

.

WHAT DO VALIDITY AND RELIABILITY MEAN? 26

HOW DOES THE PHC EVALUATION FRAMEWORK SUPPORT THE GATHERING OF CREDIBLE DATA? 26

WHAT INFORMATION CAN I GET FROM EXISTING INFORMATION SYSTEMS? 30


AANNAALLYYZZIINNGG TTHHEE DDAATTAA AANNDD RREEPPOORRTTIINNGG YYOOUURR FFIINNDDIINNGGSS 33

DATA ANALYSIS 33

REPORTING YOUR FINDINGS 34


LLEEAARRNNIINNGG FFRROOMM AANNDD SSHHAARRIINNGG YYOOUURR RREESSUULLTTSS 36

FFIINNAALL TTHHOOUUGGHHTTSS AABBOOUUTT PPHHCC EEVVAALLUUAATTIIOONN IINN NNOOVVAA SSCCOOTTIIAA 37

RREEFFEERREENNCCEESS 38

AAPPPPEENNDDIICCEESS 39

AAPPPPEENNDDIIXX 11:: EEVVAALLUUAATTIIOONN QQUUEESSTTIIOONNSS AANNDD IINNDDIICCAATTOORRSS FFRROOMM TTHHEE PPRROOVVIINNCCIIAALL PPHHCC EEVVAALLUUAATTIIOONN FFRRAAMMEEWWOORRKK 40

AAPPPPEENNDDIIXX 22:: PPRROOVVIINNCCIIAALL PPHHCC DDAATTAA CCOOLLLLEECCTTIIOONN TTOOOOLLSS 53

AAPPPPEENNDDIIXX 33:: DDAATTAA FFOORR TTHHEE PPHHCC IINNDDIICCAATTOORRSS PPOOTTEENNTTIIAALLLLYY AAVVAAIILLAABBLLEE TTHHRROOUUGGHH MMSSII 95

.

PHC Evaluation Reference Manual

Prepared by Research Power Inc., and Pyra Management Consulting Services Inc.

March 31, 2008

1

IINNTTRROODDUUCCTTIIOONN && BBAACCKKGGRROOUUNNDD

PRIMARY HEALTH CARE

Primary health care (PHC) is the first and continuing point of contact for Nova Scotians with the

health care system. It focuses on promoting health, preventing illness, managing chronic diseases

and treating people when they are sick. PHC also serves as a vehicle for ensuring continuity of care

across the health care system.

Over the past decade, Nova Scotia has been making important changes to strengthen our PHC

system. One of those changes has been to improve the way that we collect information about how

well the PHC system is performing. In 2006, Nova Scotia published a comprehensive evaluation

framework for primary health care. The PHC evaluation framework is the foundation for a

provincially-coordinated approach to collecting, analyzing and reporting information about the

PHC system.

To assist PHC administrators and providers in measuring the changes they are making to the PHC

system, this PHC Evaluation Reference Manual was created as a companion document to the PHC

evaluation framework.

AN OVERVIEW OF THE PHC EVALUATION FRAMEWORK

The document “A Primary Health Care Evaluation System for Nova Scotia” was published in 2006 and

contains the PHC evaluation framework for the province. The framework was developed through

considerable dialogue among representatives from the District Health Authorities (DHAs) and the

Department of Health via a group called the PHC Evaluation Working Group. A provincial PHC

logic model was created collaboratively, and priority evaluation questions for the outputs and

impacts of the logic model were developed. Indicators were identified for the evaluation questions,

many of which are consistent with national indicators developed through a national consensus

process coordinated by the Canadian Institutes of Health Information.

Data sources were identified for as many of the indicators as possible. For some indicators, data

was not available through current sources, so three new PHC data collection tools were developed

and pilot tested in order to support data collection (there are more details about data collection

tools later in this Reference Manual). For other indicators without current data sources, the

evaluation framework noted that appropriate data needs to eventually be captured through the

electronic Primary Health Care Information Management System.

.



March 31, 2008

2

Three Levels of Evaluation To understand the PHC evaluation framework in Nova Scotia, it is important to first understand

the underlying conceptual approach. The PHC evaluation framework is built upon the concept that

three levels of evaluation are needed in order to effectively capture the impact of changes made to

the PHC system in Nova Scotia:

1. System Level Evaluation, which includes a set of indicators that are either measured at a

provincial level or consistently measured by all DHAs. Measurement at the System Level

assesses the results of change in the primary health care system across the province. System

Level evaluation is coordinated and supported by the Department of Health, with some

data provided from the DHAs.

2. District Level Evaluation, which includes the indicators that DHAs measure as part of their

contribution to the System Level evaluation, as well as indicators unique to each district.

3. Project Level Evaluation, which includes relevant System Level and District Level

indicators as well as the indicators unique to each primary health care renewal project or

initiative taking place within each DHA.

The three-level approach to evaluation recognizes that each DHA has unique PHC initiatives

underway to meet the specific needs of their populations. It is important that each of these

initiatives is evaluated to measure the success of the initiative and to contribute to our broader

understanding of how the PHC system is changing provincially. Evaluating specific initiatives

within a DHA falls under Project Level Evaluation. The three-level approach to evaluation also

recognizes that DHAs may conduct evaluations across their district to create a district-wide picture

of the PHC system and performance. This is District Level evaluation. Both Project Level and

District Level evaluations are linked to System Level Evaluation.

To ensure consistency in PHC evaluation across the province, each DHA needs to use the System

Level resources as the foundation for their own specific District Level or Project Level evaluations.

For example, if a DHA has a specific PHC project that they wish to evaluate, using the relevant

evaluation questions from the provincial (System Level) PHC evaluation framework will ensure

that the data that they collect is consistent and hopefully at some point comparable with data

collected by similar projects in other DHAs.

For example, if four DHAs implement very different projects designed to increase access to PHC

services, there may be an opportunity to compare and contrast the impact of each of the projects if

all four of them use the same evaluation questions and indicators to evaluate their impact. The

PHC evaluation framework contains a fairly thorough question about access with associated

indicators, so it would make sense for each of the four projects to use the same questions and

measurement technique to answer the same question.

.



March 31, 2008

3

WHY AN EVALUATION REFERENCE MANUAL?

This Reference Manual was created to help people who are planning PHC evaluations at the Project,

District and System Levels. Its purpose is to build understanding about evaluation and how to plan

PHC evaluation in a way that is consistent with Nova Scotia’s PHC evaluation framework.

PHC evaluation is not without its challenges. There is a lack of data available to measure many

important indicators and limited infrastructure is in place to capture, analyze and report on data

that is specific to PHC. However, PHC evaluation in Nova Scotia is continually evolving. As

electronic medical records become more widely used throughout the province and further

development occurs to allow for better data capture, our efforts to evaluate PHC will become even

stronger. While this Reference Manual does not address those infrastructure issues, it does cover

an important step in the evaluation process. Nova Scotia will make significant advances forward in

PHC evaluation if everyone involved in planning for PHC evaluation at System, District and

Project Levels begins thinking about and planning their evaluations in a consistent manner. Asking

the same evaluation questions and using the same indicators across the province will be a huge step

forward for PHC evaluation in Nova Scotia – a step that hopefully this Reference Manual will

facilitate.

.



March 31, 2008

4

TTHHEE BBAASSIICCSS OOFF EEVVAALLUUAATTIIOONN

WHAT DOES THIS CHAPTER CONTAIN?

The following section describes the basics of evaluation including its importance, the link between

program planning and program evaluation, the types of evaluation and steps in the process.

PROGRAM EVALUATION: WHAT IS IT?

In order to answer this question, it is first important to understand what is meant by ‘program’ and

‘evaluation’. A program is a series of related activities supported by a group of resources (e.g.,

human, financial, etc.) intended to achieve specific outcomes among particular individuals, groups

and communities. Evaluation is the course of actions used to assess the value of a program.

When put together program evaluation can be described as a formalized, systematic

collection of information to make judgments about a program, improve program effectiveness and/or

inform decisions about future programming. As illustrated by the quote below, the purpose of

evaluation is not to determine whether a program is ‘good or bad’ or has ‘passed or failed’, but

rather to highlight and share a program’s successes and identify areas of improvement to better the

program and help it to meet its desired goals.

The real challenge for evaluation…is to develop methods of assessment that emphasize learning and adaptation

rather than expressing summary of judgments of pass or fail.

- Giandomenico Majone (1988)

WHY EVALUATE?

You may be thinking, “that definition of program evaluation is all well and good on paper and in

textbooks, but how does it relate to my program in the real world?” Well, there are numerous

practical reasons for program evaluation. Program evaluation can help to:

� Determine effectiveness: Help you to understand or verify the impact of your program or

services.

� Assess efficiency: Help to ensure your program is optimally using its resources.

.



March 31, 2008

5

� Be accountable: Help to account for what has been accomplished to stakeholders (e.g.,

funders, decision makers, PHC Team members, clients or patients, etc.).

� Identify ways to improve the program: Help assess how well your program and services

are doing and help you make them better (e.g., strengths, weaknesses, meeting needs).

If you don’t measure results, you can’t tell success from failure; if you can’t see success, you can’t learn from it; if

you can’t recognize failure, you can’t correct it.

- University of Wisconsin-Extension

HOW EVALUATION LINKS TO PLANNING

Planning and evaluation are interconnected and linked. Evaluation requires stakeholders to ask

questions which are critical to program planning (e.g., What is the goal of this program? How do

we intend to reach these goals, etc.). In fact, many tools used in evaluation are also useful in

program planning. For example, logic models (to be described in greater detail in the next section)

also serve as program planning frameworks and assist stakeholders to clarify the program, its scope,

its activities, and its anticipated goals. The following figure illustrates the program cycle and the

relationship between planning and evaluation:

Figure 1: Program Planning Cycle

Establish

the Program

Need

Plan the

Program

Determine

Program

Effectiveness

Implement

the

Program

Evaluation

.



March 31, 2008

6

TYPES OF EVALUATION

The timing of the evaluation and the kind of information collected will determine the type of

evaluation being conducted. There are three main types of evaluation: formative, process,

summative/outcome evaluations.

♦ Formative Evaluation: Any combination of measurements obtained before the

implementation of materials, methods, activities or programs to control, assure, or improve the

quality of performance or delivery.

♦ Process Evaluation: Any combination of measurements obtained during the

implementation of program activities to control, assure, or improve the quality of performance or

delivery.

♦ Summative Evaluation: Any combination of measurements that permit conclusions to be

drawn about the impact, outcome or benefits of a program.

When the cook tastes the soup, that’s formative evaluation, when the guest tastes it, that’s

summative evaluation.

- Robert Stake (1991)

The following table provides an overview of these three types of evaluation.

Table 1: Types of Evaluations

Formative Process Summative/Outcome

Pu

rpos

e ♦ To help plan or

implement a program

♦ To assess aspects of program

implementation

♦ To assess the outcomes of a

program

Tim

ing

♦ Used at the planning

stages of a program to

ensure its developed to

suit stakeholder needs

♦ Used early during program

implementation to ensure program

procedures and implementation is

carried out according to the program

plan

♦ Used later in the program to

explore the effects of the

program

Ex

amp

les

♦ Needs assessments

♦ Tracking services provided

♦ Measuring participant satisfaction

♦ Measuring changes in

knowledge, attitudes, skills,

behaviours

♦ Measuring changes in health

status

.



March 31, 2008

7

STEPS IN EVALUATION

As described above, program evaluation is a systematic process which can be summarized into six

key steps. Figure 2 illustrates the six steps to program evaluation. Each step will be described in

greater detail in the following sections of this report.

Figure 2: Program Evaluation Steps

EVALUATION CONSIDERATIONS

Often evaluation in the ‘real world’ is met with practical problems which may impede its

effectiveness. Some common problems are listed below:

1. Planners fail to build evaluation in to program planning

2. Limited resources often means that evaluation must be limited to ‘need to know’ rather than

‘nice to know’ information

3. Adequate procedures (e.g., tool development, data collection, etc.) cost time and resources

4. Adequate skills for evaluation are often lacking (e.g., instrument development, data

analysis) which may lead to poor evaluation design and implementation

5. Adequate data collection sources commonly are not readily available

6. Data collection is challenging in terms of measuring the ‘intangible’ (e.g., effectiveness of

partnerships, building leadership, etc.)

1

Engage

Stakeholders

6

Use & Share

Findings

2

Describe the

Program

3

Focus the

Evaluation

Design

4

Gather &

Analyze Data

5

Justify

Conclusions

Program

Evaluation Steps

.



March 31, 2008

8

7. It is often challenging to properly evaluate multilevel interventions

8. It is difficult to distinguish between cause and effect

9. Impact level changes often come slowly

FOUR STANDARDS OF PROGRAM EVALUATION

This Reference Manual addresses the quality of evaluation by asking the question: Will the

evaluation be a good evaluation? In evaluation terms, standards convey quality and include four

key areas to consider:

Four Standards of Program Evaluation

1. Utility (Is the evaluation useful?)

2. Feasibility (Is the evaluation viable and practical?)

3. Propriety (Is the evaluation ethical?)

4. Accuracy (Is the evaluation correct?)

Program evaluators view these four standards as the initial yardstick by which to judge the quality

of program evaluation efforts. Throughout this Reference Manual, you will find sections called

“Applying the Standards.” For each step in the evaluation process, you can use these sections to

reflect on the extent to which your evaluation process reflects the above standards for a quality

evaluation.

.



March 31, 2008

9

EENNGGAAGGIINNGG SSTTAAKKEEHHOOLLDDEERRSS


This chapter highlights the importance of identifying and engaging stakeholders in developing

your evaluation. It is important to engage stakeholders early in the process, so that they are

involved in identifying evaluation priorities as well as receiving and using evaluation results. It is

important to involve stakeholders as much as possible throughout the evaluation process.

Stakeholders who feel involved with the process will be able to assist in data collection processes

and will be in a better position to effectively use the results of the evaluation for program

improvement.

Defining Stakeholders

Stakeholders include organizations and people involved in the program and its evaluation, or

people who might be interested in the results of the evaluation for some reason. Stakeholders

include:

♦ Those who work with the program (e.g., program staff, volunteers);

♦ Program partners and collaborators;

♦ Those who are affected by or make use of the program (e.g., participants); and

♦ Those invested in the program (e.g., funders, decision makers).

It is important to recognize that stakeholders may be involved in the evaluation process to different

degrees. Some stakeholders will be very involved in all aspects of planning and implementing an

evaluation; others may only be involved in helping to define evaluation questions and receiving

evaluation results.

The worksheet on the following page can be used to help you reflect upon who are the stakeholders

for your PHC evaluation. Engaging stakeholders represents a process through which many voices

are heard. Completing this step helps ensure that the focus of the evaluation – and ultimately the

results of the evaluation – supports the needs of the stakeholders.

.



March 31, 2008

10

Stakeholder Identification Worksheet

For each of the three types of stakeholders, identify who the specific stakeholders are for your

particular PHC evaluation. For each stakeholder, try to identify what their main interest would be

in the evaluation – how would they realistically like to be involved; how might they use the results?

Type Examples Your Program Stakeholders Interest in the Evaluation

Those involved

in the program

development

and

implementation

(program

operation)

Program staff,

volunteers,

PHC providers,

partners in

community

organizations,

etc.

Those served or

affected by the

program

(program users)

Program

participants,

participant’s

families,

community

members, etc.

Those with

vested interests

in the program

DOH or DHA

funders, third

party funders,

clinical decision

makers in

primary,

secondary or

tertiary care,

community

leaders, DOH or

DHA managers,

directors,

advocacy

groups, etc.

.



March 31, 2008

11

APPLYING THE STANDARDS

Including stakeholders in evaluation planning and implementation is one way to ensure a quality

evaluation to meet evaluation standards. The two standards that apply most directly to - Engaging

Stakeholders – are utility and propriety. As you carry out this step, the questions presented in the

following table will help you to clarify and achieve these evaluation standards.

Questions Standards

♦ Have you included individuals and organizations

that will be affected by the evaluation in your

evaluation planning group?

♦ Have you considered adding new stakeholders as

your program evaluation is implemented?

♦ Are participants in the evaluation planning group

trustworthy and competent?

♦ How can the evaluation be planned, conducted, and

reported so that it, in turn, encourages use by

stakeholders?

Utility: Ensures that the

evaluation is useful and

answers questions that are

directly relevant to users.

♦ Are individuals clear about what is to be done, how,

by whom, and when?

♦ Is there a written understanding?

♦ Have steps been taken to assure that all stakeholders

and the population served will be respected and

their values honored?

♦ Have conflicts of interest been discussed to ensure

that the results or findings will not be compromised?

Propriety: Ensures that the

evaluation is an ethical one,

conducted with regard for the

rights and interests of those

involved.

.



March 31, 2008

12

DDEEVVEELLOOPPIINNGG YYOOUURR EEVVAALLUUAATTIIOONN FFRRAAMMEEWWOORRKK


The following chapter provides an overview of how to develop an evaluation framework for your

PHC initiative or program. Every PHC program or initiative should have its own evaluation

framework. Evaluation frameworks vary in size, scope and complexity depending on the type of

program or initiative being evaluated and the evaluation budget.

There is a provincial PHC evaluation framework that was developed to help promote consistency

in PHC evaluation across the province. Every PHC evaluation framework developed in Nova

Scotia should link closely to and build upon the provincial evaluation framework. Only if all

programs and initiatives work from the same foundation will we be able to achieve any degree of

consistency in PHC evaluation.

This chapter provides details about how to describe your program in a manner that is consistent

with the provincial PHC evaluation framework, as well as how to identify evaluation questions and

indicators that are also consistent with a province-wide approach.

An evaluation framework includes:

♦ A description of the program or initiative;

♦ An evaluation matrix, which includes evaluation questions and indicators; and

♦ A description of the data collection tools and time frames for data collection.

Each of these elements of an evaluation framework are described in this chapter.

DESCRIBE THE PROGRAM – DEVELOPING YOUR LOGIC MODEL The second step in the evaluation process involves describing your program. This may appear to

be a relatively simple task, however this can be one of the most challenging parts of evaluation.

The task becomes more complex when trying to describe PHC initiatives at the District or System

levels (see Chapter 1 for a detailed description of these levels). Yet it is not possible to evaluate an

initiative unless the planned activities and their intended changes are carefully described.

In program evaluation, programs and initiatives are commonly described using a logic model.

.



March 31, 2008

13

What is a Logic Model?

A logic model provides an overall diagrammatic representation of a program or initiative. Logic

models help to provide a broad overview of an initiative by clearly illustrating the relationship

between the initiatives’ activities, outputs, impacts and outcomes. Each of these aspects of a PHC

logic model is defined below. These definitions were created by the PHC Evaluation Working

Group that developed Nova Scotia’s PHC evaluation framework.

♦ Activities: The actions, policies, resources and strategies that are implemented to enact

changes to the PHC system. Activities are designed to meet the needs of the population.

Examples include networking, funding, service and program planning, and communications.

♦ Process and Structure Outputs: Changes that occur in the primary health care system

as a result of the activities. Examples include coordination of services, appropriate

mechanisms for individual and community participation, service and program availability and

appropriateness, skills and competencies of providers and communities.

♦ Impacts: Changes in those determinants of health and well-being that directly result from

the activities of the initiative. Examples include individual factors (e.g. protective factors,

reduced risk behaviours), community capacity, access to services, engagement with and

experience of services/programs).

♦ Outcomes: Changes in the health and well-being of the population that result in part from

the activities of the initiative. Examples include prevalence and incidence of disease, burden of

illness and disability, levels of health inequality.

A logic model organizes a program or initiative’s activities according to broad level ‘components’.

Generally the impacts and outcomes of a logic model are not linked to any one component, activity

or output but are the result of the combined effect of many or all of the activities of the program or

initiative.

Why are Logic Models Useful?

Logic models provide the foundation for the evaluation at any of the Project, District or System

Levels. Logic models help to:

♦ Describe the overall program or initiative;

♦ Demonstrate the connection between planned activities and intended outputs, impacts and

outcomes;

♦ Manage expectations about the potential effects of a program or initiative;

♦ Communicate about the program or initiative; and

♦ Clarify what needs to be evaluated.

.



March 31, 2008

14

The Provincial PHC Logic Model

As part of the process to develop a provincial PHC evaluation framework, the PHC Evaluation

Working Group developed a System Level logic model for PHC. It is a much higher level logic

model than would be developed for a specific initiative, however, every initiative developed at a

District or Project level can and should draw upon the elements of the System Level logic model

to ensure consistency across the province. The System Level Logic Model for PHC in Nova Scotia is

presented below.

Primary Health Care Logic Model For Nova Scotia

(System Level Logic Model) Elements

Activities 1. Population-based planning (including capacity and needs assessment) 2. Funding 3. Partnership development and collaboration 4. Disseminating/sharing information 5. Networking and coordination 6. Service development 7. Capacity building 8. Evaluation and research 9. Policy development and advocacy 10. Change management 11. Communications 12. Implementing information technology

Process &

Structure

Outputs

1. Planning for service delivery is informed by evidence. 2. Accountabilities within relationships in the primary health care system are clearly defined. 3. Services and programs are delivered in a coordinated and integrated way. 4. A range of services and programs provide an equitable balance between heath promotion

and provision of health care services. 5. Primary health care providers collaborate. 6. Primary health care providers’ knowledge, skills and attitudes are appropriate to the

services provided. 7. Individuals, health providers and communities have capacity to participate in PHC

planning and delivery. 8. Services are offered in ways that respond to individuals’, families’ and communities’

cultural, racial and spiritual needs. 9. Nova Scotians have a continuing relationship with a primary health care provider through

whom they can access services.

Impacts 1. Individuals and families have access to services, programs and information. 2. Individuals and families experience continuity of care. 3. The primary health care workforce is healthy and satisfied. 4. Quality primary health care services are safe, efficient, effective, affordable and acceptable

to the community. 5. Populations have better health literacy and health promoting behaviours.

Outcomes 1. Population-level health and wellness are improved. 2. Health inequalities among Nova Scotians are reduced. 3. Communities are health promoting environments. 4. The primary health care system is accountable and sustainable.

.



March 31, 2008

15

An Example: A Project Level Logic Model Developed Based on the System Level

Logic Model

Let’s imagine that fictitious DHA #15 has decided to implement a new PHC initiative that involves

developing a new collaborative to support the improvement of diabetes management by PHC

practitioners across the DHA. Collaboratives are an emerging type of knowledge translation

activity in which groups of practitioners work together to identify particular clinical practice

challenges, collectively learn about practice improvement opportunities, and work together and

mentor each other as practice improvements are implemented and practiced.

In order to begin this new project in DHA 15, the Project Coordinator, Shawna, has prepared a logic

model. Shawna began the process of developing the logic model by holding discussions with the

stakeholders who will be involved in and impacted by the work of the Collaborative, including

staff of the diabetes centres in the district, a sample of PHC providers based in the community, a

diabetes patient support group, endocrinologists in the regional hospital, the provincial Diabetes

Care Program, the Diabetes Association and the PHC Manger for the DHA. After her discussions

with stakeholders, Shawna felt as though she had a better understanding of what was required to

get the Diabetes Management Collaborative started.

To create her logic model, Shawna reviewed the PHC System Level Logic model for Nova Scotia,

knowing that her specific project needs to link with overall provincial directions in PHC renewal.

She reviewed the various activities in the first row of the System Level Logic Model to identify the

categories of activities that she would use for her logic model. These became the components of her

logic model. For example, the first element in the activities row of the System-Level logic Model is

“population based planning.” Shawna believed that this particular element represented some of

the activities that needed to be undertaken in her project, so Population-Based Planning became one

of the main components of her Project Level Logic Model.

Shawna drew upon the System Level Logic Model to help inform her decisions about what outputs,

impacts and outcomes should be included in her logic model. Shawna’s logic model for the

Diabetes Management Collaborative is on the following page.

.



March 31, 2008

16

Example Logic Model for the Diabetes Management Collaborative Project in DHA 15 Components Population Based

Planning

Networking and

Coordination

Policy

Development

Communication

Activities 1. Identify priority

diabetes

management

issues from

provincial and

district data

2. Identify from

research,

evidence-based

practice

improvements

related to priority

issues

1. Organize and facilitate

monthly meetings of the

Collaborative

2. Facilitate the development

of Collaborative teams for

mentoring

1. Adapt/create new

PHC diabetes

practice

guidelines and

procedures for

PHC providers in

the district based

on the learnings

of the

Collaborative

1. Identify potential

Collaborative

participants and invite

participation

2. Assist Collaborative

teams with internal

communication and

scheduling

3. Communicate

adapted/new guidelines

and procedures to all

PHC providers in the

District

Process &

Structure

Outputs

1. Planning for

service delivery is

informed by

evidence.

2. PHC providers

collaborate

3. PHC providers

knowledge, skills and

attitudes are appropriate

to the service provided

4. Health

providers have

capacity to

participate in

PHC planning

and delivery

5. Services and

programs are

provided in a

coordinated and

integrated way

Impacts 1. Individuals and families experience continuity of care.

2. Quality primary health care services are safe, efficient, effective, affordable and acceptable to the

community.

3. Populations have better health literacy and health promoting behaviours

Outcome 1. Population-level health and wellness are improved.

In one case, Shawna felt that the output in the System Level Logic Model was too broad for her

project, so she amended it slightly, keeping the concept the same but narrowing the scope a bit.

The output under the Policy Development component is similar to an output in the System Level

Logic Model, except that in Shawna’s logic model, it doesn’t include reference to individuals and

communities as it does in the System Level logic model.

Finally, to help with future evaluation of the project, Shawna wrote a description of the context in

which the project was being implemented. In her description, she included comments about why

the Project was started, the resources and time frame allocated to the project, a summary of the

perspectives of the stakeholders that she spoke with about the project and any issues that she

thought might be challenges to moving the project forward. She asked several of her stakeholders

to come to a meeting to review the draft logic model and context description and incorporated

feedback into her final draft.

Developing Your Logic Model

Just as Shawna did in the example logic model, every time that you develop a logic model for a

small project or a large initiative related to PHC renewal, it should directly link to the System

Level Logic Model. This will ensure that all PHC activities are categorized and made more easily

measureable in a consistent way across the province. There may be times that you believe that you

.



March 31, 2008

17

need to add an output or slightly narrow the scope of an impact, which is fine. There is a certain

degree of flexibility in the process. However, as you consider making any changes, reflect upon

what impact the change may have on the broader evaluation of PHC in Nova Scotia. Provincial

evaluation of PHC in Nova Scotia closely follows the structure of the System Level logic model. So

if you plan to make an adaptation for your logic model, ask yourself “If I change the wording of

this item, will the evaluation data from my initiative be able to be used in creating a total picture of

PHC progress in Nova Scotia?”

Logic Model Template

There are many different formats for logic models. The PHC System Level Logic Model uses a

tabular format, which is also the format recommended for District and Project Level logic models.

Logic model templates that use many arrows and multiple colours often confuse more than clarify

the linkage between activities and outcomes. The following template is recommended.

Components

Activities

Process &

Structure

Outputs

Impacts

Outcomes

Logic Model Development Checklist

The following are questions to ask yourself as you develop your PHC logic model.

Components

1. Which of the elements from the first row of the PHC

System Level Logic Model seem to be good categories

for the type of activities that are included in your

project or initiative? These will become your

components.

Activities

2. What are the high-level activities of the program or

initiative? For example, what is being created or what

services are being delivered?

Tips for Creating the Activities in

A Logic Model

♦ Don’t be tempted to list every single activity

of the project – keep the activities at a fairly high level. Remember, in most cases, even for complex initiatives, logic models generally fit on one page.

♦ Use action verbs at the beginning of each phrase describing an activity.

♦ Example action verbs: conduct, develop, identify, facilitate, educate, train, offer, refer, establish, distribute.

.



March 31, 2008

18

Process and Structure Outputs

3. Think about:

a. What services are delivered as a result of the activities? OR

b. What changes will occur in the PHC system as a result of the activities?

4. Check the PHC System Level Logic Model to identify the outputs that best match your answers

to the questions posed above in item #3.

Impacts

5. Think about:

a. What direct changes are expected to occur with the participants in the initiative? OR

b. What direct changes are expected to occur with people who receive care through the

PHC system?

6. Check the PHC System Level Logic Model to identify the impacts that best match your answers

to the questions posed above in item #5.

Outcomes

7. Check the PHC System Level Logic Model to identify the outcomes that best match the

intended outcomes from your program or initiative. Remember outcomes reflect the social and

economic consequences of a program or initiative in the broader population. It would be very

unusual for a PHC program in Nova Scotia to not have at least one of these outcomes present in

the logic model.

Applying the Standards To: Describing Your Program

You can help to ensure that your evaluation is of high quality if you review your program

description against the evaluation standards described in the first chapter. The standard that

applies most directly to -Describing Your Program- is accuracy. As you work on completing your

logic model, ask yourself these questions to help you to achieve this standard.

Questions Standards

♦ Is the program description complete?

♦ Would someone not involved in the program be able

to understand from your logic model how the

planned activities will lead to the intended change?

♦ Have you documented the context of the program so

that likely influences on the program can be

identified?

Accuracy: Ensures that the

findings can be considered

correct.

.



March 31, 2008

19

FOCUS THE EVALUATION – DEVELOPING THE EVALUATION MATRIX

After describing your program or initiative, the next step in developing your evaluation framework

is to develop your evaluation matrix. This involves deciding what evaluation questions to ask,

selecting indicators for each question and identifying how you will acquire the data to measure

each indicator.

What is an Evaluation Matrix?

An evaluation matrix provides an overview of an evaluation strategy including how the

outputs, impacts and outcomes will be measured. An evaluation matrix is presented as a table

organized according to the components of the logic model. Under each component are the outputs,

impacts and outcomes from the logic model as well as associated evaluation question(s). These

questions are “answered” by indicators (indicators are described in the next section).

What Are Indicators?

Indicators are the measures that are used to determine if or how well each output, impact and

outcome has been achieved. In most cases, the indicators in the provincial PHC evaluation

framework are numerical. For example, one of the indicators in the provincial PHC evaluation

framework is the “ratio of PHC providers

entering/leaving the workforce over the past 12

months by type of PHC provider.” This particular

indicator helps to answer the evaluation question

“Are there sufficient numbers of physicians, family

practice nurses and nurse practitioners to meet the

demand for PHC?”

In some cases however, it is not possible for a

numerical indicator to reasonably measure an

evaluation question. For example, one of the

questions in the provincial PHC evaluation

framework is “Do DHAs and the Department of

Health integrate PHC planning into strategic and

business plans for the system?” This was identified

as a very important evaluation question by the

PHC Evaluation Working Group, yet this question

does not lend itself to a useful numerical measure.

So the indicator for this question is “PHC is part of

DHA and DOH strategic and business plans”. Measurement of this indicator will require a review

of the business and strategic plans to determine the extent to which they address PHC.

An Indicator Analogy

Think about the dashboard on your car. There is often a warning light that says “Service Engine Soon.” This warning light is fed by many streams of data, such as the oil level and temperature of the engine. The light comes on when there is something not performing quite as it should be. At this point we check the car over to try and determine what has caused the warning light to come on. Because there is an ongoing monitoring system, we are able to see when performance is not as good as it could be and take corrective action to improve performance. Indicators in evaluation act the same way. They can tell us when things are going well and when things are not performing as we expected.

(NHS Institute for Innovation and Improvement, 2008)

.



March 31, 2008

20

The following are some characteristics of good indicators:

♦ Results focused: results of the activity, not the activity itself;

♦ Limited to and focused on key areas of concern;

♦ Captures relevant information;

♦ Challenging, but feasible;

♦ Involve a meaningful comparison (comparison across time, against other programs, compared

to best practice standards, etc.);

♦ Measurable using quantitative or qualitative methods;

♦ Reasonably attributed to the program or initiative;

♦ Valid and reliable: should be directly related to the work of the program and able to replicated;

and

♦ Together with other indicators provide a balanced assessment of the progress and impact of

the program or initiative.

The Provincial PHC Indicators

The provincial PHC evaluation framework contains indicators for measuring evaluation questions

associated with all of the process and structure outputs and the impacts in the System Level Logic

Model. At the time that the evaluation framework was developed, the Evaluation Working Group

agreed that although the primary health care system contributes to the achievement of the

outcomes in the logic model, the outcomes will only be achieved with contributions from both

health and non-health sectors. For this reason, indicators for the outcomes were not developed. In

the future, there may be an opportunity for an intersectoral evaluation working group to develop

appropriate indicators for the outcomes.

The provincial PHC indicators for all of the evaluation questions from the provincial PHC

evaluation framework are contained in Appendix 1, organized by outputs and impacts.

Using the Provincial PHC Evaluation Questions and Indicators

To create your evaluation matrix, you need to define your main evaluation questions. A common

way to approach this process is to identify key evaluation questions for each of the process and

structure outputs and the impacts. For each output and impact, refer to the provincial PHC

evaluation questions and indicators (see Appendix 1). Where at all possible, use the provincial

evaluation questions and indicators to help build evaluation consistency across the province. It is

also probable that you may have additional questions or indicators that are specific to your

program or initiative. These can also be added to the matrix. For each indicator, show the source of

the data that will be used.

.



March 31, 2008

21

Continuing the Example: A Sample Evaluation Matrix

The evaluation matrix presented below is a continuation of the example that we began in the

previous section about the fictitious DHA 15’s Diabetes Management Collaborative Project. The

example below is only a partial evaluation matrix. To be complete there should be at least one

evaluation question and indicators associated with each output and impact.

It is important to note that several of the outputs and impacts in the provincial PHC evaluation

framework only have questions associated with them that can be measured at the District or System

level. In these cases, specific projects will need to come up with their own questions and indicators

that align as closely as possible with the intention in the provincial framework. This was the case

for our fictitious Project Coordinator, Shawna, when she looked in the provincial framework for

example questions for her first output. Because the evaluation questions and indicators for that

particular output were all meant to be measured at the District Level, Shawna had to create an

appropriate question and indicators for her Project Level logic model. However, by keeping the

intention of the questions and the indicators conceptually similar to those in the provincial

framework, she ensured that the evaluation data from her project could be used in an upcoming

provincial evaluation of how evidence is used in planning, even though her indicators are not quite

the same as those in the provincial framework.

.



March 31, 2008

22

Example Partial Evaluation Matrix for the Diabetes Management

Collaborative Project in DHA 15

Evaluation Question Indicator Data Source

Process and Structure Output 1: Planning for service delivery is informed by evidence.

♦ Sources of population data used

(project-specific indicator)

♦ Interview: Project

coordinator

1.1 What forms of evidence were used by

the DHA and primary health care teams to

inform PHC planning and delivery?

(provincial question) ♦ Sources of data used to identify

clinical practice improvements

(project-specific indicator)

♦ Interviews: Collaborative

participants

Process and Structure Output 4: Health providers have capacity to participate in PHC planning and delivery.

♦ Number and type of learning

opportunities provided through

the Collaboratives project

(provincial indicator – narrowed

scope)

♦ Learning opportunity

record sheet (project

coordinator)

4.1 What mechanisms and/or learning

opportunities are in place to build the

capacity of PHC providers to participate in

PHC planning and deliver? (provincial

question)

♦ % of PHC providers in the

District participating in the

Collaboratives project (provincial

indicator – narrowed scope)

♦ Add a question specific to

Collaboratives on the

DHA PHC Organization

survey*

4.2 How satisfied were Collaborative

participants with their ability to influence

PHC practice around diabetes management

in the district (project-specific question)?

♦ Participant satisfaction with

ability to affect district-wide

change in practice (project-

specific indicator)

♦ Participant follow-up

survey

Impact 2: Quality primary health care services are safe, efficient, effective, affordable and acceptable to the

community.

♦ % of PHC clients/patients, 18

years and over with diabetes

mellitus who received annual

testing within the past 12 months

for all of the following:

o HbA1c testing

o Full fasting lipid profile

screening

o Diabetic nephropathy

screening

o Blood pressure measurement

o Obesity/overweight screening

(provincial indicator)

♦ Data from electronic

medical records (EMRs)

from Collaborative

participants who have

them; manual chart audit

for Collaborative

participants who do not

have EMRs (before

Collaborative activities

and 12 months after

Collaborative ends)

♦ % of PHC clients/patients, 18

years and over, with diabetes

mellitus in whom the last HbA1c

was 7.0% or less in the last 15

months (provincial indicator)

As above

2.1 What mechanisms are in place to

ensure safe patient care and to decrease

patient risk? (provincial question)

♦ % of PHC clients/patients, 18 to

75 years, with diabetes mellitus

who saw an optometrist or

ophthalmologist within the past

24 months (provincial indicator)

As above

* More details about the PHC Organization survey are provided in the next chapter

.



March 31, 2008

23

Other Content in the Evaluation Framework

As you can see from the example evaluation matrix, after defining evaluation questions and

indicators, the next step is to develop data collection tools, which is the subject of the next chapter

in this Reference Manual. Descriptions and copies of the data collection tools to be used in the

evaluation are included in the evaluation framework, as is the planned time table for data collection

and the person responsible for implementing each aspect of the evaluation.

Essentially, your evaluation framework should clearly describe:

♦ activities and intended changes of the program or initiative through a logic model;

♦ purpose of the evaluation;

♦ evaluation questions that will be addressed through the evaluation, highlighting those that

support the provincial PHC evaluation framework;

♦ indicators associated with each evaluation question, highlighting those that support the

provincial PHC evaluation framework;

♦ data collection tools, time tables for implementation; and the

♦ people responsible for data collection, entry, analysis and reporting;

Applying the Standards

As we have noted in previous steps, you can help to ensure the quality of your evaluation by

considering relevant evaluation standards throughout the evaluation process. The standards that

most directly apply to this step – Developing the Evaluation Matrix – are utility, feasibility, propriety,

and accuracy. As you participate in focusing the evaluation design, the questions presented in the

following table can help you to clarify and achieve these standards.

Questions Standards

♦ How can the evaluation be planned so that it, in

turn, encourages use by stakeholders?

♦ Are the major evaluation questions of interest to

your stakeholders identified in the evaluation

matrix?





♦ Are the evaluation procedures practical? Will they

keep disruption of daily activities to a minimum?

♦ Have you considered the political interests and

needs of various groups in planning the evaluation?

♦ Have you assessed the costs of technical resources

and time?

Feasibility: Ensures that the




.



March 31, 2008

24

Questions Standards

♦ Have you addressed issues of conflict around

evaluation and evaluation priorities openly and

honestly?

♦ Have you planned sound, ethical consistent

procedures to ensure findings are correct?

♦ Is the evaluation complete and fair in assessing all

aspects of the program, including its strengths and

weaknesses?

♦ Are individuals clear about what is to be done, how,

by whom, and when?

♦ Is there a written understanding in the form of an

evaluation framework?

Propriety: Ensures that the

evaluation is an ethical one,

conducted with regard for the

rights and interests of those

involved.

♦ Have you described the purposes and procedures of

the evaluation in detail?

♦ Can the purposes and procedures be identified and

assessed?


findings are considered correct.

.



March 31, 2008

25

DDEEVVEELLOOPP YYOOUURR IINNSSTTRRUUMMEENNTTSS AANNDD GGAATTHHEERR TTHHEE

DDAATTAA


The following chapter provides an overview of the importance of gathering quality data that is

credible (i.e., useful and accurate). How the PHC evaluation framework and specifically, the

instruments developed, can support the gathering of credible data is reviewed. This is followed by

a brief description of each instrument. The chapter concludes with a review of existing information

systems that can potentially provide information on indicators within the PHC evaluation

framework.

WHY IS IT IMPORTANT TO GATHER CREDIBLE DATA?

Stakeholders should view the information gathered as believable, trustworthy, and relevant to their

questions. Having credible data strengthens evaluation results as well as the recommendations that

follow from them. Recognizing that all types

of data have limitations, you can strengthen

the credibility of an evaluation design by

using multiple procedures for gathering,

analyzing and interpreting data – this is

called triangulation.

Selecting multiple sources of data provides an

opportunity to include different perspectives

about the initiative and mixing perspectives

provides a more comprehensive view of the

initiative. In addition, using multiple data

collection strategies including qualitative and

quantitative methods can yield data that is

more complete and useful, meeting the needs

and expectations of a wider range of stakeholders. Increased upfront participation by stakeholders

also enhances credibility because they will be more likely to accept the evaluation’s conclusions and

act on its recommendations.

Critical to collecting credible data is the development of well-defined indicators (described in the

previous chapter) as well as designing instruments that are valid and reliable. In order to conduct a

Triangulation is a process that

involves collecting information about the same

question or issue using different methods (e.g.,

questionnaires or surveys, interviews, focus

groups) or different data collection sources

(e.g., patients or clients of a program or service,

team members, Board members). In general,

the greater the variety of sources and methods

used and the greater the number of viewpoints

sought for a particular indicator, the stronger

the resulting evidence

.



March 31, 2008

26

high quality, accurate evaluation, it is important to ensure that your data collection tools and

instruments are valid and reliable. Instruments without validity and reliability may produce

evaluation results which are unfounded or not warranted.

WHAT DO VALIDITY AND RELIABILITY MEAN?

Validity refers to the extent to which the tool or instrument accurately reflects or assesses the

specific concept/activity intended to be measured. For example, if you wanted to measure levels of

aerobic activity you may ask somebody on a survey “how long did you run this week”? However

if you were trying to measure aerobic activity in terms of cardiovascular benefits, this question

would not measure variables such as the intensity of the running- a critical measurement of aerobic

activity with cardiovascular benefits. Therefore a measure must be carefully written to assess the

desired variable.

Reliability refers to the extent to which the tool or instrument yields the same result with repeat

use. A simple example of reliability may be the use of a thermometer to measure whether or not

you have a fever. Say you measure your temperature once and it reads 103 F, concerned you

decided to re-take your temperature a couple of minutes later. The thermometer now reads 100 F.

Confused by the results you re-take your temperature for a third time and you get a normal reading

of 98 F. In this scenario this thermometer would be considered an unreliable tool as it has yielded

different results upon multiple uses while other variables (such as time) were held constant. If

upon reading your temperature three times and the thermometer gives a reading of 98.3 F, 98.5 F,

98.6 F, this slight variability may not warrant the thermometer being deemed ‘unreliable’ as these

differences may include a acceptable level of error or ‘noise’. Therefore prior to using a tool, it

should be tested to determine how much random error is in its measurement. This may involve a

variety of pre-tests of the tool such as having people rate the same performance of a task or skill.

HOW DOES THE PHC EVALUATION FRAMEWORK SUPPORT THE

GATHERING OF CREDIBLE DATA?

During the development of the PHC evaluation framework it was determined that there were a lack

of data collection sources and methods to gather information related to the indicators. Therefore, to

support evaluation efforts in PHC within District Health Authorities, two data collection tools were

developed and pilot tested – a Primary Care Organization (PCO) survey and a PHC Team survey.

A third instrument, a Community/Client survey, was developed through the Primary Care

Research Unit, Department of Family Medicine at Dalhousie University and mapped well to several

indicators in the PHC evaluation framework so it was adapted and further pilot tested. All of these

instruments were developed with the input of key stakeholders and consistent with current

research in PHC. In addition, the pilot testing helped to assess the validity and reliability of the

.



March 31, 2008

27

instruments. A report of the findings of the testing of these instruments is available through the

Primary Health Section of the Nova Scotia Department of Health. These three instruments will help to support the collection of quality data that is credible: that is,

useful and accurate. In addition, when District Health Authorities use these instruments this

supports a common approach to evaluation across the province. The use of a common evaluation

approach ensures a greater likelihood of collecting consistent, reliable and valid information. The

collection of consistent data or information across Nova Scotia will also support the creation of a

provincial picture of PHC programs and services, and means that stakeholders don’t have to

reinvent the wheel each time PHC programs and services are evaluated.

It is also recognized that DHAs may need to make changes to the instruments to reflect the unique

needs and circumstances of their communities and PHC teams. For example, some questions on

the instruments may not be applicable or relevant, or wording or terminology may need to change.

The instruments are not meant to tell District Health Authorities or local practices what information

to collect or how to collect it. Rather, these instruments have been created to support local

evaluation efforts and help make the evaluation process more management.

A brief description of each instrument follows.

The Primary Care Organization Survey

The purpose of the Primary Care Organization (PCO) survey is to assess various aspects of Primary

Health Care including access to programs and services, type of programs and services, planning,

accountability mechanisms, collaboration, quality improvement initiatives, organizational polices

and procedures, and information technology and communication. The term “primary care

organization” is used to refer to any group of primary health care professionals or providers that

work together to offer primary health care services to a defined population. For example, the

phrase can refer to a community health centre or a physician group practice. The survey is to be

completed by a team of at least two people in the primary care organization who together have a

good understanding about all administrative and clinical operations of the organization, and takes

approximately 30 minutes to complete. Appendix 2 provides a copy of the survey.

The PCO survey gathers information related to both the outputs and impacts in the PHC

Evaluation Framework and helps to answer the following evaluation questions in the framework:

♦ What forms of evidence were used by the Department of Health, DHAs and Primary Health

Care teams to inform PHC planning and delivery?

♦ Are accountability relationships in the PHC system clearly defined?

♦ What processes and mechanisms are developed and implemented to coordinate PHC services

within PHC teams and between programs, levels of care and sectors?

♦ Do PHC teams offer a range of services and programs that provide an equitable balance

between health promotion and the provision of health care services?

.



March 31, 2008

28

♦ Are there mechanisms designed to support collaboration with PHC teams and district and

provincial structures?

♦ What mechanisms and/or learning opportunities are in place to build the capacity of DHAs,

PHC health providers, communities and individuals to participate in PHC planning and

delivery?

♦ How are the cultural, spiritual, racial and other diverse needs of Nova Scotians assessed and

addressed by the PHC team?

♦ Do Nova Scotians have access to timely routine care, health promotion programs and

preventive services, screening services, health care for urgent needs, and 24/7 health

information or advice?

♦ Are there sufficient numbers of physicians, FPNs and NPs, to meet the demand for PHC?

♦ What mechanisms are in place to ensure safe patient care and to decrease patient risk?

The PHC Team Survey

The purpose of the PHC Team survey is to assess various aspects of Primary Health Care including

collaboration and continuity of care, team functioning, and work satisfaction. The survey also

gathers data related to provider demographics and professional development. The survey is

intended to be filled out by members of a primary care organization or team. The survey is

completed individually by members of the primary health care team based on their experiences as a

member of the team and takes approximately 20 minutes to complete. Members of the team may

include physicians; nurses; other allied health professionals such as pharmacists, social workers,

etc.; and office administration and management. Appendix 2 provides a copy of the survey.

The PHC Team survey gathers information related to both the outputs and impacts in the PHC

evaluation framework and helps to answer the following evaluation questions in the framework:

♦ What processes and mechanisms are developed and implemented to coordinate PHC services

within PHC teams and between programs, level of care and sectors?

♦ Are there mechanisms designed to support collaboration within PHC teams, and districts and

provincial structures?

♦ What is the extent and nature of multidisciplinary teams?

♦ Are PHC providers practicing to their full scope of practice?

♦ What mechanisms and/or learning opportunities are in place to build the capacity of DHAs,

PHC providers, communities and individuals to participate in PHC planning and delivery?

♦ How are the cultural, spiritual, racial and other diverse needs of Nova Scotians assessed and

addressed by the PHC team?

♦ Do clients experience continuity of care?

♦ Are PHC providers satisfied with their work?

.



March 31, 2008

29

The Community/Client Survey

The purpose of the Community/Client survey is to assess various aspects of Primary Health Care

including access to services, programs and information; continuity of care; quality of programs and

services; client satisfaction with programs

and services; and health promoting

behaviours and health literacy. The

survey is administered through the

telephone to a random sample of

community members and takes

approximately 15 to 20 minutes to

complete. The survey has the potential to

be adapted for a local primary health care

organization (e.g., a practice or

Community Health Centre) and can be

administered through the mail to a

random sample of clients/patients.

Appendix 2 provides a copy of the survey.

The Community/Client survey gathers information related to both the outputs and impacts in the

PHC evaluation framework and helps to answer the following evaluation questions in the

framework:

♦ Do patients/clients feel satisfied that they are receiving quality PHC services from all members

of the PHC team?

♦ Do Nova Scotians have a continuing relationship with a PHC team?

♦ Do Nova Scotians have access to timely routine care, primary health care teams, health

promotion programs and preventive services, screening services, health care for urgent needs

and 24/7 health information or advice?

♦ Are individuals and families satisfied with the level of access to PHC services?

♦ Do Nova Scotians experience language barriers to accessing primary health care programs and

services?

♦ Do individuals feel like a partner in their care with their PHC team?

♦ Do clients experience continuity of care?

♦ What mechanisms are in place to ensure safe patient care and to decrease patient risk?

♦ Do Nova Scotians have improved health-promoting behaviours?

♦ Do Nova Scotians have improved health literacy?

♦ Do PHC organizations enable patients with chronic health conditions (e.g., diabetes, asthma,

coronary heart disease, depression, hypertension) to develop competencies and self-efficacy for

better managing their health?

Currently under development by Statistics Canada and the Health Council of Canada is a national

survey to obtain patient experiences with the PHC system including programs and services offered.

Sampling: Using a part of the

population in order to understand what is

occurring in the larger population. Sampling

is used to cut costs and effort while still

obtaining information from a representative

sample of the target population. It is essential

that the number of individuals providing

information for the evaluation be large

enough to produce results that are reliable and

valid and truly represent the target

population.

.



March 31, 2008

30

This survey, entitled, The Canadian Survey of Experiences with Primary Health Care gathers data on

many of the PHC indicators in the provincial evaluation framework that are collected through the

Community/Client survey. This survey was administered in 2007 at a national level and

modifications are currently being made to the survey with a second administration planned for

2008 with a larger sample size to enable provincial comparisons. There is the opportunity for

provinces to purchase larger sample sizes to enable regional (and in the case of Nova Scotia,

District) level comparisons. If DHAs are interested in implementing the Community/Client survey

it would be worthwhile to consult with the Primary Health Care section in the Department of

Health to determine the status of implementation of The Canadian Survey of Experiences with Primary

Health Care.

WHAT INFORMATION CAN I GET FROM EXISTING INFORMATION

SYSTEMS?

As previously reported, there are a lack of data sources for the majority of the indicators in the PHC

evaluation framework. There is a need to build information systems to support PHC evaluation at

the national, provincial and local levels. As these systems continue to be developed, there are

potential opportunities to gather data for some of the PHC indicators from existing information

systems, which are described below.

Existing Provincial Databases

Information for some of the indicators in the PHC evaluation framework can be gathered through

existing provincial databases. While these databases can be an efficient way to gather data for your

evaluation, there can be limitations in using existing databases that need be recognized and

reported. For example, a few of the PHC indicators can be gathered through the Medical Services

Information (MSI) database. However, because the data in the MSI database is gathered for billing

purposes it may provide incomplete or inaccurate information if you are using it to report on

quality or quantity of PHC services. In

addition, currently there is no system in place

for DHAs to access this information, and

there can be costs associated with obtaining

this information as well as variable wait times

for data to be extracted.

Appendix 3 provides a list of the PHC

indicators that can potentially be accessed through the MSI database. However, given the

limitations in using existing databases, and the expertise required to extract and analyze the data, it

is recommended that DHAs and local primary care organizations engage appropriate expertise and

support in requesting and reporting on data from the MSI and other databases.

Database: A structured file of

information or a set of related data that

are stored, sorted and retrieved, most

often using a computer.

.



March 31, 2008

31

Existing Surveys

There is the potential to gather some of the PHC indicators from existing surveys such as the

National Physician Survey and the Canadian Community Health Survey. Currently there are

issues for DHAs in accessing this data as there are often limitations in terms of sample size to allow

for data to be abstracted at a local level. In addition, only a few of the PHC indicators can currently

be collected through these surveys.

However, the Canadian Institute for Health Information (CIHI) is exploring modifying these

surveys by adapting existing questions, developing new questions or increasing the sample sizes to

capture data for many of the CIHI PHC indicators (up to 69 of the 105 indicators). Therefore, in the

future there will be the opportunity for DHAs to obtain data on more of the PHC indicators.

Although issues such as purchasing greater sample sizes to allow for local level analysis will need

to be addressed, by leveraging existing surveys DHAs may be able to expand the collection and

reporting of PHC data in a cost effective way.

Chart Audits

Another source of data to evaluate primary health care is through a review of charts from primary

care organizations (e.g., practices, community health centres). The information from patient charts

can primarily provide information on the quality of primary health care services and relate to two

impacts in the PHC evaluation framework: Quality primary health care services are safe, efficient,

affordable and acceptable to the community, and Populations have better health literacy and health promoting

behaviours. The information from the charts can help to answer two questions from the PHC

Evaluation Framework:

♦ What mechanisms are in place to ensure safe patient care and to decrease patient risk? (e.g.,

are Nova Scotians receiving PHC services based on the best available evidence); and

♦ Do Nova Scotians have improved health-promoting behaviours?

Ideally, the electronic medical record (EMR) currently being implemented in primary care

organizations across Nova Scotia will be able to provide data on these impacts and associated

indicators in the PHC evaluation framework. However, the implementation of the electronic

medical record, known as PHIM (Primary Health Care Information Management) is in the early

stages, and the functionality does not exist to extract information from the system to report on the

PHC indicators. Work is underway both nationally and provincially to build the functionality into

existing EMRs, and over time the ability to report on the PHC indicators will grow.

In the interim, for primary care organizations without an EMR or required functionality within

their EMR, another option to gather data for these indicators is to conduct a review of a sample of

written patient charts within the organization. A sample chart audit tool for a chart review is

provided in Appendix 2. It is important to understand that conducting a chart review requires

expertise in terms of selecting appropriate indicators, developing a chart audit tool, developing

criteria for review and extraction of the information, selecting a sample, reviewing and auditing the

.



March 31, 2008

32

charts, and compiling and analyzing the data. Therefore it is recommended that DHAs wishing to

conduct a chart audit consult with local or provincial experts with appropriate knowledge and

skills to support your chart review process.


As we have noted in previous steps, you can help ensure that your evaluation is a quality one by

considering relevant evaluation standards throughout the evaluation process. The two standards

that most directly apply to Developing Instruments and Gathering the Data are utility and accuracy.

As you gather the data, the questions presented in the following table can help you to clarify and

achieve these standards.

Questions Standards

♦ Have you reached an agreement on techniques to

analyze and synthesize findings before data

collection begins?

♦ Will the information collected address pertinent

issues about the program and is this responsive to

the needs of your stakeholders?





♦ Are the sources of information used in the program

described in adequate detail?

♦ Do the information-gathering procedures address

internal validity and reliability issues?

♦ Is there a system in place for identifying and

correcting errors?

♦ Has the process of analyzing quantitative and

qualitative data been effective in answering your

key evaluation questions?


evaluation findings are

considered correct.

.



March 31, 2008

33

AANNAALLYYZZIINNGG TTHHEE DDAATTAA AANNDD RREEPPOORRTTIINNGG YYOOUURR

FFIINNDDIINNGGSS

DATA ANALYSIS

After ‘raw’ data has been collected, it must be analyzed and

synthesized. Data analysis involves exploring and isolating

key or important findings from your data and synthesis

involves combining the various data collection sources to

reach a larger, richer understanding. The evaluator must

decide how they will organize, classify, compare and display

involvement.

Unobstructed access to facts can produce unlimited good only if it is matched by the desire and ability to

find out what they mean and where they lead. Facts are terrible things if left sprawlying and unattended.

- Norman Cousins (1981)

Common basic analytic synthesis strategies for qualitative and quantitative data are presented in

the following table:

Strategies Description

Describing ♦ Description of the program, its setting, staff, structure, activities, etc.

Counting ♦ Description which helps to show what is typical and what is irregular or

uncharacteristic

Clustering ♦ Pulling together like pieces of information

Comparing ♦ Judging the data against a period in time, other programs, differences between

participants, etc.

Finding Commonalities ♦ Finding common threads (e.g., common elements that characterize successful

participants, etc.)

Examining Deviant Cases ♦ Examining cases that are outliers for important clues to unanticipated phenomena

Both qualitative and quantitative data analysis can be quite complex and therefore may require

additional support, training or outside expertise. However, the following presents some instruction

on simple analysis techniques for qualitative and quantitative data.

Data: Observation or

measurement that can be

qualitative or quantitative.

.



March 31, 2008

34

Quantitative Data

Quantitative data in evaluation includes numerical information about a program or services such as

number of participants, number of patients with a particular condition, number of group education

sessions held, etc. Quantitative data analysis often involves the use of statistics. This type of

analysis can be highly complex requiring a great understanding of statistical methodologies,

however basic analysis can also be preformed and yield useful results.

Often simplified statistical analysis includes ‘descriptive statistics’ which includes basic calculations

such as means, standard deviations, averages, frequencies, etc. These operations may be easily

preformed using common software package such as Excel.

Qualitative Data

Qualitative data in evaluation often includes interviews or focus groups with program stakeholders.

Data often includes perceptions around the effectiveness, successes, challenges and facilitators of

programs and services. Presented below are simplified steps to qualitative data analysis:

♦ Read over the transcripts multiple times to build familiarity with the date

♦ Do you notice any similarities, trends, patterns or repeated ideas? Note some key terms/ideas

that are often repeated.

♦ Document these common themes and group the data (e.g., cut and paste similar transcript

passages together under the theme in a new document).

♦ Themes may be “ranked” according to strength. In qualitative data all information is

considered valuable; however you may want to present your themes according to the strength

of response (if the theme is repeated by many respondents the strength of response is higher).

♦ Describe each theme it terms of the original question. The language used to describe the theme

should reflect the strength of response (e.g., Most participants felt…, some participants noted…,

A few participants revealed…)

♦ Provide a few quotes to support your theme and its description

REPORTING YOUR FINDINGS

To get the full impact of your findings, it is important to report and share the results with the key

stakeholders. Determining the audience who the results will be shared with will dictate how the

findings should be reported and the writing style of the report. Although there is no “right” way

to present results, a sample report outline is provided below. This outline illustrates the

information that is important to include in your evaluation report.

.



March 31, 2008

35

Section Description

Executive Summary ♦ Brief summary of the report content

Background &

Purpose

♦ Provide background information on the program or initiative

♦ Purpose of the evaluation and the report

Methodology

♦ Overview of qualitative and quantitative methods used

♦ Description of tools (e.g., surveys, interview guides, etc.)

♦ Data collection procedures (e.g., sampling, number of participants, locations, etc.)

♦ Data analysis techniques

♦ Evaluation limitations

Results ♦ Presentation of findings in a logical order

- Arranged according to components of the logic model

Key Observations &

Recommendations

♦ Highlight key findings

♦ Propose next steps and recommendations for moving forward

Appendices ♦ Data collection tools and instruments (e.g., surveys, interview guides, etc.)

♦ Evaluation matrix


As we have noted in previous steps, you can help ensure that your evaluation is a quality one by

considering relevant evaluation standards throughout the evaluation process. The two standards

that most directly apply to Analysing and Reporting Your Findings are utility and accuracy. As you

analyze the data, the questions presented in the following table can help you to clarify and achieve

these standards.

Questions Standards

♦ Have you reached an agreement on techniques to

analyze and synthesize findings before data

collection begins?

♦ Will the information collected address pertinent

issues about the program and is this responsive to

the needs of your stakeholders?





♦ Are the sources of information used in the program

described in adequate detail?

♦ Do the information-gathering procedures address

internal validity and reliability issues?

♦ Is there a system in place for identifying and

correcting errors?

♦ Has the process of analyzing quantitative and

qualitative data been effective in answering your

key evaluation questions?


evaluation findings are

considered correct.

.



March 31, 2008

36

LLEEAARRNNIINNGG FFRROOMM AANNDD SSHHAARRIINNGG YYOOUURR RREESSUULLTTSS

Once the evaluation is complete it is important to share the results with your stakeholders and to

provide opportunities for program stakeholders to reflect upon the meaning of the results. It may

be useful to refer back to the Stakeholder Identification Worksheet that you created at the beginning

of your evaluation process. For each stakeholder, identify how they might be able to use the results

and the best mechanism for communicating the results to them.

To make your evaluation results most useful, it is helpful to schedule one or more meetings with

key program stakeholders to discuss the findings and their implications for the program. Key

discussion questions include:

♦ Do we need to make any changes to our program based on the results of the evaluation? If so,

what changes should be made? What changes are of highest priority?

♦ Did our evaluation process provide us with all of the information we need to make informed

decisions about the program? To satisfy our funding agencies? To demonstrate accountability

for the program objectives?

♦ Is there anything we would like to do differently in our evaluation process as we go forward?

Do we need to change any evaluation questions? Indicators? Data collection tools?

♦ Are there any other audiences we need to share these results with? If so what are the key

messages for those audiences? What method should be used to communicate key messages to

each audience?

.



March 31, 2008

37

FFIINNAALL TTHHOOUUGGHHTTSS AABBOOUUTT PPHHCC EEVVAALLUUAATTIIOONN IINN NNOOVVAA

SSCCOOTTIIAA

Considerable work has been done in Nova Scotia to build a consistent approach to PHC evaluation

across the province. There is a three level evaluation approach in place – System, District and

Project Levels. This evaluation system is designed to ensure flexibility in what is evaluated for

DHAs and specific initiatives while at the same time promoting consistency across all PHC

evaluation in the province.

Now it is up to you. The only way that Nova Scotia will begin to have a consistent approach to

evaluation is if everyone who is involved in PHC evaluation in Nova Scotia commits to using and

continuously improving upon the work that has been done to date. A consistent approach to PHC

evaluation across the province will help to build a knowledge base about the performance of the

PHC system upon which policy makers can draw to make informed decisions about the future of

the PHC system.

There is still much development to be done to support PHC evaluation in Nova Scotia. There are

currently gaps in data availability and human resources to comprehensively measure PHC in Nova

Scotia. However, we can begin to build the case for more investment in these areas if we commit to

using a consistent and coordinated approach across the province.

It is an exciting time for PHC evaluation in Nova Scotia. Thank you for being a part of the process.

.



March 31, 2008

38

RREEFFEERREENNCCEESS

Baker, Q. E., et al. (2000). An Evaluation Framework for Community Health Programs. The Centre for

the Advancement of Community Based Public Health

Enhancing the Primary Health Cara Data Collection Infrastructure in Canada - Report 2 (2006). Canadian

Institute for Health Information.

Pan-Canadian Primary Health Care Indicators (2006). Canadian Institute for Health Information,

Report 1, Vol 1.

Pan-Canadian Primary Health Care Indicators (2006). Canadian Institute for Health Information,

Report 1, Vol 2.

Pencheon, David. The Good Indicators Guide: Understanding How to Use and Choose

Indicators. Published by the Association of Public Health Observatories and the NHS Institute for

Innovation and Improvement. (no publication date is listed on the report)

Pyra Management Consulting Services Inc. (2005). Developing Effective Logic Models. Published by

author.

Pyra Management Consulting Services Inc. and Research Power Incorporated (2006). A Primary

Health Care Evaluation System for Nova Scotia. Nova Scotia Department of Health.

Sampson, Y., et al. (2003). Valuing Our Work: A Resource Kit on the Evaluation and Reporting System for

CAPC and CPNP in Atlantic Canada. Population and Public Health Branch, Atlantic Regional Office,

Health Canada.

.



March 31, 2008

39

AAPPPPEENNDDIICCEESS

.



March 31, 2008

40

AAPPPPEENNDDIIXX 11:: EEVVAALLUUAATTIIOONN QQUUEESSTTIIOONNSS AANNDD IINNDDIICCAATTOORRSS FFRROOMM TTHHEE

PPRROOVVIINNCCIIAALL PPHHCC EEVVAALLUUAATTIIOONN FFRRAAMMEEWWOORRKK

.



March 31, 2008

41

Process and Structure Output 1: Planning for service delivery is informed by evidence.

Evaluation Question Indicators

• % of PHC organizations who used information on the

composition of their practice population to allocate resources

for programs/services, over the past 12 months.

(CIHI 5)

• % of PHC organizations who currently have a PHC

client/patient registry for chronic conditions.

(CIHI 6)

• % of PHC organizations who currently have specific

programs for PHC clients/patients with specific chronic

conditions.

(CIHI 7)

• % of PHC organizations who currently do outreach to deliver

PHC services to vulnerable/special population groups.

(CIHI 9)

• % of PHC organizations who currently provide specialized

programs for vulnerable/special needs population groups.

(CIHI 10)

• % of PHC organizations who currently receive information

or process support from their health region to serve

vulnerable/special needs population groups.

(CIHI 11)

1.1 What forms of evidence were used by the

Department of Health, District Health

Authorities (DHAs) and Primary Health

Care (PHC) teams to inform PHC planning

and delivery?

• % of PHC organizations who primarily use electronic systems

to complete their professional tasks

(CIHI 100)

Process and Structure Output 2: Accountabilities within relationships in the primary health care system are

clearly defined.


• PHC is part of DHA and DOH strategic plans and business

plans.

• Number and type of human and fiscal resources dedicated to

PHC planning.

2.1 Do DHAs and the DOH integrate PHC

planning into strategic and business plans

for the health system?

• Number and type of structures for PHC delivery.

• Number and type of accountability mechanisms for PHC

teams/organizations, and partnerships.

2.2 Are accountability relationships in the

PHC system clearly defined?

• % of PHC organizations who currently have processes to

involve community input into planning the organization's

services (e.g. advisory committees, focus groups)

(CIHI 8)

.



March 31, 2008

42

Process and Structure Output 3: Services and programs are delivered in a coordinated and integrated way.


• % of PHC organizations who currently coordinate

client/patient care with other health care organizations using

standardized clinical protocols or assessment tools.

(CIHI 79)

• % of PHC organizations who currently have collaborative

care arrangements with other health care organizations.

(CIHI 80)

• % of PHC organizations who currently have collaborative

care arrangements with providers/organizations beyond the

health care sector (e.g. housing, justice, police, education,

etc.).

(CIHI 81)

• % of PHC providers who had complete information (essential

demographic and clinical information) at the point of care,

most of the time, over the past 12 months.

(CIHI 84)

• % of PHC organizations who currently use a variety of

electronic communications modalities in the exchange of

health care information with other PHC providers

(CIHI 101)

3.1 What processes and mechanisms are

developed and implemented to coordinate

PHC services within PHC teams and

between programs, levels of care, and

sectors?

• % of PHC organizations who currently have two-way

electronic communication linkages (beyond fax and

telephone) with other health care organizations (e.g.

hospitals, community mental health agencies, LTC facilities,

public health, etc).

(CIHI 102)

Process and Structure Output 4: A range of services and programs provide an equitable balance between health

promotion and provision of health care services.


• % of DOH budget allocated to health promotion

programming, screening services and acute care service

delivery.

4.1 What percentage of the provincial and

district health budget is allocated to health

promotion programming, screening

services, and acute care service delivery? • % of DHA budget allocated to health promotion

programming, screening services and acute care service

delivery.

4.2 Do PHC teams offer a range of services

and programs that provide an equitable

balance between health promotion and the

provision of health care services?

% of PHC organizations who currently provide the following

services:

• Management care for an emergent but minor health problem

(e.g., sprained ankle, unexplained rash, etc),

• Non-urgent care (e.g. well care (baby, child, woman and/or

man), chronic illness management, etc.),

• Prevention and health promotion and/or education services,

• Full maternity and child care,

• Primary mental health care,

• Psychosocial services (e.g. counseling advice for

physical/emotional/ financial problems, etc),

• Liaison with home care,

• Referral to and follow-up care from specialized agencies

such as hospitals, youth centres, specialists and/or other

providers (through formalized arrangements and/or

agreements),

.



March 31, 2008

43

• Rehabilitation services,

• Nutrition counseling services,

• Provision of home visits by PHC physicians/nurses/nurse

practitioners/ pharmacists, and

• End-of-life care.

(CIHI 12)

• % of PHC clients/patients who report that the current range

of services offered by their PHC organization meets their

needs.

(CIHI 98)

Process and Structure Output 5: Primary health care providers collaborate.


• % of PHC team/organization resources allocated for team

building.

• % of primary care physicians on alternative funding

arrangements.

• % of PHC teams/organizations with collaborative practice

arrangements.

% of PHC teams/organizations that have mechanisms to support

collaboration:

• Regular Team meetings,

• Joint goal setting,

• Shared vision.

5.1 Are there mechanisms designed to support

collaboration within PHC teams, and

district and provincial structures?

Average team effectiveness score based on:

• Strong leadership,

• Clear objectives shared by all team members,

• Mechanisms for working in and with the community,

• Focus on quality care,

• Client/Patient focused goals,

• Efficient and effective communication,

• Appropriate variety of health care providers,

• Mechanisms for conflict resolution,

• Interdisciplinary professional development,

• Shared decision-making, and

• Clear understanding of scope of practice and team role.

(CIHI 99)

• % of GPs/FPs who currently work in a solo PHC practice as

their main PHC practice setting.

(CIHI 95)

• % of GPs/FPs who currently work in a group physician PHC

practice as their main PHC practice setting.

(CIHI 96)

5.2 What is the extent and nature of

multidisciplinary teams? (CIHI)

• % of GPs/FPs/NPs who are currently working in an

interdisciplinary PHC team or network as their main practice

setting, by type of PHC provider.

(CIHI 97)

.



March 31, 2008

44

Process and Structure Output 6: Primary health care providers’ knowledge, skills and attitudes are appropriate

to the services provided.


• % of PHC providers and support staff whose PHC organization

provided them with support to participate in continuing

professional development within the past 12 months, by type of

PHC provider and support staff.

(CIHI 72)

6.1 Are PHC providers practicing to their full

scope of practice?

• % of PHC providers who are satisfied that they utilize the full

extent of their skills, by type of PHC provider.

(CIHI 88)

• % of PHC clients/patients, 18 years and over, who were satisfied

with the care received from their regular PHC provider, over the

past 12 months.

(CIHI 73)

• % of the population, 18 years and over, who were satisfied with

the telephone health information or advice line, over the past 12

months.

(CIHI 74)

• % of PHC clients/patients, age 18 years and over, who would

recommend their regular PHC provider to their family or friends.

(CIHI 75)

• % of PHC clients/patients, age 18 years and over, who were

involved in clinical decision-making regarding their health with

their regular PHC provider, over the past 12 months.

(CIHI 76)

This indicator could also be related to impact #4

• % of PHC clients/patients, 18 years and over, whose regular PHC

provider treated them in a caring manner, over the past 12 months.

(no longer on the final CIHI list)

This indicator could also be related to impact #4

6.2 Do patients feel satisfied that they are

receiving quality PHC services from all

members of the PHC team?

• % of PHC clients/patients who were satisfied with the level of

privacy provided by their PHC organization (e.g. staff in

reception, clinicians in exam room), over the past 12 months.

(CIHI 77)

Process and Structure Output 7: Individuals, health providers and communities have the capacity to participate

in PHC planning and delivery.


• Number and type of learning opportunities provided for CHBs and

DHA Boards to build their capacity to participate in PHC planning

and delivery.

• Number and type of learning opportunities provided to PHC

providers to build their capacity to participate in PHC planning

and delivery.

• % of PHC providers participating in opportunities that build their

capacity to participate in PHC planning & delivery.

7.1 What mechanisms and/or learning

opportunities are in place to build the

capacity of DHAs, PHC health providers,

communities, and individuals to participate

in PHC planning and delivery?

• % of PHC organization who currently have processes to involve

community input into planning the organization's services (e.g.

advisory committees, focus groups).

(CIHI 8)

.



March 31, 2008

45

Process and Structure Output 8: Services are offered in ways that respond to individuals’, families’ and

communities’ cultural, racial, spiritual, and other diverse needs.


• % of PHC clients/patients, 18 years and over, who experienced

language barriers when communicating with their regular PHC

provider, over the past 12 months.

(CIHI 78)

8.1 Do Nova Scotians report that the services

they receive are responsive to their

cultural, racial, spiritual and other diverse

needs?

• % of community members who report that the services they

receive are responsive to their cultural, racial, spiritual and other

diverse needs.

• % of PHC providers who report participating in learning

opportunities in the past 12 months to increase their knowledge

and skills to assess and address the cultural, spiritual, racial and

other diverse needs of clients.

8.2 How are the cultural, spiritual, racial and

other diverse needs of Nova Scotians

assessed and addressed by the PHC team?

• % of PHC teams/organizations who report that they currently use

tools and/or processes to assess the cultural, spiritual, racial &

other diverse needs of patients/clients.

Process and Structure Output 9: Nova Scotians have a continuing relationship with a primary health care

provider through whom they can access services.


• % of population who currently have a regular PHC provider, by

type of PHC provider.

(CIHI 1)

• % of population, 18 years and over, who experienced difficulties

obtaining required routine or ongoing PHC services, from their

regular PHC provider, over the past 12 months.

(CIHI 2)

9.11 Do Nova Scotians have a continuing

relationship with a PHC team?

• % of population, 18 years and over, who experienced difficulties

obtaining the required health information or advice, from their

regular PHC provider, over the past 12 months.

(CIHI 3)

Impact 1: Individuals and families have access to services, program and information


• % of population, 18 and over, who experienced difficulties

obtaining immediate care for an emergent but minor health

problem, from their regular PHC provider, over the past 12

months.

(CIHI 4).

• % of population, 18 and over, who experienced difficulties

obtaining immediate care for an emergent but minor health

problem, from their regular PHC provider, during evenings and

weekends (5:00am to 9:00pm, Monday to Friday, or 9:00am to

9:00pm, Saturday and Sundays), over the past 12 months.

(CIHI 29).

• % of PHC organizations who currently provide after hours

coverage (beyond 9:00am to 5:00pm Monday to Friday) for their

practice population.

(CIHI 30)

1.1 Do Nova Scotians have access to:

b) Timely routine care

c) Primary health care (PHC) team

d) Health promotion programs and

preventive services

e) Screening services

f) Health care for urgent needs

g) 24/7 health information or advice

• Average number of extended hours (beyond 9:00am to 5:00pm,

Monday to Friday), provided by PHC organizations per month, by

PHC organization.

(CIHI 31)

.



March 31, 2008

46


• Average length of time in days between PHC client/patient

appointment request with their regular PHC provider, and the

appointment for an emergent but minor health problem.

(CIHI 32)

• % of population who received PHC services from an

interdisciplinary PHC organization, over the past 12 months.

(CIHI 94).

• % of population 18 and over who experienced difficulties in

obtaining routine primary health care from a regular primary

health care provider over the last 12 months

(Nova Scotia unique indicator)

• % of PHC clients/patient, 18 years and over, who are satisfied

with wait time to obtain an appointment with their regular PHC

provider for an emergent but minor health problem.

(CIHI 33)

1.2 Are individuals and families satisfied with

the level of access to PHC services?

• % of PHC clients/patients, 18 years and over, who are satisfied

with wait time to obtain an appointment with their regular PHC

provider for non-urgent routine care.

(CIHI 34)

1.3 Do Nova Scotians experience language

barriers in accessing primary health care

programs and services?

• % of PHC clients/patients, 18 years and over, who experienced

language barriers when communicating with their regular PHC

provider, over the past 12 months.

(CIHI 78)

• PHC provider full time equivalents (FTEs) per 100,000

population, by type of PHC provider.

(CIHI 85)

• Ratio of PHC providers entering/leaving the workforce over the

past 12 months by type of PHC provider.

(CIHI 86)

• % PHC organizations who are currently accepting new PHC

clients/patients.

(CIHI 87)

1.4 Are there sufficient numbers of physicians,

FPNs and NPs, to meet the demand for

PHC? (CIHI)

• % of Districts that are currently implementing a plan to meet their

PHC health human resource needs.

(CIHI 93)

Impact 2: Individuals and families experience continuity of care.


2.1 Do individuals feel like a partner in their

care with their PHC team?

• % of PHC clients/patients, 18 years and over, who were involved

in clinical decision-making regarding their health with their

regular PHC provider over the past 12 months.

(CIHI 76)

• % of PHC providers who had complete information (essential

demographic and clinical information) at the point of care, most of

the time, over the past 12 months.

(CIHI 84)

• % of PHC clients/patients, 18 years and over, who felt that

unnecessary medical tests were ordered because the test had

already been done, over the past 12 months.

(CIHI 82).

2.2 Do clients experience continuity of care

(e.g. coordinated and integrated)?

• % of PHC GPs/FPs/NPs who repeated tests because findings were

unavailable, over the past month.

(CIHI 83)

.



March 31, 2008

47

• % of PHC clients/patients, 18 years and over who visited a

specialist in the past 12 months who discussed the specialist visit

with their regular PHC provider.


Impact 3: The primary health care workforce is healthy and satisfied.


• % of PHC providers who report that there are currently adequate

provisions to ensure their safety in their workplace, by type of

PHC provider.

(CIHI 89)

3.1 Are PHC providers satisfied with their

work? (this question may be more an

output than an impact)

• % of PHC providers who were satisfied with the overall quality

of work life balance over the past 12 months, by type of PHC

provider.

(CIHI 92)

• % of PHC providers who missed work due to burnout (2 weeks or

more) over the past 12 months, by type of PHC provider.

(CIHI 91)

3.2 Is the PHC workforce healthy?

• % of PHC providers who had a workplace related injury over the

past 12 months, by type of PHC provider.

(CIHI 90)

Impact 4: Quality primary health care services are safe, efficient, affordable and acceptable to the community


• Age-standardized acute care hospitalization rate for conditions

where appropriate ambulatory care prevents or reduces the need

for admission to hospital, per 100,000 population, 75 years and

under (CIHI 35)

• % of PHC clients/patients, ages 18 to 64 years, with established

diabetes mellitus (Type 1 and Type 2) who have had an acute

myocardial infarction, or above or below knee amputation or

began chronic dialysis within the past 12 months.

(CIHI 36)

• % of PHC clients/patients, ages 6 to 55 years, with asthma who

visited the emergency department in the past 12 months.

(CIHI 37)

• % of PHC clients/patients, ages 20 to 75 years, with CHF who

visited the emergency department for CHF in the past 12 months.

(CIHI 38)

• % of PHC clients/patients, 18 years and over, with diabetes

mellitus in whom the last HbA1c was 7.0% or less (or equivalent

test/reference range depending on local laboratory) in the last 15

months.

(CIHI 39)

• % of PHC clients/patients, 18 years and over, with hypertension

for duration of at least one year, who have blood pressure

measurement control (e.g. less than 140/90 mmHg).

(CIHI 40)

• % of PHC clients/patients, 65 years and over, who received an

influenza immunization within the past 12 months.

(CIHI 41)

4.1 What mechanisms are in place to ensure

safe patient care and to decrease patient

risk?

(e.g., are Nova Scotians receiving PHC service

based on the best available evidence?)

• % of PHC clients/patients, 65 years and over, who received a

pneumococcal immunization.

(CIHI 42)

.



March 31, 2008

48

• % of PHC clients/patients who received screenings for congenital

hip displacement, eye and hearing problems, by 3 years of age.

(CIHI 43)

• % of PHC clients/patients, who received required primary

childhood immunizations by 7 years of age.

(CIHI 44)

• % of women PHC clients/patients, who had a live birth and

received counselling on breast feeding, education programs, and

postpartum support to promote breast feeding.

(CIHI 45)

• % of women PHC clients/patients who are pregnant or

postpartum who have been screened for depression.

(CIHI 46)

• % of women PHC clients/patients, 50 years and over, who

received screening for colon cancer with Hemoccult test within

past 24 months.

(CIHI 48)

• % of women PHC clients/patients, ages 50-69 years, who

received mammography and clinical breast exam within past 24

months.

(CIHI 49)

• % of women PHC clients/patients, ages 18-69 years, who received

papanicolaou smear within past 3 years.

(CIHI 50)

• % of women PHC clients, 65 years and over, who received

screening for low bone mineral density at least once.

(CIHI 51)

• % of women PHC clients/patients, 55 years and over, who had a

full fasting lipid profile measured within the past 24 months.

(CIHI 52)

• % of men PHC clients/patients, 40 years and over, who had a full

fasting lipid profile measured within the past 24 months.

(CIHI 53)

• % of PHC clients/patients, 18 years and over, who had their

blood pressure measured in the past 24 months.

(CIHI 54)

% of PHC clients/patients, 18 years and over, with coronary artery

disease (CAD) who received annual testing, within the past 12 months,

for all of the following:

• Fasting blood sugar,

• Full fasting lipid profile screening,

• Blood pressure measurement, and

• Obesity/overweight screening

(CIHI 55)

% of PHC clients/patients, 18 years and over, with hypertension who

received annual testing, within the past 12 months, for all of the

following:

• Fasting blood sugar,


• Test to detect renal dysfunction (e.g. serum creatinine),

• Blood pressure measurement, and


(CIHI 56)

.



March 31, 2008

49

% of PHC clients/patients, 18 years and over, with diabetes mellitus

who received annual testing within the past 12 months, for all of the

following:

• Hemoglobin A1c testing,


• Diabetic nephropathy screening (e.g. albumin/creatinine ratio,

microalbuminuria),

• Blood pressure (BP) measurement, and


(CIHI 57)

• % of PHC clients/patients, 18 to 75 years, with diabetes mellitus

who saw an optometrist or ophthalmologist within the past 24

months.

(CIHI 58).

• % of PHC clients/patients, ages 6 to 55 years, with asthma, who

were dispensed high amounts (greater than 4 canisters) of short-

acting beta2-agonist (SABA) within the past 12 months AND

who received a prescription for preventer/controller medication

(e.g. inhaled corticosteroid -ICS).

(CIHI 59)

• % of PHC clients/patients, 18 years and over, with CHF who are

using ACE inhibitors or ARBs.

(CIHI 60)

• % of PHC clients/patients, 18 years and over, with established

CAD and elevated LDL-C (e.g. greater than 2.5 mmol/L) who

were offered lifestyle advice and/or lipid lowering medication

(CIHI 61)

• % of PHC clients/patients who have had an AMI and are

currently prescribed a beta-blocking drug.

(CIHI 62)

• % of PHC clients/patients with depression, who are taking

antidepressant drug treatment under the supervision of a PHC

provider, and who had follow-up contact by a PHC provider for

review within two weeks of initiating antidepressant drug

treatment.

(CIHI 63)

• % of PHC clients/patients, 18 years and over, with depression

who were offered treatment (pharmacological and/or non-

pharmacological) or referral to a mental health provider.

(CIHI 64)

• % of PHC clients/patients, 18 years and over, with a diagnosis of

panic disorder or generalized anxiety disorder who were offered

treatment (pharmacological and/or non-pharmacological) or

referral to a mental health provider.

(CIHI 65)

• % of PHC clients/patients with prescription or illicit drug use

problems who were offered, provided or directed to treatment by

the PHC provider.

(CIHI 66)

• % of PHC providers whose PHC organization has processes and

structures in place to support a non-punitive approach to

medication incident reduction.

(CIHI 67)

.



March 31, 2008

50

• % of PHC organizations who currently use an electronic

prescribing/drug ordering system that includes patient specific

medication alerts.

(CIHI 68)

• % of PHC organizations who implemented at least one or more

changes in clinical practice as a result of quality improvement

initiatives over the past 12 months.

(CIHI 69)

• % of PHC organizations with a process in place to ensure that a

current medication and problem list is recorded in the PHC

client/patient's health record.

(CIHI 70)

• % of PHC clients/patients who report that their regular PHC

provider (e.g. FP/GP/NP) has not explained the side effects of

medications when prescribed, within the past 12 months.

(CIHI 71)

4.2 What is the per capita operational cost of

providing primary health care services at a

practice level? At a regional health

authority level (accounting for geographic

location)? (CIHI)

Average annual per capita operational expenditures of PHC services

for:

• Health Human Resources,

o General Practitioner/Family Physicians,

o Nurse Practitioners,

o Other PHC providers,

• Supplies,

• Equipment,

• Administrative/overhead, and

• Other ______________

(CIHI 103)

% of PHC providers who were primarily remunerated by the

following method over the past 12 months by type of provider:

• Fee for service,

• Salary,

• Capitation, and

• Mixed System.

(CIHI 104)

4.3 How are PHC providers paid? (CIHI)

• Average % of PHC provider income derived from each of the

following PHC funding models for one fiscal year, by type of

PHC provider:

• Fee for service,

• Salary,

• Capitation, and

• Mixed System.

(CIHI 105)

.



March 31, 2008

51

Impact 5: Populations have better health literacy and health promoting behaviors.


% of PHC clients/patients, 12 years and over, who were screened by

their PHC provider for the following common health risks over the

past 12 months:

• Tobacco use,

• Unhealthy eating habits,

• Problem drug use,

• Physical inactivity,

• Overweight status,

• Problem alcohol drinking,

• Unintentional injuries (home risk factors),

• Unsafe sexual practices, and

• Unmanaged psychosocial stress and/or depression.

(CIHI 13)

• % of PHC clients/patients who are smokers, 12 years and over,

who received specific help or information to quit smoking from

their PHC provider, over the past 24 months.

(CIHI 14)

• % of PHC clients/patients, with problem alcohol drinking, 12

years and over, who received specific help or information to

manage alcohol consumption, over the past 24 months.

(CIHI 15)

• % of PHC clients/patients with unhealthy eating habits, 12 years

and over, who received specific help or information on healthy

dietary practices from their PHC provider, over the past 12

months.

(CIHI 16)

• % of inactive PHC clients/patients, 12 years and over, who

received specific help or information on regular physical activity

from their PHC provider, over the past 12 months.

(CIHI 17)

% of PHC organizations who currently have specific programs and/or

initiatives (including self help and self management groups) to reduce

the following health risks in their practice population:

• Tobacco use,



• Obesity, and

• Physical inactivity.

(CIHI 18)

% of health regions who currently have specific programs and/or

initiatives (including self help and self management groups) to reduce

the following health risks in the population:

• Tobacco use,



• Obesity, and

• Physical inactivity.

(CIHI 1

5.1 Do Nova Scotians have improved health-

promoting behaviours?

• % of population, 12 years and over, who are current smokers.

(CIHI 20)

.



March 31, 2008

52

• % of population, 12 years and over, who currently consume five or

more servings of fruit and vegetables daily.

(CIHI 21)

• % of population who are currently overweight or obese.

(CIHI 22)

• % of population who currently engage in regular physical activity.

(CIHI 23)

• % of population, 12 years and over, who report heavy drinking

behaviour, in the past 12 months.

(CIHI 24)

• % of PHC clients/patients with children under 2 years who were

given information on child injury prevention in the home.

(CIHI 47)

5.2 Do Nova Scotians have improved health

literacy? • % of patients/clients 18 years or over who believe that their PHC

provider helps them to understand their health problems or

required treatment.


• % of PHC clients/patients 18 years and over, with a chronic health

condition(s), and/or informal caregivers whose PHC organization

provided them with useful health care information (e.g. pamphlets,

books, tapes, videos, websites, or other community resource lists)

over the past 12 months.

(former CIHI indicator)

• % of PHC clients/patients, 18 years and over, with chronic health

condition(s), whose PHC organization provided them with

resources to support self-management or self-help groups.

(CIHI 25)

• % of informal caregivers in PHC population who received support

for their care giving role from their PHC organization over the

past 12 months.

(CIHI 26)

• % of PHC clients/patients, 18 years and over, with a chronic

condition(s), who had sufficient time in most visits to confide their

health-related feelings, fears and concerns with their PHC

provider.

(CIHI 27)

5.3 Do PHC organizations enable patient with

chronic health conditions (e.g., diabetes,

asthma, coronary heart disease, depression,

hypertension) to develop competencies and

self-efficacy for better managing their

health? (CIHI)

• % of PHC clients/patients, 18 years and over, with a chronic

condition(s), who actively participated in the development of a

treatment plan with their PHC provider over the past 12 months.

(CIHI 28)

.



March 31, 2008

53

AAPPPPEENNDDIIXX 22:: PPRROOVVIINNCCIIAALL PPHHCC DDAATTAA CCOOLLLLEECCTTIIOONN TTOOOOLLSS

.



March 31, 2008

54

Primary Care Organization Survey

Revisions based on Pilot Test, October 2006

Introduction

There are many changes taking place in Nova Scotia to improve the delivery of primary health care

services throughout the province. As part of our ongoing efforts to support changes in primary

health care, the Nova Scotia Department of Health regularly conducts evaluations to monitor

changes over time as well as assess what is working and what needs to change. This survey is part

of that ongoing evaluation initiative.

This survey is intended to be filled out from the perspective of your primary care organization. The

term ‘primary care organization’ is used to refer to any group of primary health care professionals

that work together to offer primary health care services to a practice population. For example, the

phrase can refer to a community health centre or a physician group practice.

The survey should be completed by a team of at least two people who together have a good

understanding about all administrative and clinical operations of the primary health care organization.

The survey is designed to be completed anonymously. Please do not write your name (or any other

personally identifiable information) anywhere on the survey. To ensure confidentiality, the results of

the surveys will be reported in group form and only grouped data will be presented.

It should take you approximately 30 minutes to complete this survey. Thank you for your time in

helping us to evaluate primary health care change in Nova Scotia

Section 1: Information About the Primary Health Care Organization

1. Is your primary health care organization open during the following non-business

hours? (Please do not include on-call coverage.)

A. Weekday evenings � No � Yes __ days per week from ___ to ___

B. Saturdays � No � Yes, from ________to _______

C. Sundays � No � Yes, from ________to _______

2. Does your primary health care organization provide after hours on-call coverage to

the population served?

A. Weekday evenings � No � Yes __ days per week from____ to ___

B. Saturdays � No � Yes, from _______ to ________

C. Sundays � No � Yes, from _______ to ________

.



March 31, 2008

55

3. Do patients have access to on-call services through a:

A. Telephone � No � Yes

B. Walk-in clinic � No � Yes

C. Other (please describe) ________________________________

4. Is your primary health care organization currently accepting new patients?

Please circle one answer.

Yes No

5. Please indicate which of the following types of services are provided by your

organization to individual patients by placing a check mark in either the Yes or No

column for each item. Please feel free to add comments to explain any of your answers.

Type of service Yes No Comments

a Care for an emergent but minor

problem (e.g. sprained ankle,

unexplained rash)

b Non-urgent care (e.g. well woman care,

chronic illness management)

c Pre-natal maternity care

d Intrapartum care

e Postpartum care

f Mental heath services

g Behaviour change counselling about

tobacco use

h Behaviour change counselling about

healthy eating

i Behaviour change counselling about

physical activity

j Other health promotion or prevention

services

k Psychosocial services (e.g. counselling

advice for physical, emotional,

financial problems)

l Liaison with home care services

.



March 31, 2008

56


m Referral to and follow-up care from

specialized agencies such as hospitals,

youth centres, specialists and/or other

providers (through formalized

arrangements and/or agreements)

n Rehabilitation services

o Provision of home visits

P Specialized programs (other than

outreach services) for vulnerable or

special needs population groups

q Outreach services to vulnerable or

special population groups

r Other, specify

6. Please indicate which of the following programs (e.g. self help groups, education

sessions or workshops) are offered by your primary health care organization to groups

of patients/clients (versus individual services). Please feel free to add comments to

explain any of your answers.


a Specific programs to reduce tobacco

use among your practice population

b Specific programs to reduce unhealthy

eating among your practice population

c Specific programs to reduce problem

alcohol drinking among your practice

population

d Specific programs to reduce physical

inactivity among your practice

population

e Specific programs for people with

diabetes

f Specific programs for people with

cardiovascular disease

g Specific programs for people with

cancer

h Specific programs for people with

asthma

i Other specific programs; please

describe:

Please go to next page

.



March 31, 2008

57

7. Recent work in Canada has identified four main categories of primary health care

organization models. These models are described below. Please place a check mark by

the model that best describes your primary health care organization. If your

organizational model is a mix of two or more of the models listed below, please describe

your model in the bottom row of the table.

Primary Health Care Organization Models

Model That Best

Reflects Our

Organization

1. Professional Contract Model This is a model in which care is provided primarily in a physician’s office by a

family physician in solo or group practice. It is staffed primarily by physicians

and generally reimbursement is on a fee-for-service basis. Information systems

are for internal use. There are no formal mechanisms to ensure the continuity of

care except loyalty to the physician and there are no formal mechanisms to ensure

integration with other parts of the health care system, other than physician

affiliations with other resources of care (e.g., referrals to specialists).

2. Professional Coordination Model Organizations provide continuous service over time to a defined group of

patients/population. Funding is based on a per capita, sessional fee, fee-for-

service or other model(s), or a mix of models. There are non-physicians such as

nurses on the care team and a physician or nurse is designated to provide follow-

up and continuity of care for the patient. There will generally be a nurse who

provides liaison with other parts of the health care system and coordinates the

clinical integration of services.

3. Integrated Community Model The focus of this model is to promote the integration of primary health care into

the other parts of the health care system. There is generally a range of providers

operating in a community health centre format. Information technology serves

the organization and is linked to other, external service providers serving the

population. Care responsibility is longitudinal (long term) and is provided by a

multi-disciplinary care team. There is active cooperation and coordination with

other primary health care providers and with providers of complementary services

such as hospitals to guarantee the availability of a range of services

4. Non-Integrated Community Model This model is similar to the integrated model in its structure but is differentiated

by a lack of specific integration mechanisms. For example, information systems

do not link to other parts of the health care system. Services are not provided on a

24/7 basis and there are generally no, or few, mechanisms to ensure the

longitudinal continuity of care services to individuals

5. Mixed Model or Other:

Please describe what mix of the four models described above best reflects your

organization or another model that better reflects your organization:

.



March 31, 2008

58

Section 2: Planning for Primary Health Care Services

8. Does your primary health care organizations use information about the population you

serve to guide decisions about the type of programs and services to offer?

A. No � Go to question 9

B. Yes

� If Yes:

(i) What kind of information do you use?

_____________________________________________________________________

_______________________________________________________________

_____________________________________________________________________

_______________________________________________________________

(i) What is the source of this information?

_____________________________________________________________________

_______________________________________________________________

__________________________________________________________________

__________________________________________________________________

(iii) Please provide examples of programs or services that were offered based

on the above information.

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

__________________________________________________________________

.



March 31, 2008

59

9. Does the population served by your primary health care organization have unique

cultural and/or racial characteristics or spiritual needs?

A. No, the population is typical of the province � Go to question 10

B. Yes, the population is unique culturally

� If yes:

(i) In what ways is the population unique?

_____________________________________________________________________

_______________________________________________________________

(ii) Does your primary health care organization have a good understanding

of how the cultural qualities of the population impact their health care

needs?

(a) Yes, we monitor these needs through:

_______________________________________________________________

_________________________________________________________ (b) No,

we need to:

_______________________________________________________________

_________________________________________________________

10. (i) Does your primary health care organization involve community members in

planning the services provided by the organization?


B. Yes

� If Yes:

(ii) In what ways are community members involved?

___________________________________________________________________________

___________________________________________________________________________

__________________________________________________________________

.



March 31, 2008

60

11. Does your primary health care organization have a client/patient registry for chronic

conditions? Please circle one answer.


B. Yes

�

If Yes:

Please describe:

___________________________________________________________________________

_____________________________________________________________________

Section 3: Accountability Mechanisms

12. What accountability mechanisms are used by your primary health care

organization? Please check all that apply.

� Job descriptions for all clinical and administrative people associated with the

primary care organization

� Collaborative practice agreement

� Practice plan

� Shared care arrangement

� Terms of Reference

� Other - please specify: _________________________________________

Section 4: Collaboration

13. Does your primary health care organization have a collaborative practice agreement

(e.g. description of roles, accountabilities, etc.) for the primary health care team?


B. Yes

� If Yes:

Please briefly describe your collaborative practice arrangement (e.g., how many and

what type of providers)?

.



March 31, 2008

61

14. Do the members of your primary health care team use any of the following mechanisms

to support collaboration within the team? Please feel free to add comments to explain any

of your answers.

Mechanism Yes No Comment

A Regular team meetings for

organizational administration

B Regular team meetings for case

management

C Joint goal setting

D Shared vision for the primary care

organization

E Team building sessions or

workshops

F Other; please describe:

15. Does your primary health care organization have collaborative care arrangements

with other health care providers or health care organizations? Please circle one

answer.


B. N/A � Go to question 16

C. Yes

�

If Yes:

Please briefly describe a few examples.

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

.



March 31, 2008

62

16. (i) Does your primary health care organization currently coordinate patient

care with other health care organizations?


B. Yes

� If Yes:

(ii) Does your organization use standardized clinical protocols or assessment tools to

coordinate patient care?


B. N/A � Go to question 17

C. Yes

� If Yes:

(iii) Please briefly describe a few examples.

___________________________________________________________________________

_____________________________________________________________________

________________________________________________________________________

17. (i) Does your primary health care organization have formalized partnerships

with other providers or organizations beyond the health system (e.g. housing,

police, education)?


B. Yes

� If Yes:

(ii) Please briefly describe a few examples.

___________________________________________________________________________

___________________________________________________________________________

___________________________________________________________________________

.



March 31, 2008

63

Section 5: Quality Improvement

18. (i) Has your primary care organization implemented any quality improvement

initiatives over the past 12 months?


B. Yes

�

If Yes:

(ii) Has your primary care organization implemented any changes in clinical

practice as a result of these quality improvement initiatives?


B. Yes

�

If Yes:

(iii) Please briefly describe a few examples of such changes in clinical practice. ___________________________________________________________________________

___________________________________________________________________________

Section 6: Organizational Policies and Procedures

19. Policy refers to some form of administrative statement, direction or rule. Procedures

refer to how these statements, directions or rules are to be carried out. Please complete

each of the three main shaded columns in the table below to indicate if your primary

health care organization has policies and/or procedures on any of the following issues.

Does your organization have a

policy and/or procedures on

this topic area?

If you have a policy and/or

procedures, how clearly

articulated are they?

If you have a policy and/or

procedures, to what degree do

members of your organization

adhere to the policy or procedures?

Yes

written

Yes,

unwritten

No Very clear Clear Unclear Very high

adherence

Moderate

adherence

Low

adherence

Risk management

Patient safety

Medication errors

Recording of current

medication and problem

list in all patients’ files.

.



March 31, 2008

64

Section 7: Information Technology and Communication

20. (i) Does your primary health care organization use an electronic patient

records?


B. Yes

�

If Yes:

(ii) Do you ever examine the data to assess quality of care issues (e.g. influenza

immunization rates in the elderly etc.)?


B. Yes

�

If Yes:

(iii) Please briefly describe a few examples of issues you have examined.

___________________________________________________________________________

_____________________________________________________________________

________________________________________________________________________

________________________________________________________________________

21. (i) Does your primary health care organization use an electronic

prescribing/ drug ordering system?


B. Yes

�

If Yes:

(ii) Does the system perform patient-specific medication alerts?

Please circle one answer

Yes No

.



March 31, 2008

65

22. Do you have two-way electronic links (or share an electronic record) with the following

components of the health care system and, if so, in your view how well do these links

function?

Do you have electronic

links?

If yes: How well do the electronic links

function?

Types of Organizations

Yes No Don’t

Know

Very Well Well Poorly Very

poorly

Hospitals

Hospital Emergency Departments

Primary Care Physicians in solo

practice

Other primary health care organizations

Physician specialists

Tele-Triage

Public Health

Laboratory/Diagnostic Imaging

Services

Ambulance Services

Pharmacies

Home Care

Long Term Care Facilities

Community Mental Health

Health Insurance Companies (e.g., Blue

Cross)

Other, Specify:

.



March 31, 2008

66

23. The following two pages contain a table that lists different types of communication options in each column. For each type

of organization listed in the far left column, please indicate the methods that your primary health care organization uses

for communication with these components of the health system.

Tel

eph

on

e

Tel

eco

nfe

ren

ce

Vid

eo c

on

fere

nci

ng

Em

ail

(co

mp

ute

r to

com

pu

ter

mes

sag

ing

)

Fa

x

Reg

ula

r m

ail

Inte

rnet

ch

at

site

s

Web

-ca

stin

g

Inte

ract

ive

Web

site

s

Oth

er,

Sp

ecif

y

Types of

Organizations

Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No

Hospitals

Hospital Emergency

Departments

Primary Care Physicians in

solo practice

Other primary health care

organizations

Physician specialists

Tele-Triage

Public Health

Laboratory/Diagnostic

Imaging Services

Ambulance Services

Pharmacies

Home Care

.



March 31, 2008

67

Tel

eph

on

e

Tel

eco

nfe

ren

ce

Vid

eo c

on

fere

nci

ng

Em

ail

(co

mp

ute

r to

com

pu

ter

mes

sag

ing

)

Fa

x

Reg

ula

r m

ail

Inte

rnet

ch

at

site

s

Web

-ca

stin

g

Inte

ract

ive

Web

site

s

Oth

er,

Sp

ecif

y

Types of

Organizations

Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No

Long Term Care Facilities

Community Mental Health

Health Insurance Companies

(e.g., Blue Cross)

Other, Specify

Other, Specify

Thank you for taking the time to complete this survey!

.



March 31, 2008

68

Primary Health Care (PHC) Provider/Team Survey

Revised Version October 2006

Introduction

There are many changes taking place in Nova Scotia to improve the delivery of primary health care services

throughout the province. As part of our ongoing efforts to support changes in primary health care, the Nova

Scotia Department of Health regularly conducts evaluations to monitor changes over time as well as assess

what is working and what needs to change. This survey is part of the ongoing evaluation of the primary

health care system.

This survey is intended to be filled out by members of a primary care organization. The term ‘primary care

organization’ is used to refer to any group of primary health care professionals that work together to offer

primary health care services to a practice population. For example, the phrase can refer to a community

health centre or a physician group practice.

The survey should be completed individually by members of the primary health care team based on your

experiences as a member of the team. Members of the team may include physicians; nurses; other allied

health professionals such as pharmacists; social workers, etc.; office administration and management.

The survey is designed to be completed anonymously. Please do not write your name (or any other

personally identifiable information) anywhere on the survey. To ensure confidentiality, the results of the

surveys will be reported in group form and only grouped data will be presented. A summary of the survey

results may be shared with all team members in your primary health care organization.

It should take you approximately 20 minutes to complete this survey. Thank you for your time in helping us

to evaluate primary health care change in Nova Scotia.

Section 1: Provider Demographic Information

1. Please indicate your profession and/or role on the team:

� Family Physician � Pharmacist

� Nurse Practitioner � Dietitian

� Family Practice Nurse/Clinic Nurse � Physiotherapist

� Public Health Nurse � Occupational Therapist

� Mental Health Nurse � Office Manager

� Social Worker � Clerical/Reception staff

� Psychiatrist � Psychologist

� Other (Specify): ________________

.



March 31, 2008

69

2. Please describe the type of practice you work in:

� Solo Family Physician Practice

� Group Practice (an organization of 2 or more FPs/GPs who work together, share client/ patient

records, office space, staff, technology and on-call coverage)

� Interdisciplinary Health Care Team or Network (includes a group of individuals with diverse

training who work together to deliver patient care, such as family physicians, nurses, social workers,

dieticians, and other health care providers)

� Other (Specify): ___________________________________________________

Section 2: Collaboration and Continuity of Care

Please complete the following questions related to the continuity of patient/client care by

indicating the extent to which you agree or disagree with the following statements. Please feel

free to add comments to explain your answers.

Note: The response format for this scale does not align with the other two scales in this survey. It

was changed to go from strongly disagree to strongly agree.

Statement

Str

on

gly

Dis

ag

ree

Dis

ag

ree

Ag

ree

Str

on

gly

Ag

ree

N/A

or

Do

n't

Kn

ow

Comments

1. Essential demographic and clinical

information on patients is complete

most of the time

2. Our patients/clients have to repeat

their demographics and medical

histories to multiple clinicians in

our primary health care

organization (e.g., practice,

community health centre, etc).

3. Our records show that we have

unnecessarily duplicated medical

tests over the past 12 months

Item reworded.

4. The time between when I request

lab tests and when they are received

back by me is appropriate.

Consider dropping this item.

5. We have streamlined procedures for

making referrals to other

organizations or extended care

providers (e.g., specialists,

community organizations).

Item revised.

.



March 31, 2008

70

6. We have good collaborative

relations with the other

organizations/care providers we

work with.

Item revised.

7. We always share patient/client

information, as clinically

appropriate, in a timely way with

other organizations/care providers.

8. Care providers outside of our

organization incorporate our care

plan recommendations into the care

of our patients/clients.

9. We provide a high level of care

continuity for patients between our

organization and other health care

organizations.

Reworded item – check meaning of

continuity

10. Case conferences are a valuable

means to ensure continuity of care

within our care team and should be

encouraged/supported

New item/reworded

Section 3: Professional Development Note: Deleted references to Location or mode of delivery (e.g., on-site, teleconference). This could

be added if it is thought to be important.

Following item reworded. Revised response options

1. What kind of support did your PHC organization (e.g., Practice, CHC, etc) provide to

allow your participation in continuing professional development opportunities over the

last 12 months? (Check all that apply)

� No support was provided

� Funding

� Paid time off work

� Material resources

� Mentoring by other team members

� Other (Specify): ____________________________________________________

.



March 31, 2008

71

Following item reworded. Revised response options.

2. In what kind of continuing professional development opportunities did you participate

over the last 12 months? (Check all that apply.)

� Attended a conference

� Attended a workshop

� Took a course or educational program

� Other (Specify): ____________________________________________________

Following item reworded. Response option added.

3. What topics for continuing professional development or learning opportunities have

you participated in over the last 12 months? (Check all that apply.)

� Chronic disease management � Population health

� Health promotion & disease prevention � Program planning

� Evaluation and research � Case management

� Information management/Information technology � Developmental disabilities

� Cultural competence; Diversity & social inclusion � Team building; Leadership

� Adult learning � Conflict resolution

� Patient behaviour change; Counseling � Primary Health Care

� Building community relationships; Community development

� Computer technology and training

� Other (Specify): ____________________________________________________

.



March 31, 2008

72

Section 4: Team Functioning

Please indicate your level of satisfaction with each of the following by placing a check mark in

the appropriate box. Please feel free to add comments to explain your answers.

How satisfied are you…

Ver

y D

issa

tisf

ied

Dis

sati

sfie

d

Nei

ther

Sa

tisf

ied

no

r D

issa

tisf

ied

Sa

tisf

ied

Ver

y S

ati

sfie

d

N

/A

Comments

Decision Making

1. With your participation in

decision-making within the

team?

2. With the participation of others

in decision-making within the

team?

3. With the process of care

planning decisions among

members of the team?

4. With the way your

professional expertise is

regarded within the team?

5. With the openness and

transparency of decision-

making processes?

Vision and Leadership

6. That this team has clear

leadership when that is

needed?

Item revised.

7. That this team has a shared

vision?

8. That this team has a clear

purpose and objectives?

9. That this team has clear

measures of success?

Shared Understanding

10. That team objectives are

shared by all team members?


knowledge of local health

needs?

12. That this team has a high level

of trust among its members?

.



March 31, 2008

73


Ver

y D

issa

tisf

ied

Dis

sati

sfie

d

Nei

ther

Sa

tisf

ied

no

r D

issa

tisf

ied

Sa

tisf

ied

Ver

y S

ati

sfie

d

N

/A

Comments


understanding of its

limitations?

Role Clarity

14. With your level of

understanding of your role

within the team?


understanding of the role of

others within the team?


understanding of how this

team functions?


understanding of how to

separate your team role from

your professional/clinical role?

Structure

18. With the variety of

professional disciplines

represented within the team?

19. With the frequency with which

the team is able to meet as a

group?

20. With the number of team

members who are on site?

Item reworded.

Communication

21. With processes and strategies

for resolving team conflict?

22. With communication among

professional members within

the team?

23. With the coordination of

communication with people

and organizations outside the

team?

24. With interpersonal

relationships among team

members?

.



March 31, 2008

74

Professional Development

25. With the professional

development opportunities that

you have had to support

effective team functioning?

Section 5: General Work Satisfaction

Please indicate your level of satisfaction with each of the following by placing a check mark in

the appropriate box. Please feel free to add comments to explain your answers.


Ver

y D

issa

tisf

ied

Dis

sati

sfie

d

Nei

ther

Sa

tisf

ied

no

r D

issa

tisf

ied

Sa

tisf

ied

Ver

y S

ati

sfie

d

N/A

Comments

Patient/Client Care

1. With the level of care you are

able to provide to

patients/clients?

2. With the amount of time you

have for direct patient/client

care?

3. With the balance between

patient/client care and

documentation (e.g., charting)?

4. With the time you have to do

case management?

Patient/Client Relationship

5. With the participation of

patients/clients in their own

care?

6. With your ability to develop a

rapport with patients?

7. With the guiding/directing you do

in your relationships with

patients/clients.

Item revised

Collaboration

7. With the time you have to

discuss patient/client care with

colleagues with similar

professional backgrounds?

.



March 31, 2008

75

8. With the time you have to

discuss patient/client care with

colleagues with different

professional backgrounds?

9. With the collaborative care

arrangements you have with

other health care providers?

10. With how well the team links

patients to community

resources?

11. With how well the team is

partnering with the community

(e.g., community members and

organizations) to plan and

deliver programs and services?

Information Sharing

12. With the comprehensiveness of

information available to you at

the point of care?

13. With the sharing of information

between health providers

within your health organization?

14. That the information you provide

to other health professionals is

being used?

Safety

15. With the safety of staff in your

primary work location?

16. With the safety of patients in

your primary work location?

17. With the safety of property in


18. With the security measures in


Work/Family Balance

19. With your ability to meet family

obligations?

20. With your ability to meet

personal obligations (caring for

self)?

21. With your ability to meet

professional obligations?

22. With the overall balance

between your personal and

professional commitments?

.



March 31, 2008

76

Professional Growth and Development

23. With the opportunities you have

had to self-reflect on your care

practices?


had to reflect on care practices

with colleagues?

25. With the professional

development opportunities you

have had?


had to assess the effectiveness of

the care you provide (e.g.,

evidence-based processes of care

and patient/client outcomes)?

Skill Utilization

27. With the extent to which your

clinical skills are utilized?


knowledge and expertise is

used?


interpersonal skills are used?


facilitation/collaboration skills

are used?

Support

31. With the clerical support you

have to conduct your work?

Consider dropping

32. With the material support you

have to conduct your work (e.g.,

equipment, tools, etc)?

33. With the professional support

you have to conduct your work?

Overall Work Satisfaction

34. With the job itself?

Thank you for taking the time to complete this survey

.



March 31, 2008

77

Telephone version Primary Care Practice Survey (PCPS)

Revised September 2006

General Instructions: • Interviewer must read each set of instructions for each part of this questionnaire. • Interviewer must record all responses clearly and verbatim where required. • Interviewer must avoid paraphrasing or rewording responses. RECORD FOLLOWING INFORMATION: Telephone #: ______________________ Postal Code: ______________________

Sample ID: ______________________

Hello, my name is ______________ and I am calling on behalf of ____________, your district health authority which includes all the hospitals in your region. _____________ is conducting a survey to learn about your experiences, good or bad, in getting primary health care from your regular family doctor’s office or medical clinic. Completing this survey should take about 12-15 minutes. Are you able to complete the survey?

If not, is there a better time when we may call you back? __________________________________________

A. To begin, into which age group do you fall? Are you … : READ RESPONSES IN ORDER – CODE ONE ONLY

1 Under 18 THANK, TERMINATE, AND RECORD

2 18-24

3 25-34

4 35-44

5 45-54

6 55-64, or

7 65 and over

8 Refused THANK, TERMINATE, AND RECORD

B. Gender (by observation): CODE ONE ONLY

1 Male

2 Female

Your answers will help us know what services need to be improved. We appreciate your help. Your responses are confidential. You are free to skip any questions you do not wish to answer or stop at any time. These first questions ask about your regular family doctor’s office or medical clinic. This is the place where you usually go when you are sick or need advice about your health. You would usually find family doctors and in some places nurses, nurse practitioners, or other health care providers in these offices. At this time, we are NOT asking you about your experiences with a medical specialist.

.



March 31, 2008

78

1a. Do you have a regular family doctor’s office or medical clinic you can go to if you are sick or need

advice about your health?

CODE ONE ONLY

1 Yes SKIP TO Q.2

2 No CONTINUE to 1b

8 Don’t know/No answer SKIP TO 1d

1b. Have you been trying to find one?

CODE ONE ONLY

1 Yes CONTINUE

2 No SKIP TO Q.1d

8 Don’t know/No answer SKIP TO Q. 1d

1c. [ASK IF “YES” IN Q.1b] For how long have you been trying?

CODE ONE ONLY

1 Less than six months

2 Six to twelve months

3 Over one year

8 Don’t know/No answer

1d. Over the past 12 months, where did you usually go when you were sick or needed advice about your

health? Did you …

READ AND ROTATE STATEMENTS, EXCEPT ALWAYS READ ITEM 4 LAST - CODE ONE ONLY

1 Go to the Emergency or hospital outpatient department

2 Contact a local family doctor’s office to check if they could see you

3 Go to a walk in clinic

4 Do nothing

5 Try to find help somewhere else (Specify: Where was that? ____________________)

VOLUNTEERED

97 Have not required help or advice SKIP TO Q.24 DEMOGRAPHICS


1e. In the past 12 months, how many times did you go to find help for a sickness or advice about your

health?

RECORD EXACT NUMBER – PROBE TO AVOID A RANGE

RECORD NUMBER: _________________________


SKIP TO Q.24 DEMOGRAPHICS

.



March 31, 2008

79

2. For approximately how long have you been a patient at this office or medical clinic?

RECORD EXACT TIME PERIOD IN WEEKS, MONTHS AND/OR YEARS – PROBE TO AVOID A

RANGE

RECORD TIME PERIOD: _________________________


3. In the past 12 months, how many times have you been to this family doctor’s office or medical clinic for

an illness or advice about your health?

RECORD EXACT NUMBER – PROBE TO AVOID A RANGE

RECORD NUMBER: _________________________


4. How do you rate the hours that your regular family doctor’s office or medical clinic is open for

appointments? Would you say the hours are … : READ RESPONSES IN ORDER – CODE ONE

ONLY

1 Poor

2 Fair

3 Good, or

4 Excellent

VOLUNTEERED

7 Does not apply


5. When your regular family doctor’s office or medical clinic is closed, is there a phone number you can

call when you get sick? CODE ONE ONLY

1 Yes

2 No

8 Not sure/Don’t remember

These next questions ask you to think about your experience at your regular family doctor’s office or medical clinic over the past 12 months. There are 2 types of visits we are interested in … The first type of visit we call Routine care. These include visits for reasons such as a physical examination, pap tests, BP checks and other routine type care for a chronic condition. The second type we called a visit for an Urgent but minor health problem. These are problems that come up suddenly like a fever, headache, sprained ankle or rashes. (They are not serious enough to make you go immediately to a hospital emergency.) First we want to ask you about the ROUTINE CARE visits you’ve made in the last 12 months. Remember ROUTINE CARE visits are for things such as a physical examination, pap test or BP checks….

.



March 31, 2008

80

6. When you need to make a routine care visit with your regular family doctor or health care provider … Do you find it: [READ RESPONSES IN ORDER] … to do so? CODE ONE ONLY

4 Very Easy

3 Easy

2 A bit difficult

1 Very difficult

VOLUNTEERED 8 Don’t know/No answer 9 Have not made any routine visits – GO TO Q7.

6a. [And], How quickly are you able to see your regular family doctor or health care provider when the

appointment you need is for routine care? Would you say it is usually:

READ RESPONSES – CODE ONE ONLY

05 the Same day

04 the Next working day

03 Within three working days

02 Within four working days

01 Five or more working days …

VOLUNTEERED 98 Don’t know/No answer

99 Other (SPECIFY: __________________________________)

6b. [DO NOT ASK Q.6b IF “DON’T KNOW/NO ANSWER” IN Q.6a] How do you rate this? Would you

say this is… : READ RESPONSES IN ORDER – CODE ONE ONLY

1 Poor

2 Fair

3 Good, or

4 Excellent

VOLUNTEERED

7 Does not apply

8 Don’t know/No answer Now, we want to ask you about visits you may have made for URGENT BUT MINOR HEALTH PROBLEMS, such as a fever, headache, sprained ankle or rash … 7. When you need to make this kind of urgent visit with your regular family doctor or health care provider

… do you find it [READ RESPONSES IN ORDER] to do so? CODE ONE ONLY

4 Very Easy

3 Easy

2 A bit difficult

1 Very difficult

VOLUNTEERED 8 Don’t know/No answer 9 Have not made any urgent type visits – GO TO Q8.

.



March 31, 2008

81

7a. [And], How quickly can you usually get to see your REGULAR family doctor or health care provider if

the appointment you need is for an urgent but minor health problem? Would you say it is usually:

READ RESPONSES – CODE ONE ONLY

05 the Same day

04 the Next working day

03 Within three working days

02 Within four working days

01 Five or more working days

97 Does not apply


99 Other (SPECIFY: __________________________________)

7b. [DO NOT ASK Q.7b IF “DON’T KNOW/NO ANSWER” OR IF “DOES NOT APPLY” IN Q.7a] How do

you rate this? Would you say this is…: READ RESPONSES IN ORDER – CODE ONE ONLY

1 Poor

2 Fair

3 Good, or

4 Excellent

VOLUNTEERED

7 Does not apply


7c. When you need to see your regular family doctor or health care provider during the evening (after 5

pm) or on weekends for an urgent but minor health problem, … Do you find it …[READ RESPONSES IN ORDER ] to do so? CODE ONE ONLY

4 Very Easy

3 Easy

2 A bit difficult

1 Very difficult

9 Or, you have not required care during the evening or on weekends for an urgent but minor health

problem

VOLUNTEERED 8 Don’t know/No answer

8. Where did you go the last time you needed care for an urgent but minor health problem …and your

regular family doctor’s office or medical clinic was OPEN. Did you … :

READ AND ROTATE STATEMENTS, EXCEPT ALWAYS READ ITEM 06 LAST - CODE ONE

ONLY

01 Contact your regular family doctor’s office & see someone that same day

02 Go to the Emergency department

03 Go to a walk-in clinic that was open

04 Contact your regular family doctor’s office & make an appointment for another day

05 Do nothing

06 Or did you try to find help somewhere else? (SPECIFY: Where was that? ______________)

VOLUNTEERED

97 Have not required care for an urgent but minor problem when the clinic was open


.



March 31, 2008

82

9. Where did you go the last time you needed care for an urgent but minor health problem and your

regular family doctor’s office or medical clinic was CLOSED? Did you … :

READ AND ROTATE STATEMENTS, EXCEPT ALWAYS READ ITEM 06 LAST - CODE ONE

ONLY

01 Contact your regular family doctor’s office and follow the instructions given on the phone telling you where to go for out of hours care? 02 Go to the Emergency department

03 Go to a walk-in clinic that was open

04 Wait until your regular doctor’s office was open and then contact them

05 Do nothing

06 Or did you try to find help somewhere else? (SPECIFY: Where was that? ______________)

VOLUNTEERED

97 Have not required care for an urgent but minor health problem when the clinic was closed


10a. How long do you usually have to wait at your regular family doctor’s office or medical clinic until your

visit begins?

RECORD EXACT AMOUNT OF TIME IN HOURS AND/OR MINUTES– PROBE TO AVOID A RANGE

AMOUNT OF TIME: _________________________



say this is …: READ RESPONSES IN ORDER – CODE ONE ONLY

1 Poor

2 Fair

3 Good, or

4 Excellent

VOLUNTEERED

7 Does not apply


11. In the past 12 months, have you received care from …:

READ STATEMENTS IN ORDER - CODE ONE ONLY PER STATEMENT

a. A family physician b. A nurse c. A nurse practitioner d. A mental health worker e. A social worker f. A nutritionist/Dietician g. An addictions counsellor i. Anyone else? (SPECIFY: Who was that? ___________________________)

1 Yes

2 No

8 Not sure/don’t remember

.



March 31, 2008

83

12. Who do you usually see for your general health care needs? Do you usually see a … :

READ LIST IN ORDER, EXCEPT ALWAYS READ ITEM 09 LAST CODE ONE ONLY

01 Family physician

02 Nurse

03 A nurse practitioner

09 Or do you usually see someone else? (SPECIFY & PROBE: Who would that be?

____________)

VOLUNTEERED


[IF THE PARTICIPANT MENTIONED ONLY A ‘FAMILY DOCTOR’ IN Q.12, THEN CONTINUE WITH …] For

this survey, we will refer to this person as your ‘USUAL FAMILY DOCTOR’

[IF THE PARTICIPANT IN Q.12 MENTIONED SOMEONE OTHER THAN A ‘FAMILY DOCTOR’, THEN

CONTINUE WITH …] For this survey, we will refer to this person as your ‘USUAL HEALTH CARE

PROVIDER’

These next questions ask about your usual FAMILY DOCTOR [OR HEALTH CARE PROVIDER].

READ THE FOLLOWING ONLY IF NO FAMILY DOCTOR OR HEALTH CARE PROVIDER IS MENTIONED IN Q.12: If you don’t have a [usual family doctor]/[health care provider], please answer about the one family doctor [health care provider] at your regular family doctor’s office or medical clinic who you know best.

SKIP TO Q.24 IF RESPONDENT SAYS NO DOCTOR [HEALTH CARE PROVIDER] KNOWS HIM/HER BEST.

13a. In general, when you go to your regular family doctor’s office or medical clinic, how often do you see

your usual family doctor [health care provider]? Would it be … :

READ RESPONSES IN ORDER – CODE ONE ONLY

1 Always

2 Almost always

3 A lot of the time

4 Some of the time

5 Almost never, or

6 Never

VOLUNTEERED



say it is… : READ RESPONSES IN ORDER – CODE ONE ONLY

1 Poor

2 Fair

3 Good, or

4 Excellent

VOLUNTEERED


.



March 31, 2008

84

14. When you go to your usual family doctor [health care provider], how do you rate the following? [READ

AND ROTATE STATEMENTS] Would you say it was [READ RESPONSES IN ORDER]?

CODE ONE ONLY PER STATEMENT

a. How thoroughly the doctor [provider] asks about your symptoms and how you are feeling

b. How well the doctor [provider] listens to what you have to say

c. How well the doctor [provider] puts you at ease during your physical exam

d. How much the doctor [provider] involves you in decisions about your care and treatment

e. How well the doctor [provider] explains your problems or any treatment that you need

f. How thoroughly the doctor [provider] explains the side effects of medication being prescribed

g. The amount of time the doctor [provider] spends with you

1 Poor

2 Fair

3 Good, or

4 Excellent

VOLUNTEERED

7 Doesn’t apply


15. Thinking about how well your usual family doctor [health care provider] knows you, how do you rate the

following? [READ AND ROTATE STATEMENTS] Would you say it is … :

READ RESPONSES IN ORDER – CODE ONE ONLY PER STATEMENT

a. His or her knowledge of your medical history

b. His or her knowledge of what worries you most about your health

c. His or her knowledge of your responsibilities at home, work or school

1 Poor

2 Fair

3 Good, or

4 Excellent

VOLUNTEERED 7 Doesn’t apply


16. After a visit to your regular family doctor’s office or medical clinic, does the advice or assistance provided …:

READ AND ROTATE STATEMENTS - READ RESPONSES IN ORDER - CODE ONE ONLY PER STATEMENT

a. Help you to cope better with your health problem or illness

b. Help you to understand your health problem or illness better

c. Help you to keep yourself healthy

3 To a great extent 2 To some extent, or 1 To little or no extent VOLUNTEERED 8 Don’t know/No answer

.



March 31, 2008

85

17. In the last 12 months, has your family doctor [health care provider] offered [READ RESPONSES IN ORDER] help, information or advice on [READ AND ROTATE STATEMENTS]? CODE ONE ONLY PER STATEMENT

A1. Eating a healthy diet

A2. Regular physical activity appropriate for you

3 A lot 2 A little, or 1 No VOLUNTEERED 8 Don’t know/No answer

18a. Have you smoked over the past 2 years?

CODE ONE ONLY 1 Yes CONTINUE 2 No GO TO Q19 8 Don’t know/No answer GO TO Q19 18b. Over the past 2 years, has your family doctor [health care provider] offered [READ RESPONSES IN

ORDER] help, information or advice to help you quit smoking? CODE ONE ONLY


19. Do you, or anyone around you, feel you need to reduce the amount of alcohol you drink?

CODE ONE ONLY

1 Yes CONTINUE 2 No SKIP TO Q.20 8 Don’t know/No answer SKIP TO Q.20 19a. Over the past 2 years, has your family doctor [health care provider] offered [READ RESPONSES IN

ORDER] help, information or advice on how to manage the amount of alcohol you drink? CODE ONE ONLY


20a. Moving along … In the past 12 months, have you ever visited any kind of medical specialist?

CODE ONE ONLY

1 Yes CONTINUE

2 No SKIP TO Q.20c

8 Not sure/Don’t remember SKIP TO Q.20c

.



March 31, 2008

86

20b. After you went to the specialist, did you have an opportunity to talk about the specialist visit when you

went back to your regular family doctor’s office or medical clinic?

CODE ONE ONLY – PROBE FOR SPECIFIC “NO” RESPONSE

1 Yes

2 No, I haven’t been back to my regular family doctor’s office or medical clinic since

3 No, it was not discussed


21. Do you experience problems communicating with your usual family doctor [health care provider] due to

language differences?

CODE ONE ONLY

1 Yes

2 No


22a. In some clinics there are nurses who work with physicians. Is there a nurse available in your regular

family doctor’s office or medical clinic?

CODE ONE ONLY

1 Yes

2 No


22b. In the future, how willing would you be to see a nurse who is working with your doctor [health care

provider] for some of your health care needs (such as blood pressure checks, immunizations, minor

illness such as a cold or flu)? Would you be … :

READ RESPONSES IN ORDER - CODE ONE ONLY

4 Very willing to see a nurse

3 Somewhat willing to see a nurse

2 Would prefer not to see a nurse, or

1 Would NOT be willing to see a nurse

VOLUNTEERED


23. How would you rate the level of privacy provided by staff in the reception area? READ RESPONSES IN ORDER – CODE ONE ONLY

1 Poor 2 Fair 3 Good, or 4 Excellent VOLUNTEERED 8 Don’t know/No answer

.



March 31, 2008

87

23a. All things considered, are you [READ RESPONSES IN ORDER] with your regular family doctor’s office

or medical clinic? CODE ONE ONLY

4 Completely Satisfied

3 Fairly Satisfied

2 Fairly Dissatisfied

1 Completely Dissatisfied

VOLUNTEERED

7 Neutral/neither satisfied nor dissatisfied


23b. If your family doctor [provider] was taking on new patients, would you [READ RESPONSES IN ORDER] recommend him/her to family or friends? CODE ONE ONLY

4 Definitely

3 Probably 2 Probably not 1 Definitely no VOLUNTEERED 8 Don’t know/no answer It will help us to understand your answers if you could tell us a little about yourself. 24. Overall, how has your health been over the last 12 months? Would you say it has been … :


1 Poor

2 Fair

3 Good

4 Very good, or

5 Excellent

VOLUNTEERED


25. Do you have any long-term illnesses, health problems, health issues related to your age, or any

disabilities that limit your daily activities or the work you can do?

CODE ONE ONLY

1 Yes

2 No


.



March 31, 2008

88

26. Do you have any of the following?

READ LIST IN ORDER - CODE ONE PER STATEMENT

a. High blood pressure

b. Diabetes

c. Heart disease

d. Lung disease

e. Cancer

f. Arthritis

h. Asthma

i. Depression

j Any other chronic illness/condition? (SPECIFY: What would that illness or condition be?

________)

1 Yes

2 No

8 Refused/No answer

27. Which of the following best describes you? Are you … :


01 Employed (full-time or part-time, including self-employed or on a work training program)

02 Unemployed and looking for work

03 At school or in full-time education

04 Unable to work due to long-term sickness or disability

05 Looking after your home/family

06 Retired from paid work

99 Or, some other status (SPECIFY: What would that be? _______________________)

VOLUNTEERED

98 Refused

28. What is the highest level of education that you have completed? CODE ONE ONLY

01 Grade 8 or less

02 Grade 9 or 10

03 Grade 11

04 Completed high school

05 Some university/community college/technical college

06 Completed community college/technical college

07 Completed an undergraduate university degree program (for example, BA, BSc)

08 Completed a graduate university degree program (for example, Masters, PhD)


99 Other (SPECIFY: ___________________________)

.



March 31, 2008

89

29. What is your best estimate of your total household income, before taxes and deductions, for the past 12

months? Please stop me when I reach the correct amount. Would it be … :

READ RESPONSES IN ORDER - CODE ONE ONLY

01 Less than $10,000

02 $10,000 to less than $20,000

03 $20,000 to less than $30,000

04 $30,000 to less than $40,000

05 $40,000 to less than $50,000

06 $50,000 to less than $60,000

07 $60,000 to less than $80,000, or

08 $80,000 or more

VOLUNTEERED

98 Not sure/Don’t know/Refused

30. Do you consider yourself a visible minority? CODE ONE ONLY

1 Yes

2 No

7 Unsure


31. What is your postal code? RECORD, EVEN IF PARTIAL

RECORD: B __ __ __ __ __

97 Refused/Don’t know

On behalf of _____ we thank you very much for taking time to complete this survey. It is greatly appreciated.

.



March 31, 2008

90

Sample Chart Audit Worksheet

(Developed through Primary Care, Capital Health)

1. Auditor: ______________________________________________________________

2. Date: _________________________________________________________________

3. Chart ID: _____________________________________________________________

4. Is this patient a regular patient of the practice?

� Yes

� No (e.g., only ER record; address is from away; physician is not from MVFP)

5. Gender: M F

6. DOB: ________________________________________________________________

7. Age group: _______ 7-18 years of age

_______ 19-49 years of age

_______ 50-64 years of age

_______ 65+ years of age

8. Health category:

Charts have been pulled according to health category (billing codes). For each patient chart please specify the appropriate health

category. If the patient’s health category is not supported by evidence in the chart please explain.

____ Hypertension (unless otherwise indicated, please explain: _____________________________)

____ Diabetes (unless otherwise indicated, please explain: _________________________________)

____ CAD (unless otherwise indicated, please explain: ____________________________________)

____ Asthma (unless otherwise indicated, please explain: __________________________________)

____ AMI (unless otherwise indicated, please explain: _____________________________________)

____ CHF (unless otherwise indicated, please explain: _____________________________________)

____ Depression (unless otherwise indicated, please explain: _______________________________)

____ Healthy (does not have any of the conditions listed above)

.



March 31, 2008

91

1 2

Part A: All Patients

Please complete Part A according to patient’s age group for all charts regardless of patient health category. Please

clearly circle either ‘yes’ or ‘no’ in the space provided.

All Patients Aged 7-18 Years

9. Has the patient been a patient of the practice from 0-7 years of age?

(if no, skip to question 13)

- Date of initiation into the practice _______________________________________

Yes No

10. Has the patient been screened for congenital hip displacement and eye and hearing

problems by 3 years of age? Yes No

11. Has the patient received required primary childhood immunizations and related

boosters by 7 years of age including:

- Course of Diphtheria-tetanus-pertussis-polio (DPTP) including one shot at 2, 4 and 6

months, one shot at 18 months and one shot at 4-6 years?

- Course of measles-mumps-rubella (MMR) including one shot at 12 months, and one

shot at either 18 months or 4-6 years of age?

- Course of Hemophilus Type B including a shot at 2, 4, 6 and 18 months?

Not applicable (e.g., contraindication, parent refusal)

Please describe: __________________________________________________________

Yes

Yes

Yes

No

No

No

12. Has the patient’s parent been given information on child injury prevention in the home

by 7 years of age? Yes No

13. Has the patient received specific help or information on regular physical activity over

the past 12 months? Yes No

All Patients Aged 18 and over

14. Has the patient received specific help or information on regular physical activity over

the past 12 months? Yes No

15. Has the patient received at least one full fasting lipid profile (e.g., total cholesterol, high-

density lipoprotein cholesterol, triglyceride and calculated density lipoprotein

cholesterol levels) during the past 2 years?

Yes No

16. Has the patient’s blood pressure been measured at least once during the past 24

months? Yes No

17. What was the patient’s blood pressure measurement at last visit? ____ SBP

____ DBP

For All Female Patients Aged 18 and over

18. Has the female patient received at least one papanicolaou smear within the past three years? Yes No

.



March 31, 2008

92

All Patients Aged 50 Years and over

19. Has the patient received screening for colon cancer with a Hemoccult test at least once

within the past 2 years?

Not applicable (e.g., patient refusal)

Please describe:___________________________________________________________

Yes No

For All Female Patients Aged 50 Years and over

20. Has the female patient received any of the following within the past 24 months:

- Mammography?

- Clinical breast examinations?

Not applicable (e.g., patient refusal)

Please describe:___________________________________________________________

Yes

Yes

No

No

All Patients Aged 65 Years and over

21. Within the last year, has the patient been offered or received an influenza immunization?

Not applicable (e.g., contraindication, patient refusal).

Please describe:___________________________________________________________

Yes

No

22. Has the patient been offered or received a pneumococcal immunization?

Not applicable (e.g., contraindication, patient refusal)

Please describe:___________________________________________________________

Yes

No

For All Female Patients Aged 65 Years and over

23. Has the female patient’s bone density been screened for low mineral density? Yes No

.



March 31, 2008

93

Part B: Health Condition

Please complete Part B according to patient’s health category. Please clearly circle either ‘yes’ or ‘no’ in spaces provided.

Diabetes

24. Within the last 15 months has the patient had an HbA1c less than or equal to 7%? Yes No

25. What was the most recent recorded HbA1c result? ________

26. Within the last 24 months has the patient visited an optometrist or ophthalmologist? Yes No

27. Within the past year, has the patient received testing for:

- Hemoglobin A1c testing (HbA1c)?

- Full fasting lipid profile screening?

- Nephropathy screening (e.g. albumin/creatinine ratio, microalbuminuria)?

- Blood pressure (BP) measurement?

- Obesity/overweight screening (at least one of the following: a weight measurement,

body mass index or waist circumference)?

Yes

Yes

Yes

Yes

Yes

No

No

No

No

No

Asthma

28. Within the last year has there been emergency department visit for asthma exacerbation? Yes No

CHF

29. Within the last year has there been an emergency department visit for CHF

exacerbation? Yes No

30. Has the patient been prescribed an ACE inhibitor or ARBs within the past year?

Not applicable (e.g., contraindication)

Please describe:___________________________________________________________

Yes

N/A

No

Hypertension

31. Was the most recent blood pressure measurement within one of the targets below:

- Target for non-diabetics: <140/90?

- Target for diabetics or renal disease: <130/80?

- Target for proteinuria: <125/75?

Yes

Yes

Yes

No

No

No

32. What was the patients’ blood pressure measurement at last visit? ____ SBP

____ DBP

.



March 31, 2008

94


- Fasting blood sugar?

- Full fasting lipid profile screening?

- Test to detect renal dysfunction (e.g., serum creatinine)?

- Blood pressure measurement?

- Obesity/overweight screening (e.g., at least one of the following: a weight

measurement, body mass index or waist circumference)?

Yes

Yes

Yes

Yes

Yes

No

No

No

No

No

Coronary Artery Disease


- Fasting blood sugar?

- Full Fasting lipid profile screening?

- Blood pressure measurement?

- Obesity/overweight screening (at least one of the following: a weight measurement,

body mass index or waist circumference)?

Yes

Yes

Yes

Yes

No

No

No

No

35. Does the patient have elevated LDL-C (>2.5 mmol/L) and offered or received:

- Lifestyle advise (e.g., counselling for physical activity and diet)?

- Lipid lowering medication?

Yes

Yes

No

No

Acute Myocardial Infarction

36. Has the patient been prescribed a beta-blocking drug within the past year?

Not applicable (e.g., contraindication)

Please describe:___________________________________________________________

Yes

No

Depression

37. Has the patient been offered:

- Pharmacological treatment?

- Non-pharmacological treatment such as services of a mental health provider (e.g.,

psychologist, psychiatrist, social worker, mental health nurse, etc.)?

Yes

Yes

No

No

38. Has the patient had a follow up visit with a PHC provider within two weeks of starting

anti-depressant drug treatment? Yes No

.



March 31, 2008

95

AAPPPPEENNDDIIXX 33:: DDAATTAA FFOORR TTHHEE PPHHCC IINNDDIICCAATTOORRSS PPOOTTEENNTTIIAALLLLYY

AAVVAAIILLAABBLLEE TTHHRROOUUGGHH MMSSII

PHC Indicator Comments

• % of PHC organizations who currently provide after

hours coverage (beyond 9:00 a.m. to 5:00 p.m. Monday

to Friday) for their practice population. (CIHI 30)

• Need to define PHC organizations

• At the present time, there is no requirement to submit time

of day with FFS billing

• Premiums are paid for work outside of 9 to 5 M to F, so all

FFS doctors use those specific billing codes, however those

that shadow bill may not

• The strategy could be to sort all MSI billing by GPs by

billing number (provincial licensing number) and see what

percentage of these billed a service out of hours

• Average number of extended hours (beyond 9:00 a.m. to

5:00 p.m., Monday to Friday) provided by PHC

organizations per month, by PHC organizations. (CIHI

31)

• Need to define PHC organizations

• At the present time, there is no requirement to submit time

of day with FFS billing

• Sort billing codes by time codes

• % of population who received PHC services from an

interdisciplinary PHC organization, over the past 12

months. (CIHI 94)

• Would need a list of physicians to include in this query.

• PHC provider full time equivalents (FTEs) per 100,000

population, by type of PHC provider. (CIHI 85)

• Define PHC provider. If GP’s are all considered to be

providing primary health care, one would need to clarify the

approach taken as some GP’s may provide more sub-

specialized services (palliative care, geriatric care, allergy,

etc).

• Ratio of PHC providers entering/leaving the workforce

over the past 12 months by type of PHC provider. (CIHI

86)

• Could provide what physicians are billing and extrapolate

that no billing or less than a certain amount may indicate

they have relocated. Some physicians will continue to be

licensed in NS and maintain a billing number but not

practice here on a regular basis. Would need to confirm

parameters to be used.

• % of PHC clients/patients, ages 18 to 64 years, with

established diabetes mellitus (Type 1 and Type 2) who

have had an acute myocardial infarction, or above or

below knee amputation or began chronic dialysis within

the past 12 months. (CIHI 36)

• The challenge for many of the quality indicators that try to

link diagnosis with treatment/testing is that the diagnosis in

MSI is generally the primary reasons for visiting the

physician at that time. For example, the patient with

Diabetes may be seen 6 times in a given year but each time

is for a separate and distinct diagnosis and diabetes may not

be noted as the diagnosis at any visit. It is possible to get

some data on patient populations (i.e. with hypertension,

diabetes) then the complexity is linking this with the MIS

(hospital information) or LIS (lab information) systems.

• Algorithms for administrative data have been developed

that can identify some chronic conditions

• % of PHC clients/patients, ages 20 to 75 years, with

CHF who visited the emergency department for CHF in

the past 12 months. (CIHI 38)

• Complex links with LIS. See comments above.

• % of PHC clients/patients, 18 years and over, with

diabetes mellitus in whom the last HbA1c was 7.0% or

less (or equivalent test/reference range depending on

local laboratory) in the last 15 months. (CIHI 39)


.



March 31, 2008

96



coronary artery disease (CAD) who received annual

testing, within the past 12 months, for all of the

following:

� Fasting blood sugar,

� Full fasting lipid profile screening,

� Blood pressure measurement, and

� Obesity/overweight screening

• (CIHI 55)



hypertension who received annual testing, within the

past 12 months, for all of the following:

� Fasting blood sugar,


� Test to detect renal dysfunction (e.g. serum

creatinine),

� Blood pressure measurement, and


• (CIHI 56)

Complex links with LIS. See comments above.

% of PHC clients/patients, 18 years and over, with diabetes

mellitus who received annual testing within the past 12

months, for all of the following:

� Hemoglobin A1c testing,


� Diabetic nephropathy screening (e.g.

albumin/creatinine ratio, microalbuminuria),

� Blood pressure (BP) measurement, and


(CIHI 57)

• % of PHC clients/patients, 65 years and over, who

received an influenza immunization within the past 12

months. (CIHI 41)

*Provincial Immunizations are broken down into:

� ADAC-Tetanus, Diphtheria, Pertussis

� EXEM

� *INFL-Influenza

� MENC-Meningococcal Type C

� MMAR-Measles, Mumps, Rubella

� PENT-Diphtheria, Pertussis, Tetanus, Polio &

haemophilus influenza Type B

� QUAD-Diphtheria, Pertussis, Tetanus, Polio

� TEDI – Tetanus & Diphtheria (adult)

VARI- Varicella

• % of PHC clients/patients, 65 years and over, who

received a pneumococcal immunization. (CIHI 42)

Yes-PNEC

• % of PHC clients/patients who received screenings for

congenital hip displacement, eye and hearing problems,

by 3 years of age. (CIHI 43)

Cannot breakdown by specific areas of screening. Would be

included under regular check-ups/office visits.

Billing codes do not capture

• % of PHC clients/patients, who received required

primary childhood immunizations by 7 years of age.

(CIHI 44)

* see above codes for immunizations that could be reported on.

• % of women PHC clients/patients, ages 18-69 years,

who received papanicolaou smear within past 3 years.

(CIHI 50)

Yes, can be obtained

.



March 31, 2008

97


• % of PHC providers who were primarily remunerated by

the following method over the past 12 months by type of

provider: Fee for service, Salary, Capitation, and Mixed

System. (CIHI 104)

• % of PHC organizations who currently provide the

following services:

� Management care for an emergent but minor health

problem (e.g., sprained ankle, unexplained rash,

etc),

� Non-urgent care (e.g. well care (baby, child, woman

and/or man), chronic illness management, etc.),

� Prevention and health promotion and/or education

services,

� Full maternity and child care,

� Primary mental health care,

� Psychosocial services (e.g. counseling advice for

physical/emotional/ financial problems, etc),

� Liaison with home care,

� Referral to and follow-up care from specialized

agencies such as hospitals, youth centres, specialists

and/or other providers (through formalized

arrangements and/or agreements),

� Rehabilitation services,

� Nutrition counseling services,

� Provision of home visits by PHC

physicians/nurses/nurse practitioners/ pharmacists,

and

� End-of-life care.

(CIHI 12)

Define PHC organization.

Run/sort codes to see range of services provided, however there

would be significant limitations in the data

• % of primary care physicians on alternative funding

arrangements.

Yes, can be obtained

primary health care · 2014. 5. 12. · primary health care (phc) is the first and continuing point...

Documents