monitoring and evaluating community system strengthening current css indicators and approaches round...

Monitoring and Evaluating Community System Strengthening

Current CSS Indicators and Approaches

Round 11 regional Workshop, Nairobi, KenyaAugust 19, 2011

Monitoring & Evaluation

Outline of the presentation

1. Overview of CSS indicators review; rational, objectives, process and approach;

2. The indicator set;

3. Developing M&E frameworks for CSS programs and linking these to the Global Fund proposal;

4. Program review and evaluations

Review rational and process

27 indicators from May 2010 release not incorporated into RD 10 proposals;

Preparing for Round 11 launch; Solicit implementer’s feedback

Assess applicability of the indicators at regional and country level;

To assess the comprehensiveness in measuring community systems strengthening;

Determine whether existing alternatives can be recommended; Assess existing or anticipated challenges measuring the

recommended indicators; Coordinated by TGF but conducted by a partner representative

Technical working group Regional consultations with constituencies

Results overview

1. A set of 29 indicators – all process and output level

2. Two categories - CORE and additional

3. While many are process and output level, core indicators will indicate improvements in the system over time as proxy measures of system/service quality

4. Shows the linkage to health outcomes as presented in a holistic health system approach

5. Clearly identified gaps in the CSS M&E framework for further development;

1. Field testing and Validation

2. QoS framework development – Programs and Community system

CSS Output/Process Level Indicators by SDA

10 CORE INDICATORS

Core component 1: Enabling environments and advocacy

SDA 1: Monitoring and documentation of community and government interventions

Number of community based organisations and/or networks that have meaningfully participated in joint national programme reviews or evaluations in the last 12 months (1.1)

SDA 2: Advocacy, Communication and Social mobilisation

Number of community-led advocacy campaigns that saw a targeted policy change or can clearly document improved implementation of an existing (targeted) policy within 2 years of the start of the advocacy campaign (2.1)

Core component 2: Community networks, linkages, partnerships and coordination

SDA 3: Building community linkages, collaboration and coordination

Number and percent of community based HIV, TB and malaria service organisations with referral protocols in place that monitor completed referrals according to national guidelines (3.1)


10 CORE INDICATORS

Core component 3: Resources and capacity building

SDA 4: Human resources: skills building for service delivery, advocacy and leadership

Number and percentage of staff members and volunteers currently working for community based organisations that have worked for the organisation for more than 1 year (4.1)

SDA 5: Financial resources Number and percentage of community based organisations that have a sound financial management system, which is known and understood by staff and consistently adhered to (5.1)

SDA 6: Material resources – infrastructure and ` (including medical and other products & technologies)

Number and percentage of community based organisations reporting no stock-out of HIV, TB or Malaria essential commodities according to program implementation focus during the reporting period (6.1)


10 CORE INDICATORS

Core component 4: Community activities and service delivery

SDA 7: Community based activities and services – delivery, use and quality

Number and percentage of community based organisations that deliver services for malaria, TB and HIV according to national or international accepted service delivery standards (7.1)

Core component 5: leadership and organizational strengthening

SDA 8: Management, accountability and leadership

Number and percentage of staff members of community-based organisations with written terms of reference and defined job duties (8.1)


10 CORE INDICATORS

Core component 6: Monitoring & Evaluation and Planning

SDA 9: Monitoring & evaluation, evidence-building

Number and percentage of community based organisations that submit timely, complete and accurate financial and programmatic reports to the nationally designated entity according to nationally or internationally recommended standards and guidelines (where such guidelines exist) (9.1)

SDA 10: Strategic planning Number and percentage of community based organisations with a developed strategic plan covering 2 to 5 years (10.1)


19 ADDITIONAL INDICATORS

Core component 1: Enabling environments and advocacy

SDA 1: Monitoring and documentation of community and government interventions

Number of community based organisations and/or networks that have documented and publicized barriers to equitable access to health services and/or implementation of national AIDS, TB and Malaria programming during the last 12 months (1.2)

Core component 2: Community networks, linkages, partnerships and coordinationSDA 3: Building community linkages, collaboration and coordination

Number and percentage of community based organisations that are represented through membership in national or provincial level technical or coordination policy bodies of disease programmes and providing feedback to communities (3.2)Number and percentage of community based organizations that implemented at least one documented feedback mechanism with the community they serve in the last 6 months (3.3)




SDA 4: Human resources: skills building for service delivery, advocacy and leadership

Number and percentage of community health workers currently working with community based organisations who received training or re-training in HIV, TB or malaria service delivery according to national guidelines (where such guidelines exist) during the last national reporting period (4.2)

Number and percentage of community based organisations that received supportive supervision in accordance with national guidelines (where such guidelines exist) in the last 3/6 months (4.3)

Number and percentage of volunteers working with community based organisations that are provided with incentives (4.4)




SDA 5: Financial resources Number and percentage of community based organisations that have core funding secured for at least 2 years (5.2)

SDA 6: Material resources – infrastructure and ` (including medical and other products & technologies)

Number and percentage of community based organisations that keep accurate data for inventory management according to national or international policy (6.2)

Number and percentage of community based organisations with staff or volunteers that are responsible for stock management trained or re-trained in stock (inventory) management in the past 12 months (6.3)Number and percentage of community based organisations that maintain adequate storage conditions and handling procedures for essential commodities (6.4)



Core component 4: Community activities and service delivery

SDA 7: Community based activities and services – delivery, use and quality

Number and percentage of community based organisations that implemented activities contributing to the national disease strategic plan as documented by their plans and reports to the national designated entity (7.2)Number and percentage of people that have access to community-based HIV, TB or malaria services in a defined area (7.3)

Core component 5: leadership and organizational strengthening

SDA 8: Management, accountability and leadership

Number and percentage of community based organisations with staff in managerial positions who received training or re-training in management, leadership or accountability during the last reporting period (8.2)Number and percentage of community based organizations that received technical support for institutional strengthening in accordance with their requests the last 12 months (8.3)




SDA 9: Monitoring & evaluation, evidence-building

Number and percentage of community based organisations with at least one staff member in charge of M&E (9.2)

Number and percentage of community based organisations with at least one staff member in charge of M&E who received training or re-training in M&E according to nationally recommended guidelines (where such guidelines exist) during the last national reporting period (9.3)

Number and percentage of community based organisations using standard data collection tools and reporting formats to report to the national reporting system (9.4)

Number and percentage of community based organisations conducting documented reviews of their own programme performance according to their strategic plan in accordance to national reporting cycle (9.5)




SDA 10: Strategic planning Number and percentage of community based organisations that are implementing a budgeted annual work plan (10.2)

Developing M&E frameworks for CSS programs

• In country consultations to include the CSS programs and M&E into national strategies – national M&E plan

• View CSS programs as part of the overall health system Refer to HSS/CSS complementarity

• Link CSS interventions to health results and therefore have a comprehensive results framework refer to - CSS results framework

– Show direct linkage to health service delivery

• Link M&E framework to the Global Fund grant through the proposal Performance Framework (select from the national M&E plan or an equivalent)

Example of the complementarity between the Health and community System

INPUTS (HSS/CSS) INPUTS (HSS/CSS)

ACTIONSACTIONS

% of facilities with % of facilities with

capacity (ready) to capacity (ready) to

provide ACT provide ACT

treatment to treatment to

malaria cases malaria cases

(drugs, trained (drugs, trained

staff & guidelines, staff & guidelines,

& diagnostic s& diagnostic s))

Facilities reporting Facilities reporting

no stock outs of no stock outs of

ACTs/LLINsACTs/LLINs

Increase “in-Increase “in-

patient & patient &

outpatient visits” outpatient visits”

(e.g., (e.g., people people

accessing ACTsaccessing ACTs))

% of children % of children

under under

five/pregnant five/pregnant

women that women that

slept under a slept under a

LLIN the LLIN the

previous night previous night

% of % of

uncomplicated uncomplicated

malaria cases malaria cases

receiving receiving

appropriate appropriate

treatment treatment

Annual Annual

parasite parasite

incidence rateincidence rate

HSS SDA: Health HSS SDA: Health

WorkforceWorkforce

-Health worker retention-Health worker retention

Training & educationTraining & education

HSSHSS SDA: Procurement & SDA: Procurement &

Supply Supply Chain ManagementChain Management

-Distribution of ACTs, -Distribution of ACTs,

LLINs and laboratory LLINs and laboratory

equipment & diagnosticequipment & diagnosticss

CSS SDA: Service CSS SDA: Service

availability, use and availability, use and

qualityquality

-Demand creation through -Demand creation through

community community mobilizationmobilization

for LLIN use, involvement for LLIN use, involvement

in PMTCT, follow up of in PMTCT, follow up of

mothers and babies in mothers and babies in

communities and creating communities and creating

awareness of available awareness of available

PMTCT servicesPMTCT services

OUTPUTOUTPUT OUTCOMEOUTCOME IMPACTIMPACT

Outcome Level Indicators

Malaria

1. Suspected malaria cases that have laboratory diagnosis/ Children under-5 with fever (household survey) that received laboratory diagnosis

2. Uncomplicated malaria cases receiving appropriate treatment

3. Women who received two or more doses of IPT during their last pregnancy

Tuberculosis

1. Case notification rate

2. Treatment success rate

Outcome Level IndicatorsMaternal, New-born and Child Health

1. Proportion of women attending antenatal care

2. Proportion of institutional deliveries

3. Children who received DTP3

4. Children fully immunised

5. Drop out between DTP1 and DTP3 coverage (%)

6. Equity in immunisation coverage (Difference in DTP3 coverage in lowest wealth quintile and coverage in highest wealth quintile) (%)

HIV/AIDS

1. Proportion of adults/children with advanced HIV infection receiving ART

2. HIV-positive pregnant women who receive ART for PMTCT

Impact Level Indicators

Health Status

1. Neonatal mortality

2. Mortality due to major cause of death

3. Child mortality

4. Maternal mortality ratio

5. ART survival rate

6. PMTCT transmission rate

7. Confirmed malaria cases

8. Inpatient confirmed malaria cases

9. Future deaths averted

Financial Risk Protection

The ratio of household out-of-pocket as payments for health to total expenditure on health

CSS indicators recommended data sources • Administrative records: organizational routine source documents.

– The records will defer by indicator and organisation and therefore need to be defined at the planning stage.

– Examples: Activity reports, policy documents, monitoring reports, supervision checklists, client registers, training records.

• Institutional Surveys/Assessments: periodic data collection exercises

– gather information on defined aspects of the organisation usually related to performance and or quality standards.

– These are commonly implemented by the national level designated entity with oversight responsibility for community based activities and services.

– either as part of routine supervision or as an exclusive exercise.

– Require predefined service standards and checklists

• Population-based survey - representative sample from population

– commonly implemented by the national level designated program authority with the involvement of all partners.

– CSS implementers are encouraged to participate as much as possible in the planning and execution of the surveys.

20

Program reviews and evaluation

•CSS programs need to plan and conduct periodic program evaluations

•CSS programs encouraged to include impact outcome indicators to demonstrate contribution to health results

•Tracking of relevant core indicators to track improvement in the community system

– For Periodic reviews, the core indicators will be assessed along with the grant impact outcome indicator performance

monitoring and evaluating community system strengthening current css indicators and approaches round...

Documents