Component 2The Certification of Graduates Using

a National Competency-based Examination

Tri Hanggono Achmad 2nd HPEQ Conference

Denpasar, 4 December 2011

Outline of presentation • Overview of Component 2 • Report and key performance indicator • Lesson learned • Challenges and Future plan

OVERVIEW OF COMPONENT 2

Health Professions Education Quality (HPEQ) Project

• Rationale: Indonesia’s current health indicators, the current status of Indonesia’s health professional education programs, and on the increasing demand for larger numbers of and more highly qualified health professionals.

• Goals: strengthen quality assurance policies governing the education of health professionals in Indonesia through 1. rationalizing and assuring competency focused accreditation

of public and private health professional training institutions,2. developing national competency standards and testing

procedures for certification and licensing of health professionals, and

3. building institutional capacity to employ results based grants for encouraging use of accreditation and certification standards in the development of medical school quality.

Objectives

Sub-component 2.1: Establishing an Independent National Agency for Competence Examination of Health Professionals

Sub-component 2.2: Improving the Methodology and Management of the National Competency-based Examination

Sub-component 2.3: Developing an Item Bank Networking System to Support the National Competence Examination

Milestones for Component 2Main Program 2010 2011 2012 2013 2014

2.1 National Agency kick offmid-term

evaluationfinal

evaluationtake off sustainable

2.2.a Computer-based Testing (CBT)

Paper-based, Preparation and

Try out

Implementation

Implementation

Implementation

Implementation

Paper-based, CBT Preparation and

Try Out

Paper-based, CBT

Preparation and Try Out

Implementation

Implementation

Implementation

PreparationBenchmarking

TestPreparation and Try Out

Implementation

Implementation

2.2.b Objective Structured Clinical

Examination (OSCE)

Preparation Try-out Implementati

onImplementati

onImplementati

on

PreparationTry out

Implementation

Implementation

Implementation

Preparation Preparation Preparation Try outImplementati

on

2.3 National Item Bank Networking for

Assessment (NIBNA)

Preparation Establishment

Enhancement for Pre-clinical

Formative Test

Enhancement for Clinical Formative

Test

Enhancement for Progress

Test

Preparation Establishment

Enhancement for Pre-clinical

Formative Test

Enhancement for Pre-clinical

Formative Test

Enhancement for Clinical Formative

Test

Preparation Preparation Preparation PreparationEstablishmen

t

Medicine Dentistry Nursing/Midwifery

HPEQ +

• Pharmacy • Nutrition • Public Health • Other health professionals related to MTKI

KPI PROJECT per YEAR

Key Performance Indicator Successfull rates on the first take ofNational Competence Examination

Health Professional

Baseline (2009)

Target Value

2010 2011 2012 2013 2014

Medicine 71,67 % 71,67 % 73,25 % 75,50 % 79,60 % 84 %Dental 79 % 80 % 80 % 81 % 82 % 83 %Nursing 60 % 65 %Midwifery 60 % 65 %

ACHIEVEMENT OF COMPONENT 2

Objectives

Sub-component 2.1: Establishing an Independent National Agency for Competence Examination of Health Professionals

Sub-component 2.2: Improving the Methodology and Management of the National Competency-based Examination

Sub-component 2.3: Developing an Item Bank Networking System to Support the National Competence Examination

Subcomponent 2.1

Lembaga Pengembangan Uji Kompetensi (LPUK)• An independent national agency that responsible in

developing assessment strategy, methodology, and tools to evaluate competency of graduation from health professional education institution (medicine, dentist, nurse, midwife)

Function: 1. Assessment service: as resource person and

facilitator to help the management of examination for organization in each health professions.

2. Academic: to improve assessment system in health professional education with evidence based practice

Organization

• Tim Task Force 1300a/E3-HPEQ/SK/08.11• Director: Iwan Dwi Prahasto• Secretary: YulherinaMember: • Moh. Ghozali (Medicine) Gandes Rahayu (Medicine)• Tatong Haryanto (Medicine) Mei Syafriadi (Dentist)• Iwan Dewanto (Dentist) Mia Damayanti (Dentist)• I Made Kariasa (Nurse) Pramita Iriana (Nurse)• Yetty L Irawan (Midwife) Ani Kusumastuti (Midwife)• Rahayu Endah Astuti (HPEQ)

Advisory board

National

Regional

Examination Center

Chariman

Examination Division

Assess Management

Division ICT Division Item Bank

Division R & D Division

Executive secretary

The Milestone LPUK

Before 2010: First phase of development

2010 – 2014: Program Acceleration and Expantion

After 2014: Sustainability Phase

Establishment of KBUKDI & KDGI

Development of KNUKP

Various workshop in item development & review, standard setting

Implementation of UKDI & UKDGI, and competence examination in some MTKPs

Academic paper & various guidelines for workshop & implementation of examination

Resource Sharing

International Benchmarking

Human resource

development

Regional & International collaboratio

n

Recertification & assessment of professional behaviourResearch

grant & scientific publicati

onResource & ICT support for assessment Establishme

nt of LPUK & quality standard

Collaboration w/ other HP

Subcomponent 2.1

• Benchmarking to National Board Examination for medicine, nurse & dentist

• Technical assistance from Prof. Gordon Page from Univ. British Columbia – Canada

• Agreement with Ministry of Health MoH Regulation Number 1796/Menkes/Per/VIII/2011 about Registration of health professional

• Guidelines to conduct various workshop on assessment• Item development for CBT • IT and CBT management • Item development for OSCE• OSCE Examiner • Trainer for Standardized Patient• Item bank administration• OSCE Coordinator

• Research grant on assessment

Objectives

Sub-component 2.1: Establishing an Independent National Agency for Competence Examination of Health Professionals

Sub-component 2.2: Improving the Methodology and Management of the National Competency-based Examination

Sub-component 2.3: Developing an Item Bank Networking System to Support the National Competence Examination

Subcomponent 2.2

• Medicine & Dentist: • Standard setting with Modified Angoff • Try out & CBT Implementation• Try out OSCE

• Nurse & Midwife: • Try out CBT in November 2011

Try out & CBT Implementation

Activity Number of CBT Center Medicine Dentist Nurse Midwife

Try Out 1 6 1 12 (809) 13 (610)Try Out 2 10 6Try Out 3 12 11Try Out 4 14 13Try Out 21CBT 1 10CBT 2 14CBT 3 27

Try Out OSCE

Medicine

OSCE Center

Candidates

Try Out 1 1 14Try Out 2 12 160

Dentist OSCE Center

Candidates

Try Out 1 1 9Try Out 2 13 274

Standard setting

Standard Setting

Medicine Dentist

Number of judges Cut Score Number of judges Cut Score

I 52 53 21 49

II 52 58 21 49

III 46 58

IV 48 58

V 50 58

VI 66 61

• Judges: from health professions institutions • Implementation of standard setting in some institutions

Total= 32 CBT CenterTotal workstations= 2892 PC/laptop(842 : 2140)

Number of CBT Center & Workstations

Average result of UKDI & UKDGI in 2010 – 2011

2010 201170%

72%

74%

76%

KPI HPEQ UKDI

2010 201169%71%73%75%77%79%81%83%85%

UKDGI KPI

Comparison of UKDI result based on Accreditation

ANOVA p=0,000

Objectives

Sub-component 2.1: Establishing an Independent National Agency for Competence Examination of Health Professionals

Sub-component 2.2: Improving the Methodology and Management of the National Competency-based Examination

Sub-component 2.3: Developing an Item Bank Networking System to Support the National Competence Examination

Subcomponent 2.3 Establishment of NIBNA • Item development for knowledge based assessment –

CBT bank masing2 profesi• Item development for skill based assessment - OSCE• Development of IT support system for student assessment • Technical assistance for Item Bank Administration: Prof. Clarke Hazlet from Univ. Alberta – Canada

Number of workshops & participants in 2011

No. Workshop Frequency Participant Institution

1 Item Review and Development (CBT)

6 (4 M, 2 D)8 (4 N, 4 Mid)2 (other HP)

389 (M)45 (D)168 (N)132 (Mid)

65 (M)26 (D)168 (N)129(Mid)

2 CBT Coordinator 2 26 (M)8 (D)

26 (M)8 (D)

3 Standard Setting 7 65 (M)26 (D)

65 (M)26 (D)

4 OSCE Examiner 3 (2 M, 1 D) 131 (M)28 (D)

65 (M)26 (D)

5 Item Review and Development (OSCE)

3 (2 M, 1 D) 130 (M)53 (D)

65 (M)26 (D)

6 OSCE Coordinator 2 12 (M) 12 (M)

7 Standardized Patient 2 65 (M)26 (D)

65 (M)26 (D)

M: Medicine; D: Dentist; N: Nurse; Mid: Midwife; HP: Health Professions

Item Bank

Medicine Dentist Nurse Midwife 0

500

1000

1500

2000

2500

3000

3500

4000

Number of MCQ Item in Banks

OSCE: Medicine: 12 stations Dentist: 9 stations

Jumping Hurdles SuccessfullyDeveloping adequate quantity of high quality items: achievable target

If development is collaboratively produced & organized, realistic timelines for achieving a “large-enough” bank are

- 5 yrs for Medicine (given 72 medical schools & 40,000 items req’d)

- 10 yrs in Dentistry (if 26 dental schools & 30,000 (?) items req’d)

- 4 yrs in Nursing (if 57 nursing schools & 25,000 (?) items req’d)

- 3 yrs in Midwifery (if 60 midwifery schools & 20,000 (?) items req’d)

SIPENA Development Road Map

© PadmaNet 2010 – ss100722

IT Modules - interconnectivity

Examinee OL-Reg

Examinee On Line Registration

Information Web

Exam Location Adm

Local Exam Supervisor

Registration

National Exam Supervisor

Registration

Exam Process

PBT

CBT

OSCE

BAU

Summative - Scoring

Absolut

Normatif

Formative - Institution

Selected Item Bank

Tested Item Bank

New Item Bank

Summative - Assesment

Bank 1 : New Item

Bank 2 : Reviewed Item

Bank 3 : Tested Item

Bank 4 : Selected Item

Call for Item

Item Writer / Reviewer Registration

Peer Review

Panel Expert

Blinding

Review

Blinding – Review

Lesson learned • Building awareness of the role of assessment in improving the

quality of life (health) and driving in prioritizing quality• Support and Commitment from, and collaboration among various

stakeholders• More potencies being identified• Resources sharing (human, infrastructure, items, system)

efficiency & close the gap• NCE support the improvement of assessment method & learning

process in institutions assessment drives learning (standard of competence)

• Assessment as a learning / education materials and management process

• Progress in adopting evidence based protocol security means strong committment

Challenges & Future plan

• The dynamic of the project continuity of the program based on the road map with possible modification

• Maintaing quality in bigger scope & quantity of project prioritizing & synchronize & coordination among Project Components & stakeholdersresources sharing among health professionals

• Closing the gap of development & implementation paralel implementation of assessment development & learning response (case : OSCE & Clinical teaching)

• Customers friendly item banking & evidence based security system• Credit for item citation index academic merit system• Future research in professional assessment (incl. Publications)• Collaboration with other international examination agency

This Project Can be & Will be

Collaboration is Project’s Mode of Operation

Since collaboration among health care educators also involves endorsement & participation of government & professional bodies in health care

Successful !

Terima Kasih

tachmad@yahoo.com

spdewi@yahoo.com