Medication Aide Certification Examination (MACE™) and Standard Setting Process

Ada Woo, PhD

Senior Psychometrician, NCSBN

Outline

Overview of test development process MACE item development 2010 national MACE standard setting 2010 medication aide job analysis 2011 medication aide knowledge survey

MACE Test Development Process

1. Job Analysis

2. Content Outline

3. Item Writing and Item Review

4. Subject Matter Expert (SME) Panel Review

5. Editorial Review

6. Building Test Forms

7. Setting the Passing Standard

8. Pretesting of Items

9. Administering Examinations

NCSBN and MACE: Historical Background

At the 2005 NCSBN Delegate Assembly, following a motion from the floor, the delegates passed the resolution to investigate the possibility of a competency examination for medication assistive personnel.

In 2007, the delegates adopted the Medication Assistant-Certified (MA-C) Model Curriculum.

MA-C Model Curriculum

Provide the boards with a curriculum for MA-Cs that will assist the boards in their mission of public protection

Allow for uniformity in MA-C education across boards and other regulating agencies

Consist of 60 hours of didactic training and 40 hours of supervised clinical practicum

Content modules comprising the didactic portion are provided in five modules, including recommended times for each

MACE Test Development

In August 2008, NCSBN acquired exclusive ownership of the MACE examination.

NCSBN is responsible for the development of MACE content.

Two MACE item writing and review workshops were conducted during May 12-14 and May 19-21, 2009 in NCSBN offices.

Fifteen subject matter experts (SMEs) representing six states and a wide range of expertise participated in the workshops.

Participants from May 12 – 14, 2009 meeting Name NCSBN

AreaState Job Title

Julie Bucher, RN

I OR Program Director

Bernadette Murphy, MBA, BSN, RN I OR Nurse Educator

Nancy Stephen, RN I OR Nurse Manager

Anne Citarella, RN

II IA Assistant Professor

Victor Palmer, RN

II IA Assistant Program Manager for Allied Health and Long-Term Care

Mary Jameson, RN II MO Staff RN/CNA-CMT-CPR-Insulin Instructor

Jill Layne, BSN

II MO Director of Nursing

Mary Stassi, RNC II MO Health Occupations Coordinator

Anne Citarella, RN

II IA Assistant Professor

Penny Worthley, RN,C, LNHA II MO Staff Education Coordinator

Avril Marshall, BA, RN IV DC Instructor

Dorothy Kritsch, BSN, RN,

IV NJ Professional Service Rep.

Penny Worthley, RN,C, LNHA II MO Staff Education Coordinator

Avril Marshall, BA, RN IV DC Instructor

Participants from May 19 – 21, 2009 meeting

Name NCSBN Area

State Job Title

Lisa Bare, RN III NC Assistant Director of Nursing

Bethany Davis, BSN, RN III NC Nursing Instructor

Lynn Thorpe, MSN, RN III NC Competency Evaluator/examiner for nurse aide candidates

Procedures of Item Writing and Review Meetings/Workshops

SMEs discussed a proposed content outline constructed based on the requirement listed in MA-C Model Curriculum.

After adopting the content outline, the SMEs established weights/percentages for each content category.

SMEs reviewed items administered on recent MACE test forms based on item statistics. In all, 180 items were identified for review. SMEs reviewed items for accuracy and appropriateness of entry-level medication aide range of authorized duties.

Workshop OutcomeMACE™ Item Writing and Review--Summary of Activity

  Item disposition

  Total items written/reviewed Approved for pretest Future rewrites

Activity at Meeting      

Newly written (total number) 91    

Newly written approved for pretest   52  

Client specific items with stats  180    

Client specific items approved for pretest

  57  

Items identified as problematic     14

     

Total Items Approved for Pretest 109

Total Items Reviewed 271

MACE Content Outline

I. Authorized DutiesA. Building RelationshipsB. Role of Delegating/Supervising NurseC. Role of MA-C

1. Permitted Duties2. Restrictions/Limitations

D. The Process of Delegation1. Responsibility of MA-C When Accepting Delegation Tasks

E. Specific Legal and Ethical Issues

(con’t) MACE Content Outline

II. Medication AdministrationA. Administering and Charting Medications

1. Medication Orders2. Documentation of Medication Administration3. Storage4. Disposal

B. Safety and Rights of Medication AdministrationC. Preparation and Actual Medication AdministrationD. Prevention of Medication ErrorsE. Routes of AdministrationF. Factors Affecting How the Body Uses MedicationG. Classifications/Categories of Medications Related to Body Systems and

Actions (e.g., antimicrobials, cardiovascular, dermatological, endocrine…)

H. Rights of Individuals

(con’t) MACE Content Outline

III. Medication Concepts and Measurements A. Medication Basics

1. Terminology and abbreviations2. Dosage Range3. Actions and Implications4. Therapeutic and other side effects (e.g. idiosyncratic,

paradoxical, antagonist)5. Precautions6. Allergic and Adverse Reactions7. Interactions

B. Forms of Medication1. Liquid2. Solid and semi-solids

C. Mathematics, Weights and Measures

(con’t) MACE Content Outline

IV. Observation, Care and Reporting

A. Causes and Reporting of Medication Errors

B. Reporting of Symptoms and Side Effects

C. Reporting Any Change From Client’s Normal Condition

D. Location of Resources and References (e.g., nurse, pharmacist, physician, package/drug insert, drug reference manuals)

National MACE Standard Setting

The purpose is to determine a passing score for the National MACE examination in compliance with the Standards for Educational and Psychological Testing

Used criterion-referenced standard setting method

Used modified-Angoff procedures Used two test forms (total of 78 unique items)

Subject Matter ExpertsSME Area State Work Setting

Bernadette Murphy, MBA, BSN, RN

I OR State Hospital

Nancy Stephen, RN I OR State Hospital

Jill Layne, BSN, RN II MO Continuing Care Retirement Community (CCRC)

TyAnn Bell, CNA, CMT II MO CCRC

Victor Palmer, RN II IA Community College

Lynn Thorpe, MSN, RN III NC Competency Evaluation

Martha Holloway, MS, RN III AL Public Schools

Jeanne Delicata, MSN, RN, ACNS, BC

IV ME Veterans Home

Lynn Johnston, MS, BSN, RN

IV ME Veterans Home

Maryann Ingram, LPN IV MD Nursing Home

Standard Setting Procedures

1. Received overview of standard setting process2. Discussed qualifications for taking the MACE

examination and define minimally competent candidates

3. Reviewed content outline4. Received standard setting training5. Rated practice items6. Reviewed and discussed practice item ratings7. Rated first batch of actual items8. Rated remaining items9. Discussed results from round 1 ratings10. Conducted round 2 ratings

Results

Form and Round % Cut-score Raw Cut-score (50 item test)

Standard Deviation

Intraclass Rating

Reliability

Form 1 Round 1 77.94% 38.97 (39) 2.71 0.74

Form 2 Round 1 77.31% 38.66 (39) 2.45 0.70

Form 1 Round 2 79.45% 39.77 (40) 3.12 0.77

Form 2 Round 2 78.94% 39.47 (40) 3.17 0.75

Ratings became more consistent among judges from round 1 to round 2.

Judges unanimously agreed that the national MACE passing standard should be set at 40 (out of 50) for 2010 national MACE test forms.

Post-Workshop Questionnaire

Question 1: I feel the training was adequate. Average rating: 3.73

Question 2: I understood the definition of a “minimally competent examinee”.

Average rating : 3.73

Question 3: I understood the rating process. Average rating : 3.82

Question 4: I feel my opinion was considered. Average rating : 3.91

Question 5: The practice exercises helped me understand the process.

Average rating : 3.82

Rating scale:1 = strongly disagree2 = disagree3 = agree4 = strongly agree

Some Feedback from SMEs Staff was wonderful about explaining the process of

standard setting. It was a great experience. Great experience. The exercise validated the process

of the cut score. This was a wonderful opportunity to be able to

participate in this workshop. I look forward to participating again soon in other projects.

From a state evaluator’s and nurse educators perspective, the workshop was relevant in understanding the concepts and dynamics involved in establishing a minimum for safe medication administration of entry level certified medication aides.

(con’t) SME Feedback

Excellent process of standard setting! Everyone’s opinion was valued and respected. Thorough explanations.

It was a great process! Invite me back! Thank you for allowing me to participate in this

process. I see and better understand how the NCSBN works to keep the public safe.

I enjoyed the experience and feel the process was very fair and accurate.

Very informative and enjoyable! Staff was most helpful. Thanks for the opportunity.

2010 Medication Aide Job Analysis Proposed Timeline and Next Steps

October 2010 – August 20111. Develop methodology plan2. Host SME workshop3. Develop surveys and correspondence4. Mail pre-survey notice and surveys5. Collect data 6. Analyze data7. Conduct nonresponder study8. Finalize report9. Publish research brief10. Host SME Test Specification workshop

2010 Medication Aide Knowledge Survey Proposed Timeline and Next Steps

April 2011 – November 20111. Develop methodology plan2. Host SME workshop3. Develop surveys and correspondence4. Mail pre-survey notice and surveys5. Collect data 6. Analyze data7. Conduct nonresponder study8. Finalize report9. Publish research brief

How you can assist in the 2010 Medication Aide Surveys

No National Registry Challenges in obtaining mailing addresses for

a nationally representative sample Examples from other nursing survey studies:

1. Nurse Aide 2010 JA and KSA Surveys – Randomly selected 6,500 nursing directors from hospitals, nursing homes and long term care facilities based on a facility mailing list

2. NCLEX-RN 2008 practice analysis – Randomly sampled 6,000 NCLEX-RN candidates who passed the examination from January 1 to April 13, 2008.

MACE Publications

1. MACE Item Writing and Review Workshop Report - https://www.ncsbn.org/MACE_Item_Writing_Workshop_Report.pdf

2. MACE Standard Setting Workshop Report - https://www.ncsbn.org/Standard_Setting_Report_for_MACE.pdf

3. Medication Assistant-Certified (MA-C) Model Curriculum - https://www.ncsbn.org/07_Final_MAC.pdf

4. Quick Reference to MA-C Model Curriculum - https://www.ncsbn.org/07_new_MACChecklist.pdf

Thank You!