core evidence-/based regulation: a regulatory performance measurement system

COREEvidence-/based Regulation:

A Regulatory Performance Measurement System

Goal

To help State Boards of Nursing improve their management and

service delivery

Need for Study

Multiple stakeholders were demanding accountability

Lack of clarity among stakeholders about Board roles and responsibilities

Trend toward outcome measurement at state level

Purpose of the Study

Incorporating data from internal and external sources

Using benchmarking strategies Identifying best practices

Establishment of a Performance Measurement System

History

In 1998, the NCSBN Board of Directors appointed a project advisory group to provide oversight and guide development of a performance measurement system that incorporates data collection from internal and external sources, identification of best practices, and the use of benchmarking strategies.

History

Twelve dedicated member boards of nursing (Kentucky, Louisiana-RN, Maryland, Missouri, Nebraska, New Mexico, North Carolina, North Dakota, Ohio, Tennessee, Texas-RN and West Virginia-PN) contributed time and resources to the development of 10 instruments used to gather data in five areas: discipline, licensure, practice, education and governance.

Phase 1 of the Project

1. Validation of Board Roles

Study conducted by The Urban Institute

Methods to Gather Information

Interview– Focus Groups– Individuals

Review of Secondary Documents

Results of Phase 1:

Roles Identified– Establish scope of practice for nurses– Issue licenses to qualified nurses– Assure continued competence– Investigate complaints and impose

disciplinary sanctions as appropriate

Phase 2 of the Project

1. Validation of Board Roles

2. Identification of Performance Indicators

Results of Phase 2:

Technical Work Group developed– Performance indicators– Outcome Indicators– Output Indicators– Efficiency Indicators

Examples

Performance Indicator– Timeliness of complaint handling

Outcome Indicator– Average time for complaint resolution

Examples continued

Output Indicator– Number of complaints resolved in FY

Efficiency Indicator– Average cost per completed complaint

Phase 3 of the Project

1. Validation of Board Roles

2. Identification of Performance Indicators

3. Tool Development

Tool Development Process

Original tools developed and piloted– Because the science of regulation was not formed

enough to guide their efforts, they did not know which data would prove to be the crucial evidence and so they included all data.

Original tools revised and further tools developed– 6 data collection tools for boards of nursing– Surveys to collect data from 6 stakeholder groups

Board Surveys

1. Discipline

2. Licensure

3. Education Program Approval

4. Practice

5. Governance (Executive Staff)

6. Governance (Board President)

Board Surveys Included

Processes used, e.g.:– Investigator caseloads– Use of site visits or self-reports for

education programs

Timeliness issues, e.g.:– Days needed to processes license request

Board Surveys Included

Outcomes achieved, e.g.:– Number of discipline cases closed

Opinions, e.g.:– From Executive Staff and Board President

Stakeholder Groups

1. Licensed nurses

2. Health care employers

3. Nurses who had been the subjects of complaints

4. Persons who had lodged complaints

5. Nursing associations

6. Nursing education programs

Stakeholder Surveys Included

Perceptions of board’s– Timeliness,– Fairness,– Adequacy of regulation, etc.

Satisfaction with board’s– Communication with stakeholder group,– Nursing program approval process, etc.

Phase 4 of the Project

1. Validation of Board Roles

2. Identification of Performance Indicators

3. Tool Development

4. Data Collection

Data Collection

Stakeholder contact information submitted by boards– Random samples selected from those

submitted

6 data collection tools sent to boards of nursing

Data Collection Wisdom

Not all boards routinely collected the data asked for– Many boards used this as an opportunity to

improve/modify amount and types of data collected

Language/definitions (i.e., financial data, board processes) differed among boards

Phase 5 of the Project

1. Validation of Board Roles

2. Identification of Performance Indicators

3. Tool development

4. Data Collections

5. Reports of Findings

Report Format

I. Aggregate findingsA. Data results

1. From board surveys

2. From stakeholder surveys

B. Relationships among variables

II. State-specific findingsA. Comparison of state with all states

B. Comparison of state with “like” boards

Example of Comparison with Aggregate Data, i.e., Ed. Program Perceptions

Approval Process State Rating

Aggregate Rating

Interval betweenboard visits

2.66 1.40

Preparation time for board visits

2.45 1.41

Feedback/evaluation provided by board

2.16 1.38

Comparisons with Similar Boards

Boards evidenced a wide variety of resources, structures & processes

Boards were compared to other boards similar in a number a variables

Comparison Variables

Size of staff Staff assigned to specific functions Numbers of investigators Whether or not state mandated

reporting of errors Processes related to complaint review

Comparison Variables continued

Board structure Standard of proof Staff autonomy Number of board meeting per year Timeliness of discipline processes Timeliness of licensure processes

Comparisons of Sample Board with Other Boards with Similar Numbers of Staff

Sample Board

Similar Boards

Number of nursing programs 78 60.53

Rate of complaints resolved 0.78 0.66

Complaints per investigator 116.9 95.78

Nurses' perceptions of courteousness 1.23 1.23

Number of staff involved with investigations 12 7.92Number of board meetings/year 10 6.4

Phase 6 of the Project

1. Validation of Board Roles

2. Identification of Performance Indicators

3. Tool development

4. Data Collections

5. Reports of Findings

6. Search for “best practices”

The Search for Best Practices

Data were used to identify boards with consistently high ratings in– Outputs– Effectiveness

Ratings were explored in 5 functional areas– Discipline, licensure, education program

approval, practice and governance

The Search for Best Practices

Selected boards were interviewed to discover– Practices common among boards with

consistently high ratings– Differences from boards with lower ratings

Discipline Best Practices

Boards with the highest ratings on discipline outcomes– Delegated authority to board staff– Communicated well with stakeholders– Hired investigators and attorneys &

actively managed discipline process– Trained and mentored investigative staff– Applied discipline sanctions consistently

Licensure Best Practices

Boards with the highest ratings on licensure outcomes– Secured essential human and other

resources– Made an aggressive commitment to

customer service

Education Best Practices

Boards with the highest ratings on education outcomes– Provided consultative, as well as

evaluative services to education programs– Took a leadership role in establishing

congruence between education and regulation

Practice Best Practices

Boards with the highest ratings on practice outcomes– Facilitated understanding of legal scope of

practice– Made an aggressive commitment to customer

service– Established a high level of involvement with the

statewide nursing community– Delegated authority to board staff

Governance Best Practices

Boards with the highest ratings on governance outcomes– Promoted an understanding of the respective

roles of staff and board members– Built an effective working relationship and a high

level of trust between board and staff– Facilitated an effective working relationship

among board members– Demonstrated a commitment to board member

development

Phase 7 of the Project

1. Validation of Board Roles2. Identification of Performance Indicators3. Tool development4. Data Collections5. Reports of Findings6. Search for “best practices”7. Development of Ongoing System of

Performance Measurement

CORE

Nursing Boards educated on CORE– Manuals prepared, distributed and

explained– Ongoing presentations and publications

Best Practice “Tool Kit”– Submissions by boards of systems and

processes that have facilitated best practice

CORE

Ongoing improvement of data collection system– All survey items linked to outcome and/or

best practice– Data collection streamlined– Additional tools created and piloted

• Information Technology• Finance• Board Member

CORE Timeline

2006

Advisory Committee

Formed

2000

Stakeholder Surveys

Board Surveys

20021998

Urban InstituteStudy

Stakeholder Surveys

1999 2001 2003 2004 2005

RevisedQuestionnaires

Schedule

Jan Feb March April May June July Aug Sept Oct Nov Dec

Jan Feb March April May June July Aug Sept Oct Nov Dec

Conduct Stakeholder Surveys

Analyze Data

Report Results in Aggregate & by State

Conduct Board Surveys

Analyze Data

Report Results in Aggregate & by State

X

X

X

X

X

X X

X X

X

Board Participation2006 Stakeholder Surveys

Nurses

Nurses Subject of a Complaint

Persons Who Made

a Complaint About a Nurse Associations Employers

Education Programs

30 24 17 20 25 26

Total = 33 different boards participatedin 2006 Stakeholder Surveys

Utilization of Data by Boards of Nursing

Data has been used to: Support decision-making Develop mandated reports Provide information to legislators Change data management processes Improve stakeholder satisfaction Streamline processes Determine priorities

Benchmarking

"How are we doing?" "Are we tracking the right measures?" "How do we compare with others?" "Are we making progress fast enough?" "Are we using the best practices?"

CORE

PerformanceBenchmarking

Process Benchmarking

Identifying best practices

Adopting best practices

Performance benchmarking

The collection of (generally numerical) performance information and making comparisons with other compatible organizations.

Benchmarking of Board Performance

Quality Efficiency (cost effectiveness & timeliness)

Stakeholder survey Stakeholder surveyBoard survey (?) Board survey

Criteria of Item Selection

Measurable Comparable between states Objective, if possible

Licensure– Cost per licensee– Cost per license processed– Timeliness to process licensure

• by initial examination

• by endorsement

• by renewal

• by reinstatement

Benchmarking Board Efficiency

Practice– Cost per licensee– Number of practice questions handled per FTE

Benchmarking Board Efficiency

Education Program Approval– Rate of education site visits per staff

Benchmarking Board Efficiency

Discipline– Average number of investigations completed by each

investigator– Rate of investigations completed– Rate of complaints resolved– Rate of nurses disciplined– Timeliness of complaint handling – Cost per resolved complaint– Cost per investigation– Cost per licensee– Cost per discipline case

Benchmarking Board Efficiency

Mock Benchmarking Report

My board Similar boards All boards Complaints per investigator

117 95 112

Rate of complaints resolved

.78 .66 .66

Process Benchmarking

The comparison of practices, procedures and performance answering the question of What the best practice is in this topic and how to adopt the best practice in my board

Identification of Best Practice

How to define best practice How to identify best practice

Level I – Properly conducted RCT, systematic review or meta-analysis

Level II –Quasi-experimental, correlational, descriptive, survey, evaluation, and qualitative

Level III –Expert opinion or consensus statements

Levels of Evidence

From Nancy Spector, who adapted from Polit and Hungler (2004) & Gallagher (2003)

Identifying best practice

Choose a process to study: •One process at a time•If desire to study more than one process, it is best to set up separate studies for each process.

Identification of best practice

– With clear research questions– With clearly defined conceptual model

For example

What is the best practice for a discipline process?

A Conceptual Model

Board Practice Authority of staff Communication with PMCs

& CNAs Actively Managing

disciplinary process Mentoring investigative staff Consistent discipline

sanctions

Desired outcomes

Cost effectiveness

Perceived fairness and timeliness of the discipline process

Perceived quality of the board’s disciplinary process

Perceived effectiveness of board’s disciplinary process

Operationalize Variables

Authority of staff• Staff autonomy in resolving selected discipline cases

• Staff/Board relationship

Communication with PMCs and CNAs• Board communication to PMCs and CNAs before initiation

of investigative process and after the completion of the disciplinary process

• Board communication to PMCs and CNAs on rationale for the board action

• Availability of real person for questions

• Attitude of board toward CNAs

Operationalize Variables

Actively Managing disciplinary processa)Authority to hire or directly contract with the

investigators and attorneys involved with the disciplinary process.

b) Investigators being an RNc) Consistent process (including case

management and resolution)d) Investigators involved in the entire

disciplinary process

Operationalize Variables

Mentoring investigative staff Training for investigators that may or may

not include CLEAR training. Training materials available to novice

investigators. Mechanism of mentoring investigators

Operationalize Variables

Consistent discipline sanctions Enforcement processes are reviewed on a regular

basis to ensure consistency of discipline sanctions. The board develops policies and procedures to

ensure consistent handling of complaints. Formal mechanics are in place to review proposed

sanctions for consistency with previous action taken for similar cases.

Possible Areas of Interest

For example: What is the best time to renew a license? Are there any difference in outcomes

between using a RN as investigator vs. an non-RN investigator?

Performance Benchmarking

ChooseMeasuresChoose

MeasuresCollectData

CollectData

AnalyzeData

AnalyzeData

ProduceReport

ProduceReport

ReportAnnuallyReport

Annually

Process Benchmarking

ChooseProcessesChoose

ProcessesFormTeamFormTeam

DevelopBaselineDevelopBaseline

CompareProcessesCompare

ProcessesPlan forChangePlan forChange

Manage forChange

Manage forChange

Asked A Lot of Questions

? ? ????

????

? ?? ? ?

????

? ? ??? ?

??

?

?

Relationships

Timeliness of handling complaints # of FTEs or staff workload

Example:

More PrecisionQ: Were you treated fairly?

ORConsider the matters under review without any existing prejudice towards the people or organizations involved ( the decision-maker should act in a manner that is unbiased, fair and open-minded)

Communicating to the parties involved what is at issue, what decisions might be made adversely to the parties involved and what evidence and information the board may use in reaching its decision (the person should be informed of the matter)

Offering a procedure by which the parties involved may test, challenge or contradict the information and evidence used by the board in reaching its decision

Offering a procedure by which the evidence, information and submissions from the parties involved can be submitted to the board (opportunity to make oral or written submissions to the board)

The reasons for the board’s decisions must be based on the facts of the case under consideration  The person should be told the reasons for the decision There is consistency between previous actions and proposed sanctions for similar cases

Next Steps

Identify and remove barriers to participation

Identify and remove barriers to participation

Performance Measures

Support member boards’ adaptations of best practices

Best Practices Identified by Focus

Groups

OR Committee Identified Best

Practice