substantially equivalent competency assessment ... project - sec... · substantially equivalent...

Substantially Equivalent Competency Assessment

Competencies in Relation to Internationally

Educated Nurses’ Applications for Registration

February 2016

1

Learning From Experience Project

SEC Assessment Data Analysis Report

Executive Summary

The employment of internationally educated nurses (IENs) is a widely-used strategy to

address the global nursing shortage. As a regulated profession, all registered nurses

(RNs) are required to obtain licensure/registration before they can practice, with all

Alberta RN applicants required to provide evidence of competence to practice in the

profession. Yet differences in nursing education programs, health care systems, and

nursing scopes of practice between countries can cause challenges in assessing an

IEN’s nursing competencies.

The Learning from Experience: Improving the Process of Internationally Educated

Nurses’ Applications for Registration (LFE) project sought to use evidence to increase

the efficiency of the IEN application for registration process, and included a systematic

analysis of IEN application data which led to adoptions of policy and practice revisions to

support a better assessment process.

The Substantially Equivalent Competency (SEC) Assessment is a competency-based

assessment designed to evaluate nursing knowledge, skills, attributes, values and

judgments to determine if an IEN possesses substantially equivalent competencies to an

Alberta RN. This statistical analysis linked SEC Assessment data and LFE registration

process data to examine the SEC Assessment competencies in relation to each other, to

application characteristics, and to subsequent outcomes in the IEN application for

registration process. The project team used a variety of statistical analysis methods,

including statistical modeling, to investigate research questions.

In analyzing the strongest and most challenging competency categories, both

Aggregated Competency Mean Scores and statistical modeling identified Professional

Responsibility and Accountability competencies as being strongest amongst the study

population, followed by Ethical Practice, Self-Regulation, Service to the Public, and

Knowledge-Based Practice: Specialized Body of Knowledge. As well, there was

statistically significant evidence to show that all of the applicants had the most

challenges in the Knowledge-Based Practice: Competent Application of Knowledge

competency category.

Applicants had similar SEC Assessment results regardless of their age, gender, basic

education credential, number of countries of experience or practice currency. However,

applicants educated in a country with a similar scope of nursing practice to Canada and

those with a Canadian licensed practical nurse registration had higher mean scores on

the SEC Assessment than applicants who did not have those characteristics.

The trend across competency categories was also generally the same for all of the

education country groups represented in the study population with applicants

demonstrating stronger competencies in Ethical Practice, Professional Responsibility

and Accountability and Self-Regulation, and more challenges in the Service to the

Public, Knowledge-Based Practice: Specialized Body of Knowledge and Knowledge-

Based Practice: Competent Application of Knowledge competency categories. Across all

2



of the competency categories applicants educated in the United Kingdom and United

States education country groups had better odds of meeting competencies whereas

applicants educated in the India and Asia education country groups had the lowest odds

of meeting competencies.

In relation to courses assigned, applicants who were referred to each type of course had

more challenges meeting all of the competencies than the applicants who were not

referred to that type of course. The exception was for Nursing Skills courses where there

was no discernable difference between the results of applicants who were referred to

Nursing Skills courses compared to applicants who were not referred. This may be

attributable to the educational requirement to complete Nursing Skills courses as

prerequisites to Clinical Practice courses.

Applicants who passed the national entry-to-practice exam had higher competency

scores across all of the categories than applicants who failed the exam and statistical

modeling confirmed that applicants with high SEC Assessment competency scores were

more likely to pass.

Conversely, ineligible applicants displayed noticeable lower scores across all of the

competency categories compared to applicants who obtained an initial RN registration.

The results of this analysis will be used to support evidence-informed policies at CARNA

surrounding the assessment based on the SEC Assessment report and application

documentation. The findings also have the potential to inform policy and practice

adaptations at SEC Assessment Centres, educational institutions, and other regulatory

bodies.

3



Contents

1.0 Introduction ............................................................................................................ 4

1.1 Background ................................................................................................... 4

1.2 Project Purpose and Objectives ..................................................................... 5

1.3 Research Questions ...................................................................................... 5

2.0 Methods ................................................................................................................. 5

2.1 Data Collection and Variables ........................................................................ 5

2.1.1 SEC Assessment Data ......................................................................... 5

2.1.2 IEN Application Data ............................................................................ 6

2.1.3 Data Cleaning and Linkage .................................................................. 7

2.2 Data Analysis................................................................................................. 7

2.2.1 SEC Assessment Competency Analysis ............................................. 7

2.2.2 IEN Application Characteristics and Outcomes Data Analysis ............. 8

2.2.3 SEC Assessment Competencies and Outcomes Data Analysis .......... 9

3.0 Results ................................................................................................................. 10

3.1 Applicant Demographics .............................................................................. 10

3.2 Strongest and Most Challenging SEC Assessment Competencies .............. 11

3.3 SEC Assessment Competencies in Relation to Application Characteristics . 14

3.4 SEC Assessment Competencies in Relation to Outcomes........................... 16

3.4.1 SEC Assessment Competencies in Relation to Courses ................... 16

3.4.2 SEC Assessment Competencies in Relation to Exam Results .......... 17

3.4.3 SEC Assessment Competencies in Relation to Ineligibility ................ 18

4.0 Discussion ............................................................................................................ 19

4.1 Summary ..................................................................................................... 19

4.2 Limitations ................................................................................................... 20

4.3 Moving Forward ........................................................................................... 21

Appendix A: Data Analysis Framework.......................................................................... 22

Appendix B: Research Team and Acknowledgments .................................................... 23

4



1.0 Introduction

1.1 Background

The employment of internationally educated nurses (IENs) is a widely-used strategy to

address the global nursing shortage1. As a regulated profession, all registered nurses

(RNs) are required to obtain licensure before they can practice, with all Alberta RN

applicants required to provide “evidence of competence in the practice of the

profession”2. Yet differences in nursing education programs, health systems, and nursing

scopes of practice between countries can cause challenges in assessing an IEN’s

nursing competencies.

Upon receiving an application for registration from an IEN, the College and Association

of Registered Nurses of Alberta (CARNA) assesses the applicant’s education,

experience and other qualification documentation to determine if the applicant’s

professional knowledge, skills, attributes, values and judgments (competencies) can be

deemed substantially equivalent to the competencies required to fulfill the roles and

responsibilities expected of RNs in Alberta. If a determination cannot be made based on

the application documents, an applicant may be referred to a Substantially Equivalent

Competency (SEC) Assessment.

The SEC Assessment uses a variety of strategies to assess RN competencies including:

multiple choice and short answer exams, interview-based case management situations,

clinical lab role play situations and self-assessment. It identifies whether the applicant’s

competencies are substantially equivalent to an entry-level RN in Alberta or if any

competency gaps exist and to what extent. The resulting SEC Assessment report

informs CARNA as to whether the applicant can be deemed substantially equivalent, if

he or she needs to complete bridging education to address competency gaps or if the

gaps are too extensive and the applicant is not eligible to practice in Alberta.

The Learning from Experience: Improving the Process of Internationally Educated

Nurses’ Applications for Registration (LFE) project sought to use evidence to increase

the efficiency of the IEN application for registration process. One component of the

project analyzed SEC Assessment competency data linked with data from the analysis

of application variables to develop an understanding of the relationship between SEC

Assessment competencies, and application characteristics and outcomes.

1 Canadian Nurses Association (2009). Tested Solutions for Eliminating Canada’s

Registered Nurse Shortage. Canadian Nurses Association: Ottawa, ON. 2 Government of Alberta (2000). Health Professions Act. Government of Alberta:

Edmonton, AB. Section 28 (2).

5



The research project was led by the College and Association of Registered Nurses of

Alberta and funded by Alberta Health through Health Canada’s Internationally Educated

Health Professionals Initiative. The views expressed herein do not necessarily represent

the views of Health Canada or Alberta Health.

1.2 Project Purpose and Objectives

The LFE research project sought to improve the efficiency of the internationally educated

nurse application for registration process while upholding CARNA’s commitment to

public safety and to the principles of fairness and transparency. The LFE project strived

to:

Develop an evidence-based model for the assessment of IEN applications for

registration.

Make CARNA processes as transparent, simple and efficient as possible from

receipt of a complete application to eligibility for RN registration.

Clearly articulate requirements and expectations of applicants.

Build capacity and leadership in the area of nursing regulation.

The SEC Assessment data analysis sought to:

1. Provide an understanding of the SEC Assessment competencies in relation to

each other.

2. Clearly understand if and how the SEC Assessment competencies relate to IEN

application for registration process characteristics and outcomes.

1.3 Research Questions

The research questions for the SEC Assessment data analysis were as follows:

1. What competencies did applicants have the most challenges with?

2. What competencies were applicants the strongest in?

3. Are there any trends in competencies that applicants were strongest in or had the

most challenges in according to education country group?

4. How did the competencies relate to the courses assigned?

5. Were there competencies that were related to passing or failing the national

entry-to-practice exam?

6. Were there competencies that were related to applicants being deemed

ineligible?

2.0 Methods

2.1 Data Collection and Variables

2.1.1 SEC Assessment Data

SEC Assessment data was collected at Mount Royal University (MRU) as part of routine

reporting in the IEN application for registration process. The SEC Assessment data in

6



this study included competency scores from applicants who completed an SEC

Assessment between May 2008 and September 2012. MRU SEC Assessment Centre

staff retroactively entered the data manually into an electronic database, between April

and June 2013.

There are multiple types of SEC Assessments categorized as:

General – assessment of general nursing entry-to-practice competencies in six

competency categories:

o Professional Responsibility and Accountability (“Professional Responsibility”);

o Knowledge-Based Practice: Specialized Body of Knowledge (“Knowledge”);

o Knowledge-Based Practice: Competent Application of Knowledge

(“Application of Knowledge”);

o Ethical Practice;

o Service to the Public (“Service to Public”); and

o Self-Regulation.

Focused (Maternal/Newborn, Child Health, Mental Health) – assessment of

specialized nursing entry-to-practice competencies in the specified area of focus.

General plus Focused – general assessment with one or two focused areas.

Complete – general assessment with all three focused areas.

Individual competencies in each competency category are measured for all the types of

SEC Assessments, however many of the competencies permeate throughout the

different focused SEC Assessments and therefore are only recorded under the General

competency category when a “Complete” or “General plus Focused” assessment is

performed. Therefore, all the applicants in the data had general SEC Assessment

competency information, but not everyone had focused area assessment results.

2.1.2 IEN Application Data

The LFE Project baseline data included 3504 complete application files received from

IEN applicants between January 1, 2008 and December 31, 2011. Information was

collected as part of routine data collection during the application for registration process

and was reported within the CARNA operational database and/or paper applicant files.

Data available only from paper files was entered manually into the database and data

entry was audited to ensure accuracy.

Data from applicants meeting the following criteria were included in the LFE Project:

Internationally Educated Nurse (nursing entry-level education completed outside

of Canada).

Completed application (all application documents received) between January 1,

2008 and December 31, 2011.

Assessed under the Health Professions Act.

Applications that were active, assessed and lapsed (24 months of inactivity) after

the application was considered completed.

7



The following consideration was also applied:

Re-applications: The applicant’s most current data was used in place of the

original application data.

Each file was individually reviewed to ensure consistent data entry. The project team

entered and cleaned the data from the files from March to November 2012. IEN

application for registration outcome data for the applicants captured in the SEC

Assessment data analysis was updated in June 2015.

2.1.3 Data Cleaning and Linkage

The SEC Assessment data and LFE Project baseline data were linked based on unique

stakeholder numbers. Data from 1425 SEC Assessment reports were received from

MRU. After linkage, there were 1104 SEC Assessment reports identified for 1076

applicants in the LFE baseline data. Some applicants completed more than one SEC

Assessment. For applicants with multiples reports, only the most complete SEC

Assessment results were included for analysis therefore 28 SEC Assessment reports

were excluded from the linked data. Ultimately, data from 1076 SEC Assessment reports

were analyzed.

2.2 Data Analysis

SEC Assessment competency data was analyzed to identify areas of strengths and

challenges between competencies and competency categories. Demographic and

outcome statistics were also identified for the study sample. Modeling was conducted to

link the competency data and the LFE data, analyzing relationships between the

competencies, and characteristics and outcomes. Analyses were conducted using SAS

9.3. Appendix A outlines the data analysis framework for the SEC Assessment data

analysis.

2.2.1 SEC Assessment Competency Analysis

A three point rating scale comprised of “Met”, “Partially Met”, and “Not Met” ratings was

used for the SEC Assessment until January 2011. It was revised to a four point rating

scale to offer further delineation of the “Partially Met” rating category. This leveling of the

partially met category allowed for greater definition of the degree of a candidate’s

performance in the areas of knowledge, skills, attitude, and judgment requiring growth.

After February 2011, ratings used to describe candidate performance were categorized

according to “Met”, “High Partially Met”, “Low Partially Met” and “Not Met”.

In order to accommodate the differences between the 3-point scale and 4-point scale,

the SEC Assessment competency data was standardized or weighted to allow for a one

scale system as described in Table 2.2.1. It is important to note that “Partially Met”

(score 3) in the analysis scale was not necessarily better than “Low Partially Met” (score

2) or worse than “High Partially Met” (score 4). The analysis scale scores (5 point scale)

were used for statistical analysis.

8



Table 2.2.1: Rescaling of SEC Assessment Scales

Not Met Low

Partially Met

Partially Met

High Partially

Met Met

3 Point Scale 1 - 2 - 3

4 Point Scale 1 2 - 3 4

Analysis Scale 1 2 3 4 5

SEC Assessments are conducted in four major areas: General, Maternal-Newborn, Child

Health and Mental Health. The latter three areas are considered specialty areas while

General assessments focus on general nursing competencies. All of the applicants in

the study had General assessment competency data. Some applicants also had

competencies in specialty areas assessed. In the General assessment, there were 78

individual competencies which were assessed in the six aggregated competency

categories described in Section 2.1.1. In order to identify the most challenging and

strongest individual competencies, mean scores were used for analysis. Comparisons of

aggregated competencies were based on Cochran-Mantel-Haenszel tests and

proportional odds models.

Specialty assessments are often interconnected with the General assessment

competencies, with assessors focusing on specialty competencies that are typically not

demonstrated in the General assessment. Therefore specialty assessment data is

typically focused in the Knowledge and Application of Knowledge competency

categories. Since specialty area competency data is inconsistent, statistical models were

only built on General SEC Assessment results.

Means for all individual competencies were calculated and compared within each

aggregated competency category. The highest mean individual competency score

indicated the strongest individual competency in the category for all applicants. In

contrast, the lowest mean individual competency score indicated the most challenging

individual competency for all applicants.

Proportional odds models were used to estimate the odds ratios of aggregated

competency categories. The SEC Assessment competency scores were grouped into

three groups based on practice: scores 1 and 2 were grouped together as “Not Met”;

score 3 remained separate as “Partially Met” and scores 4 and 5 were grouped together

as “Met”. The consideration of the 5 scores as 3 groups reflected the typical practice of

CARNA registration assessors when they reviewed an IEN's SEC Assessment results.

2.2.2 IEN Application Characteristics and Outcomes Data Analysis

Descriptive statistics (frequency, proportion, mean, histograms) were generated for each

application characteristic and outcome variable with continuous variables evaluated and

further categorized to describe frequency and conduct cross tabulations. The variables

represented the applicants’ demographic, education, employment and external

registration information, as well as variables representing courses, process outcomes

9



and competency scores. The p-values based on variance from cross tabulation and chi-

square tests were used to assess significant associations between variables and

outcomes in the bivariate analyses. Only General SEC Assessment results were used

for the linked analysis.

Aggregated Competency Median and Average Scores were calculated for each of the

applicant characteristic variables based on the 78 individual competencies for each

applicant. Therefore, every applicant would have one overall score to represent all of

their SEC Assessment competencies. Descriptive statistics (mean, median, minimum,

maximum, 90th percentile) were generated for each variable.

Trends in competencies by education country group were modeled by using a random

intercept proportional odds model. Aggregated competency groups and education

country groups were used as covariates, and their interaction term was added to the

model.

2.2.3 SEC Assessment Competencies and Outcomes Data Analysis 2.2.3.1 SEC Assessment Competencies and Courses Data Analysis

Descriptive statistics (frequency, mean, histograms) were generated for each course

type by aggregated competency category. Spearman correlations between the

aggregated competency category and each course type were also produced to evaluate

the independently paired associations between the competency categories and courses.

2.2.3.2 SEC Assessment Competencies and National Entry-to-Practice Exam

Results Data Analysis

Cross tabulation between the exam results and the SEC Assessment competency

categories was conducted using chi-squared tests. In addition, the bivariate analysis

between the exam results and the mean score of aggregated SEC Assessment

competency categories was conducted.

Logistic regression models were used to explore the independent associations between

the exam results (pass or fail) and SEC Assessment multiple choice and short answer

percentage scores, and the mean scores of the SEC Assessment competencies within

each aggregated competency category. The model diagnostic was based on Hosmer-

Lemeshow tests and a Receiver Operating Characteristic curve was produced to assess

the predictive power of each variable.

2.2.3.3 SEC Assessment Competencies and Ineligibility Analysis

Aggregated Competency Mean Scores were calculated for applicants who were deemed

ineligible as compared to applicants who received an initial RN registration.

10



3.0 Results

Following a description of key applicant demographics, the data analysis results are

presented in response to each of the six research questions.

3.1 Applicant Demographics

Of the 1076 applicants included in the data analysis:

The mean age of applicants was 32 with 86.43% of applicants younger than age

40 when they applied.

82.16% of the applicants were female and 17.84% were male.

The majority of applicants were educated in the Philippines (61.52%) and India

(12.08%).

16.54% of applicants had a basic education credential similar to an Alberta

baccalaureate (3-4 years post-secondary generalist nursing education after 12

years of elementary and secondary education) and 82.99% were considered to

have a diploma or degree not similar to an Alberta baccalaureate.

10.50% were educated in an area where the scope of nursing practice is similar

to that of Canada.

15.06% were registered with a licensed practical nurse regulatory body in

Canada at the time of application.

30.48% of applicants had transition (migration) experience, working in additional

countries other than the one they were educated in prior to applying to CARNA.

Only 6.78% of the applicants did not have practice currency (graduation or 1125

hours of practice in the last 5 years) at the time of application.

The average amount of time since the applicants last practiced was 0.71 years.

Figure 3.1.1 outlines the applicants’ pathway through the IEN application for registration

process including the frequency of outcome decisions at each phase. Following the SEC

Assessment, the majority of applicants (66.17%) were referred to bridging education

while 30.02% were deemed ineligible. At the time of data extraction, 40.71% had been

deemed eligible to write the national entry-to-practice exam and work in Alberta with

temporary registration to fulfill the reference requirement (“TP Eligible”).

Subsequent to being deemed TP Eligible, 34.01% of applicants had written the national

entry-to practice exam and 27.88% of applicants had obtained an initial RN registration.

Over the course of the process 31.13% of the applicants had been deemed ineligible for

RN registration in Alberta, and 7.16% of the applicants had no forward movement in two

years and were lapsed.

11



Figure 3.1.1: SEC Assessment Data Analysis Population Flow Chart

3.2 Strongest and Most Challenging SEC Assessment

Competencies

Mean competency scores were calculated for each of the 78 general competencies with

the highest mean scores indicating the strongest competencies and the lowest mean

scores indicating the most challenging individual competencies. Therefore, research

questions one and two were addressed together.

Research Question 1: What competencies did applicants have the most

challenges with?

Research Question 2: What competencies were applicants the strongest in?

Table 3.2.1: Strongest and Most Challenging Individual Competencies in the General SEC Assessment

Category

Strongest Individual Competency

Most Challenging Individual Competency

Professional Responsibility

Accountable and responsible for own actions, decisions, including personal safety.

Demonstrates critical inquiry processes in relation to new knowledge and technologies.

12



Category

Strongest Individual Competency

Most Challenging Individual Competency

Knowledge Has knowledge of relational communication and understands that relational practice is the foundation for all nursing practice.

Understands the role of primary health care in health delivery systems (accessibility, health promotion, public participation, appropriate use of technology, intersectoral collaboration) and its significance for population health.

Application of Knowledge

Completes assessment in a timely manner.

Verifies when clients have essential information and skills.

Ethical Practice Understands ethical and legal considerations related to maintaining client confidentiality in all forms of communication.

Takes into consideration the spiritual and religious beliefs and practices of clients.

Service to Public Manages physical resources to provide effective and efficient care (equipment, supplies, medication, linen).

Participates and contributes to nursing and health care team development by: building partnership, knowing and supporting the full scope of practice of various team members, and using appropriate channels of communication.

Self-Regulation Understands the significance of the concept of fitness to practice in the context of individual self-regulation and public protection.

Distinguishes between the legislated scope of practice for the RN individual scope of practice based on own level of competence.

13



Figure 3.2.1: Score Distribution Among General Aggregated Competency Categories

Figure 3.2.1 shows the mean aggregated competency scores for all of the competency

categories. The Professional Responsibility category has the highest percentage of

“Met” scores and lowest number of “Not Met” scores, whereas the Application of

Knowledge category has similar percentages of “Met” and “Not Met” scores. Focusing on

the “Met” scores, the analysis of the aggregated mean competency scores would

indicate that the competency categories from strongest to most challenging are

Professional Responsibility, Ethical Practice, Self-Regulation, Service to Public,

Knowledge and Application of Knowledge. The trend in scores for all of the competency

categories is similar.

Proportional odds modeling also shows that the strongest aggregated competency

category for all the applicants was Professional Responsibility, followed by Ethical

Practice, Self-Regulation, Service to the Public, Knowledge and Application of

Knowledge. There was statistically significant evidence to show that all applicants had

more challenges in Knowledge and Application of Knowledge compared to other

competency categories. This aligns with the findings from the aggregated mean

competency scores.

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

45.00

1-Not Met 2-Low PartiallyMet

3-Partially Met 4-High PartiallyMet

5-Met

Pe

rce

nt

Score

Professional Responsibility Knowledge Application of KnowledgeEthical Practice Service to Public Self Regulation

14



3.3 SEC Assessment Competencies in Relation to Application Characteristics

Of the 1076 applicants, the Median Competency Mean Scores were similar amongst the

age groups, genders, basic education credential, number of countries of experience

other than country of education, and practice currency. However, applicants with a

Canadian LPN Registration had a higher Median Competency Mean Score than

applicants who did not report an LPN registration in Canada.

Research Question 3: Are there any trends in competencies that applicants were

strongest in or had the most challenges in according to education country group?

In analyzing applicants’ education countries, some countries had very few observations

therefore, in order to have more balanced data, countries were grouped into 13

Education Country Groups. The Research Team identified several countries where

larger numbers of applicants were educated (e.g. the Philippines, India, the United

States) and where there was interest in individual education country analysis. Other

countries were grouped by geographical region with the assumption that education

programs were similar in each region. Applicants who received their entry-level

education in the United Kingdom, the Middle East, Australia/New Zealand, and the

United States had high Median Competency Mean Scores compared to the other

education country groups. Applicants who were educated where the scope of nursing

practice is similar to Canada also had a higher Median Competency Mean Score than

applicants who were educated in other areas.

15



Figure 3.3.1: Aggregated Competency Mean Scores by Education Country Group

The trend across competency categories was similar for all of the education country

groups with higher Aggregated Competency Mean Scores in Ethical Practice,

Professional Responsibility and Self-Regulation, and lower scores in Service to Public,

Knowledge and Application of Knowledge. There were a few exceptions. The

Australia/New Zealand country group had high Service to Public scores, similar to those

of Ethical Practice and Self-Regulation. As well, the Eastern Europe, the

Tropics/Caribbean, Asia and India country groups had Self-Regulation scores lower than

their Service to Public scores. The study population included only two applicants

educated in the South America education country group therefore the country group was

not included in this analysis due to the small sample size.

The random intercept proportional odds model confirmed the trends identified by the

Aggregated Competency Mean Scores. Throughout each competency category, the

United Kingdom and United States had the highest Aggregated Competency Mean

Scores and highest odds ratios indicating that applicants educated in those country

groups had better odds of meeting all of the competencies. Applicants educated in the

India country group had the lowest mean scores and lowest odds ratios. The Asia

education country group also had low odds ratios in Knowledge, Professional

Responsibility, Application of Knowledge, and Self-Regulation.

16



3.4 SEC Assessment Competencies in Relation to Outcomes

3.4.1 SEC Assessment Competencies in Relation to Courses

Research Question 4: How did the competencies relate to the courses assigned?

Following an SEC Assessment, all of the types of courses, except for Nursing Skills and

Ethics, had more than half of applicants referred to that type of course. The highest

percentages of applicants were referred to Health Assessment (80.53%), Mental Health

(78.33%) and Professional Nursing in Canada (76.87%) courses.

For all the courses except Nursing Skills courses, applicants who were referred to the

courses had discernibly lower Aggregated Competency Mean Scores than applicants

who were not referred to the course. There were no individual competency categories

that were associated with referral to any of the courses except the Ethics courses, rather

referral to a particular course was related to lower Aggregated Competency Mean

Scores across all of the competency categories. The largest differences between scores

for competency categories were around 1.00-1.07 for most of the courses. Applicants

referred to Ethics courses had an Aggregated Competency Mean Score in the Ethical

Practice competency category that was 1.70 lower than applicants who were not

referred to Ethics courses.

There was no discernible difference between the Aggregated Competency Mean Scores

of applicants who were referred to Nursing Skills courses compared to applicants who

were not referred. This may be related to the educational requirement to complete

Nursing Skills courses as prerequisites to Clinical Practice courses. Assessors may

determine that the applicant needs a Clinical Practice course even if the applicant’s

nursing skills are satisfactory in which case they would be referred to both. This may

occur if a) the applicant can demonstrate competencies related to nursing skills but has

challenges with critical thinking competencies; b) the applicant has been out of practice

a number of years or is close to losing currency as Clinical Practice courses are

considered to meet the currency requirement, or c) the applicant has limited clinical

experience.

The Spearman Correlation indicated that applicants referred to Professional Nursing in

Canada, Health Alterations and Therapeutics, Clinical Practice, and Pediatrics courses

were all moderately negatively correlated to all of the competency categories. Thus the

better applicants did on the SEC Assessment; the less likely they were to be assigned to

take those four types of courses.

17



3.4.2 SEC Assessment Competencies in Relation to Exam Results

Research Question 5: Were there competencies that were related to passing or

failing the national entry-to-practice exam?

Figure 3.4.1: Aggregated Competency Mean Scores in Relation to Exam Results

Applicants who passed the national entry-to-practice exam had higher Aggregated

Competency Mean Scores than applicants who failed the exam across all competency

categories. The range in differences between mean scores shows the smallest

difference for Service to Public and the largest difference for Self-Regulation.

Logistic regression modeling showed that all if the competency categories have a

significant impact on the exam results. Applicants with strong SEC Assessment results

were more likely to pass the national entry-to-practice exam. The strongest predictive

competency categories for exam results are Ethical Practice, Knowledge and

Professional Responsibility; however the six competency categories are highly

correlated.

4.29

3.80 3.69

4.19 3.97

4.14

3.74

3.20 3.14

3.61 3.54 3.50

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

ProfessionalResponsibility

Knowledge Application ofKnowledge

Ethical Practice Service to Public Self-Regulation

Agg

rega

ted

Co

mp

ete

ncy

Me

an S

core

Competency Category

Pass (N=325) Fail (N=41)

18



3.4.3 SEC Assessment Competencies in Relation to Ineligibility

Research Question 6: Were there competencies that were related to applicants

being deemed ineligible?

Figure 3.4.2: Aggregated Competency Mean Scores for Ineligible Applicants and Applicants who Received an Initial RN Registration

Ineligible applicants had discernibly lower Aggregated Competency Mean Scores than

applicants who received their initial RN registration across all competency categories.

There were no individual competency categories related to ineligibility, rather ineligible

applicants had low scores across all competency categories. Ineligible applicants had

Aggregated Competency Mean Scores lower than 2.50 across all competency

categories whereas applicants who received an initial RN registration had Aggregated

Competency Mean Scores higher that 3.50 in all of the competency categories. The

range in differences between mean scores shows the smallest difference for Service to

Public and the largest difference for Self-Regulation.

2.37

1.93 1.79

2.44 2.26

2.15

4.27

3.78 3.67

4.17 3.96

4.13

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

ProfessionalResponsibility

Knowledge Application ofKnowledge

Ethical Practice Service toPublic

Self-Regulation

Agg

rega

ted

Co

mp

ete

ncy

Me

an S

core

Ineligible Applicants RN Registered Applicants

19



4.0 Discussion

4.1 Summary

To our knowledge, this is the first study to examine SEC Assessment competencies in

relation to the larger IEN application for registration process. Typically characteristics of

the IEN applicants are withheld from SEC Assessment assessors to prevent bias during

the assessment. As well, CARNA registration assessors make the decision on the SEC

Assessment review outcome based on the SEC Assessment report and the applicant’s

education and experience documentation. This study used data from the SEC

Assessment Centre and from CARNA’s applicant database to explore the relationship

between SEC Assessment competencies, and application characteristics and outcomes.

The analysis revealed that trends in the competency categories were similar, with

applicants demonstrating stronger competencies in Professional Responsibility, Ethical

Practice and Self-Regulation, and having the most challenges in demonstrating

Application of Knowledge competencies across all of the data analyses. Both the

Aggregated Competency Mean Scores and statistical modeling showed that the

competency categories from strongest to most challenging were Professional

Responsibility, Ethical Practice, Self-Regulation, Service to Public, Knowledge and

Application of Knowledge. There was statistically significant evidence to show that all

applicants had the most challenges in the Application of Knowledge competency

category. Difficulties demonstrating skills and abilities in the presence of an assessor

may contribute to the factors that cause challenges in this competency category.

Similar Median Competency Mean Scores were observed regardless of IEN applicant

demographic characteristics including age, gender, basic education credential, number

of countries of experience, and practice currency. However, those educated in a country

with a similar scope of nursing practice to Canada and those with a Canadian LPN

registration had higher Median Competency Mean Scores than those who were

educated elsewhere or who did not have a Canadian LPN registration.

The trend across competency categories was also generally the same for all of the

education country groups with higher Aggregated Competency Mean Scores in Ethical

Practice, Professional Responsibility and Self-Regulation, and lower scores in Service to

Public, Knowledge and Application of Knowledge. Across all of the competency

categories the United Kingdom and United States education country groups had the

highest Aggregated Competency Mean Scores and had better odds of meeting

competencies according to statistical modeling. This finding is not surprising considering

the similarities of nursing scopes of practice and health care systems between these

countries and Canada. In contrast, applicants educated in the India and Asia education

country groups had the lowest mean scores and lowest odds ratios for meeting

competencies.

20



In the analysis of the SEC Assessment competencies in relation to courses assigned,

there was no individual competency category related to referral to any of the courses,

rather referral to each course was related to lower Aggregated Competency Mean

Scores across all of the competency categories. For all courses except Nursing Skills,

applicants who were referred to each type of course had discernibly lower Aggregated

Competency Mean Scores than those who were not referred to that type of course. The

lack of detectable difference between the Aggregated Competency Mean Scores of

applicants who were referred to Nursing Skills courses compared to applicants who were

not referred may be attributable to the educational requirement to complete Nursing

Skills courses as prerequisites to Clinical Practice courses.

The Aggregated Competency Mean Scores were higher for applicants who passed the

national entry-to-practice exam than for applicants who failed the exam across all

competency categories. Logistic regression modeling confirmed that applicants with

strong SEC Assessment results were more likely to pass. The strongest predictive

competency categories for exam results were Ethical Practice, Knowledge and

Professional Responsibility. However the six competency categories are highly

correlated and modeling showed that all of the competency categories have a significant

impact on the exam results.

There was no individual competency category that was related to applicant ineligibility,

rather across all of the competency categories, ineligible applicants had Aggregated

Competency Mean Scores that were less than 2.50 whereas applicants who received

their initial RN registration had Aggregated Competency Mean Scores above 3.50.

4.2 Limitations

LFE Project data was collected as part of routine data collection during the IEN

application for registration process. As the data were extracted from an active database

which was not designed for research purposes, it was not entered and cleaned in

isolation and variations in the data extraction may exist. Following extraction from the

CARNA database, the Research Team, Data Entry Clerks and Data Analyst thoroughly

reviewed and cleaned the data used for this analysis.

In May 2013, CARNA transitioned to a new client management database with different

functionality and coding. This required the development of a new data query to extract

the project data. As the LFE Project data in this analysis spanned a time period which

included both databases, an alignment of data codes was required.

The timeframe for the SEC Assessment data analysis was different than the timeframe

for the LFE Project baseline data, therefore data linkage was complex with some of the

SEC Assessment records falling outside of the baseline data period. Data from 1425

SEC Assessment reports was received from MRU however, after linkage with the LFE

Project data the study population included data from 1076 applicants.

21



As well, due to the various education and employment experiences for each applicant,

not every applicant was referred to a complete SEC Assessment (both general and

specialty areas). Therefore data in specialty areas were only available for some

applicants.

The shift in the SEC Assessment scale from a 3 point scale to a 4 point scale in the

midst of the data set required both scales to be converted to a 5 point scale for analysis.

Therefore in order to interpret these findings in relation to either scale a conversion was

made.

4.3 Moving Forward

Findings from the SEC Assessment data analysis provide an enhanced understanding of

SEC Assessment competencies in relation to IEN application for registration

characteristics and process outcomes.

Knowledge surrounding strong and challenging competency areas may inform SEC

Assessment Centres in the development of assessment tools and approaches, and

guide IENs in preparing for the SEC Assessment. Additionally, findings surrounding

strong and challenging competency categories, and the course related analysis may

provide guidance to registration assessors in assigning courses, or to educators in the

development of bridging education curriculum. Registration assessors may also apply

findings related to the exam results and ineligibility to decision-making surrounding the

review of SEC Assessment competency scores.

Overall, the results of this analysis can be used to support evidence-informed policies at

CARNA to improve the decision-making processes surrounding the SEC Assessment.

The findings also have the potential to inform policy and practice adaptations at SEC

Assessment Centres, educational institutions offering bridging education and other

regulatory bodies.

22



Appendix A: Data Analysis Framework

23



Appendix B: Research Team and Acknowledgments

Working Group Jennifer Kwan, LFE Project Manager Mengzhe (Maggie) Wang, Data Analyst Winston Qui, Biostatistician Dr. Pam Nordstrom, Director, School of Nursing and Midwifery, Mount Royal University College and Association of Registered Nurses of Alberta Cathy Giblin, Registrar/Director, Quality Assurance Loreta Suyat, Deputy Registrar Nan Horne, RN Consultant Nancy MacPherson, RN Consultant Roxanne Ferrier, RN Consultant Barry Bissoondatt, Manager Operations, Registration Services Gillian Lemermeyer, RN Policy Associate Crystal Komanchuk, Communications Coordinator University of Alberta Dr. Greta Cummings, Professor, Faculty of Nursing

ACKNOWLEDGEMENTS

The LFE Research Team would like to acknowledge the dedicated work of the Data Entry Clerks in entering and cleaning IEN application data, and Jessica Schaub in the development of this report. We would like to thank the Mount Royal University SEC Assessment Centre for entering and providing the SEC Assessment competency data, as well as valuable information on the assessments. Additionally, we recognize the ongoing support of the CARNA Registration Services Department and Leadership Team. The Learning from Experience: Improving the Process of Internationally Educated Nurses’ Applications for Registration project was funded by Alberta Health through Health Canada’s Internationally Educated Health Professionals Initiative. The views expressed herein do not necessarily represent the views of Health Canada or Alberta Health.

substantially equivalent competency assessment ... project - sec... · substantially equivalent...

Documents