msw evaluation report: 2011-2012 submitted by: w. jay ...€¦ · contained in this report is ......

1

MSW Evaluation Report: 2011-2012 Submitted by: W. Jay Gabbard, MSW, Ph.D. To: Dr. Dean May, Department Head-Social Work-Western Kentucky University Dr. Amy Cappiccie, MSW Program Director Date: 7/7/12 Contained in this report is a summary of evaluation data gleaned from the MSW program for the academic year 2011-2012. This information can be used to supplement departmentaland college reports, as well as for CSWE Accreditation purposes. ACAT In order to assess the mastery of foundation level knowledge of MSW students at WKU, the ACAT measures standard curricular areas within accredited graduate social work programs. Curriculum A, the one administered to “traditional” WKU MSW students, assesses knowledge in the following curricular areas:

Human Behavior in the Social Environment Social Policy Social Work Practice Research Methods Diversity Populations at Risk Social and Economic Justice Values and Ethics

Results from the ACAT are provided to individual students in the form of standard scores for each curricular area as well as for overall performance for the cohort taking the test. These curricular areas and the standard scores associated with them can be linked directly to program objectives. The ACAT specifically assesses concepts from the following Foundation level objectives: 1.1.1f, 1.1.2f, 1.1.3f, 1.1.4f, 2.1.6f, 2.1.7f, 4.1.10f, 4.1.11f Results-2 Year Traditional Cohort (Class of 2012) The ACAT is normally administered at the end of traditional students first year of coursework. However, due to a delay, 24 students in the 2 Year Traditional Program (Class of 2012) did not take their ACAT exam until the beginning of Fall Semester 2011 (their second year of classes). Here are the results of this administration:

2

Area Standard Score %ile Reference Group Size Diversity 508 53 7941 Populations at Risk 508 53 7941 Social/Economic Justice 503 51 7941 Values and Ethics 530 62 7941 Policy & Services 531 62 8240 Social Work Practice 547 68 8240 HBSE 516 56 8240 Research Methods 555 71 7941 OVERALL PERFORMANCE 536 64 7941 The results of this administration indicated that overall, this cohort of traditional MSW students performed at the 64th percentile compared to similar MSW students nationwide. This was one of the highest overall percentiles achieved during the time that the ACAT has been administered to traditional MSW students at WKU. Students scored highest in the areas of research and social work practice. They also performed well in the areas of values and ethics and policy and services. The weakest scores were in the domains of social/economic justice, diversity, and populations at risk. The results here were a bit puzzling, as there is a flagship diversity course (SWRK 501) that all students take their first semester and content on diversity and populations at risk is infused throughout the foundation year. However, these results will be reviewed (see below) to see if additional diversity content needs to be added to the curriculum. At the next MSW program meeting in Fall 2012, the results of this administration will be discussed with faculty and a strategy devised to maintain strengths and address deficiencies in the aforementioned areas. As well, ACAT results will be shared at the next Departmental Advisory Committee (DAC) meeting during the 2011-2012 academic year, in order to solicit feedback from community members and stakeholders. Results were also provided to students, in order that they might improve in areas where they were deficient.

3

Results-2 Year Traditional Cohort (Class of 2013) The traditional MSW class of 2013 (24 students) took the ACAT at the end of their first year of classes in Spring 2012. Here are the results of this administration (individual and aggregate results have already been sent to students): Area Standard Score %ile Reference Group Size Diversity 532 63 8293 Populations at Risk 513 55 8293 Social/Economic Justice 497 49 8293 Values and Ethics 503 51 8293 Policy & Services 505 52 8740 Social Work Practice 553 70 8740 HBSE 541 66 8740 Research Methods 545 67 8740 OVERALL PERFORMANCE 534 63 8293 The results of this administration indicated that overall, this cohort of traditional MSW students performed at the 63th percentile compared to similar MSW students nationwide. Thus, the overall scores from this cohort remained relatively stable when compared to the prior cohort, who scored at the 64th percentile. Similar to the prior cohort, students scored highest in the areas of social work practice and research methods. The areas of greatest improvement (at least in terms of relative percentile ranks) from the prior administration were in the areas of diversity and HBSE, with both areas up 10 percentile points. The weakest scores were in the domains of social/economic justice (the lowest from the last administration), values and ethics, and policy and services. It appears that social/economic justice needs to be targeted and discussed during the next MSW program meeting in Fall 2012, as it has been the lowest scoring percentile on the last two tests. As well, these ACAT results will be shared at the next Departmental Advisory Committee (DAC) meeting during the 2011-2012 academic year, in order to solicit feedback from community members and stakeholders.

4

Cultural Awareness Inventory One of the primary goals of the MSW program at WKU is that graduates emerge with the knowledge, skills, and self-awareness required for culturally sensitive practice. Towards this end, both advanced standing and traditional students take a flagship diversity course (SWRK 612 and 501 respectively) and diversity content is infused throughout the curriculum. In order to assess student’s cultural awareness, the MSW program at WKU utilizes an instrument that has been employed in numerous evaluative studies nationwide. The Cultural Awareness Inventory is adapted from Lum’s Social Work Cultural Competencies Self-Assessment (2003). The Cultural Awareness Inventory consists of a number of sections, including an introduction, background information, 44 statements that are answered on a 4-point scale, and two open-ended items at the conclusion of the instrument. The majority of the instrument is made up of the 44 self-reported statements that encompass the cultural domains of: cultural awareness, knowledge acquisition, skill development, and inductive learning. Students respond to each statement based on the following directions:

Rate yourself on your level of competency on a scale of 1 – 4: 1 = Unlikely 2 = Not very likely 3 = Likely 4 = Definitely Circle the appropriate number.

Again, given the MSW program’s mission “To educate and prepare students for professional social

work practice to meet the needs of increasingly diverse rural populations,” the Cultural Awareness Inventory is used to assess a critical component of this aspect of the mission statement.

Cultural Awareness Inventory-How the Instrument and Data are Used: The Cultural Awareness Inventory is employed as a pre-and post-test assessment procedure, statistically evaluated with dependent t-tests. It is administered at the beginning and conclusion of the student’s MSW program. The goal is to assess for significant changes (or lack thereof) in students’ perceived level of cultural competency. While self-perceived change does have its obvious limitations (diversity content is also measured more concretely on the ACAT), it does allow for students to reflect back on the knowledge and skills they have acquired in courses and in fieldwork, which is a valuable exercise for them to engage in at the conclusion of their study. Results-Class of 2012 (Advanced Standing and Traditional) Below are the t-test results of this inventory for both advanced standing and traditional graduates in 2012. There were also a number of students who were not present for the post-test the day it was administered who are not reflected in the data below.

5

Results (2012 Graduates)

Paired Samples Statistics

Mean N Std. Deviation Std. Error Mean

Pair 1 pre cc score 139.90 42 15.540 2.398

post cc score 158.79 42 13.530 2.088

Paired Samples Correlations

N Correlation Sig.

Pair 1 pre cc score & post cc score 42 .142 .368

Paired Samples Test

Paired Differences

t df

Sig. (2-

tailed) Mean

Std.

Deviation

Std. Error

Mean

95% Confidence Interval

of the Difference

Lower Upper

Pair

1

pre cc score - post

cc score

-18.881 19.097 2.947 -24.832 -12.930 -6.407 41 .000

6

Here is a visual display of these results on a scatterplot (notice the upward significant trend):

Looking at the results of the dependent t-test for all 2012 graduates who completed both the pre

and post ratings, there was a significant (at the .001 level) increase in all MSW students’ cultural

competency scores from when they started the program to the end of their course of study. This

trend is further confirmed by the positive upward trend displayed on the scatterplot above.

The average increase of 19 points on the inventory provides evidence that students (from their

perspective) became more sensitive to culturally diverse groups and more skilled in providing

culturally sensitive interventions in practice during their time in the MSW program at WKU.

This is an important finding, as one of the primary goals of the social work profession and MSW

7

program at WKU is to produce graduates who are highly skilled in assessing and intervening

with diverse client populations.

Next, here are the results of the dependent t-test for the 29 advanced standing (e.g. one year)

students in the graduating 2012 cohort who completed the pre and post assessment:



Pair 1 pre cc score 141.72 29 15.659 2.908

post cc score 160.24 29 11.792 2.190


N Correlation Sig.

Pair 1 pre cc score & post cc score 29 .411 .027

Paired Samples Test

Paired Differences

t df

Sig. (2-

tailed) Mean

Std.

Deviation

Std. Error

Mean


of the Difference

Lower Upper

Pair

1

pre cc score - post

cc score

-18.517 15.242 2.830 -24.315 -12.719 -6.542 28 .000

The results of the advanced standing cohort’s dependent t-test also indicated a significant (at the

.01 level) increase in self-reported cultural competency from orientation to graduation. The

average increase for this cohort (19 points) was similar to that of the overall gain for the

combined graduating class discussed above. Also, the correlation between the scores was

statistically significant at the .05 level (p = .03), unlike that of the overall graduating class.

In sum, the significant increase in cultural competency for advanced standing students supports

the program and professional goals of producing graduates who are sensitive to the needs of their

diverse clients and skilled in providing culturally sensitive interventions. The infusion of

diversity content throughout the curriculum and flagship diversity courses contribute to students’

continual growth in these areas while in the program.

8

Finally, here are the dependent t-test results for the students in the 2 year traditional cohort that

graduated in 2012 (pretest was obtained at orientation in 2010). Please note that there were a

number of students who did not complete the post-test or who left the program (or who shifted to

part-time status), which contributed to the lack of post-assessment data with a number of

students in this cohort.



Pair 1 pre cc score 135.85 13 15.071 4.180

post cc score 155.54 13 16.865 4.678


N Correlation Sig.

Pair 1 pre cc score & post cc score 13 -.376 .205

Paired Samples Test

Paired Differences

t df

Sig. (2-

tailed) Mean

Std.

Deviation

Std. Error

Mean


of the Difference

Lower Upper

Pair

1

pre cc score - post

cc score

-19.692 26.512 7.353 -35.713 -3.671 -2.678 12 .020

In terms of the traditional two year graduates, they manifested a slightly larger average increase

in cultural competency scores than their advanced standing colleagues and the combined class of

2012 with an average increase of 20 points in their two years in the program. This increase was

significant at the .05 level (p = .02).

Thus, students in both graduating cohorts and the overall group indicated a statistically

significant (at the .05 level) increase in cultural competency during their tenure at WKU. These

results indicated success in reaching a stated goal of the MSW program, with regards to

cultural competency and diversity:

“More specifically, dependent t-tests (students put the last few digits of their student id numbers

on the instruments so pre-post results can be paired) after graduation each year. An increase in

overall positive trend, confirmed by a p value of less than benchmark standard of .05 (traditional

in social science research) is the rejection level for statistically significant findings and an

indication of success in this area.”

9

Foundation and Concentration Assessments

The MSW program faculty and administration are interested in knowing students’ assessments of

their educational outcomes before and after each program sequence. At the beginning and end of

the foundation curriculum for students in the traditional two-year program, and orientation and

end of the MSW program for advanced standing students, assessments are administered to

monitor students’ performance based on a self-efficacy report. On these assessments, students

rank their learning outcomes in relation to all 12 foundation performance indicator. An important

feature (and limitation) of this instrument is that it is self-reported.

Upon beginning and completing the concentration sequence, all students are administered a

concentration assessment to evaluate their performance, based on a self-efficacy report. Students

rank their learning outcomes in relation to all 13 concentration performance indicators (specific

indicators are at the beginning of the document and in the instrument section of Standard 8). An

important feature (and limitation) of this instrument is that it is self-reported.

Below is the pre-test data for the 2011-2012 traditional program cohort (e.g. graduating class of

2013), with regards to the 12 performance indicators identified above:

Descriptive Statistics-Foundation Year Pre

N Range Minimum Maximum Mean Std. Deviation Variance

utilizes critical thinking 13 3 2 5 3.15 .987 .974

applies generalist

perspective

13 4 1 5 2.85 .987 .974

synthesizes theoretical

frameworks

13 4 1 5 2.92 1.188 1.410

utilizes communication skills 13 3 2 5 3.54 .776 .603

demonstrates

professionalism

13 3 2 5 3.69 .855 .731

understands values and

ethics

11 3 2 5 3.64 .924 .855

identifies value conflicts 11 3 2 5 3.45 1.128 1.273

critiques research 11 4 1 5 2.64 1.027 1.055

integrates research

techniques

11 4 1 5 2.91 1.044 1.091

interprets historical events 11 3 2 5 3.18 .982 .964

analyzes policies 11 4 1 5 3.00 1.095 1.200

applies strategies and

interventions

11 3 2 5 3.27 .905 .818

pre score 13 45 15 60 35.46 11.900 141.603

Valid N (listwise) 11

10

Looking at the descriptive statistics above, the average score for all 12 indicators was a 36 (out

of 60 total points), with a standard deviation of 11.90 and variance of 141.60. Students in this

cohort rated themselves highest on the foundation pretest in the areas of demonstrating

professionalism, understanding values and ethics, and utilizing communication skills. They self-

rated lowest in the areas of critiquing and integrating research. It is not surprising that entering 2

year MSW students would self-rate lower in these areas, as research is an area that often yields

the most apprehension for entering graduate students.

Here were the post-results on the 12 foundation indicators for this same group of students:

Descriptive Statistics-Foundation Year Post


utilizes critical thinking 19 2 3 5 3.79 .535 .287

applies generalist

perspective

19 2 2 4 3.47 .612 .374

synthesizes theoretical

frameworks

19 3 2 5 3.37 .684 .468

utilizes communication skills 19 2 3 5 4.00 .577 .333

demonstrates

professionalism

19 3 2 5 4.21 .918 .842

understands values and

ethics

19 1 4 5 4.53 .513 .263

identifies value conflicts 19 2 3 5 4.16 .688 .474

critiques research 19 3 2 5 3.63 .831 .690

integrates research

techniques

19 2 2 4 3.58 .692 .480

interprets historical events 19 3 1 4 3.32 .885 .784

analyzes policies 19 4 1 5 3.42 1.071 1.146

applies strategies and

interventions

19 2 3 5 3.68 .671 .450

pre score 19 13 39 52 45.16 4.004 16.029


Note: A dependent t-test comparison was not possible with this data due to the lack of

coding/id numbers on the post-test. After the traditional MSW cohort’s first year of classes, the

average post score was 45 (out of 60 points), a 9 point overall mean increase from their pre-test

average score at orientation. This was indicative of increased perceptions of competence across a

broad spectrum of foundation indicators.

Once again, the domains of values/ethics and professionalism were the areas where students

reported being the most competent. The areas of critiquing and integrating research techniques

were no longer the lowest (possibly due to the completion of introductory research methods) and

11

the new lowest areas were interpreting historical events and understanding theoretical

frameworks.

The area of greatest perceived growth in the foundation year was in students’ ability to critique

research (plus .99) and that least perceived growth was in their ability to interpret historical

events (plus .14). Also, there was a notable increase (plus .89) in students’ ability to understand

values and ethics within the context of the social work profession.

Here is the standard for evaluating these foundation and concentration level objectives in the

MSW program:

Each objective is evaluated on a five-point continuum:

1 (59-below, F) Severely Lacking Competency

2 (60-69, D) Somewhat Inadequate Competency

3 (70-79, C) Adequate Competency

4 (80-89, B) More than Adequate Competency

5 (90-100, A) Outstanding Competence

On all of the post-test objectives, students reported they achieved adequate competency (and in

four domains above average competency). This was indicative of a positive upward trend in

student’s perceptions of their competency in key foundation level areas. These findings were, of

course, tempered by the fact that the data was self-reported.

Shifting to concentration objectives, below are the pretest results across the 13 various

program indicators for the Advanced Standing Class of 2012 (only advanced standing is included

here as no pretest concentration objectives data for traditional students in their second year was

gathered):

12

Descriptive Statistics-Concentration Year Pre (Advanced Standing)


generalize foundation

knowledge

28 3 2 5 3.46 .693 .480

integrates specialized

knowledge

28 3 2 5 3.54 .693 .480

implements community

organization

28 4 1 5 3.21 .833 .693

demonstrates effective

leadership

28 3 2 5 4.00 .720 .519

demonstrates professional

use of self

28 3 2 5 3.93 .766 .587

integrates understanding

rurality

28 3 2 5 3.79 .833 .693

manages value and ethical

conflicts

27 2 3 5 3.81 .681 .464

evaluates compliance cc

standards

27 2 3 5 3.93 .675 .456

evaluates compliance code

ethics

27 2 3 5 4.15 .602 .362

research competencies

rural

27 3 2 5 3.15 .818 .670

analyzes practice research 27 3 2 5 3.19 .786 .618

performs policy rural setting 27 3 2 5 3.04 .706 .499

applies admin skills 27 3 2 5 3.44 .892 .795

pre score 28 35 23 58 45.86 7.658 58.646


Looking at the results of the pretest scores for the 2011-2012 Advanced Standing cohort, they

rated out an average competency score of 45.86 (out of 65 points), with a standard deviation of

7.66 and a variance of 58.65. Similar to foundation year students, these students rated research

competencies and analysis (as well as policy) as the lowest areas of perceived competency. This

was not a surprise, as graduate social work students traditionally have the most apprehension in

these areas before beginning coursework. This is also after having had BSW courses in these

areas.

The highest ratings were in (similar to foundation year students) ethics, professionalism, and

leadership. In all of the areas surveyed, respondents rated themselves as having “adequate”

competency (e.g. greater than a score of 3) before commencing their MSW studies.

13

Below are the post scores on concentration level objectives for this same Advanced Standing

cohort:

Descriptive Statistics-Concentration Year Post (Advanced Standing)



knowledge

30 2 3 5 4.33 .711 .506


knowledge

30 3 2 5 4.30 .750 .562


organization

30 3 2 5 3.77 .898 .806


leadership

29 2 3 5 4.34 .670 .448


use of self

30 2 3 5 4.57 .679 .461


rurality

30 2 3 5 4.57 .626 .392


conflicts

29 2 3 5 4.62 .622 .387


standards

29 2 3 5 4.72 .591 .350


ethics

29 2 3 5 4.72 .591 .350


rural

29 3 2 5 4.24 .830 .690




pre score 30 41 24 65 55.53 9.358 87.568


Looking at the data, similar to foundation level students, there was a marked overall increase in

perceived competency on concentration level objectives, increasing from an average of 45.86 on

the pretest to 55.53 on the posttest, for a total increase of 9.67 points.

For advanced standing students, the domains of greatest perceived growth were in that of

research competency (plus 1.09), analyzing research (plus 1.02), and performing policy in a rural

setting (plus 1.03). These were areas where students rated fairly low on the pretest, so this

development indicated an increased perception of mastery of two areas that these students

perceived themselves as weak in at the beginning of their studies.

14

In terms of areas of least perceived growth, leadership was the first (plus .34) but this was also

the highest domain rated on the pretest. Again, notwithstanding the limitations of self-reported

data, students in this cohort reported being more skilled and competent in all 13 domains

assessed by the instrument and “more than adequate competency” in most domains. The highest

reported areas of competency were in ethics and the ability to evaluate cultural competency

standards, while the lowest in the advanced standing cohort was in the ability to implement

community organization strategies.

In terms of traditional students completing their second concentration year of the MSW program,

because no pre-test of concentration level objectives was obtained at the beginning of the 2011-

2012 academic year, only posttest data was obtained for this cohort during their final year. This

data is displayed below

Descriptive Statistics-Concentration Year Post (Traditional)



knowledge

15 2 3 5 4.20 .561 .314


knowledge

15 2 3 5 4.20 .676 .457


organization

15 2 3 5 4.00 .655 .429


leadership

15 2 3 5 4.27 .594 .352


use of self

15 2 3 5 4.60 .632 .400


rurality

15 1 4 5 4.60 .507 .257


conflicts

15 2 3 5 4.20 .676 .457


standards

15 1 4 5 4.73 .458 .210


ethics

15 1 4 5 4.67 .488 .238


rural

15 2 3 5 4.13 .743 .552




pre score 15 21 44 65 55.07 5.587 31.210


15

Looking at the posttest results, the average perception of competency for traditional students

completing their two-year program was 55.07 (compared to an average of 55.53 for advanced

standing students). Their highest rated domains were (similar to other cohorts at both the

foundation and concentration levels) in the areas of ethics and compliance with cultural

competency standards. Their lowest perceived areas of competency were in the areas of research,

policy, and community organization, similar to the results of the advanced standing cohort. On

all domains, students self-rated as performing at “more than adequate competency.”

msw evaluation report: 2011-2012 submitted by: w. jay ...€¦ · contained in this report is ......

Documents