annual beap presentation

Annual BEAP PresentationBPD Annual ConferencePortland, ORMarch 2012

The BEAP TeamVicky BuchanColorado State Universityvictoria.buchan@colostate.edu

Brian ChristensonLewis-Clark State Collegeblchristenson@lcsc.eduhttp://www.lcsc.edu/bchristenson

Ruth Gerritsen-McKaneUniversity of Utahruth.gerritsen-mckane@socwk.utah.edu

Tobi DeLong HamiltonLewis-Clark State College, CDAtadelong-hamilton@lcsc.edu

Roy (Butch) RodenhiserBoise State UniversityRoyRodenhiser@boisestate.edu

Marshall Smith (Retired)docsmith@umich.edu

Phil NgBEAP@PHILNG.NET

What BEAP Can Do For Your Program:•Student demographics•Assessment on Knowledge, Skills, &

Values•Employment information•Advising feedback•Professional development of students &

alumni/ae•Peer comparison by region, program type,

auspice & nationally•…and more

Matrix ExampleCompetency Alumn

iExit Employer FCAI FPPAI

2.1.7—Apply knowledge of human behavior and the social environment.-

A. utilize conceptual frameworks to guide the processes of assessment, intervention, and evaluation;

B. critique and apply knowledge to understand person and environment.

A. C7-12

B. C12

A. D7-12

B. D12

A. Q7-10

B. None

A. 17, 19

B. 14,22,23,56,57

A. 39-40

B. 39-40

The BEAP Matrix Can Be Found in:DeLong-Hamilton, T., Buchan, V. V., Hull, G. H., Christenson, B., Gerritsen-McKane, R., Rodenhiser, R. W., & Smith, M. L. (2011). Responding to the 2008 EPAS: Baccalaureate education assessment direct and indirect measurement. Journal of Baccalaureate Social Work, 16(1), 17-31.

Sample ProfileOverview of all Respondents

Instrument

2000-2009 2010 2011 Overall

Entrance 31,107 1,374 2,432 34,913

Exit 19,768 623 1,710 22,101

Alumni 5,445 101 212 5,758

Employer 1,963 40 74 2,077

Totals 56,483 2,138 4,428 64,849

Frequencies for Auspices of College or University, 2011Instrument Public Private

Denominational

Private Non-denomination

al

Entrance 1,442 700 286

Exit 947 477 286

Alumni 66 68 78

Employer 36 30 8

Frequencies for Program Type, 2011Instrument BSW Only Combined

BSW/MSW

Entrance 520 144

Exit 1,228 482

Alumni 164 48

Employer 66 8

Entrance Survey – Student EthnicityEthnicity 2000-2010

%N = 32,481

2011 %N = 2,620

Overall %N =

35,101European / Caucasian 62.6 53.6 58.1

African American / Black

19.8 21.8 20.8

Native American 7.5 5.0 6.3

Other Hispanic / Latino

4.9 5.4 5.2

Other 3.8 4.2 4.0

Asian / Pacific Islander 3.3 4.3 3.8

Chicano / Mexican 3.8 3.9 3.9

Puerto Rican 2.3 1.8 2.1

Exit SurveyKnowledge, Values, Skills*2010 Likert Scale changed from 0-10 to 1-9

Mean Score

2000-2009

2010 2011N=1,655

Overall

Knowledge 7.34 7.15 7.35 7.28

Values 7.42 7.34 8.34 7.70

Skills 8.67 8.26 7.55 8.16

Exit SurveyAdvising Scores*2010 Likert Scale changed from 0-10 to 1-9

Question 2000-2009N=

19,768

2010N= 623

2011N= 1,644

OverallN=

22,035

Course Selection / Curriculum Planning

6.88 6.93 7.11 6.98

Professional Advising

6.59 6.39 6.59 6.52

Career Planning

6.17 5.85 6.16 6.06

Foundation Curriculum Assessment Instrument

Purpose of the FCAI1. Provides Pre/Post test in seven major

curricular areas of a BSW program2. Provides a direct measure to assist

programs with evaluation of their curriculum

3. Assists with identification of curricular areas that may need attention

4. Provides national comparative data

Curricular Components Curriculum Area Number of

Questions

Practice 13

Human Behavior & Social Environment 10

Policy 9

Research 9

Ethics and values 8

Diversity 8

Social and Economic Justice 7

Sample HBSE Question•The concept “person-in-environment”

includes which of the following:a. Clients are influenced by their

environmentb. Clients influence their environmentc. Behavior is understood in the context of

one’s environmentd. All of the above

Sample Practice Question•Determining progress toward goal

achievement is one facet of the _____ stage.

▫a. Engagement ▫b. Evaluation ▫c. Assessment ▫d. Planning

Testing History – 2006-2011 TEST #questions N• Version 3 65 n= 305• Version 4 55 n= 381• Version 5 82 n= 286• Version 6 74 n= 36• Version 7 72 n= 318• Version 8 64 n=2231• Version 9 64 n=5839

Reliability Testing• Version 8

–Tested in two junior practice classes–Students tested twice, 2 weeks apart–Pearson’s correlation coefficient

•r = .86– a

Item difficulty index• Overall difficulty or average should be

around .5 (Cohen & Swerdlik, 2005)

• FCAI = .523 (n=415)

• “This is a very good difficulty level for the test.Not likely to misrepresent the knowledge level of test takers”.

Reliability & Effect Size

•Cronbach’s alpha = .784

•Effect Size d = 6.87

Current Data•Number of Schools: 41 from 28 States•Number of Respondents:

▫Pre-test: 3721▫Post-test: 2118

Schools By Type•BSW Only: 22•BSW/MSW Combination: 8•Not Reported: 1

Schools By Auspice•Public: 32•Private Denomination: 13•Private Non-Denomination: 6

Foundation Curriculum Assessment Instrument (FCAI)Program Name: Students are: Entering BSW Program ______ Exiting BSW Program _______Month ____, Year ____, Semester ____

I. Program Cumulative Scores Compared with all Student Scores

N = Score Range

SD t-test Value

p

Program

National

School X / National Comparison

PRE TEST

POST TEST

SD SIG

School X 33.83 (52.87%)

45.85 (71.61%)

6.18 .000

National 33.11 (51.73%)

38.96 (60.88%)

8.85 .000

Overall Scores Pre-PostBSW Entering

BSW Exiting

T-test

S.D. Sig

Raw Score

33.1 38.96 43.4 8.21 .000

% Correct

51.7 60.8

Scores By Curricular Area:Practice and HBSE

PRE-TEST

POST-TEST

S.D. SIG.

PRACTICE

7.29 8.98 2.28 .000

HBSE 5.47 6.46 1.76 .000

Scores by Curricular Area:Ethics, Diversity, and Soc/Eco Justice

PRE TEST

POST TEST

SD SIG

ETHICS 4.43 5.30 1.57 .000

DIVERSITY 4.01 4.53 1.65 .000

SOC/ECO JUSTICE

4.14 4.62 1.60 .000

Scores by Curricular Area:Policy and Research

PRE TEST

POST TEST

SD SIG

POLICY 3.94 4.59 1.81 .000

RESEARCH 3.73 4.48 1.79 .000

III. Program: BSW Student Scores by Curricular Area 1 Program : Entrance & Exit

Curricular Area Question

SuggestedCompetency

Correct responses and % Entrance

Correct responses and %

Exit

Practice

Question 1 2.1.1C 4/735.48%

39/7750.65%

Question 2 2.1.10A 46/7363.01%

65/7784.42%

Question 4 2.1.10H 21/7328.77%

43/7755.84%

Expansion beyond BSW•Based upon CSWE assertions related to

educational levels in social work education, we expanded testing to three additional groups: ▫MSW foundation students:

entering exiting

▫Advanced standing students: entering

1. Purpose of this instrument: to review and improve curriculum

2. Program will want to “monitor” scores over several years (or several cohorts) for trends.

3. The FCAI can be considered a measure of “value added” from program entry to exit.

Points to keep in Mind

Field Practicum/Placement Assessment Instrument

(FPPAI)

• 55 Likert Scale questions measuring practice behaviors linked to the EPAS 2008 competencies.

• Qualitative feedback form for each domain available for program use.

• Available online and in print format.• Individual program outcomes report with

national comparisons for EPAS 2008 Competencies & Practice Behaviors (coming next year)

• Can be used as a final field assessment and mid-test/post test design.

Assessment Methodology

FPPAI Scale

FPPAI Reporting and EPAS 2008 Overview

• FPPAI Administration Overview (Online & Paper versions)

• FPPAI Individual Reporting

• FPPAI Aggregate Reporting

• Resources available on BEAP website

Field/Practicum Placement Instrument Reporting

• More than 50 Schools utilizing the FPPAI instrument currently

• More than 1,100 administrations to date• National data comparisons available next year• Cronbach’s Alpha reliability test of internal

consistency was completed (mid-point), (n=340, .969)

• Cronbach’s Alpha reliability test of internal consistency was completed (final), (n=778, .975)

Field/Practicum Placement Instrument Reporting

Table 2: Field Placement Assessment Instrument (FPPAI) Evaluation Results: (n=340)

Section Average Minimum Maximum 1. Professional Social Work Skills and Supervision 6.56 1.71 9.00 2. Professional Communication 6.68 2.20 9.00 3. Social Work Values and Ethical Practice 6.55 2.00 9.00 4. Critical Thinking 6.33 1.50 9.00 5. Diversity 6.99 2.60 9.00 6. Human Rights, Social and Economic Justice and Policy Practice 6.44 2.00 9.00 7. Research 6.31 2.00 9.00 8. Human Behavior in the Social Environment 6.34 1.50 9.00 9. Generalist Practice 6.51 2.00 9.00 10. Social Work Practice with Individuals, Families, Groups, Organizations, & Communities 6.37 2.45 9.00

Average 6.51

EPAS Scoring

2.1.1 Identify as a professional social worker and conduct oneself accordingly

A. advocate for client access to the services of social work; 6.41 B. practice personal reflection & selfcorrection to assure continual professional development; 6.37 C. attend to professional roles and boundaries; 6.63 D. demonstrate professional demeanor in behavior, appearance, and communication; 6.84 E. engage in careerlong learning; 6.53 F. use supervision and consultation. 6.39 Total Section Score 6.53

2.1.2 Apply social work ethical principles to guide professional practice.

A. recognize and manage personal values in a way that allows professional values to guide practice; 6.49

B. make ethical decisions by applying standards of the NASW Code of Ethics and, as applicable Statement of Principles 6.62

C. tolerate ambiguity in resolving ethical conflicts; 6.65 D. apply strategies of ethical reasoning to arrive at principled decisions. 6.62 Total Section Score 6.60

2.1.3 Apply critical thinking to inform and communicate professional judgments

A. distinguish, appraise, and integrate multiple sources of knowledge, including research based knowledge, and practice wisdom; 6.31

B. analyze models of assessment, prevention, intervention, and evaluation; 6.38 C. demonstrate effective oral and written communication in working with individuals, families, groups, and colleagues. 6.68

Total Section Score 6.46

2.1.4 Engage diversity and difference in practice.

A. recognize the extent to which culture"s structures & values may oppress, marginalize, alienate, or create/ enhance privilege/ power; 6.99

B. gain sufficient self-awareness to eliminate the influence of personal biases and values in working with diverse groups; 6.86

C. recognize and communicate their understanding of the importance of difference in shaping life experiences; 6.69

D. view themselves as learners and engage those with whom they work as informants. 6.65 Total Section Score 6.80

2.1.5 Advance human rights and social and economic justice.

A. understand the forms and mechanisms of oppression and discrimination; 6.46 B. advocate for human rights and social and economic justice; 6.41 C. engage in practices that advance social and economic justice. 6.31 Total Section Score 6.39

2.1.6 Engage in research informed practice and practice informed research.

A. use practice experience to inform scientific inquiry 6.31 B. use research evidence to inform practice. 6.31 Total Section Score 6.31

2.1.7 Apply knowledge of human behavior and the social environment.

A. utilize conceptual frameworks to guide the processes of assessment, intervention, and evaluation; 6.34

B. critique and apply knowledge to understand person and environment. 6.34 Total Section Score 6.34

2.1.8 Engage in policy practice to advance social and economic wellbeing and to deliver effective social work services.

A. analyze, formulate, and advocate for policies that advance social wellbeing; 6.42 B. collaborate with colleagues and clients for effective policy action. 6.48 Total Section Score 6.45

2.1.9 Respond to contexts that shape practice.

A. continuously discover, appraise, and attend to changing locales, populations, scientific and technological relevant services; 6.33

B. provide leadership in promoting sustainable changes in service delivery and practice to improve the quality of social services. 6.28

Total Section Score 6.31

2.1.10(a)-(d) Engage, assess, intervene, and evaluate with individuals, families, groups, organizations, and communities.

A. substantively and affectively prepare for action with individuals, families, groups, organizations, and communities; 6.41

B. use empathy and other interpersonal skills; 6.72 C. develop a mutually agreedon focus of work and desired outcomes. 6.37 D. collect, organize, and interpret client data; 6.56 E. assess client strengths and limitations; 6.49 F. develop mutually agreedon intervention goals and objectives; 6.34 G. select appropriate intervention strategies. 6.37 H. initiate actions to achieve organizational goals; 6.31 I. implement prevention interventions that enhance client capacities; 6.37 J. help clients resolve problems; 6.31 K. negotiate, mediate, and advocate for clients; 6.61 L. facilitate transitions and endings. 6.30 M. critically analyze, monitor, and evaluate interventions. 6.21 Total Section Score 6.41

Table 2: Field Placement Assessment Instrument (FPPAI) Evaluation Results: (n=778)

Section Average Minimum Maximum 1. Professional Social Work Skills and Supervision 7.25 2.14 9.00 2. Professional Communication 7.36 2.33 9.00 3. Social Work Values and Ethical Practice 7.33 2.00 9.00 4. Critical Thinking 7.07 1.33 9.00 5. Diversity 7.61 2.00 9.00 6. Human Rights, Social and Economic Justice and Policy Practice 7.15 2.00 9.00 7. Research 6.77 1.00 9.00 8. Human Behavior in the Social Environment 6.99 1.00 9.00 9. Generalist Practice 7.26 2.00 9.00 10. Social Work Practice with Individuals, Families, Groups, Organizations, & Communities 7.04 1.00 9.00

Average 7.18

EPAS Scoring

2.1.1 Identify as a professional social worker and conduct oneself accordingly

A. advocate for client access to the services of social work; 7.12 B. practice personal reflection & selfcorrection to assure continual professional development; 7.18 C. attend to professional roles and boundaries; 7.38 D. demonstrate professional demeanor in behavior, appearance, and communication; 7.48 E. engage in careerlong learning; 7.23 F. use supervision and consultation. 7.12 Total Section Score 7.25

2.1.2 Apply social work ethical principles to guide professional practice.

A. recognize and manage personal values in a way that allows professional values to guide practice; 7.25

B. make ethical decisions by applying standards of the NASW Code of Ethics and, as applicable Statement of Principles 7.41

C. tolerate ambiguity in resolving ethical conflicts; 7.43 D. apply strategies of ethical reasoning to arrive at principled decisions. 7.41 Total Section Score 7.37

2.1.3 Apply critical thinking to inform and communicate professional judgments

A. distinguish, appraise, and integrate multiple sources of knowledge, including research based knowledge, and practice wisdom; 6.77

B. analyze models of assessment, prevention, intervention, and evaluation; 6.99 C. demonstrate effective oral and written communication in working with individuals, families, groups, and colleagues. 7.36

Total Section Score 7.04

2.1.4 Engage diversity and difference in practice.

A. recognize the extent to which culture"s structures & values may oppress, marginalize, alienate, or create/ enhance privilege/ power; 7.61

B. gain sufficient self-awareness to eliminate the influence of personal biases and values in working with diverse groups; 7.53

C. recognize and communicate their understanding of the importance of difference in shaping life experiences; 7.40

D. view themselves as learners and engage those with whom they work as informants. 7.37 Total Section Score 7.48

2.1.5 Advance human rights and social and economic justice.

A. understand the forms and mechanisms of oppression and discrimination; 7.17 B. advocate for human rights and social and economic justice; 7.16 C. engage in practices that advance social and economic justice. 7.05 Total Section Score 7.12

2.1.6 Engage in research informed practice and practice informed research.

A. use practice experience to inform scientific inquiry 6.77 B. use research evidence to inform practice. 6.77 Total Section Score 6.77

2.1.7 Apply knowledge of human behavior and the social environment.

A. utilize conceptual frameworks to guide the processes of assessment, intervention, and evaluation; 6.99

B. critique and apply knowledge to understand person and environment. 6.99 Total Section Score 6.99

2.1.8 Engage in policy practice to advance social and economic wellbeing and to deliver effective social work services.

A. analyze, formulate, and advocate for policies that advance social wellbeing; 7.14 B. collaborate with colleagues and clients for effective policy action. 7.15 Total Section Score 7.14

2.1.9 Respond to contexts that shape practice.

A. continuously discover, appraise, and attend to changing locales, populations, scientific and technological relevant services; 6.95

B. provide leadership in promoting sustainable changes in service delivery and practice to improve the quality of social services. 6.97

Total Section Score 6.96

2.1.10(a)-(d) Engage, assess, intervene, and evaluate with individuals, families, groups, organizations, and communities.

A. substantively and affectively prepare for action with individuals, families, groups, organizations, and communities; 7.09

B. use empathy and other interpersonal skills; 7.38 C. develop a mutually agreedon focus of work and desired outcomes. 7.09 D. collect, organize, and interpret client data; 7.25 E. assess client strengths and limitations; 7.20 F. develop mutually agreedon intervention goals and objectives; 7.03 G. select appropriate intervention strategies. 7.08 H. initiate actions to achieve organizational goals; 7.00 I. implement prevention interventions that enhance client capacities; 7.09 J. help clients resolve problems; 7.06 K. negotiate, mediate, and advocate for clients; 7.30 L. facilitate transitions and endings. 6.89 M. critically analyze, monitor, and evaluate interventions. 6.79 Total Section Score 7.10

How to Access BEAPhttp://beap.utah.edu/Orhttp://www.bpdonline.org•“Resource Center”•BEAP

▫Resource Center▫Accreditation Resources

A quick look at the BEAP website▫Navigating the site▫Locating the results of your instruments

Ordering BEAP Instruments▫It is very important for each batch of

forms to have a face sheet!▫See handout for ordering BEAP for the

first time.▫See handout for re-ordering BEAP.