Power up your Program EvaluationsMarketing, Development, & Assessment

Evaluation Form (paper or online) Responses reviewed by SAO Responses shared with faculty If paper, some reviews placed in the

resource center for prospective students to review

If online (Zoomerang, SurveyMonkey, etc.) reviews may be downloaded & stored.

Other actions taken on a case by case basis.

Traditional Approach

Definition: A systematic exploration to discover and judge something that we think has value.

Evaluation is “Ex Valore” in Latin, which means to “strengthen” or “empower” Ex – “out of, from” Valor – “strength, value, valor”

It was originally a “means” rather than an “end”

A means to “empowerment” synonymous to positive growth and development

A means that involves people, processes, and results associated with what we do

What is evaluation?

Practices - Offices that manage the process, third-party subcontracts, host institution services, information sessions, student advising, pre-departure, policies, procedures, etc.

Programs - Direct enroll, faculty-led, third-party, consortium, student teaching, internships, etc. (their administration, academics, housing, social, expenses, etc.)

Projects - Freshmen orientation, study abroad fair, marketing workshop for faculty, study abroad ambassadors project, welcome back reception, and other special events.

to… Prove (show) something is working or needed Improve the practice, program, or project Discover something new for development

We can evaluate…

Coordinate evaluation process for multidimensional goals Student Satisfaction Data (regarding Administration, Academics,

Housing, Social, Expenses) helps us in marketing and development Learning outcomes, intercultural development, progress towards

global citizenship guides our questions regarding assessment Shrewdly and efficiently share information with stakeholders.

Power up your evaluations…

Stakeholders Why? Purpose/Goal

Prospective Students-Parents, Alumni, potential Donors

Earn respect, convince of value-benefits-outcomes, and grow participation

Marketing

OSA, Offices, Partner Institutions, Organizations, Faculty, & Academic Departments

Monitor, grow, improve, foster goals & objectives in programs and operations.

Development

OSA, the field of study abroad, Faculty, Academic Departments, Accreditation Bodies, Upper Administration, Donors

To show that stakes are valid, gain support, lobby, grow the education abroad as a unique & important form of education.

Assessment

AbroadScout 2.0Mission: to empower students, schools, and the education abroad industry with quality information and Web 2.0 tools for marketing, development, and assessment.

AS 2.0 Marketing, Development, Assessment Tools

Add your programs and scholarships to the main AS directory (also edit/delete)

Create your own PROGRAM FINDER (a customized directory of the sending institution’s approved list of programs with scholarships and reviews)

Create your own COURSE FINDER (a course articulation database tied to your customized PROGRAM FINDER) - Co managed

Customize the AbroadScout Program Evaluation alongside your program partners. One survey per student works for multiple parties (home, provider, host).

Create and manage new surveys . Share surveys and results with select partner institutions, offices, etc. Great for development and assessment projects.

AS2.0 Funding Model FREE – All Marketing-Development-Assessment tools. PRO ACCOUNT – Allows you to upload a logo. Cost $300/yr.

Marketing - It gives your programs more visibility, as they are randomly sorted among all pro-account programs in the first tier of any search (in the main directory and customized program finders). Also, more information is included in the initial search results.

Added Customization - In your Program Finder and Course Finder, it gives your pages a more personalized feel with the logo from your institution on top.

ADVERTISING POLICY –To keep information unbiased, and avoid conflicts of interest, the directories (programs, scholarships, and reviews) are completely free of advertising, aside from the pro-account option we offer to all program sponsors equally.

FUNDING MODEL – We fund the site through pro-accounts in the directories and banners-links in continent/country sections of the website. These sections contain articles and news, relevant to student interests, many of which are written by students.

Study Abroad @Abilene Christian University

Private Master’s Level University 4000 Undergraduate and 700 Graduate 3 Permanent sites, 2 Temp sites, 8 consortium sites

Semester and Summer Programs Oxford, England Montevideo, Uruguay Leipzig, Germany

Approx. 220 students abroad split evenly between semester-long and summer programs

General Education during semesters Discipline specific during summers

ACU On-site directors, ACU visiting faculty, local adjuncts and partner universities

Study Abroad @ Abilene Christian University

University Mission and Vision Increase Student Learning Increase participation to reach 50% by 2020

(currently 28%) Expectation that we balance our budget

Accountability to constituents Students, faculty, parents,

donors, board of trustees, home and host communities

Regional Accrediting Body – SACS

Institutional Drivers of Marketing, Development and Assessment

Abilene Christian University

What, How and Why Global Perspective Inventory

Pre-departure and post completion Embedded in an orientation course

Course evaluations Paper and pencil Online

From Assessment to Marketing

Online student program evaluation Yes/No and open ended-text box questions obtained

at the close of the program Reflection papers embedded in a course Faculty debriefing

From Assessment to Marketing

Internal Global Perspectives Inventory (GPI) and

online program evaluation results to Program Directors, Senior University Administrators, PR/marketing and Board of Trustees (BOT)

Getting from A (Assessment) to M (Marketing)

Course Evaluations – faculty, Dept. Chairs and College Deans, PR/marketing offices

Student Reflection papers –faculty, PR/marketing offices

Student numbers and demographics - Senior Administrators, BOT – sign leases, purchase property etc…

Getting from Assessment to Marketing – Internal (continued)

External Quotes incorporated into

Web page articles, ads etc… Information relayed to

current and prospective students and their families via promotional and recruiting materials

Host community is provided feedback – i.e. volunteer organizations, Property/facility owners, neighbors, partner institutions

Getting from A to M (continued)

Increased student learning and self-efficacy Value Added Holistic education

Increased student enrollment opportunities for growth Paying the bills

Increased funding to improveand expand services and sites

C *e*l*e*b*r*a*t*i*o*n*s

Making every one happy vs. creating a culture of quality enhancement and improvement

Closing the Loop – Follow up assessment to maximize continual improvement

Biting off more than we can chew – trying to do too much

C-h-a-l-l-e-n-g-e-s

4-1-4 calendar (5-week January term), abroad and on-campus courses offered

About 1,400 UD students abroad in 2010/11; 75% on January programs (1060 in 2011); 90% on short-term programs (summer and winter programs)

Over 40% of undergraduates (population 15,000) study abroad

About 70 UD programs annually, 50 in January, all faculty led, representing 75% of academic departments

About 20 semester/exchange programs (about 130 students total)

Study Abroad at Univ. Delaware

Paper evaluation (satisfaction survey) unwieldy to administer data hard to retrieve/summarize/distribute

Office of Educational Assessment created 2005 institutional interest in assessing undergraduate

learning campus resource for expertise

Discovery Learning Experience required for all undergrads as of 2006

good reason for mandatory assessment study abroad counts as DLE!

Where We Were

Given that: the number of students and programs is very large; students are enrolled in a wide variety of courses; most are not enrolled in a foreign language course

(though many are); programs are located across the globe in both rural and

urban settings; the range of program conditions is great (housing,

mobility, interaction with host culture); and faculty directors have different program goals:

What can we measure that applies to all students?and

How can we measure it?

Thoughts on How to Move Forward

What We Do – Data CollectionLook for general impacts (changes from pre to post) in areas of knowledge, skills, and attitudes (applicable to all programs)Design a short instrument (31 Likert scale items, 2 short answer)Incorporate into required online pre-departure orientationInstitute a required post-program assessment (linked to University-wide Discovery Learning ExperienceInclude standard satisfaction survey (subjective feedback on program quality)Use existing online course evaluation system (via registration for UNIV 370, pass/fail, 0- credit,

study abroad marker)

What We Do – Data Organization

Download satisfaction survey feedback; send as pdf to faculty after grades are submitted

Download course feedback; send as pdf to dept. chairs

Program coordinators prepare summaries of each program, addressing budget, faculty organization and performance, incidents abroad, student feedback (numerical ratings, selected quotations), and suggestions for next time

Summaries sent with course syllabi to dept. chair and dean

Assessment results stored and analyzed internally

Largely responsibility of faculty for short-term programs

Responses to short-answer questions can be used in publications

What We Get and Use: Marketing

What We Get: Development Masses of data: 98% return

rate Electronic data collection,

retrieval, storage using UD’s course evaluation system

Student satisfaction survey results helpful for faculty and program administrators

Used in program summary sent to faculty, chairs, deans

Documentation of problem cases

What We Get: Assessment Quantitative and qualitative pre- and post-

sojourn data from about 1400 students on 70 programs annually

98% return rate Electronic data collection, retrieval, storage,

using UD’s existing online course evaluation system

Ongoing data collection Improve academic

programming and student learning

Friends in Assessment Office

Positive contribution to institutional re-accreditation

assessment is self-reported

on-campus control group difficult to maintain

in-house instrument (untested beyond UD)

data backlog (lack of time for analysis)

Limitations/Disadvantages summaries very time-

consuming and labor-intensive to compose and send

not completely objective

impact of summaries unknown (Do chairs/ deans read them?)

What kind of assessment or evaluation of education abroad programs is done at your institution?

Who sees the results? How are the results used? What prevents results from being used

for marketing purposes, for implementing programmatic or procedural changes, or for better understanding student learning?

Questions/Discussion

Lisa Chieffo, Ed.D.Associate Director

Institute for Global StudiesUniversity of Delaware

Newark, Delaware, USA302-831-2852

lchieffo@udel.eduwww.udel.edu/international

How to Find UsKevin Kehl, Ed.D.Executive Director

Center for International EducationAbilene Christian University

Abilene, Texas, USA325-674-2710

kehlk@acu.eduwww.acu.edu/academics/studyabroad

Wendy WilliamsonDirector

Office of Study AbroadEastern Illinois UniversityCharleston, Illinois USA

217-581-3390wswillliamson@eiu.eduwww.eiu.edu/~edabroad