evaluation methods and findings of the field-tested learning assessment guide

Evaluation Methods and Findings of the Field-tested

Learning Assessment Guide

Sue Daffinrud

Olga Lucia Herrera

The Motivation for the FLAG

Motivated by needs of faculty who wanted centralized place to learn about already-verified assessment techniques and tools (Third Annual NISE Forum)

Does the FLAG satisfy part of this need?

FLAG Structure

Five major sections Getting Started Matching Goals to CATs CATs Searchable Tools Resources

FLAG target audience

Instructors who:

teach freshman and sophomore-level science, math, and engineering courses

and are interested in learning about new

approaches to assessment

FLAG Evaluation

Does the FLAG website fulfill some of the assessment needs of its target audience?

Formative and Summative Evaluation

Identifying target audience sample

Selected subset of SME disciplines Biology, chemistry, engineering, mathematics,

and physics Selected range of institutions

By Carnegie classification Associate of Arts, Baccalaureate, Masters,

Doctoral, and Research Region of country

Identifying target audience sample

Selected faculty through their participation in some initiative which has student learning as its focus. bioQUEST, Worskhop Biology, OCEPT Chemistry Coalitions Engineering Coalitions, EESP Project NExT Workshop Physics

50 faculty who have demonstrated an interest in “learning, not teaching”

Bio Chem Engr Math Phys

Assoc 2 2 2 2 2

Bacc 2 2 2 2 2

Masters 2 2 2 2 2

Doctoral 2 2 2 2 2

Research 2 2 2 2 2

Contacting participants

Called and emailed faculty requesting their participation

Participation involved: Browsing through FLAG for 30 minutes Providing feedback in a 30 minute interview

High participation rate

Interviews

Participants were asked about: Their current teaching practice How they learn about new

assessment/teaching techniques Impressions of FLAG, both specific and

general Whether would recommend FLAG to others

Data Gathering Status

Have interviewed 32 faculty Will interview the remaining 18 in

December

Preliminary findings

Demographics of respondents

Technical issues

Themes of Use and Need

Dissemination

Demographics

Range of teaching experience (2-22 yrs) Critical of the value of lecture Goals: improving students’ interest in science Role: facilitator for their learning Info on assessment gained through

conversations with fellow faculty, conferences in their discipline

Few “surf” the web

Technical Issues – Navigation patterns

Participants spent between 30 minutes to 4 hours in the FLAG, most about an hour.

Many moved “left to right”

Technical Issues

Almost all found the web site Easy to read Structured well Could find what they needed Enjoyed being able to download documents

Suggestions Provide more obvious description of site structure Provide better navigation instructions

Themes of Use/Need - Overall

Overall response: Positive

Almost all respondents had already or had plans to recommend the site to other faculty, faculty developers, or other groups

Themes of Use/Need - Sections

Has teaching component Matching Goals to CATs Primer

Availability of links to other web sites Annotated bibliography References

“Empty” sections create expectation Searchable tools “Experts” by state

Themes of Use/Need - CATs

Time is a factor in browsing and in applicability First page of CATs (why, how, what’s

involved) People selected:

CATs which were familiar to them CATs which best fit their goals and time

Themes of Use/Need – CATs (cont)

People want a complete picture Like the case study approach When and how to use the CAT What to do with the information How to get students “on board” Clear and useful examples

People want “tools” from their discipline to varying degrees

Dissemination of FLAG

Evaluation AS dissemination Hope to find out how best to reach people

like our sample Discipline-specific professional meetings Workshops People on campus who do assessment Email to appropriate email lists

Final Notes

Submitted abstract for presentation at AAHE 2000

Write monograph or paper Provide more formative information

to Tony To ILT group as they develop site