october 27, 2005ece department meeting1 assessment plan and notes on the abet program evaluator...

October 27, 2005 ECE Department Meeting 1

Assessment Plan and Notes on the ABET Program

evaluator training

J. Fernando Vega-Riveros, Ph.D.Dept. of Electrical and Computer

EngineeringUniversity of Puerto Rico


Objective of presentation• Present issues and comparative

analysis of assessment approaches and ABET criteria

• Promote discussion of issues as they pertain to Department INEL/ICOM

• Make a decision about department assessment instruments and methods


Rationale• A unified assessment approach for

our two programs would would reduce administrative and academic burden on faculty, students and staff.

• Same instruments and tools used for the whole department


Accreditation timelineIN

EL

an

d I

CO

M a

ccre

dit

ed t

ill

2009

Ass

essm

ent

Pla

n

Conduct assessment and close the loop

Req

ues

t re

-acc

red

itat

ion

to

AB

ET

Su

bm

it S

elf-

Stu

dy

Rep

ort

AB

ET

Vis

it

TodayDecember 2

005

Summer 2006

Summer 2007

January 2008

July 2008

December 2009

Def

ine

asse

ssm

ent

inst

rum

ents

Des

ign

met

rics

, th

resh

old

s an

d

sam

pli

ng

met

ho

d


Assessment and evaluation1

• Assessment: act of collecting data or evidence that can be used to answer classroom, curricular. Or research questions.

• Assessment method: Procedures used to support the collection of data for assessment purposes.

• Evaluation: refers to the interpretations that are made of the evidence collected about a given question.

1 Olds, B.M.; Moskal, B.M. and Miller, R. Assessment in engineering education: evolution, approaches and future collaborations. ASEE Journal of Engineering Education. Vol. 94 No. 1. Jan. 2005. pp 13-25


Types (components?) of assessment

Longitudinal Design

Tra

nsve

rsal

Ass

essm

ent

Assessment of outcome achievement in a given academic term (snapshots)

Tracks outcome achievement over time

Formative Assessment

Su

mm

ativ

e A

sse

ssm

ent


Goal of assessment• Measure outcome achievement (a-k) and

how these outcomes are developing along and across the curriculum

• Validate interventions, e.g. curricular revisions, improved or novel pedagogies, new courses, course restructuring, etc.

• Collect data for quality assurance and continuous quality improvement

• Realistic and minimize burden


Program Educational ObjectivesProgram will allow the graduates of the program to:

1. Obtain a broad educational experience necessary to understand the impact of electrical/computer engineering problems and solutions within a global and societal context;

2. Possess a combination of knowledge and analytical, computational, and experimental skills necessary to solve practical electrical/computer engineering problems;

3. Have adequate communication skills both as an individual and as part of a team;

4. Value the importance of lifelong learning

5. Be aware of contemporary issues and thus be able to make decisions taking into consideration professional and societal needs and ethical implications


Program educational outcomesGraduates of the program will have the:

a) Ability to apply knowledge of mathematics, science and engineering necessary to carry out analysis and design appropriate to electrical/computer engineering problems;

b) Ability to design and conduct experiments as well as analyze and interpret data;

c) Ability to design computer systems to meet desired needs;

d) Ability to function in multidisciplinary teams;

e) Ability to identify, formulate, and solve engineering problems;


Program educational outcomes (cont)f) Understanding of professional and ethical responsibility

g) Ability to communicate effectively;

h) Broad education necessary to understand the impact of engineering solutions in a global/societal context;

i) Recognition of the need for and ability to engage in lifelong learning

j) Knowledge of contemporary issues

k) Ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.


Current Assessment Approach (ICOM)• Instruments

• GPA (Transcript Analysis)• Student, alumni and employer surveys

• Metrics and satisfaction thresholds• Items related to grades are considered satisfactory it

they are above 3.0 (B grade)• Items related to satisfaction rating are satisfactory if

• Above 3.0 (weight of a “satisfied” response in surveys); or

• Above average satisfaction of all items in same category


Issues of current approach• Minimum burden on faculty

• High burden on accreditation team (coordinator)

• Uses surveys, practice that ABET discourages for outcome assessment. Surveys can be used as supportive information but depends on perception not on actual outcome achievement3

• Some thresholds move depending on population and may not be set according to more explicit program criteria.

3 ABET 2005 Program Evaluator Training Materials (EAC Training, Approved December 2004).


Issues of current approach (Continued)• GPA does not discriminate among outcomes of a given

course; it gives some indication of achievement of the set of outcomes set for a given course or courses.

• ABET has some concerns about use of course grades4.

4 Guidelines to Institutions, Team Chairs and Program Evaluators on Interpreting and Meeting the Standards Set Forth in Criterion 3 of the Engineering Accreditation Criteria. http://www.abet.org/Linked Documents-UPDATE/Program Docs/EAC Guidelines or Criterion3.pdf.

http://www.abet.org/Linked%20Documents-UPDATE/Program%20Docs/EAC%20Guidelines%20or%20Criterion3.pdf




Assessment methods2

• Descriptive designs• Surveys• Interviews and focus groups• Observation• Conversational analysis• Ethnographic studies

• Experimental designs• Randomized controlled trials• Matching• Baseline data• Longitudinal design• Post-test-only design

2 Olds et-al Op. cit.


Approaches to outcome assessment

Alternative

Pros Cons

Student Portfolios

Unified assessment for all students and all academic termsComprehensive

Manageable with small student populations.How do we store and organize portfolios?Who is responsible for portfolio repository?

Periodic comprehensive tests

StandardizedUnified assessment for all students and all academic termsProfessional licensure could be one of the testsWhat if student achieves outcome but fails the course or vice versa?

How do we assess criteria d, g and i?How and when would we give them?How can we motivate students to take them seriously?Who designs the tests?Standardized tests measure how well students learned information but may not demonstrate how well they solve problems


Approaches to outcome assessment (Continued)

Alternative Pros Cons

Cornerstone and capstone courses

ComprehensiveAssessment instruments could be more standardized than in content courses (no need to come up with new questions for tests or exams)

What are the cornerstone courses? Do we have any?We need to identify candidate courses and/or create them

Course material-based assessment

We may already have many necessary elements and evidence materialsProbably the smallest additional burden on facultyAllows student progress monitoringAllows multiple measures of each outcome

Need to do a mapping between materials and outcomesHow to sample materials (statistically significant)Not comprehensive


Approaches to outcome assessment (Continued)

Alternative

Pros Cons

Professional Licensure test (Reválida)

Almost comprehensiveCan be used for comparative analysis with rest of the nationShows outcomes of the program

Does not allow assessment of outcomes d, g and i.Only one point measurement and does not let us see how students develop skills to achieve outcomesLong response time to any curricular changes


Assessment Proposal• Establish cycles of assessment for courses

• Sampling of course material (no need to be exhaustive, just representative and statistically significant)

• Mapping of course material and educational outcomes (may need collaboration of course instructor)

• Measurement of outcome achievement using course material and outcome mapping.

• No course is assessed every semester (unless something exceptional is found). Round-Robin strategy.

• Not all sections of a course need to be assessed. Assessment plan will establish cycle of sections assessment to minimize burden on faculty members and distribute assessment load as evenly as possible.

• Will use professional licensure examination as a component of summative assessment and benchmarking

• Results reviewed by area committees and final results send to steering committees


Assessment Proposal – developing the assessment plan Determine who we will assess:

Educational objectives: graduates between 1 and 3 years. Educational outcomes: current undergraduate students in the Department.

Establish a Schedule for Assessment. Proposed re-accreditation timeline.

Determine who will interpret results: Area committees.

Interpret how results will inform teaching/learning and decision making: Steering committees.

Determine how and with whom we will share interpretations: Department, College, Office of Continuous Improvement and Assessment, ABET, Middle States, CES through periodic reports and self studies.

• Decide how our department and institution will follow-up on implemented changes: periodic assessment and continuous program quality improvement.


Assessment Schedule

January 2006

June 2006

December 2006

June 2007

December 2007

Req

ues

t re

-acc

red

itat

ion

to

AB

ET

Su

bm

it S

elf-

Stu

dy

Rep

ort

January 2008

July 2008

Pub

lish

Ass

essm

ent

Sch

edul

e

1 st A

sse

ssm

ent

Sam

plin

g

2 nd

Asm

t S

mpl

and

sta

rt c

losi

ng lo

op

3 rd A

smt

Sm

pl a

nd c

ontin

ue c

losi

ng lo

op

4 th A

smt S

mpl

& fi

nish

asm

t. cy

cle

AB

ET

Vis

it

December 2009

CommitteeEvaluations




Questions, discussion and decision

october 27, 2005ece department meeting1 assessment plan and notes on the abet program evaluator...

Documents

department inelicommake

assessment method

assessment purposes

program evaluator trainingj

collection of data

experimental skills

societal needs

societal contextpossess