abet processes what needs to be done? how can it all be accomplished? what are the benefits?

58
ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits? Fred DePiero Associate Dean & EE Professor

Upload: sook

Post on 11-Jan-2016

18 views

Category:

Documents


1 download

DESCRIPTION

ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?. Fred DePiero Associate Dean & EE Professor. Outline. Overview of ABET Requirements What needs to be done? Specific assessment methodologies & program improvements Suggested approaches - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

ABET ProcessesWhat Needs to Be Done?

How Can It All Be Accomplished?What Are the Benefits?

Fred DePiero

Associate Dean & EE Professor

Page 2: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Outline• Overview of ABET Requirements

– What needs to be done?– Specific assessment methodologies & program

improvements

• Suggested approaches– Centralized (most) indirect measures– Decentralized (most) direct measures– Draw from university sources (direct & indirect)– College-wide improvements

• College-wide overview– Lessons learned, general comments

Page 3: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Organization of Talk

• In Order of Criteria (‘C2’, ‘C3’, ‘C4’)

• ‘What?’– What are requirements?

• ‘How?’– Suggestions on how to accomplish tasks

• ‘Benefits!’– What are institutional benefits?

Fred DePiero [email protected] What?

Page 4: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Limited DisseminationOf Materials

• Distribute freely within your faculties

• Do NOT post on public web sites– BlackBoard (e.g.) fine

Fred DePiero [email protected]

Page 5: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Audience? / Status?

• Familiar with:– Criteria 2, 3, 4, 5? A-K?– ‘Indirect’ vs ‘Direct’ measure?

• Status:– Have PEOs? Reviewing and revising?– Measurable skills identified for A-K?

Rubrics?– Any direct measures for A-K?– Curriculum mapping?

Fred DePiero [email protected]

Page 6: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Recent ABET Visit, ‘Went Well’

• Visit: October 26-28, 2008• Simultaneous visit (CAC & EAC)

• 12 Programs evaluated for reaccreditation – Including 2 outside CENG

• 1 Program evaluated for first accreditation• (Specific results confidential)• Biomedical Engr going for 1st accr in ~1.5 years• Five Self Studies selected for display at ABET

Symposium 2010 (Las Vegas).– ME, CE, IE, AERO, CSC

Page 7: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

My Role?• Provide centralized services

– On-Going: run surveys of seniors, alumni– Gather data from university-wide sources– Visit Prep: Additional data services (course

offerings, teaching assn), transcripts, schedule, coach to coordinator, chair, faculty

• Specific guidance to programs– Self Study template. Compare recent S-S examples

• Provide suggestions– Share best practices. Report from BAP, ASEE

• Also an ABET Program Evaluator - Invaluable!– 5 team members are PEVs

Page 8: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Overview of All Criteria• C1, Students

– Advising, transfer credit, grad evaluations

• C2, Program Educational Objectives– Career accomplishments ~3-5 years after

graduation– Review and revise PEOs

• C3, Student Outcomes, A-K– Skills, knowledge, abilities acquired by

time of graduation

Fred DePiero [email protected]

Page 9: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Overview of All Criteria…• C4, Continuous Improvement

– Use evaluation of C2, C3 to improve program

• C5, Curriculum– Basic math & science, eng science, design

• C6, Faculty (Sufficient & Competent)

• C7, Facilities

• C8, Institutional Support

• Program CriteriaFred DePiero [email protected]

Page 10: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Criterion 2: Objectives• ‘Objectives’ define career accomplishments

of graduates, ~3-5 years after graduation. • Programs define their own Objectives.• Examples: Graduates will…

– Continue education (graduate school or other)– Work in an ethical manner– Become leaders on project team– Be good members of the community, do

volunteer work

• Examples: MATE, ME, CE, (IE)C2 What?

Page 11: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Criterion 2: Objectives• Assessment, Typical Approaches

– Use alumni surveys– Feedback from Industrial Advisors

• Evaluation– My interpretation, validated by ABET– All alumni should attain one or more

Objectives. ‘1-of-N’ requirement.– All Objectives should be attained by at

least some alumni.

C2 What?

Page 12: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Improving Survey Processes

Fred DePiero [email protected], C3 How?

Page 13: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Creating Better Surveys

• Three most important aspects:–Short, Short, SHORT!

How?

Page 14: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Get More ValueFrom Surveys

• Alumni surveys – 2 styles of questions• Include likert scale questions on each PEO.• Asked specific questions (‘yes/no’), for more

accurate data - fewer misinterpretations– On specific career accomplishments of alumni– Drop ‘Leader in field?’, rather ‘Award for work?’

• (Examples: MATE, ME, CE, IE)

C2 How?

Page 15: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Example Alumni Survey (MATE)(PEOs listed on survey)

– 1. Apply materials engineering principles to analyze and solve real-world engineering challenges.

– 2. Communicate and perform as effective engineering professionals in both individual and team-based project environments.

– 3. Develop intellectually through continuous learning.

– 4. Work in an individual or team environment in a socially responsible manner.

Please rate the extent to which you feel that the career objectives are appropriate for CalPoly MATE graduates? (Accomplished within 5 years of graduation)

• Very Appropriate

• Somewhat Appropriate

• Neutral

• Somewhat Inappropriate

• Not Appropriate

Fred DePiero [email protected]

Page 16: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Example Alumni Survey (MATE)(PEOs listed on survey)

– 1. Apply materials engineering principles to analyze and solve real-world engineering challenges.

– 2. Communicate and perform as effective engineering professionals in both individual and team-based project environments.

– 3. Develop intellectually through continuous learning.– 4. Work in an individual or team environment in a socially responsible manner.

(The following specific accomplishments could be presented as Yes/No questions)• I routinely analyze and solve real-world engineering challenges;• I have been promoted from an entry-level position;• I have been given a raise and/or a bonus;• I have started or now own a successful business;• I am a team member of a successful organization;• I am part owner of a patent;• I have been given positive performance reviews;• I have management/leadership responsibilities;• I have learned new skills;• I have acquired knowledge beyond my B.S. degree;• I have earned graduate degrees;• I have earned other degrees and/or certifications;

Fred DePiero [email protected]

Page 17: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Get More ValueFrom Surveys

• Alumni surveys – 2 styles of questions• To ensure ABET compliance

– Ask seniors ABET Criteria verbatim– Issue/Limitation: Varied interpretation by respondents,

others

• For more useful data– Translate ABET, local / specific terms, e.g. describing skills– Map results into PEOs

C2 How?

Page 18: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Using Electronic ToolsTo Help with Surveys

• Web-based, via SurveyMonkey.com– Alumni: Social networking site ‘PolyLink’, helps build on-

line community– Seniors: Email addresses from univ data sources– SurveyMonkey provides simple statistics

• Organize documents (survey instruments) and results electronically…

C2, C3 How?

Page 19: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Encouraging BetterSurvey Response Rates

• Email Prompt / Message – Alumni Surveys– Modified subject line - less likely flagged as spam– An energetic email message– Sender appears to be Dept Chair

• Motivating respondents– Either a course requirement for senior surveys,

or instructor’s polite request– Prize and gift offered for alumni surveys

C2 How?

Page 20: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Survey Return Rates Improved

• Alumni: – 20%-40% (2x - 3x improvement)

• Seniors: – 100%, if a strict course requirement– 75% if requested & encouraged by

instructor, nagged via re-prompts

C2, C3 How?

Page 21: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Criterion 3: Student Outcomes• Outcomes define students’ abilities by

graduation.• Recommend: Use ABET’s standard 3A-3K• Requirements: (Challenging!)

– Programs define more specific measurable skills. (Action verb - Preferably high on Bloom’s scale)

– Curricula must ensure that every student has an opportunity to learn these skills.

– Recommend both direct & indirect measures in a sustainable process.

C3 What?

Page 22: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Criterion 3: Outcomes ‘A-K’(a) an ability to apply knowledge of mathematics, science, and engineering(b) an ability to design and conduct experiments, as well as to analyze and

interpret data(c) an ability to design a system, component, or process to meet desired needs

within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

(d) an ability to function on multidisciplinary teams(e) an ability to identify, formulate, and solve engineering problems(f) an understanding of professional and ethical responsibility(g) an ability to communicate effectively(h) the broad education necessary to understand the impact of engineering

solutions in a global, economic, environmental, and societal context(i) a recognition of the need for, and an ability to engage in life-long learning(j) a knowledge of contemporary issues(k) an ability to use the techniques, skills, and modern engineering tools

necessary for engineering practice.

Note: Over half are non-technical, non-traditional for engineering programs!Engage GE Community for a better curricular mapping (win-win)!

C3 What?

Page 23: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

‘Measurable’ Skills - 3D

• 3D) Multidisciplinary Teams • Students will be able to…

– Recognize value of a MD team: broad skill set.– Communicate effectively with colleagues in other

disciplines and listen well.– Employ flexible styles and behaviors and

recognize the style of another.– Identify when problems occur due to poor team

member interaction. Identify ways to improve team dynamics.

• Assess: Via reflection after (or mid) project.

C3 What?

Page 24: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

‘Measurable’ Skills - 3H

• 3H) Societal Impact • Students will be able to…

– Identify societal impact associated with a project– Awareness of response of science and

technology to needs driven by society, health & safety, or the environment.

– Awareness of the potential for a negative impact of science and technology on society, perhaps resulting from an unintended consequence.

• Assess: Short essay, or ‘Give an example’

C3 What?

Page 25: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

‘Measurable’ Skills - 3J• 3J) Contemporary Issues • Students will be able to…

– Demonstrate awareness of current or recent events in nation or world that could influence a student's career path or their field.

– Identify sustainability issues associated with a project

– Identify a disruptive technology.– Demonstrate awareness of the way technology

impacts and is impacted by public policy making [local, state, federal].

• Assess: Short essay, or ‘Give an example’

C3 What?

Page 26: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

‘Measurable’ Skills - 3I

• 3I) Life-Long-Learning • Students will be able to…

– Learn new techniques, tools, or devices outside the classroom environment.

– Find appropriate technical resources, compare quality and availability of sources

– Organize info, and evaluate applicability / usefulness.

• Assess: Via reflection after (or mid) project.

C3 What?

Page 27: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

‘Measurable’ Skills - Examples• For IE, See SS pg 44

3b) Students will attain an ability to design and conduct experiments, as well as to analyze and interpret data

• SKILLS

1. Students will be able to identify the experimental and control variables to conduct DOE

2. Students will be able to check various data collection assumptions

3. Students will be able to select proper DOE models to correctly analyze the obtained sample data

4. Students will be able to conduct experiments (prepare materials and equipment, use and specify measuring and test equipment, collect experimental data for analysis)

5. Students will be able to perform inferential statistical analysis through confidence intervals, test of hypothesis, ANOVA, Chi-squared, and regression

6. Students will be able to assess the risk of making wrong statistical recommendations

7. Students will be able to interpret (explain) data and make recommendations based on those interpretations

• For ME, See SS Criterion 3

Fred DePiero [email protected] What?

Page 28: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Effort to Identify Measurable Skills - Helpful

• Finding common ground on what is most important in your discipline– Provide focus & reinforcement

• Provide closer ties to curriculum.– Get better traction with faculty

• Skills seem trite? – Make more meaningful!

• Raise the bar, as Program improves

C3 How?

Page 29: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Broad Spectrum Of Outcomes Valuable

• Example: MD Team Skills (Help creative process)– A. Black, 1990 identified 32 traits of creative people.

Divergent thinking, flexible, risk taker, sensitive, severely critical…

– Less likely to have all 32 traits in a homogeneous team.– Hence MD teams are more likely to include qualities which

are in opposition.• Csikszentmihalyi, M.1999

• MD teams foster creativity, which plays a crucial role in concept generation and eng’g design.

• Would broad, non-traditional Outcomes be included, without ABET requirement?

C3 Benefits!

Page 30: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Direct Measures…

• Some centralized, some decentralized

• Some methods scalable

Fred DePiero [email protected] How?

Page 31: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Scalability Improves Sustainability for Direct Measures

• Decentralized Efforts• Involving course projects / assignments / exams

– Faculty Assessors examine sampling of student work, using rubrics. (Common approach). Requires rubric… All A-K?

– Capstone projects, group presentations, peer evals (rubric)– Embedded questions on common finals

• Student Analysis of Own Senior Project– Project styles may vary, but analysis can be consistent. All A-K!– LLL: ‘Identify new tools, abilities learned outside classroom’.

C3 How?

Page 32: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Scalability Improves Sustainability for Direct Measures

• Centralized Efforts• Writing Proficiency Exam

– Graduation requirement– Institutionalized process. Results in student’s record– Rubric/norming used by faculty, campus-wide

• Fundamentals in Eng’g Exam– A-K coverage limited. Used by some programs.

C3 How?

Page 33: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Assessment Instruments -Consider Worst-Case Measures

• “Locally-developed” exams– Inspired by ABET & IAB comments– Multiple-choice (scalable) or essay. – Required in course, contributes ~5% to grade– Summative, 4 years. All Outcomes covered. – Yields new information, beyond coursework.

• Results for Local Exams– Students: Fine or Bomb. No need to split hairs.

C3 How?

Page 34: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Assessment Instruments -Consider Worst-Case Measures

• Shock of summative direct measures!– Faculty provided questions that students failed to

answer, in a summative setting.– Results unexpected, hence faculty engaged.

• Student learning can be compartmentalized.– Often faculty view is in terms of courses. Need

verification of summative abilities.– Industrial Advisors raised this concern (disciplinary

skills, quiz given during interviews).

C3 How?

Page 35: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Criterion 4: Improvement• (C4-ABET for 2010/11-Proposed) The program must regularly use

appropriate, documented processes for evaluating the extent to which both the PEOs and student outcomes are being attained. The results of these evaluations must be utilized as input for the continuous improvement of the program. Other available information may also be used to assist in the continuous improvement of the program.

• My read:– Assessment data used to justify improvements– Continuously measure and improve– Other info OK (Good!)

• Does not say…– A given change must (+) impact the measure that had originally

inspired the change

C4 What?

Page 36: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Criterion 4: Improvement

• Logic of Justification: Many-to-Many Mapping– Based on many sources of info (typical, across univ)

• Justification could include:– Measurements AND– Faculty observation, IAB observation, surveys / exit

interviews (open ended)

• Don’t put ‘cart before horse’ regarding changes. – Look for a control group, and validating measurement– Deploy quiz/survey for validation purpose

C4 What?

Page 37: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

C4: Example from IE• Improvement Area #1: Statistics sequences: prerequisites and redundancy• Opportunity:

Data of 2006 senior exit interview showed that statistics sequence (STAT312 --> IME326 --> IME430) was much overlapped.

Instructors who teach IME326 also found IME326 course description was similar to STAT312.• Responsible Party: John ...• Root Causes:

Prior curriculum had a fault that IME326 and STAT312 had similar course outlines.

Lack of curricular flow chart showing length of sequences: Students realize too late that sequences of prerequisite courses are long

• Planned Activities and approximate timeline:

Action #1. Change pre-requisite of IME326 from STAT312 to STAT321 in catalog 2009-2011.

Action #2. Compare students’ performance in IME326 between who took STAT312 and STAT321 as pre-requisite through a pre-requisite quiz in the first week of IME326.

Action #3. Flow chart for IE and MfgE to be developed and made available to students

Action #4. Prerequisites to be enforced by computer

Action #5. Review Exit Exam and modify Stat questions, if necessary

Fred DePiero [email protected]

Page 38: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

C4: Example from IE• Improvement Area #1: Statistics sequences: prerequisites and redundancy…

• Actions Completed:

Action #1. Completed.

Action #2. two pre-requisite quizzes were conducted in Fall 2007 and Winter 2008. The effect of the pre-requisite course (STAT312 vs. STAT321) on the student performance in IME326 was studied in Fall 2007 and Winter 2008 quarters.

Action #3. Completed May 2007 (Pan)

Action #4. Completed November 2007

Action #5. Exit exam reviewed and several additional questions on stats knowledge added November 2007

• Evidence of Improvements:

Action #1. It seems that student performance in IME326 of those who took STAT321 is slightly better than that of those who took STAT312, though the improvement is not statistically significant. Note that the study was an observational study and the sample size of STAT321 was very small. Expected evidence in exit exams in 2008-2009 after catalog change takes effect regarding stat background consistency

• Other Examples: IE (pg 77), Iowa State (pg 58), ME (pg 131)

Fred DePiero [email protected]

Page 39: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

C4: Example from IE• Evaluating the level of attainment of Outcome 3A

• 3a) Students will attain an ability to apply knowledge of mathematics, science, and engineering

Alumni and industrial advisory board members both agree that 3(a) applying math/science/engineering knowledge is a fairly important outcome (ranked 6th out of 11 with a composite importance score of 4.12).

Advisory board members judge this outcome to be achieved at an acceptable level (3.9 out of 5). This is also considered an appropriate level based on its relative importance as rated by alumni and IAB members.

Employers of IE graduates also see the application of math/science/ engineering knowledge as being achieved at an acceptable (4.09) and appropriate (4.4 for importance) level. In addition, “general engineering skills” are identified as one of the most desirable skills from employer job postings.

Senior exam scores related to this outcome show that our graduates achieved better than acceptable level (62.37% correct answers on the exit exam). Overall our students’ performance related to 3(a) is acceptable and at an appropriate level given its importance.

Graduating seniors consistently rate their level of attainment of this outcome as well above an acceptable level (4.31 out of 5 on average). They are more confident with general math/science/engineering knowledge than with materials knowledge. Relatively, the scores are somewhat low given their importance, but they do not stand out from the other Outcomes based on the survey responses.

Senior Exit Interviews show that the curriculum may have an opportunity for improvement. Several comments related to how engineering statics and dynamics courses are related to the curriculum. It is not clear however, if this was primarily an issue for Industrial Engineering majors. The issue will continue to be monitored. Although few Industrial Engineering students take the FE/EIT exam, the exam results will continue to be monitored for signs of obvious trends in performance.

Based on these findings, this outcome appears to be achieved at an acceptable level … continuous improvement efforts are better focused elsewhere.

Fred DePiero [email protected]

Page 40: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Criterion 4: Use Dedicated Survey to Close Loop

• To help program improvement process, for quick loop closure!– Ask directly about impact in classroom – Possibilities? Priorities? Benefits?– Students, Faculty, Industry Advisors all partake & appreciate

giving input - Rank importance of changes.– Impact of some improvements diffuse, assess locally with a

control group. Rather than trying to measure benefit in a summative fashion (with seniors)!

C4 How?

Page 41: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

(My) Rubric Describing Quality of Improvements

• Very StrongMany improvements with clear ties to assessment data and at least

one follow-on measurement to close the loop using identical assessment instruments before and after the change

• StrongMany improvements with clear ties to assessment data and at least

one follow-on measurement to close the loop

• Good Improvements with ties to assessment data and some loose efforts

to close the loop

• OK (Still ‘OK’, but thin ice)Improvements with some ties to assessment data

C4 What?

Page 42: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Examples of Improvements• Added numerical methods based on FE Exam (direct, computers area) and

other indications. Since have observed a (slight) improvement in these scores. A similar change was justified/validated in the area of ethics.

• Increased emphasis on design in curriculum based on senior survey input. Surveys have since indicated better abilities in this area.

• Stepped up undergrad advising by assigning individual faculty to students, based on survey input. Surveys have since shown an improvement.

• Adjusted support courses and prerequisites related to statistics area, based on exit interviews.

• Improved advising of students regarding benefits of a graduate degree, based on input from Senior Surveys.

C4 What?

Page 43: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

C4: Centralize Common Improvements?

• One common improvement is gold!

• Students ‘Multidisciplinary Certified’– Pools co-curricular experiences to handle

throughput and establish solid mapping

• Tech Writing, Psychology– Strengthened curricular mapping

C4 How?

Page 44: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Criterion 5: Curriculum

• Prerequisite Issues– ‘Basic science’ vs ‘Engineering Science’

vs ‘Engineering Design’– Watch prereqs of capstone courses

• Identify standards and constraints in Self Study

Fred DePiero [email protected] What?

Page 45: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Culture Change: Data-Driven Decision Making

• “Others are doing X. Let’s do that too!”– What determines success? Use an anecdotal start & finish?

Or use multi-year measures.– How to prioritize possible options for changes?

• “We know water is wet. Why measure?”– Consistent benchmarks over many years.– Avoid anecdotal decisions

• For ABET Evaluator: Prove water is wet.– Assessment evidence beneficial when preparing publication

on innovative teaching methods for peer review.

• Goal: Data with ‘pedagogical significance’ rather than emphasizing statistical significance. Use judgment.

Benefits!

Page 46: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Use Technology Appropriately

• “Where can I put all this info?”• Document mgmt - (e.g. via BlackBoard)• One Bb course per Program

– Create template with slots to fill

• Posted items document time line (e.g. surveys)

• Bb system supported by university, with training for users

• Provided PEVs with access prior to visit

How?

Page 47: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Use Technology Appropriately

• Wish List• New survey tool:

– Link to campus data for email addresses– Program-specific questions & deployment– Tag questions to outcomes (ABET & university)– Agglomerate results at program, college &

university levels (for ABET & WASC)

How?

Page 48: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Seek Partners Across Campus

• Career Services: Employer surveys (on C3)– New likert scale (*)

• Not important for entry‐level• Weak –but no concern• Weak –needs strengthening• Sufficient. No Concerns• Strong• Strong –needs more emphasis

– Example provided– Careful: ‘Abilities at start of employment…’

• (*) From Bob Warrington

How?

Page 49: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Seek Partners Across Campus

• Institutional Planning & Assessment

• Writing Proficiency Exam, grad requirement

• NSSE + FSSE… (via WASC) – A 4th, 5th leg of stool? – Might reveal a problem worth exploring? – Might confirm an improvement under consideration?

How?

Page 50: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Seek Partners Across Campus

• GE Community– Help firm-up curricular mapping (Benefits all

programs!)– Win-win, liberal arts instructors emphasize need to

engineering students– Math/Science: Understand common needs and

expectations.• Who has ‘quantitative reasoning’ on their syllabus (None)• Who helps introduce, develop & master this (Many)

– Liberal Arts: ‘Transferable skills’

How?

Page 51: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Seek Partners Across Campus

• Campus-wide assessment of ULOs– University Learning Objectives– Map well with A-K– Part of WASC / Regional accreditation efforts

How?

Page 52: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Seek Synergy with Regional Accreditation Efforts

– WASC: Regional accreditation for campus– Defining measurable skills, associated with

University Learning Objectives• Written communication• Life-Long Learning (incl. information competency)• Others in subsequent years

– Teams examining • Curriculum map (incl. gen. ed.)• Measurable skills• Defining rubrics• Augmenting instruction in required gen ed courses• Institutionalizing direct measure in gen ed course

How?

Page 53: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Seek Synergy with Regional Accreditation Efforts

• WASC self-study less structured than ABET. Define themes.

• Theme: Integrated nature of student learning– Contributions of Major + GE + Co-Curricular– Co-Curricular contribution:

• Housing / student life have learning objectives!• (GE has learning objectives)

– Survey on student employment surprising• Students identified campus employment as

contributing to ULO attainment.

How?

Page 54: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

ABET Offers Many Resources

• ABET Symposium 2010 (was BAP XII, XI…)• ‘Institutional Representatives Day’

– Pre-, Post-Visit

• ABET web site & contact form• Become an ABET Evaluator

– Attend Evaluator training (in Hawaii!)– Keep an eye on ABET.org (requirements evolve)

• Programs work together, central coordination. Share best practices.

How?

Page 55: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Finally - Perspective

• “We can still learn from an imperfect assessment process.”– Bok, ‘Our Underachieving Colleges’

• A ‘Learn-by-Doing’ process!

How?

Page 56: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Goal: Data-Driven Curriculum RevisionAdopted by Faculty

• Imagine a faculty member saying…• Past

– ‘I changed my course. I know its better.’

• Now– ‘I changed my course. I need to collect data for ABET.’

• Future– ‘I think I need to improve my course. I need data to…– … assess severity of perceived problems.’– … assess impact of possible changes.’– … assess benefits of improvements.’

Benefits!

Page 57: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

What is the Value of ABETTo Faculty & Students?

• Encourages Scholarly Teaching Practices• ABET Evaluators are trained to ask,

“How do you know good (communication) skills when you see them?”

• Good Answers, for ABET– “We defined skills and use a (test/quiz…)” – “We have a rubric to evaluate student work”

• Bad Answer, for ABET– “I know it when I see it!”

• Beneficial, provides consistent evaluation.

Benefits!

Page 58: ABET Processes What Needs to Be Done? How Can It All Be Accomplished? What Are the Benefits?

Colle

ge o

f En

gin

eerin

g

Fred DePiero [email protected]

Conclusion: Develop Sustainable Processes

• Centralize effort where appropriate– Clearly articulate expectations for

centralized / decentralized efforts

• Leverage campus data sources and processes

• Seek partners in assessment across campus