a streamlined, quantitative, web-based, course-level assessment process jörg moßbrucker glenn...

A Streamlined, A Streamlined, Quantitative, Web-based,Quantitative, Web-based,Course-Level Assessment Course-Level Assessment

ProcessProcess

Jörg MoßbruckerJörg MoßbruckerGlenn WrateGlenn Wrate

Mike O’DonnellMike O’DonnellMilwaukee School of EngineeringMilwaukee School of Engineering

Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 22

OverviewOverview

Do what you say

Prove itMeasure what you do

Implement changes

to improve

Say what you do


Say what you do, do what you Say what you do, do what you saysay

Detailed course descriptionsDetailed course descriptions– Course outlineCourse outline– Course prerequisites by topic and Course prerequisites by topic and

coursecourse– Course outcomesCourse outcomes– Course topicsCourse topics– Day-by-day descriptions of lecture and Day-by-day descriptions of lecture and

lab sessionslab sessions


Example: Circuits IExample: Circuits I

Prerequisite by topics:Prerequisite by topics:

Differentiation of algebraic and Differentiation of algebraic and trigonometric functions trigonometric functions

Solution of a system of linear Solution of a system of linear equations using a calculator and/or equations using a calculator and/or computer computer


Example cont: Course Example cont: Course OutcomesOutcomes

Upon successful completion of this course, the student will:Upon successful completion of this course, the student will:

Demonstrate problem solving skills Demonstrate problem solving skills Demonstrate a standard of expertise in the understanding Demonstrate a standard of expertise in the understanding

of circuit laws and in the analysis of electrical circuit of circuit laws and in the analysis of electrical circuit Write and solve KCL and KVL equations using standard Write and solve KCL and KVL equations using standard

methods of circuit analysis for CD circuits methods of circuit analysis for CD circuits Simplify networks using source transformations and Simplify networks using source transformations and

Thevenin's/Norton's theorems Thevenin's/Norton's theorems Use the superposition principle in circuit analysis Use the superposition principle in circuit analysis Demonstrate calculator skills to solve circuit equations Demonstrate calculator skills to solve circuit equations Demonstate the ability to analyze DC circuits using SPICE Demonstate the ability to analyze DC circuits using SPICE Demonstrate circuit laboratory skills and perform DC Demonstrate circuit laboratory skills and perform DC

measurements measurements


Example cont: Course Example cont: Course TopicsTopics

DC circuit laws and concepts (10 DC circuit laws and concepts (10 classes) classes)

DC circuit analysis (9 classes) DC circuit analysis (9 classes) Op-Amps (3 classes) Op-Amps (3 classes) Inductors/Capacitors (3 classes) Inductors/Capacitors (3 classes) Review (3 classes) Review (3 classes) Tests and quizzes (2 classes) Tests and quizzes (2 classes)


Example cont.Example cont.

All the above data plus day-by-day All the above data plus day-by-day description of lecture sessions and description of lecture sessions and suggested laboratory sessions are suggested laboratory sessions are stored in a MS Access databasestored in a MS Access database

Web description is automatically Web description is automatically created from databasecreated from database

Course coordinator is responsible for Course coordinator is responsible for datadata


Measure what you doMeasure what you do

Course assessment measures IN/OUTCourse assessment measures IN/OUT

IN: Assessment of student IN: Assessment of student performance data of prerequisite performance data of prerequisite topicstopics– Prerequ-exam, homework etc.Prerequ-exam, homework etc.

OUT: Assessment of student OUT: Assessment of student performance data of course outcomesperformance data of course outcomes– Final examFinal exam


AssessmentAssessment

Assessment of student performance Assessment of student performance data generated out of Prerequ. Exam data generated out of Prerequ. Exam and Final Examand Final Exam

Must generate meaningful dataMust generate meaningful data Must generate only a marginal Must generate only a marginal

increase in workload for facultyincrease in workload for faculty


Assessment cont.Assessment cont.

How can we generate assessment How can we generate assessment data out of already existing (exam) data out of already existing (exam) data?data?– Two-dimensional grading schemeTwo-dimensional grading scheme

How can we generate meaningful How can we generate meaningful data (i.e. is there a difference data (i.e. is there a difference between 80% and 83%)?between 80% and 83%)?– Reduction of input data rangeReduction of input data range


Assessment DataAssessment Data Data reduction by grouping student Data reduction by grouping student

performance into three categories:performance into three categories:

– AdequateAdequate understanding (excellent to understanding (excellent to satisfactory, 100% to app. 75%)satisfactory, 100% to app. 75%)

– MarginalMarginal understanding (below understanding (below satisfactory or poor)satisfactory or poor)

– InadequateInadequate understanding (failure) understanding (failure)

Good Just passed Clueless


Generation of dataGeneration of data

Student 1

Student 2

Problem 1 Problem 2 Problem 3 Grade

90% 75% 80%

50% 60% 75%

82%

63%

A

M

I

A AM

MI I

1

0

1

0

1

1

1

1

0


Generation of data cont.Generation of data cont.

A

M

I

1

0

1

0

1

1

1

1

0

Outcome 1

Outcome 2

Outcome 3

Problem 1 Problem 2 Problem 3

*

* *

*

A M I

1 0 1

1 2 1

1 1 0


MS Excel ExampleMS Excel Example


Cont.Cont.


Database EntryDatabase Entry

Web-based interfaceWeb-based interface– Allows direct entry of assessment dataAllows direct entry of assessment data– Enters data into MySQL databaseEnters data into MySQL database– Allows also entry of commentsAllows also entry of comments


Data Flow OverviewData Flow OverviewAccess DBWorkload

Access DBCourse Descr.

Progress DB

Online CourseDescriptions

MySQL DB forCourses/Year/

Quarters/Sections

Web DataInput

MySQLDatabase

AccessDatabase

AdobeAcrobat

Report in pdf

Multiple ScriptsIn AccessReports/Queries


Result ExampleResult Example


Implement Changes to Implement Changes to ImproveImprove

What?What? How?How? When?When? Who?Who?

What?What?

– Course outcomes with the lowest Course outcomes with the lowest “Adequate” scores over one academic “Adequate” scores over one academic yearyear

– Collated and weighted results over all Collated and weighted results over all sections over one academic year sections over one academic year reduces noise considerablyreduces noise considerably


Cont.Cont.

How?How?

– Change of course topics/focusChange of course topics/focus– Change of lab sessionsChange of lab sessions

……– Removal of unnecessary course Removal of unnecessary course

outcomesoutcomes


Cont.Cont.

When and who?When and who?

– At end of academic year assessment team At end of academic year assessment team assembles results and determines “critical” assembles results and determines “critical” course outcomes requiring actioncourse outcomes requiring action

– Assessment team focuses on “tracks” in EE, Assessment team focuses on “tracks” in EE, such as:such as: Digital, Analog, Circuits, Communications etc.Digital, Analog, Circuits, Communications etc.

– Course coordinators implement changes over Course coordinators implement changes over summersummer


ExampleExampleChange of a 2-course sequence in circuits without lab to a 3-course sequence with labChange of a 2-course sequence in circuits without lab to a 3-course sequence with lab


ConclusionsConclusions Assessment data generated directly from student Assessment data generated directly from student

performance data already availableperformance data already available Data reduction by grouping into three categoriesData reduction by grouping into three categories Instant graphical feedback to teaching faculty on Instant graphical feedback to teaching faculty on

performance of their sectionperformance of their section Simple, web-based interface to input assessment Simple, web-based interface to input assessment

datadata One-click calculation of assessment data of multiple One-click calculation of assessment data of multiple

sections/quarters/yearssections/quarters/years Assessment team determines annually action neededAssessment team determines annually action needed Course coordinators implement changesCourse coordinators implement changes Data provides numerical feedback of changesData provides numerical feedback of changes Program director has overview over course Program director has overview over course

outcomes data for program outcomesoutcomes data for program outcomes


Issues (past and present)Issues (past and present)

No. of prerequisites usually small -> No. of prerequisites usually small -> noise exceptionally largenoise exceptionally large– Common prerequisite exam for all sections Common prerequisite exam for all sections

of a course determined by course of a course determined by course coordinatorcoordinator

Different sections behave very Different sections behave very differentlydifferently– No common final examNo common final exam– Instructor understanding of performanceInstructor understanding of performance– Section/time/student etc. dependentSection/time/student etc. dependent– Adds to overall noiseAdds to overall noise


Cont.Cont.

Reduction to a very small set of Reduction to a very small set of outcomes:outcomes:

breadth vs. depthbreadth vs. depth


OverviewOverview Things I need:Things I need: General process (do what you say, measure what you do, improve)General process (do what you say, measure what you do, improve) Do what you sayDo what you say

– Course descriptionsCourse descriptions– Course outcomesCourse outcomes– Course topicsCourse topics– Role of the course coordinatorRole of the course coordinator– Example of course on-lineExample of course on-line– Where is the data storedWhere is the data stored

Measure what you doMeasure what you do– Different methods of assessmentDifferent methods of assessment– In/Out measuresIn/Out measures– Advantages/disadvantagesAdvantages/disadvantages– Numerical/non-numerical dataNumerical/non-numerical data– Common exams yes/noCommon exams yes/no– How can we use data already collected for a class?How can we use data already collected for a class?– Is there a difference between 85% and 80%? If so how muchIs there a difference between 85% and 80%? If so how much– -> reduction of in data (adequate/marginal/inadequate)-> reduction of in data (adequate/marginal/inadequate)– What do these mean?What do these mean?– How to get it out of final pre-requ?How to get it out of final pre-requ?– Multiple sources for a single course outcome from different questionsMultiple sources for a single course outcome from different questions– Computed automatically in Excel (example)Computed automatically in Excel (example)– How do we get the computed data and resultsHow do we get the computed data and results– Databases usedDatabases used– Examples of resultsExamples of results

ImproveImprove– Different possibilities how to improveDifferent possibilities how to improve– When to improveWhen to improve– Who decides when whatWho decides when what– Assessment processAssessment process

IssuesIssues– HonestyHonesty– Integrity of the course coordinatorIntegrity of the course coordinator– NoiseNoise– Still too much dataStill too much data

a streamlined, quantitative, web-based, course-level assessment process jörg moßbrucker glenn...

Documents

classes example cont

classes dc circuit analysis

existing exam data

understanding of circuit

facultyassessment cont

circuit laboratory skills

analysis of electrical

course topicsdc circuit