a streamlined, quantitative, web-based, course-level assessment process jörg moßbrucker glenn...
Post on 16-Jan-2016
212 Views
Preview:
TRANSCRIPT
A Streamlined, A Streamlined, Quantitative, Web-based,Quantitative, Web-based,Course-Level Assessment Course-Level Assessment
ProcessProcess
Jörg MoßbruckerJörg MoßbruckerGlenn WrateGlenn Wrate
Mike O’DonnellMike O’DonnellMilwaukee School of EngineeringMilwaukee School of Engineering
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 22
OverviewOverview
Do what you say
Prove itMeasure what you do
Implement changes
to improve
Say what you do
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 33
Say what you do, do what you Say what you do, do what you saysay
Detailed course descriptionsDetailed course descriptions– Course outlineCourse outline– Course prerequisites by topic and Course prerequisites by topic and
coursecourse– Course outcomesCourse outcomes– Course topicsCourse topics– Day-by-day descriptions of lecture and Day-by-day descriptions of lecture and
lab sessionslab sessions
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 44
Example: Circuits IExample: Circuits I
Prerequisite by topics:Prerequisite by topics:
Differentiation of algebraic and Differentiation of algebraic and trigonometric functions trigonometric functions
Solution of a system of linear Solution of a system of linear equations using a calculator and/or equations using a calculator and/or computer computer
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 55
Example cont: Course Example cont: Course OutcomesOutcomes
Upon successful completion of this course, the student will:Upon successful completion of this course, the student will:
Demonstrate problem solving skills Demonstrate problem solving skills Demonstrate a standard of expertise in the understanding Demonstrate a standard of expertise in the understanding
of circuit laws and in the analysis of electrical circuit of circuit laws and in the analysis of electrical circuit Write and solve KCL and KVL equations using standard Write and solve KCL and KVL equations using standard
methods of circuit analysis for CD circuits methods of circuit analysis for CD circuits Simplify networks using source transformations and Simplify networks using source transformations and
Thevenin's/Norton's theorems Thevenin's/Norton's theorems Use the superposition principle in circuit analysis Use the superposition principle in circuit analysis Demonstrate calculator skills to solve circuit equations Demonstrate calculator skills to solve circuit equations Demonstate the ability to analyze DC circuits using SPICE Demonstate the ability to analyze DC circuits using SPICE Demonstrate circuit laboratory skills and perform DC Demonstrate circuit laboratory skills and perform DC
measurements measurements
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 66
Example cont: Course Example cont: Course TopicsTopics
DC circuit laws and concepts (10 DC circuit laws and concepts (10 classes) classes)
DC circuit analysis (9 classes) DC circuit analysis (9 classes) Op-Amps (3 classes) Op-Amps (3 classes) Inductors/Capacitors (3 classes) Inductors/Capacitors (3 classes) Review (3 classes) Review (3 classes) Tests and quizzes (2 classes) Tests and quizzes (2 classes)
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 77
Example cont.Example cont.
All the above data plus day-by-day All the above data plus day-by-day description of lecture sessions and description of lecture sessions and suggested laboratory sessions are suggested laboratory sessions are stored in a MS Access databasestored in a MS Access database
Web description is automatically Web description is automatically created from databasecreated from database
Course coordinator is responsible for Course coordinator is responsible for datadata
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 88
Measure what you doMeasure what you do
Course assessment measures IN/OUTCourse assessment measures IN/OUT
IN: Assessment of student IN: Assessment of student performance data of prerequisite performance data of prerequisite topicstopics– Prerequ-exam, homework etc.Prerequ-exam, homework etc.
OUT: Assessment of student OUT: Assessment of student performance data of course outcomesperformance data of course outcomes– Final examFinal exam
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 99
AssessmentAssessment
Assessment of student performance Assessment of student performance data generated out of Prerequ. Exam data generated out of Prerequ. Exam and Final Examand Final Exam
Must generate meaningful dataMust generate meaningful data Must generate only a marginal Must generate only a marginal
increase in workload for facultyincrease in workload for faculty
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1010
Assessment cont.Assessment cont.
How can we generate assessment How can we generate assessment data out of already existing (exam) data out of already existing (exam) data?data?– Two-dimensional grading schemeTwo-dimensional grading scheme
How can we generate meaningful How can we generate meaningful data (i.e. is there a difference data (i.e. is there a difference between 80% and 83%)?between 80% and 83%)?– Reduction of input data rangeReduction of input data range
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1111
Assessment DataAssessment Data Data reduction by grouping student Data reduction by grouping student
performance into three categories:performance into three categories:
– AdequateAdequate understanding (excellent to understanding (excellent to satisfactory, 100% to app. 75%)satisfactory, 100% to app. 75%)
– MarginalMarginal understanding (below understanding (below satisfactory or poor)satisfactory or poor)
– InadequateInadequate understanding (failure) understanding (failure)
Good Just passed Clueless
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1212
Generation of dataGeneration of data
Student 1
Student 2
Problem 1 Problem 2 Problem 3 Grade
90% 75% 80%
50% 60% 75%
82%
63%
A
M
I
A AM
MI I
1
0
1
0
1
1
1
1
0
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1313
Generation of data cont.Generation of data cont.
A
M
I
1
0
1
0
1
1
1
1
0
Outcome 1
Outcome 2
Outcome 3
Problem 1 Problem 2 Problem 3
*
* *
*
A M I
1 0 1
1 2 1
1 1 0
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1414
MS Excel ExampleMS Excel Example
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1515
Cont.Cont.
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1616
Cont.Cont.
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1717
Cont.Cont.
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1818
Cont.Cont.
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 1919
Database EntryDatabase Entry
Web-based interfaceWeb-based interface– Allows direct entry of assessment dataAllows direct entry of assessment data– Enters data into MySQL databaseEnters data into MySQL database– Allows also entry of commentsAllows also entry of comments
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2020
Data Flow OverviewData Flow OverviewAccess DBWorkload
Access DBCourse Descr.
Progress DB
Online CourseDescriptions
MySQL DB forCourses/Year/
Quarters/Sections
Web DataInput
MySQLDatabase
AccessDatabase
AdobeAcrobat
Report in pdf
Multiple ScriptsIn AccessReports/Queries
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2121
Result ExampleResult Example
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2222
Implement Changes to Implement Changes to ImproveImprove
What?What? How?How? When?When? Who?Who?
What?What?
– Course outcomes with the lowest Course outcomes with the lowest “Adequate” scores over one academic “Adequate” scores over one academic yearyear
– Collated and weighted results over all Collated and weighted results over all sections over one academic year sections over one academic year reduces noise considerablyreduces noise considerably
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2323
Cont.Cont.
How?How?
– Change of course topics/focusChange of course topics/focus– Change of lab sessionsChange of lab sessions
……– Removal of unnecessary course Removal of unnecessary course
outcomesoutcomes
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2424
Cont.Cont.
When and who?When and who?
– At end of academic year assessment team At end of academic year assessment team assembles results and determines “critical” assembles results and determines “critical” course outcomes requiring actioncourse outcomes requiring action
– Assessment team focuses on “tracks” in EE, Assessment team focuses on “tracks” in EE, such as:such as: Digital, Analog, Circuits, Communications etc.Digital, Analog, Circuits, Communications etc.
– Course coordinators implement changes over Course coordinators implement changes over summersummer
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2525
ExampleExampleChange of a 2-course sequence in circuits without lab to a 3-course sequence with labChange of a 2-course sequence in circuits without lab to a 3-course sequence with lab
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2626
ConclusionsConclusions Assessment data generated directly from student Assessment data generated directly from student
performance data already availableperformance data already available Data reduction by grouping into three categoriesData reduction by grouping into three categories Instant graphical feedback to teaching faculty on Instant graphical feedback to teaching faculty on
performance of their sectionperformance of their section Simple, web-based interface to input assessment Simple, web-based interface to input assessment
datadata One-click calculation of assessment data of multiple One-click calculation of assessment data of multiple
sections/quarters/yearssections/quarters/years Assessment team determines annually action neededAssessment team determines annually action needed Course coordinators implement changesCourse coordinators implement changes Data provides numerical feedback of changesData provides numerical feedback of changes Program director has overview over course Program director has overview over course
outcomes data for program outcomesoutcomes data for program outcomes
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2727
Issues (past and present)Issues (past and present)
No. of prerequisites usually small -> No. of prerequisites usually small -> noise exceptionally largenoise exceptionally large– Common prerequisite exam for all sections Common prerequisite exam for all sections
of a course determined by course of a course determined by course coordinatorcoordinator
Different sections behave very Different sections behave very differentlydifferently– No common final examNo common final exam– Instructor understanding of performanceInstructor understanding of performance– Section/time/student etc. dependentSection/time/student etc. dependent– Adds to overall noiseAdds to overall noise
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 2828
Cont.Cont.
Reduction to a very small set of Reduction to a very small set of outcomes:outcomes:
breadth vs. depthbreadth vs. depth
??
Dr.-Ing. Joerg Mossbrucker, MSOE Dr.-Ing. Joerg Mossbrucker, MSOE 20062006 3030
OverviewOverview Things I need:Things I need: General process (do what you say, measure what you do, improve)General process (do what you say, measure what you do, improve) Do what you sayDo what you say
– Course descriptionsCourse descriptions– Course outcomesCourse outcomes– Course topicsCourse topics– Role of the course coordinatorRole of the course coordinator– Example of course on-lineExample of course on-line– Where is the data storedWhere is the data stored
Measure what you doMeasure what you do– Different methods of assessmentDifferent methods of assessment– In/Out measuresIn/Out measures– Advantages/disadvantagesAdvantages/disadvantages– Numerical/non-numerical dataNumerical/non-numerical data– Common exams yes/noCommon exams yes/no– How can we use data already collected for a class?How can we use data already collected for a class?– Is there a difference between 85% and 80%? If so how muchIs there a difference between 85% and 80%? If so how much– -> reduction of in data (adequate/marginal/inadequate)-> reduction of in data (adequate/marginal/inadequate)– What do these mean?What do these mean?– How to get it out of final pre-requ?How to get it out of final pre-requ?– Multiple sources for a single course outcome from different questionsMultiple sources for a single course outcome from different questions– Computed automatically in Excel (example)Computed automatically in Excel (example)– How do we get the computed data and resultsHow do we get the computed data and results– Databases usedDatabases used– Examples of resultsExamples of results
ImproveImprove– Different possibilities how to improveDifferent possibilities how to improve– When to improveWhen to improve– Who decides when whatWho decides when what– Assessment processAssessment process
IssuesIssues– HonestyHonesty– Integrity of the course coordinatorIntegrity of the course coordinator– NoiseNoise– Still too much dataStill too much data
top related