kevin matthews thomas janicki university of north carolina wilmington
TRANSCRIPT
DEVELOPMENT AND PRELIMINARY RESULTS OF AN ADAPTIVE GRADING/LEARNING SYSTEM (AGLS)
Kevin MatthewsThomas Janicki
University of North Carolina Wilmington
Background
Literature Review
Benefits and Need
System Development
Hypothesis
Limitations
Conclusions
Overview
Increased need for computer literacy skills
◦ National Science Foundation (NSF)◦ National Research Council (NRC)◦ College requirements for literacy
Background
Increased class size as budgets are cut, may lead to:
◦ Fewer assignments◦ Simpler assignments◦ Less personalized feedback◦ Longer delay in returning grading◦ Increase plagiarism
Problems
Question?How could technology assist the instructor to increase student learning in computer literacy courses?
Literature reviewNine elements that should be present in order for learning to occur (Gagne, Briggs, Wager)
Gain attention
Inform learner on the objective
Stimulation recall of prior learning
Presenting the stimulus
Provide learning guidance
Eliciting performance
Providing feedback
Assessing performance
Enhancing retention and transfer
*most related to success(Martin Klein, Sullivan)
Contiguity
Repetition
Feedback
(Gagne, Briggs, Wager)
Behavior Learning Theory
Traditional Learning Resource Based Learning
Teacher as expert model Teacher as facilitator / guide
Textbook as primary source Variety of sources / media
Facts as primary Questions as primary
Information is packaged Information is discovered
Emphasis on product Emphasis on process
Assessments is quantitative Assessment is qualitative / quantitative
Resource Based Learning
Rakes
Need for automation
Using course management software
◦ 30% of graded assignments had NO feedback Excluded perfect score assignments
◦ Average of 28 days to score an assignment!
◦ Various ‘grading keys’ when done manually
◦ These do not lead to learning
Feedback comments/response
To be effective, the student’s response to concepts should immediately follow instruction.
Multiple assignments of similar nature should be presented repetitively to reinforce new material presented during a lesson.
Quick and customized feedback allows a student to identify correct answers and see errors in incorrect answers.
The best way to reach this goal of student success is to adopt a resource-based approach to learning.
Key criteria to increase learning
Build or ‘buy’ a solution
Benefits/Features AGLS Case-based Procedural Test-BankChallenging, real-world problems ■ ■
Automated grading ■ ■ ■ ■
Consistent grading ■ ■ ■ ■
Instant feedback ■ ■ ■ ■
Customized feedback ■
Web interface/portal ■ ■ ■ ■
Multiple skills assessed concurrently ■ ■
Hands-on experience ■ ■ ■
Smaller one-skill problems ■ ■ ■
Question/assignment library ■ ■ ■
Reduced preparation/paperwork time for instructor ■ ■ ■ ■
Availability of student reporting ■ ■ ■ ■
Expandable answer banks ■
Repository for file submissions ■ ■
Plagiarism detection ■ ■Instructor created exercises ■ ■
LimitationsAGLS
Case-based
Procedural
Test-Bank
Answers must be exact matches
■ ■ ■
Limited number of cases ■ ■
Textbook/supplemental required
■ ■
Software must be installed ■
“Simulated” environment ■
Other purchases required ■ ■ ■
System design
AGLS(Grading Modules)
Grading Key
Student Files
Service Requested
Score
Correct Answers
Incorrect Answers
Feedback
INPUT OUTPUT
Key (ideal) correct answer given
Student answer differs
Instructor prompted with correct answer and student answer
Instructor deems an answer as correct or incorrect
An answer is only seen once
Excel – alternative answers
Key (ideal) names given for tables, fields, etc.
Students may have typos or misspellings
Instructor prompted with all table names, field names, etc.
Instructor deems a name acceptable.
A penalty may be given.
Access – naming alternative
Data Gathering
◦ Existing grade book of feedback comments and response time
◦ Expert panel to evaluate quality of ‘feedback comments’
Hypothesis and experiment
Hypotheses
H0 (μ1 – μ2 = 0): The use of the AGLS will not affect the amount of feedback provided to students.
H1 (μ1 – μ2 < 0): The use of the AGLS will increase the amount of feedback provided to students.
1. Affect on Quantity of Feedback
Control Experiment
Mean 45.10 71.36
Sample Size 628 3138
t statistic -6.57
Two tail p value 5.8 x 10-11
H0 :The use of the AGLS will not affect the quality of feedback provided to students.
H1 : The use of the AGLS will increase the quality of feedback provided to students.
2. Affect on Quality of Feedback
Control Experiment
Mean 3.27 3.45
Sample Size 49 51
t statistic -1.13
Two tail p value .26
H0 :The use of the AGLS will not affect the time for an assignment to be graded.
H1 (μ1 – μ2 < 0): The use of the AGLS will decrease the time for an assignment to be graded.
3. Affect on Response Time
Control Experiment
Mean 28.59 8.13
Sample Size 628 3138
t statistic 33.43
Two tail p value 9.7353 x 10-215
The use of the AGLS will increase the amount of feedback provided to students.
The use of the AGLS will NOT affect the quality of feedback provided to students.
The use of the AGLS will decrease the time for an assignment to be graded.
Conclusions
Only Access and Excel 2007
Only certain items were initially included to be graded
Limitations
Web Service
Refined library of assignments and correct/incorrect answers
Additional features to be graded
Testing outside of introductory computer course domain
Future work
Questions