some scaling laws for mooc assessments - aspiring minds · some scaling laws for mooc assessments...
TRANSCRIPT
Some Scaling Laws for
MOOC Assessments
Nihar B. Shah
Joint work with: J. Bradley, S. Balakrishnan,
A. Parekh, K. Ramchandran, M. J. Wainwright
MOOCs
Information Dissemination Scales Well
?
?
??
Assessment & Feedback Not Easy
Auto-Grading
What is the name of this workshop?
○ Assess○ Recess○ Digress○ Matress
✓
Restricted Applicability
Need human participationfor subjective topics
Peer-Grading
Peer-Grading
A+
B-
C+
B+
B-
Aggregate
Peer-Grading
A+
B-
C+
B+
B-
Aggregate
Potential to scale: Number of graders scales automatically with the number of students!
Coursera HCI 1
A+
B-
C+
B+
B-
Median
Many Errors Observed
Other Aggregation Algorithms
Piech et al. ‘13
Gutierrez et al. ‘14
Walsh ‘14
Díez et al., ‘13
Other Aggregation Algorithms
No Guarantees
Piech et al. ‘13
Gutierrez et al. ‘14
Walsh ‘14
Díez et al., ‘13
Scalable Peer-grading?
Which peer-grading algorithms can guaranteethat the expected fraction of students misgradedgoes to zero (as the class size becomes large)?
None – no aggregation algorithm can give such a guarantee.
(when peer-grading is used as a standalone)
Impossibility Result
THEOREM
If average grading ability of students is invariant to d then theexpected fraction of students misgraded under any peer-gradingalgorithm is lower bounded by a constant c > 0 (independent of d).
Let d = number of students
• The constant c depends on the ability of the graders
• The results holds even if instructor grades a constant fraction of submissions
Impossibility ResultLet d = number of students
Intuition:• Due to noisy graders, many errors when d is small
• When d is large, want to use largeness of system to combat noise
• Although #graders increases with d, the number of submissionsto be graded also increases proportionally
• For any individual student, there is no “improvement” in thepeer-grading system as d increases
How to Make Peer-grading Scalable?
Dimensionality reduction!(Clustering)
And then peer-grade.
Cluster Submissions…
Cluster Submissions… Then Peer-grade
Theoretical Guarantee
THEOREM
If the d submissions can be clustered into at most d/log(d)clusters with at most o(d) errors, then the expected fraction ofstudents misgraded goes to zero as d gets large.
Theoretical Guarantee
Intuition:• Each submission graded by log(d) students. Grows as d increases.• d is large, so aggregate reliable even if graders extremely noisy.
THEOREM
If the d submissions can be clustered into at most d/log(d)clusters with at most o(d) errors, then the expected fraction ofstudents misgraded goes to zero as d gets large.
Clustering: In Practice…Active topic of research
“Powergrading”Basu et al. ‘13
Brooks et al. ’14 “ACES”Rogers et al. ‘14
Essay GradingLarkey ‘98
“Codewebs”Nguyen et al. ‘14
“Overcode”Glassman et al. ‘14
Clustering: In Theory…
Do they belong tothe same cluster?
Submission 1
Submission 2Yes/No
Correct with probability ≥ ½ + δ(for some δ > 0)
Suppose there are d/log(d) or fewer underlying clusters. Suppose there is an algorithm such that:
Then the expected fraction of students misgraded goes to zero as the number of students becomes large.
THEOREM
Clustering: In Theory…
Takeaway: Suffices to design a just-better-than-random comparator
Do they belong tothe same cluster?
Submission 1
Submission 2Yes/No
Correct with probability ≥ ½ + δ(for some δ > 0)
Suppose there are d/log(d) or fewer underlying clusters. Suppose there is an algorithm such that:
Then the expected fraction of students misgraded goes to zero as the number of students becomes large.
THEOREM
Summary: Peer-grading in MOOCs
• Most literature is empirical, we take a statistical approach
• Takeaways:
1) Peer-grading as a standalone does not scale
2) Dimensionality reduction + peer-grading can scale
3) Any better-than-random comparator suffices