labrador: a tool for automated grading support in multi-section courses
DESCRIPTION
Labrador: A Tool for Automated Grading Support in Multi-Section Courses. Drexel University Programming Learning Experience (DUPLEX) http://duplex.mcs.drexel.edu Christopher D. Cera, Robert N. Lass, Bruce Char, Jeffrey L. Popyack, Nira Herrmann, Paul Zoski Drexel University - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/1.jpg)
Labrador: A Tool for Automated Grading Support in Multi-Section Courses
Drexel University ProgrammingLearning Experience (DUPLEX)http://duplex.mcs.drexel.edu
Christopher D. Cera, Robert N. Lass, Bruce Char, Jeffrey L. Popyack, Nira Herrmann, Paul Zoski
Drexel UniversityMathematics and Computer Science
![Page 2: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/2.jpg)
2
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
![Page 3: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/3.jpg)
3
The Group
Bruce Char– Professor, Computer Science
Nira Herrmann– Professor and Head, Mathematics
Jeffrey L. Popyack– Associate Professor, Computer Science
Paul Zoski– Instructor, Math and Computer Science
Christopher D. Cera– Computer Science Graduate Student
Robert N. Lass– Computer Science Undergraduate Student
![Page 4: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/4.jpg)
4
Who Am I
First TA in MCS to experiment with WebCT in December 2000
Courses I have TA’ed using WebCT– 3 x Introductory Programming I, II– 1 x Object Oriented Programming
Migrated course content to WebCT from previous course website– HTML assignments and labs into question database– Gradebook maintainer
Wrote demo’s and documentation to train other TAs
Developer of software supplements to WebCT to support course administration
![Page 5: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/5.jpg)
5
The Course
Intro Programming I, II
Variety of Majors – Computer Science– Computer
Engineering– Information Systems– Mathematics– Digital Media
Class– 1 x 1 hour lecture– 2 x 1 hour labs
People– 250-300 Students– 2-3 Professors– 10-12 Teaching
Assistants
![Page 6: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/6.jpg)
6
The Duplex Project: An Overview
Take advantage of advances in Information Technology to improve instruction and reduce costs for computer programming courses
– Modular Structure Multiple Entry Points Multiple Audiences Multiple levels of knowledge (Bloom’s Taxonomy)
– Computer Supported Cooperative Work (CSCW) in student labs
– Online Services – Today’s Topic
![Page 7: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/7.jpg)
7
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
![Page 8: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/8.jpg)
8
Before WebCT
Hundreds of people involved in paper exchanges of handwritten assignments and quizzes
Testing programs required floppy exchanges No Chat No newsgroup-style threads Feedback and grades are not online
![Page 9: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/9.jpg)
9
Before WebCT
Large Classes
– Managing several hundred students is difficult due to the logistics involved with organizing paperwork and people.
![Page 10: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/10.jpg)
10
WebCT
Service from Information Resources & Technology (IRT) department (v3.1)
MCS started Dec. 2000 (v3.5)
Upgraded to v3.7 recently
Transition to v3.8 planned in Fall
![Page 11: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/11.jpg)
11
Noteworthy Features
General Course Website
Labs: Online Quizzes with Automated Grading
Homework: Online Assignments
Joint Staff--Student Chat
Discussion Threads
![Page 12: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/12.jpg)
13
Software Design Goals
Bulk Assignment Downloading (prior to 3.7)
Bulk Quiz Downloading
Section Sorting for bulk downloads, or only one section
![Page 13: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/13.jpg)
14
Software Design Goals
Post-Processing student files– Archive Extraction (tar, zip)– Decompress (gz, zip)– Decode (uue)
Minimal staff intervention when transferring submissions between systems, eg. Plagiarism Detection Systems (PDS)
Automatically collate source code
Generate electronic documents to facilitate grading and archiving
Remote Execution– downloading to computer x (on
campus) while operating at computer y (off campus)
![Page 14: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/14.jpg)
15
The Bigger Picture
Course-Specific Tasks– Not all processing should be done on the server
Select files have to be transferred to a different system for further processing and analysis by staff
Client-side support is needed to perform this, preferably automated and not necessarily using a web browser.
![Page 15: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/15.jpg)
16
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
![Page 16: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/16.jpg)
17
Labrador: Our Solution
Client-side WebCT Supplement
Implemented in Perl
Works for users with TA and Designer access to WebCT
![Page 17: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/17.jpg)
18
Perl
Practical Extraction and Reporting Language
Created in 1987 to replicate Unix shell functionality
Powerful text manipulation
High-level: rapid prototyping and development
Large library of modules and active development community
Perl is used by many applications, including WebCT
![Page 18: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/18.jpg)
19
A Taste of Labrador Perl Programming
![Page 19: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/19.jpg)
20
Labrador works on Windows, Mac, Linux, Other Unix Variants, etc.
Perl has been ported to all major operating systems
Perl programs can be compiled:
– No need for the end user to install Perl
– Perl2exe for Windows support (www.perl2exe.com)
![Page 20: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/20.jpg)
21
User Interface
Different users / Different UI preferences
Command-line
Interactive
Configuration File
GUI (in development)
![Page 21: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/21.jpg)
22
Required Info
http://webct.drexel.edu/SCRIPT/CS164_Fall2002/scripts/designer/dropbox_edit.pl?DROPBOX_ASSN_VIEW+_side_nav++1006285909
Username/Password
Server URL webct.drexel.edu
Course ID CS164_Fall2001
Submission Name or ID 1006285909
Optional Username List
![Page 22: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/22.jpg)
23
Components
SubmissionDownloader
SectionSorting
PostProcessing
Moss
JPlag
PDFGenerator
![Page 23: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/23.jpg)
24
Bulk Downloader
Web-Crawler: simulates clicks of actual staff member
– Downloads actual web content (HTML)
– Parses HTML to find desired text or URLs to crawl to next
– Uses HTTP operations
Works on assignments and quizzes
Only component in Labrador which interacts with WebCT
![Page 24: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/24.jpg)
25
Organizing Submissions by Section
Not supported in original Labrador prototype, added later on.
Organizes files into directories
3 possible input methods:– Creating a Section column in
gradebook– “Username, Section” CSV File– Username file
![Page 25: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/25.jpg)
26
PDF Generation for Electronic Mark-up
Adobe Portable Document Format can be viewed on all major platforms
With Adobe Acrobat, PDFs can be annotated by graders
Sony VAIO Slimtop PC (PCV-LX920)
Wacom Pen Tablet.
![Page 26: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/26.jpg)
27
VAIO / Wacom
![Page 27: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/27.jpg)
28
PDF Markup Example
![Page 28: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/28.jpg)
29
Redistribution
How can we return annotated PDF’s back to students using WebCT?
WebCT Mail
WebCT Groups
Version 3.8 now addresses this issue
![Page 29: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/29.jpg)
30
Heterogeneous Systems
Different systems want different formats
Plagiarism Detection Systems– Moss has been used most extensively– Began experimenting with JPlag more recently
Future work: Automatic program compiling and testing
![Page 30: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/30.jpg)
31
Automated Plagiarism Detection
Digital formats make "borrowing" easy
– Browsing similar works needs a simple and quick user interface.
– Careful review by faculty to assess results and present to students
![Page 31: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/31.jpg)
32
Moss
Processes C, C++, Java, ML, Lisp, Scheme, Pascal, and Ada programs.
Common code feature reduces false positives
![Page 32: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/32.jpg)
33
Moss Interface
![Page 33: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/33.jpg)
34
JPlag
Processes C, C++, Scheme, and Java programs
For plain text files, it matches a user specified number of words appearing in succession
Could be used for any course grading written (text) documents
![Page 34: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/34.jpg)
35
Command Line Invocation
![Page 35: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/35.jpg)
36
Interactive
![Page 36: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/36.jpg)
37
Quiz Download with Config File
![Page 37: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/37.jpg)
38
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
![Page 38: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/38.jpg)
39
Recent WebCT Enhancements
Relevant to this talk:
– 3.7 Addressed bulk download issue for assignments
– 3.8 Attaching documents to an assignment
![Page 39: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/39.jpg)
40
Future WebCT Enhancements
Power users will need functionality not yet supported
Every domain will also require additional functionality
Not feasible for all domain-specific functionality to run on the WebCT server
![Page 40: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/40.jpg)
41
Wanted
API for non-administrators
Stateful protocol so clients can be built by 3rd parties
![Page 41: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/41.jpg)
42
HTTP: Insufficient Protocol
Inconvenient content interchange Heavy client interaction
A HTTP based approach will be sensitive to the exact location of web pages, and format of text within them.
![Page 42: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/42.jpg)
43
Labrador Availability
Contact Us
http://duplex.mcs.drexel.edu
![Page 43: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/43.jpg)
44
Project Support
The Pew Learning and Technology Program at the Center for Academic Transformation
National Science Foundation, Division of Undergraduate Education, DUE-#0089009
The Ramsey-McCluskey Family Foundation, Margaret Ramsey, '84
Drexel University
![Page 44: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/44.jpg)
45
References
[1] Alex Aiken. Moss: A system for detecting software plagiarism (unpublished), http://www.cs.berkeley.edu/~aiken/moss.html.
[2] Jeffrey L Popyack, Bruce Char, Paul Zoski, Nira Herrmann, and Christopher D. Cera. Managing course management systems. In Proceedings of the thirty-third SIGCSE technical symposium on Computer Science Education, Birds-of-a-Feather Sessions, page 423, 2002.
[3] L. Prechelt, G. Malpohl, and M. Philippsen. Jplag: Finding plagiarisms among a set of programs. Technical Report 2000-1, Fakultat fur Informatik, Universitat Karlsruhe, Germany, March 2000.
[4] Larry Wall, Tom Christiansen, and Jon Orwant. Programming with Perl. O’Reilly and Associates, 3rd edition, 2000.
[5] Michael J. Wise. Yap3: Improved detection of similarities in computer program and other texts. In Proceedings of the twenty-seventh SIGCSE technical symposium on Computer Science Education, pages 130–134. ACM Press, 1996.
![Page 45: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/45.jpg)
46
Roadmap
Introduction
Problems and Solution Goals
Labrador
Discussion
Bonus Slides
![Page 46: Labrador: A Tool for Automated Grading Support in Multi-Section Courses](https://reader035.vdocument.in/reader035/viewer/2022062517/568137de550346895d9f7f70/html5/thumbnails/46.jpg)
47
Blooms Taxonomy
Knowledge– remembering of previously learned material; recall (facts or whole theories); bringing to mind.
Comprehension– grasping the meaning of material; interpreting (explaining or summarizing); predicting outcome and
effects (estimating future trends). Application
– ability to use learned material in a new situation; apply rules, laws, methods, theories. Analysis
– breaking down into parts; understanding organization, clarifying, concluding. Synthesis
– ability to put parts together to form a new whole; unique communication; set of abstract relations. Evaluation
– Ability to judge value for purpose; base on criteria; support judgment with reason. (No guessing).