Hua Ai, Xiaojun Peng, Rosa Ren, Nan ZhouHua Ai, Xiaojun Peng, Rosa Ren, Nan Zhou

http://dream.sims.berkeley.edu/4hmwk/

Overview

Introduction Introduction Problem StatementProblem Statement GoalsGoals

SystemSystem ProjectProject

Research and AnalysisResearch and Analysis Needs AssessmentNeeds Assessment Competitive SystemsCompetitive Systems

DesignDesign Pilot ImplementationPilot Implementation

PrototypesPrototypes DemoDemo

Future WorkFuture Work AcknowledgementAcknowledgement

myHomework@SIMS

For: For:

About:About:

What: What:

SIMS professors and studentsSIMS professors and students

Assignments and related informationAssignments and related information

A solution that enablesA solution that enables Effective document managementEffective document management

Effective information useEffective information use

What’s the Problem? We have no standard way to manage assignments for all We have no standard way to manage assignments for all

courses. courses. Every professor does things a little differently Every professor does things a little differently Various submission and feedback methods can be confusing, each Various submission and feedback methods can be confusing, each

with their own advantages and disadvantages with their own advantages and disadvantages IT has to support a variety of ways to do electronic submission, IT has to support a variety of ways to do electronic submission,

extra time is spent customizing and managing solutions for extra time is spent customizing and managing solutions for individual coursesindividual courses

Assignment related information not used toAssignment related information not used to enable better communication between professors and studentsenable better communication between professors and students allow efficient workload management by professors and studentsallow efficient workload management by professors and students

Practice Inconsistent with SIMS’s Mission Practice Inconsistent with SIMS’s Mission Advance information management and the use of information Advance information management and the use of information

technology to enable more efficient flow of information, better technology to enable more efficient flow of information, better communication and cooperation among individuals and institutions communication and cooperation among individuals and institutions

System Goals

Provide consistent mechanism for all SIMS coursesProvide consistent mechanism for all SIMS courses manage assignments and related informationmanage assignments and related information encourage better use of the documents and their metadataencourage better use of the documents and their metadata

Document ManagementDocument Management Information UseInformation Use

ProfessorsProfessors -Easily obtain and manage submissions by students & groups

-Assignment planning

-Easily grade & give feedback

-Track student performance

StudentsStudents -Simplify submission process

-Share assignments

-Archive previous work

-Better plan & manage work

-Receive grade & feedback

-Track overall performance

Project GoalsLimited by time, we focused our effort onLimited by time, we focused our effort on Research and AnalysisResearch and Analysis

Understand the Understand the current “system”current “system” and its and its problem areasproblem areas Thoroughly understand the Thoroughly understand the needs needs of each user group of each user group Evaluate Evaluate existingexisting systemssystems

DesignDesign Develop high level design of a system that would Develop high level design of a system that would

address the problemsaddress the problems Pilot implementationPilot implementation

Develop pilot interface and databaseDevelop pilot interface and database

Needs Assessment Overview

Purpose:Purpose: Understand overall process and context of issuesUnderstand overall process and context of issues Understand specific user and organizational needsUnderstand specific user and organizational needs

MethodologyMethodology

Stake

Holders/Users

Professors / Professors / TA’sTA’s

StudentsStudents SIMS & SIMS & Campus ITCampus IT

Method Semi-structured Semi-structured InterviewsInterviews

Structured Structured SurveySurvey

Open-ended Open-ended InterviewsInterviews

Participants 7 professors7 professors

2 TA’s2 TA’s

35 students35 students KevinKevin

ETS staffETS staff

Overall Context

On averageOn average Professors teach 1-2 courses (including seminars) each Professors teach 1-2 courses (including seminars) each

termterm Students are enrolled in 2-3 SIMS courses each termStudents are enrolled in 2-3 SIMS courses each term Many courses require 6-10 assignments and 3-4 course Many courses require 6-10 assignments and 3-4 course

project related submissionsproject related submissions Almost all students have submitted assignment Almost all students have submitted assignment

electronically, especially using email with attachmentselectronically, especially using email with attachments 57% experienced problems w/ electronic submission57% experienced problems w/ electronic submission Few professors and students have used a “Learning Few professors and students have used a “Learning

Management System”Management System”

Professors’ Context Most teach at least 5 years and many 15+ Most teach at least 5 years and many 15+ A lot of course preparation before term startsA lot of course preparation before term starts

Scope of courseScope of course Material: review books and articles, do readers…Material: review books and articles, do readers… Course schedule: topics, readings assignments, projectsCourse schedule: topics, readings assignments, projects

Real goal is to help students learnReal goal is to help students learn Submission management not a pleasant but necessary task (among others)Submission management not a pleasant but necessary task (among others) Most have tried everything: paper, email, web pages, directoriesMost have tried everything: paper, email, web pages, directories

Grades are evil but necessaryGrades are evil but necessary Really represent individuals and their overall effort, progress? Really represent individuals and their overall effort, progress? Same grade for different students really mean the same thing?Same grade for different students really mean the same thing? Grades have their places and not always the most importantGrades have their places and not always the most important

Professors’ Concerns

Balance of needs when designing assignmentsBalance of needs when designing assignments Purpose of specific assignments Purpose of specific assignments Kinds of equipment, tools and material requiredKinds of equipment, tools and material required Amount of time involvedAmount of time involved Flexible submission methodsFlexible submission methods Appropriate formatAppropriate format Manageable artifactsManageable artifacts

Group situations most complicatedGroup situations most complicated Track progress by groupTrack progress by group Feedback for group and individual membersFeedback for group and individual members

Using student workUsing student work permission is biggest concern and difficult to managepermission is biggest concern and difficult to manage

Student Preferences

35 responses35 responses 15 115 1stst, 15 2, 15 2ndnd, 5 others, 5 others 16 female, 17 male16 female, 17 male Almost all used some form Almost all used some form

electronic submission electronic submission 57% (20) experienced 57% (20) experienced

problems w/ e-submitproblems w/ e-submit Many preference questions Many preference questions

1-7 Likert scale1-7 Likert scale 1 least prefer, 7 most1 least prefer, 7 most

ElectronicElectronic submission 5.7submission 5.7 grade 5.3grade 5.3 feedback 4.3feedback 4.3

More concerned w/ receiving More concerned w/ receiving notification when professor notification when professor plan to share (5.5) than plan to share (5.5) than giving actual permission giving actual permission (5.0)(5.0)

51% (18) archive 2-5 yrs, 51% (18) archive 2-5 yrs, 22% (8) indefinitely22% (8) indefinitely

Willingness prepare files for Willingness prepare files for long term archive = 4.8long term archive = 4.8

IT Concerns

Current support provided by SIMS IT Current support provided by SIMS IT Create course homework email addressCreate course homework email address Create and manage share directories for courses, project groupsCreate and manage share directories for courses, project groups Set up various services to support websites, databases, many special Set up various services to support websites, databases, many special

software supportsoftware support Recently custom upload web interface for Document EngineeringRecently custom upload web interface for Document Engineering

IT concernsIT concerns Authentication, security of database and filesAuthentication, security of database and files Storage of documentsStorage of documents Long term archive issuesLong term archive issues

ETS StaffETS Staff Majority of support helping professors become familiar w/ the Majority of support helping professors become familiar w/ the

supported systems, setting up accounts, handholding alone the waysupported systems, setting up accounts, handholding alone the way

Competitive Systems

SystemSystem StrengthsStrengths Not suitable for SIMS because:Not suitable for SIMS because:

Web-CTWeb-CT Currently supported by Currently supported by Berkeley (ETS) Berkeley (ETS)

Free for each professorFree for each professor

Stove pipe model: handle assignment Stove pipe model: handle assignment spec, collection, grading isolately. spec, collection, grading isolately.

Developed elsewhere, no plug in for Developed elsewhere, no plug in for extension. (e.g., e-grade)extension. (e.g., e-grade)

Lack support for planning or archivingLack support for planning or archiving

BlackBoard BlackBoard (CourseInfo)(CourseInfo)

Currently supported by Currently supported by Berkeley (ETS)Berkeley (ETS)

Free for each professorFree for each professor

Many features for course management, yet Many features for course management, yet each part too simple. each part too simple.

Assignment management does not meet Assignment management does not meet SIMS and not easy to use enough.SIMS and not easy to use enough.

WebcrossingWebcrossing Few professors are Few professors are experimenting experimenting

Focuses on discussion forum and Focuses on discussion forum and online live lectures.online live lectures.

MonsterTrack & MonsterTrack & NSF Fast Track NSF Fast Track Proposal SystemProposal System

Both document Both document management system management system used by manyused by many

Different application and audienceDifferent application and audience

Limited types of documentsLimited types of documents

Used for schedulingUsed for scheduling interviewinterview

Halftime

Introduction Introduction MotivationMotivation GoalsGoals

SystemSystem ProjectProject

Research and AnalysisResearch and Analysis Needs assessmentNeeds assessment Competitive systemsCompetitive systems

DesignDesign Pilot ImplementationPilot Implementation

PrototypesPrototypes DemoDemo

Future WorkFuture Work AcknowledgementAcknowledgement

Design & Pilot Implementation

JSP

Web Interface

Architecture

Application(dynamic processing)

JDBC API

Databasemysql

Interface Prototype

Implementation

Some Implementation

From ER to Data Load-up

Interface Prototype PurposePurpose

Iteratively develop a Iteratively develop a potential interface with potential interface with the involvement of the involvement of users from early onusers from early on

MethodologyMethodology

ProfessorsProfessors StudentsStudents

Round 1Round 1 3 participants3 participants 3 participants3 participants

Round 2Round 2 2 participants2 participants 3 participants3 participants

Prototyping Usability Testing

Interface Prototypes – Tasks

Professor Professor Plan and specify submission detailsPlan and specify submission details View, grade assignments, and give feedback View, grade assignments, and give feedback Calculate final gradeCalculate final grade

StudentStudent Submit assignmentsSubmit assignments Receive grade and feedbackReceive grade and feedback Adjust permission setting Adjust permission setting

Interface Prototype - Issues TerminologyTerminology

assignment, project, homework, submission, deliverable assignment, project, homework, submission, deliverable Sequential Process ModelingSequential Process Modeling

Assignment specification and submissionAssignment specification and submission PlanningPlanning

Force professors to enter one assignment at a time without overall Force professors to enter one assignment at a time without overall context context

Homepage does not provide helpful overviews for all coursesHomepage does not provide helpful overviews for all courses Group ManagementGroup Management

Who set up and change, how to keep track and reuseWho set up and change, how to keep track and reuse Final GradesFinal Grades

Handle by system or defer to Excel?Handle by system or defer to Excel?

Demo Post an assignment

Submit an assignment

Grade it!

Future Work More management functions More management functions

Group management: Feedback from fellow group membersGroup management: Feedback from fellow group members Security: Access rights for TAsSecurity: Access rights for TAs Sophisticated grades manipulation Sophisticated grades manipulation

More information-use functionsMore information-use functions Archiving / Reusing (content management)Archiving / Reusing (content management) Planning and ReportingPlanning and Reporting

MoreMore implementationimplementation supportingsupporting functions by underlying JSP and database so SIMS has functions by underlying JSP and database so SIMS has

a real system!a real system!

SIMS “Learning Management System”SIMS “Learning Management System” course content (website) managementcourse content (website) management

Why We Choose This Project

Problem can be easily constrained so we can arrive at a Problem can be easily constrained so we can arrive at a potential solution with the available resourcespotential solution with the available resources

Chance for group members to use various skills and Chance for group members to use various skills and training gained at SIMS training gained at SIMS

Easy access to all user groupsEasy access to all user groups Professors (chance to get to know them better and get advice for Professors (chance to get to know them better and get advice for

our project!)our project!) Cool IT staffCool IT staff Awesome SIMS students – (easily bribed by food :)Awesome SIMS students – (easily bribed by food :)

Most importantly, we wanted a chance to contribute to Most importantly, we wanted a chance to contribute to SIMS by helping to address a real SIMS problem SIMS by helping to address a real SIMS problem

Lessons Learned

Understanding user needs is iterative, iterative, iterative Understanding user needs is iterative, iterative, iterative Project focused on front end issues should have more than Project focused on front end issues should have more than

1 person with experience1 person with experience Vision is important when too immerged in tasks and Vision is important when too immerged in tasks and

detailsdetails Need another term to implementNeed another term to implement SIMS students like rice puddingSIMS students like rice pudding Frozen burritos and a big bottle of hot sauce can go a long Frozen burritos and a big bottle of hot sauce can go a long

wayway Spouses and boyfriends make great tech support but need Spouses and boyfriends make great tech support but need

to prevent them from interrupting our offsite meetingsto prevent them from interrupting our offsite meetings

Thank You! Professors: Professors:

Ray LarsonRay Larson Marti HearstMarti Hearst David Messerschmitt David Messerschmitt Bob GlushkoBob Glushko John ChuangJohn Chuang Peter LymanPeter Lyman Nancy Van HouseNancy Van House

SIMS IT: SIMS IT: Kevin HeardKevin Heard

ETS Staff: ETS Staff: Fred BeshearsFred Beshears Howie LanHowie Lan Charmaine FalcaoCharmaine Falcao

MonsterTrak: MonsterTrak: Scott HolcombScott Holcomb

Students:Students: All Survey RespondentsAll Survey Respondents Moryma AydelottMoryma Aydelott Steve KafkaSteve Kafka Mary ThompsonMary Thompson Thuan NguyenThuan Nguyen Luis VillafanaLuis Villafana Linda HarjonoLinda Harjono Mike KimMike Kim Craig RixfordCraig Rixford Calvin SmithCalvin Smith Ramona MartinezRamona Martinez Linda DuffyLinda Duffy Peter RoesslerPeter Roessler

Final Survey, Questions

Visit us at : http://dream.berkeley.edu/4hmwk