int. conf. e-learning 2011 project management and assessment of distributed teams in an online...

Int. Conf. e-Learning 2011

Project Management and Assessment of Distributed Teams in an Online Capstone

Masters-Level Information Technology Course

Charles Tappert and Allen StixPace University, New York


Five Previous Conference Papers on Real-World Student

Projects

1. Integrating Real-World Projects for Actual Customers into Capstone Courses (E-Learn 2002)

2. Security-Related Real-World Projects (ISECON 2004)3. Interplay of Student Projects and Student-Faculty

Research (E-Learn 2007)4. Pedagogical Issues in Managing Information

Technology Projects Conducted by Geographically Distributed Student Teams (SITE 2009)

5. Assessment of Student Work on Geographically Distributed Information Technology Project Teams (E-Learn 2009)


Real-World Student Projects

Conducted in capstone courses for 10 years

Student teams build real-world computer information systems for actual customers

Project systems serve the community internal university community at Pace greater university community external non-profit local community

Real-World Student Projects (cont)

Real-world projects are a stellar learning experience for students

Win-win situation for all Students Customers Instructors and other involved faculty School of CSIS University


Migrate to Online Format

Migrated from traditional face-to-face format to online format in Fall 2006

To be progressive Technology for online courses adequate Online preferred by employed students –

no scheduling conflicts & no commuting To expand the population of students

beyond the greater NYC area


Challenges of Online Format

Uncertainties of how traditional course methods port to the online environment and what new methods might be required

Teams lacking co-presence require higher level of organizational and process skills

No weekly classroom meetings as safety net for teams’ interaction and functioning


Team Projects – Categories


Project CategoryNumberProjects

ProjectSemesters

ProjectRelated

Pubs

OffshootPubs

Web Applications 21 25 21

Pervasive Systems 15 25 18

PC Applications 11 18 13

Artificial Intelligence 9 11 12

Pattern Recognition 9 12 34 19

Biometric Systems 32 35 39 19

Quality Assurance 5 9 5 5

Totals 102 135 142 43

Table 1. Summary of projects and publications.

Team Projects – Sources


Project Source Number

Faculty Ideas or Research 42

Student Ideas or Research 36

External Community 13

Internal University Needs 11

Totals 102

Table 2. Project sources.

Team Projects – Publication Types


Publication Type Number

External Conference Papers 53

Journal Articles 7

Book Chapters 2

Doctoral Dissertations 17

Masters Theses 4

Internal Conference Papers 98

Internal Technical Reports 4

Totals 185

Table 3. Publication types.

Journal 2010

Team Projects – Spring 2011

Course website “Projects” page Fall 2009


Team Website

Project title and description Project members and customers All deliverables posted

Weekly status reports Midterm & final presentation slides User manual Technical paper


Team Project – Example Website

Personality Assessment from Handwriting Project

Journal 2010

Team Project – Example Website

Team Projects – Example Systems

Handwriting Forgery Quiz System Rare Coin Grading System Biometric Systems Projects (2010-2011)

Keystroke biometric system Stylometry system Biometric product investigation

Freeware – face & voice Licensed products – face & fingerprint


Keystroke & Stylometry Systems

The 2008 U.S. federal Higher Education Opportunity Act (HEOA) requires institutions of higher learning to make greater access control efforts to assure that students of record are those actually accessing the systems and taking exams in online courses by adopting identification technologies as they become more ubiquitous

To meet these needs, keystroke and stylometry biometrics were investigated towards developing a robust system to authenticate online test takers

Performance statistics on keystroke, stylometry, and combined keystroke-stylometry systems were obtained on data from 40 test-taking university students


Keystroke Biometric System

Keystroke biometric systems measure typing characteristics believed to be unique to an individual and difficult to duplicate

Most previous work on passwords – two commercial products Keystroke biometric is appealing for online student verification

not intrusive to computer users inexpensive – only hardware required is a computer with keyboard text continues to be entered for potential repeated checking after an

initial authentication phase (referred to as dynamic verification) Our keystroke system developed over last 6-7 years is unique

collects raw keystroke data over the Internet focuses on long-text input where sufficient keystroke data are available to

permit the use of powerful statistical feature measurements


Keystroke Biometric SystemSpring 2011 Results

The best equal-error-rate performance on the keystroke system was 0.55% which is an improvement over earlier reported results on this system

Receiver Operating Characteristic (ROC) curves: False Reject Rate (FRR) vs False Accept Rate (FAR)

Experiments 1 and 2:Open systems – trained and tested on different students

Experiment 3:Closed system – trained and tested on all studentsInt. Conf. e-Learning 2011

Stylometry Biometric System

Stylometry is the study of determining authorship from the linguistic styles of the authors

Traditionally, it has been used to attribute authorship to anonymous or disputed literary documents

More recently, computer-based communication and digital documents have been the focus of research, sometimes with the goal of identifying perpetrators or other malicious behavior

We are just beginning to apply stylometry to verify online student test takers, early results poor


Stylometry Biometric System

The performance of the stylometry system a poor 33% EER, indicating that stylometry is not suitable for text lengths of short-answer tests (50-100 words per question)

Receiver Operating Characteristic (ROC) curves: False Reject Rate (FRR) vs False Accept Rate (FAR)

Experiments 1 and 2:Open systems – trained and tested on different students

Experiment 3:Closed system – trained and tested on all studentsInt. Conf. e-Learning 2011

Biometric Product Investigation

Freeware systems (usually 30-day trial) – fall 2010 project Face recognition

KeyLemon (KeyLemon) & FaceSDK (Luxand) Voice recognition

VoiceCipher (VoiceLatch) & bio-iVault/iWallet (myBiodentity)

Inexpensive licensed products – spring 2011 project Face recognition SDK (Neurotechnology) Fingerprint recognition SDK (Neurotechnology)

Results - freeware and cheap licensed systems equally poor Eigenface algorithm (coded in 2003 project) worked better Recent info exposes poor accuracy of biometric products


Simple Eigenface Algorithm coded in 2003 Project


?Query

Face DB

Face Recognition: SystemFace Recognition: System


Biometric IdentificationBiometric Identification

Tom Cruise in Minority Report, 2002Int. Conf. e-Learning 2011

Inspirational Portrait of IndividualityInspirational Portrait of Individuality


Face Recognition: National SecurityFace Recognition: National Security


Speaker Individuality: “My name is …”Speaker Individuality: “My name is …”


“My name is” from Two Different Speakers

Speaker IndividualitySpeaker Individuality


“My name is” divided into seven sound units.

Speaker IndividualitySpeaker Individuality


biomouse Fingerprint scanner

Digital

Camera

LCD Pen

tabletMicrophone

Multi-modality Biometric AuthenticationMulti-modality Biometric Authentication

Embeded & Hybrid User Verification system

System that requires user verification


Issues/Solutions Stemmingfrom Scattered Teams

Project stakeholder communication Issue – communication gets difficult

For example, scattered team members more likely to feel isolated and want to communicate directly with instructor or customer

Solution Communication between team and instructor/customer

must be through team leader Email distribution lists for whole class and for each team Project team leaders must be local to facilitate

communication/meetings with instructor and customers Course website provides central source of course

information Blackboard discussion forum for each project (see below)


Issues/Solutions Stemmingfrom Scattered Teams (cont)

How to handle quizzes, deliverables, etc. Issue – classroom meetings not available Solution – use Blackboard educational software

Quizzes Collecting digital deliverables Discussion forums

Forum for archiving instructor email Forum for student introductions Forum for textbook and other course material Forum for each team project


Issues/Solutions Stemmingfrom Scattered Teams (cont)

Provide some face-to-face interaction Issue – no weekly classroom meetings Solution – three classroom meetings for

local students/customers1. Near beginning of course

1. Face-to-face introductions, nature of course, specifics of course, student team project meetings

2. Midterm1. Project status presentations

3. End of semester1. Final project presentations


Current Assessment of Online Students

Individual quizzes (20%) Blackboard educational software system

Team initial assignment (10%) Students learn to function as a team

Team project midterm checkpoint (20%) Team project final checkpoint (20%) Team technical paper (30%) Strong emphasis on projects

No midterm/final exams (used in two-semester course)


Team Member Self and Peer Evaluations

Issue – lack of classroom meetings makes it difficult to determine individual team members’ contribution to the project work

Peer evaluations critical for distributed teams

Minimal team member/customer contact Minimal team member/instructor contact

Literature indicates Various granularity levels in peer evaluations Some automated systems reported


Team Member Self and Peer Evaluations

Three times during the semester After initial assignment to learn the process At the midterm checkpoint At the final end-of-semester checkpoint

Process for a graded team event First assign a team grade Adjust individual grades up/down based on

self/peer, customer, and instructor evaluations


Example Team Peer Evaluationand Grade Chart (4 member

team)

Team Member

Eval 1 Eval 2 Eval 3 Eval 4 Summary

Grade

1 + = + ++ + + + + 93

2 = = – – – – – – 79

3 – = + – – 83

4 = = – + = 85

Average = = = = = 85

+/- 2% for each summary +/- sign, showing only peer evaluations.


Pedagogical Course Evaluations

Issue – lack of classroom meetings makes it difficult for instructor to determine relative value of the course methodologies

Solution – semester-end survey (Survey Monkey)

Procedures/methods that worked well, or did not work well, and why


Pedagogical Customer Evaluations

Issue – instructor is often not aware of the quality of team-customers interactions

Solution – semester-end survey Obtain student feedback on customer

interaction Were customer requirements clear? Was amount of contact/interaction adequate? Was help on the project work appropriate?


Something New in 2010-2011Compared Projects to Reality Game

Shows

Capitalize on student interest in TV game shows Like Survivor and The Apprentice

Common with project teams & game show groups

Work toward common goals with division of responsibilities Acquire and share new knowledge Harness skills of different teammates Adjust to different personalities Exhibit initiative without disruptiveness

Differences between projects & game shows Survivor “voted out” & The Apprentice “you’re fired” Game show participants compete against each other


Conclusions

Over five year’s experience in face-to-face mode

Over four year’s experience in online mode

Techniques for managing and assessing distributed teams have been successful and they continue to evolve


int. conf. e-learning 2011 project management and assessment of distributed teams in an online...

Documents

elearning 2011sheet1table

capstone courses e

studentfaculty research

interplay of student

projects isecon

stellar learning experience

summary of projects

team projects spring