Assessment of Web-based Education Tools: Are your Students Learning Through Technology?

Andrea IrbyUndergraduate Academic Programs

North Carolina State University

Background

NC State is a large, public, research

one institution. 19,000 undergraduates; 9,000 grads. Under pressure in late 1990s to grow Possible expansion of Distance Education Virtual Advising Center created 1999

Why Technology? Enables us to transcend many barriers

Physical campusRaceCountry, nationality“Saving Face”

Student development theory – meet them where they are

Enables us to be more effective and efficient especially in times constrained resources

Why Technology?

Types of Information Exchange

Synchronous vs. AsynchronousData - ways of expressing things Information - the arrangement of data into meaningful patternsKnowledge - the application and productive use of informationInformation versus Knowledge

Technology in EducationCan transform key relationshipsPersonalization: Targeting the student vs. the departmentEngaging the student: making applications interactiveStudent Decision Support: Information becomes knowledgeCreate community: cultivate student relationships

Carl Jacobson, 2000

Engaging Students On-Line

Web-sites On-line Course Management SystemsEmail/ListservsOn line Chat or Instant Messenger type toolsBulletin Boards and BlogsSocial Networking sites

“Students will gravitate toward the technology medium that best meets their needs” Karen Thurmond- Univ of Memphis

Focus on Web-sites2nd Generation Web-sites: Information based3rd Generation are interactive, knowledge based

How do we know if/ how many students come to our website?Are they able to find the information they need?Is it clear and meaningful to them?Do the receive our intended information?

How to Know

Log Data Analysis (data mining)

Usability Studies

Accessibility Testing

Log Data Analysis

Data (log) is created each time a person visits a web-page.

Analytic Tools then either pull data from the server that hosts the web-site or read it from html code on each web-page.

Sample tools Google Analytics (reads code)- free

www.google.com/analytics

SAS Web Analytics (pulls from server)www.sas.com/solutions/webanalytics

TopShareWare (pulls from server)www.topshareware.com/LogSuite-download-38225.htm

Easy to Read Reports (in real life )

Information ReceivedVisitors/UsersRepeat Visitors/Cached VisitorsHow long on siteTraffic by time of day and monthWhere enter, exit, or drop out of siteBrowsers, operating systemsWhere they are logging in fromCampaign/Action Tracking

Virtual Advising Center data mining

2000:

72,419 annual visitors 26,795 repeat visitors176 avg. visitors per day

1,021,947 hits to site

2006:

285,053 annual visitors 108,320 repeat visitors780 avg. visitors per day

8,269,215 hits annually

Virtual Advising Center data mining

Monday-Thursday busiest days. Visitors on site for avg. of 2 minutes now about 5.Path in and out; pages turned.IP locations –whose accessing site on campus, off campus, around the world.Heaviest traffic times per day.

How we used this information

Sent surveys out on busiest daysUsed busiest times of day to determine staff coverage for chatUsed IP address/location information to help IT staff on campus place kiosks in high demand locationsTo enhance relationships with local feeder community colleges

Usability Testing

Observed “Focus” Groups for web-sitesReal data versus self reported dataExamines how people use some human-made object Controlled experimentSmall, cross section of users

Trained users or “Hallway” users

Usability Testing

Pre-test questionnaires get input on subject’s familiarity with web-siteActual test involves a series of short, simple tasks the subject is asked to complete Observations are recorded by “tester”Patterns of how subject completes each task are notedEmotional responses key to watch

Set up

Location can be:In a computer lab with multiple teams

In a usability lab

At your own computer desk

Usability Testing

Post tests – record overall impressions of web-site and experience; reflectionVery Informative Web-site on Usability Testing: www.usability.govIf testing uncovers difficulties, redesignTest in small groups and oftenFor virtual advising center, its how we keep students at the forefront of our design

Accessibility

Inclusive web-communities allow people with diverse abilities to be engaged in on-line learningWeb-Content Accessibility Guidelines

http://www.w3.org/TR/1999/WAI-WEBCONTENT-19990505/#def-checkpoint

3 Priorities for Website Accessibility

Accessibility

Series of CheckpointsPriority 1- Must satisfy these checkpoints or one or more groups will find it impossible to get and use the informationPriority 2- Should satisfy these design elementsPriority 3- May satisfy design elementsResults in A, AA, AAA website ratings

Accessibility

Test your site with on-line toolsTest your site with screen readers/JAWsOn-line web resources to help you:

http://atrc.utoronto.ca/ http://ncsu.edu/it/access/development/software/tools.phphttp://www.accessible.org/bobby-approved.html

Evaluating Learning

Web-site must be interactivetests, essays, portfolios, chats, email

Determine what we want students to learn Get student input Variety of ways: focus groups, check lists, rubrics and content analysis Have more than one staff member working on assessment

What we’ve tried and learned

Focus Groups- what we may see as learning, they see as “help”. Over time, they may recognize what they have learned: raises question of timing of assessmentElectronic communication trails are rich with evidence of learning. Students are readmitting, graduating, learning about themselves, picking majors, staying, expanding their academic portfolios.

Evaluating Learning

Need to develop a rubric for evaluating rich content.

George Mason Business School example

Databases/Microsoft Access tool

Technology is a tool – through which students can learn and for helping us with assessment.

Discussion? Questions?