Defining and Implementing Quality Assurance Standards for

Online Courses

Lawrence C. Ragan, Director, Instructional Design/Development, The Pennsylvania State University

Christina M. Sax, Professor & Assistant Dean, Social, Behavioral, Natural, & Mathematical Sciences, University of Maryland

University College

Presentation Agenda• Session Objectives• Participant Activity• QM Overview: Circle and Rubric• Penn State Approach/Status• Story of Convergence• QM Attractive Features• PSU Adoption/Adaptation• QM Opportunities• a. Models• b. Institution/Institute

Session Objective

• Provide an overview of the development, implementation and administration of an inter-institutional quality assurance system and the application within a single institution.

Participant Activity

• 2-minute drill– Turn to person next to you and discuss:– What characteristics would you use to assess the

quality of the vehicle you may be purchasing?

• Feedback from Teams

• Issues of Definition--single vs group

How do we …

• identify & recognize it?

• motivate & instill it?

• assess & measure it?

• insure it?

• assure it?

Quality Matters

• Quality does matter to …• students• faculty• administrators• institutions• consortia• accrediting agencies• legislators• tax-payers

Quality Matters: Inter-Institutional Quality

Assurance in Online Learning• Grantor: FIPSE

– Grant period: 9/03 – 8/06– Award: $509,177

• Grantee: MarylandOnline– Statewide consortium: 14 community colleges, 5

senior institutions

• http://www.QualityMatters.org

FIPSE Interested Because …

• Quality assurance of online courses is important

• Voluntary inter-institutional assurance has never been done before

• This can serve as a national model

Quality Matters!

Peer Course Peer Course ReviewReview

FeedbackFeedback

CourseCourse

Course Meets Course Meets Quality ExpectationsQuality Expectations

Course Course RevisionRevision

Instructional Designers

Institutions• CAO’s• AR’s

Faculty Course Developers National Standards &

Research Literature

Rubric

Faculty ReviewersTraining

Incoming process

For Our Purposes, Quality Is…

• More than average; more than “good enough”

• An attempt to capture what’s expected in an effective online course at about an 85% level

• Based on research and widely accepted standards

85 %

Major Themes• develop inter-institutional consensus - criteria &

process online course QA• assure & improve course quality• positively impact student learning• faculty-centered activities• promote voluntary participation and adoption• ensure institutional autonomy• replicable, reliable, and scalable processes• foster sharing of materials and expertise• create opportunities for training and professional

development

What Quality Matters is NOT

• Not about an individual instructor

(it’s about the course design)

• Not about faculty evaluation

(it’s about course quality)

• Not a win/lose, pass/fail test

(it’s about a continuous improvement process in a supportive environment)

Quality Matters Rubric

• Based on – research literature

– nationally recognized standards of best practice

– instructional design principles

• Used by review teams to:– assess course quality in 8 key areas (40 review elements)

– provide feedback to faculty course developer

– provide guidance to instructional design support team

Rubric Scoring

Standards Points Relative Value

14 3 Essential

12 2 Very Important

14 1 Important

TOTALS

40 80

• Team of three reviewers

• One score per standard based on majority

• Two criteria to meet quality expectations:• “Yes” to all 14 Essential Standards• Receive at least a total of 68 points

Review Teams

• Teams composed of 3 reviewers:– 1 from home institution, 2 from others – 1 from same discipline, 2 from others– May be either faculty and or ID/IT (practitioners)– mix of CC & 4 yr schools– mix of large & small schools– mix of public & private schools

• Course author--resource for review team

Reviewer Rubric Training

• Focus on:– Application of rubric to course review– Interpretation of review elements– Providing constructive feedback– Competency-based

Course Reviews

• To date, 50% meet expectations on initial review– Instructional design support provided

• Identified 11 common areas for improvement– Target for course development/revision, faculty training

• instructor's self-introduction

• netiquette expectations

• learning objectives stated at the module/unit level

• self-check/practice activities with feedback

• Interaction

• links to school's academic support

• ADA issues

What’s In It For Institutions …

• External validation process

• Strengthens institution’s accreditation package

• Raise QA as a priority activity

• Gain access to a sustainable, replicable, scalable QA process

• Inform online course training & practices

• Provide professional development activities

What’s In It For Participants …

• improve your online course• instructional design support• external quality assurance • expand professional community• chance to review other courses• gain new ideas for your own course• participation useful for annual evaluations,

promotion applications, professional development plan/requirements

National Participation

• Scope:– individuals from 70 different institutions (including the 19

MOL schools) in 14 different states

– Over 250 faculty trained to review online courses using the rubric

– National external partners & advisory board

• Use of QM System:– online course development, review, and revision, faculty

training, formation of distance learning policies & steering committees, institutional reaccredidation packages

Awards - 2005

• WCET Outstanding Work (WOW) Award

• USDLA 21st Century Best Practice Award

• Maryland Distance Learning Association (MDLA) Best Program Award

PSU Approach to Quality Standards

• History: WC started in 1998 with 4 courses grew to 150 by 2002

• Course design and development initiatives from units external to WC

• “Some” variability of design and development practices

• Compromise of “PSU Quality”• If delivered via WC then held responsible for

ensuring quality

Definition of “Quality”

The user (learner) of a system (educational) has a reasonable opportunity for success.

PSU Online Quality Standards

• Charge by Provost to eLearning Council to identify standards of quality for online learning

• Built upon technical standards and pedagogical guidelines developed by WC

• Forwarded to Provost and accepted as University policy by the University Web Task Force

Implementation

• Distributed to Deans as policy (probably stopped there)

• Do exist as record somewhere and accessible if you can find them

• No enforcement “teeth”-- no implementation system established

• No mechanism for reviewing and updating• Some may claim “academic freedom” and

standards need not apply

PSU Interest in QM Standards

• PSU eLearning Council examining the potential of QM model at PSU

• Subcommittee organized to look at various adaptation options

• Primary points of interest:– Externally validated– Well defined rubric– Established “system”– Research-based criteria

Potential Options1. PSU can fully accept and adopt the QM system for quality

assurance. • Courses would be submitted to QM for evaluation and ultimately

the QM seal of approval.2. PSU can adopt the QM system to the PSU environment including

approval of reviewers and QM process.• PSU would negotiate with the QM board on the adaptation

process in order to maintain the QM seal of approval.3. PSU can adopt the QM system to the PSU environment including

approval of reviewers and QM process.• PSU would not seek the QM seal of approval.

4. Penn State would maintain the current model of Technical Standards and Pedagogical Guidelines.

QM: Looking Ahead

• Adapt rubric & process for other formats– Hybrid/blended, face-to-face, continuing

education, commercial, professional training

• Adapt rubric & process for specific institutional needs

• Explore the “QM Institution” concept

• Assess the impact on student learning through research projects

QM: Looking Ahead

• Diversify Training Program– Additional review training & train-the-trainer

• Annual rubric update cycle

• Sustainability plan

• Develop partnerships & business opportunities

www.qualitymatters.org

For more information:

Kay Kane

Kkane@pgcc.edu