Quality? Who Quality? Who Says?Says?

WCET Annual ConferenceWCET Annual Conference

November 9, 2007November 9, 2007

ObjectivesObjectives

Walk away with multiple views on Walk away with multiple views on rubrics for assessing quality of rubrics for assessing quality of online instructiononline instruction

Take ideas from three different Take ideas from three different quality assurance methodsquality assurance methods

Recognize the impact and value of a Recognize the impact and value of a quality assurance programquality assurance program

Our PanelOur Panel

Kay Kane, Quality Matters, Kay Kane, Quality Matters, MarylandOnlineMarylandOnline

Allison Peterson, Texas Woman’s Allison Peterson, Texas Woman’s UniversityUniversity

David Curtis, Park UniversityDavid Curtis, Park University

Michael Anderson, UT TeleCampusMichael Anderson, UT TeleCampus

AgendaAgenda

Introduce QA programsIntroduce QA programs

Address Key Issues:Address Key Issues:Impact on participationImpact on participationStudy outcomesStudy outcomesValue of the processValue of the processLessons LearnedLessons Learned

Q&AQ&A

Final ThoughtsFinal Thoughts

Quality Quality Assurance Assurance ProgramsProgramsWhat’s it all about?What’s it all about?

How do we do it?How do we do it?

Texas Texas Woman’s Woman’s UniversityUniversityA Brief Introduction to the A Brief Introduction to the Quality Enhancement PlanQuality Enhancement Plan

Texas Woman’s Texas Woman’s UniversityUniversity

Texas Higher Education Texas Higher Education Coordinating Board (THECB)Coordinating Board (THECB) Principles of Good Practice (PGP)Principles of Good Practice (PGP)

Institutional Context & CommitmentInstitutional Context & Commitment

Quality Enhancement Plan (QEP)Quality Enhancement Plan (QEP) Quality Matters (QM)Quality Matters (QM)

Quality MattersQuality Matters

Peer ReviewPeer Review

FeedbackFeedback

CourseCourse

Course Meets Course Meets Quality ExpectationsQuality Expectations

Course Course RevisionRevision

The RubricThe Rubric

Eight standards:Eight standards: Course Overview and IntroductionCourse Overview and Introduction Learning Objectives Learning Objectives Assessment and MeasurementAssessment and Measurement Resources and MaterialsResources and Materials Learner InteractionLearner Interaction Course TechnologyCourse Technology Learner SupportLearner Support ADA ComplianceADA Compliance

Key components must align.

QM StrengthsQM Strengths

Grounded in:Grounded in: research literatureresearch literature national standards of best national standards of best

practicepractice instructional design principlesinstructional design principles

www.QualityMatters.orgwww.QualityMatters.org

Park UniversityPark UniversityBy the Numbers!By the Numbers!

Online Students = 52,000+Online Students = 52,000+ This counts all students taking all classes, so a This counts all students taking all classes, so a

student taking four online class is counted as student taking four online class is counted as four (4) students.four (4) students.

Online Courses = 322 Discrete CoursesOnline Courses = 322 Discrete Courses A total of 524 SectionsA total of 524 Sections

Online Faculty = 455 +Online Faculty = 455 +

TrainingTraining

DevelopingDeveloping

ReviewingReviewing

Quality Starts…Quality Starts……and doesn’t stop

Key IssuesKey Issues Impact on participationImpact on participation Study outcomesStudy outcomes Value of the processValue of the process Lessons LearnedLessons Learned

Impact of Impact of Participation Participation

on Faculty on Faculty Peer Peer

ReviewersReviewers

Year One – Spring 2007Year One – Spring 2007

January 2007 – August 2007January 2007 – August 2007 10 Faculty10 Faculty 6 Staff6 Staff QM Reviewer TrainingQM Reviewer Training Revise 1 Course – 14 Essential Revise 1 Course – 14 Essential

StandardsStandards $400 Stipend$400 Stipend

Year One – ResultsYear One – Results

3 Course Reviews3 Course Reviews 1 Review and Recognized1 Review and Recognized 1 in Review1 in Review 1 Preparing for Review1 Preparing for Review

2 Course Reviewers2 Course Reviewers 1 Summer1 Summer 1 Fall1 Fall

3 Potential Reviewers3 Potential Reviewers

Year Two – Fall 2007Year Two – Fall 2007

September 2007 – August 2008September 2007 – August 2008 25 Faculty25 Faculty 1 Staff1 Staff QM Reviewer TrainingQM Reviewer Training Revise 1 Course – All 40 StandardsRevise 1 Course – All 40 Standards Review 1 CourseReview 1 Course $400 Stipend$400 Stipend

Year Three – Fall 2008Year Three – Fall 2008

September 2008 – August 2009September 2008 – August 2009 35 Faculty35 Faculty QM Reviewer TrainingQM Reviewer Training Revise 1 Course – All 40 StandardsRevise 1 Course – All 40 Standards Review 1 CourseReview 1 Course $400 Stipend$400 Stipend

Training ChangesTraining Changes

Online Educator SymposiumOnline Educator Symposium Online CourseOnline Course PresentationsPresentations One-to-one consultationsOne-to-one consultations

Study OutcomesStudy Outcomes

Quality MattersQuality Matters

FY07 Course Reviews FY07 Course Reviews

85 courses reviewed FY0785 courses reviewed FY07

23 different institutions23 different institutions

45 QM Managed45 QM Managed

40 Independent40 Independent

FY07 Course ReviewsFY07 Course Reviews

Upon initial review:Upon initial review:

38% met expectations38% met expectations

62% do not yet meet expectations62% do not yet meet expectations 30% of these were revised and met 30% of these were revised and met

expectationsexpectations

Currently Recognized by QM: 56%Currently Recognized by QM: 56%

Common ThemesCommon Themes

Common areas for course Common areas for course improvementimprovement

These are potential targets for: These are potential targets for: faculty trainingfaculty training special attention in initial course special attention in initial course

developmentdevelopment

Common Areas for Common Areas for ImprovementImprovement

Module objectives measurable/consistent with course objectives 45%

"Self-check“/practice assignments for timely feedback 42%

Instructions on how to meet the learning objectives 40%

Equivalent alternatives to auditory and visual content 39%

Instructor response and availability 38%

Requirements/skills/prerequisites clearly stated 35%

Navigational instructions 32%

Criteria for the evaluation of students' work and participation 31%

Impact on Faculty Impact on Faculty and Reviewers during Grantand Reviewers during Grant

FacultyFaculty said as a result of the review: said as a result of the review:

91% made changes in the course91% made changes in the course

89% felt quality of course design improved89% felt quality of course design improved

ReviewersReviewers said as a result of the review: said as a result of the review:

73% made changes to own online course73% made changes to own online course

100% said valuable professional 100% said valuable professional development activitydevelopment activity

Student Learning & Student Learning & InteractionInteraction

College of Southern MDCollege of Southern MD General education IT course (100+ General education IT course (100+

students)students)

Revisions made in all learning modules:Revisions made in all learning modules: created Learning Guides (explicit roadmap)created Learning Guides (explicit roadmap) reorganized presentation and designreorganized presentation and design added classroom assessment techniques added classroom assessment techniques

(CATs)(CATs)

Increase in:Increase in: the frequency of content access by studentsthe frequency of content access by students % of A grades awarded in the course% of A grades awarded in the course

Impact of Navigation Impact of Navigation DirectionsDirections

Prince George’s CC, MDPrince George’s CC, MD Standard I.1 – course navigationStandard I.1 – course navigation

After revision of navigation bar:After revision of navigation bar: Students asked fewer questions: Students asked fewer questions:

course navigation, locating course navigation, locating information, course requirementsinformation, course requirements

Less student concern about what Less student concern about what they needed to do to succeedthey needed to do to succeed

Student Perceptions & Student Perceptions & PrioritiesPriorities

SUNY CantonSUNY Canton

3 standards most often noted as not 3 standards most often noted as not being met:being met: Course navigation directionsCourse navigation directions Assessments & measurement provide Assessments & measurement provide

feedback to studentsfeedback to students Clear standards for instructor response & Clear standards for instructor response &

availabilityavailability

Park UniversityPark University

Value of the ProcessValue of the Process

Value of the ProcessValue of the Process

The Administration has always been The Administration has always been concerned with maintaining concerned with maintaining academic integrity in all programs, academic integrity in all programs, on-ground and online.on-ground and online.

The goal was never to be “the The goal was never to be “the biggest”, just the best. And by being biggest”, just the best. And by being the best, we might be the biggest the best, we might be the biggest

Value of the ProcessValue of the Process

The adoption of the Quality Matters program The adoption of the Quality Matters program and ideology assures that the design and and ideology assures that the design and development of online classes will meet the development of online classes will meet the highest quality standards of academic highest quality standards of academic excellence.excellence.

The “master course” structure for online The “master course” structure for online courses, combined with using the QM rubric for courses, combined with using the QM rubric for online reviews, assures that a course AND all online reviews, assures that a course AND all sections of that course will met QM Standards sections of that course will met QM Standards and Expectations.and Expectations.

Value of the ProcessValue of the Process

SO 141 – Introduction to Sociology SO 141 – Introduction to Sociology exampleexample

Quality (at) TrainingQuality (at) Training…get off on the right foot

Training workshop required one year Training workshop required one year outout

Course examplesCourse examples Content designContent design Interactivity (communication)Interactivity (communication) AssessmentAssessment

Multiple online coursesMultiple online courses Principles of Good Practice Self-StudyPrinciples of Good Practice Self-Study Instructional Design TutorialInstructional Design Tutorial

Quality DevelopmentQuality Development…from the first word

Developing in stagesDeveloping in stages Checkpoint #1Checkpoint #1

Syllabus, sample lessonSyllabus, sample lesson

Checkpoint #2Checkpoint #2 ½ of the course, Mac½ of the course, Mac Content, interaction, assessmentContent, interaction, assessment

Copy editingCopy editing

Quality ReviewsQuality Reviews…before the course runs

Reviewing from multiple perspectivesReviewing from multiple perspectives Technical review (external)Technical review (external)

ContentContent CommunicationCommunication AssessmentsAssessments ServicesServices LinksLinks

Final Check (internal)Final Check (internal)

Quality EvolvesQuality Evolves…or becomes extinct

Critical factorsCritical factors Faculty buy-inFaculty buy-in

Student satisfactionStudent satisfaction

ObjectiveObjective Pre-tech reviewsPre-tech reviews Tech reviewsTech reviews

SubjectiveSubjective CheckpointsCheckpoints RubricsRubrics

1 Help Desk tickets as a percent of courses

2 Student negative comments on design as a percent of evaluations

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

SP00 FA00 SP01 FA01 SP02 FA02 SP03 FA03 SP04 FA04 SP05 FA05 SP06 FA06 SP07

Technical issues

in courses1

Student design complaints2

0

20

40

60

80

100

120

140

160

180

200

SP00 FA00 SP01 FA01 SP02 FA02 SP03 FA03 SP04 FA04 SP05 FA05 SP06 FA06 SP07

Courses

Technical calls & Technical calls & student evaluationsstudent evaluations

Quality Looks AheadQuality Looks Ahead…or falls behind

Planned changesPlanned changes TrainingTraining

Best practicesBest practices Advanced workshopsAdvanced workshops

DevelopmentDevelopment Community of PracticeCommunity of Practice Template/technique sharingTemplate/technique sharing

ReviewingReviewing cQual applicationcQual application

Wrap Up / Q&AWrap Up / Q&A

Final Thoughts…Final Thoughts…

Thank you!Thank you!