the online academy (toa) john baek michelle didier dan feinberg lisa herbert danalyn robinson...

The Online Academy(TOA)

John BaekMichelle Didier

Dan FeinbergLisa Herbert

Danalyn RobinsonHeather White

Overview

The project– An online high school– A partnership between GMU and Northern VA

county school systems

Overview

Review of last semester– Focus on performance analysis, design &

development

Mission Statement• Develop high quality courses, correlated to the

Virginia SOL and COPLS model• Support the expert mentors

Overview

• Redirection of focus

• Focus this semester– Framework for continued development: code library,

site structure, consistent design using style guidelines and templates

– Field testing– Evaluation & usability testing– Complete the Intro module/ Resource guides

Overview

• Our process– Team meetings – SME (teachers) meetings– Project management– Divide & conquer

Presentation Outline

• Evaluation– Evaluation needs & goals– Methods– Findings & recommendations

• Future of TOA– Framework for future development– Project recommendations

Evaluating TOA

Need for Evaluation

Methods of Evaluation

Findings and Recommendations

Need for Evaluation• Inform and direct current and future development

• Examine the usability of the checklist– Organization and self-regulation

• Determine effectiveness of navigation– Locating resources– Intuitiveness—do the learners know where to go next?

• Review the course modules– Clarity of instruction– Engagement and motivation of instruction – Adequacy of the resources, skills, and content for challenge– Pedagogical re-engineering (approach and format)

Evaluation Goals

• Analyze instructional strengths • Determine the effectiveness of the design

and transfer of learning • Evaluate the usability and the benefits of

the checklist to the learner • Locate ease of use problems (navigation)

Methods of Evaluation

• Quality assurance

• Field testing

• One-to-One evaluation

• Expert review

• Case study

Meeting the Evaluation Goals

Instructional strengths

•One-to-one evaluation•Expert review•Case study

Effectiveness of the design and transfer

•Quality assurance•Case study•Expert review

Checklist Usability

•Expert review•Field test

Locate ease of use problems

•Field test•One-to-one evaluation

Quality Assurance

For each course, at least two modules were field tested.

•Chemistry•Mars Matters•Sappy Solutions

•Earth Science•Passing Gas…For Effect Only•Like It…Weather or Not

•Algebra•Graphing – Slopes

•Geometry•Right Triangles/ Trigonometry•Points, Lines, Planes, Segments, and Angles

•English 11•The Two Macks: A Harlem Renaissance Club•Regionalism, Realism, and Nationalism

•World History I•India and China•African Civilizations

•World History II•Imperialism•Between the Wars

Before anything was field tested, the modules were evaluated for Quality Assurance.

• Forms that don’t work• Typing mistakes• JavaScript errors• Form elements not properly labeled• Pages that don’t print correctly• Consistent styles• Missing page titles or code comments

The result is a much more polished looking product.

Field Testing

Field Testing

• 14 Modules

• Technical problems

• Student response

• Self regulation

• Synchronous discussion (Digichat)

Digichat• Encourages peer

interaction• Encourages

mentor-learner interaction

• Mentor office hours

Intro Module

Intro Module• Need for Intro Module• Elements of most

modules• Use basic TOA

functions

Mentor Resources

• Overview• Objectives• Answer keys

Challenge• Problem based context for entire module• Background building knowledge• Culminating activity

The Checklist•Self regulation

•Mentor-student negotiation

•Links to activities

TOA Introductory Module

• Research for the module was gathered from several sources– Teachers’ feedback from their field testing– Results from one-to-one evaluation– Review of related literature

• Maintain theme of the story introduced in the Challenge

Working with a Mentor

Working with a MentorActivity

Being an Online Learner

Being an Online LearnerActivity

Online Research

Online Research

Online Research

One-To-One Evaluations

A face to face meeting with members of our target audience where we ask them to use TOA and give us their opinion on

the site.

• Are the instructions clear?

• Can the user locate resources?

• Can learners navigate?

• Are the resources, skills, and content adequate for completing the challenge?

•Is the instruction engaging and motivating?

•Will the student know the references?

•Is the content at the appropriate level?

•Does the checklist provide self-regulatory benefits?

One-To-One Evaluations

1. Recruit students.

2. Prepare evaluation questions.

3. Design data collection tools.

4. Write script.

5. Set up testing area.

6. Conduct evaluation.

7. Analyze results.

8. Write evaluation report.

One-To-One Evaluation Process

1. Recruit Students

• We were provided access to two students at a Washington, DC private school

• “Ralph” – A 16 year old white male currently enrolled in Chemistry

• “Kelly” – A 16 year old white female who had taken Chinese History

2.Prepare Evaluation Questions

3. Design Data Collection Tools• Tape recorded and

transcribed• Field notes were

taken by interviewer and observer

• Follow up questionnaire was given

4. Write Script

5. Set Up Testing Area6. Conduct Evaluation

7. Analyze Results

8. Write Evaluation Report

World History I Module

World History I Module

World History I Module

World History I Module

World History I Module

World History I Module

World History I Module• Summary of students’ impressions:

– Kelly:• Liked the story presented in the challenge and carried

through the activities• Liked the PowerPoint activities• Thought the activities were explained well• Liked self paced aspects of course

– Ralph:• Navigated using pull-down menus• Liked self paced aspects of course• Unsure of being motivated enough to complete activities• Liked the layout and color scheme of the site

Expert Reviews

Expert Review Steps

1. Recruit experts

2. Prepare questions

3. Design data collection tools

4. Set up testing area

5. Conduct evaluation

6. Analyze results

7. Write evaluation

Dr. Dabbagh’s Interview

• Earth Science Module on Gas

• Open questions based on likes and dislikes

• How learning theories are applied

• Does this address multiple learning styles?

Earth Science

Passing Gas…

The Checklist

An Activity

Dr. Dabbagh’s Comments

• Thought the content was very situated in real life scenarios

• Liked the sub activities under the challenge, how main challenge is umbrella

• Liked the graphics, thought they were appropriate to content

• Liked the embedded links to additional information

Dr. Dabbagh’s Recommendations

• Main module page needs to reflect flow of lessons (ex. If there is an order to a lesson, then having all the links available makes users think they can skip around.)

• Students need to have a better understanding of where they are in the order of things.

• Unclear how the module works, ex. the challenge is actually done last.

• Needs to closely match the checklist with the verbiage of the lessons.

• Include the amount of time available to complete module and the number of hours students are expected to devote to the module.

• Should provide opportunity for reflection.

TOA Recommendations

• We suggest that curriculum writers strive to write clear, specific instructions for activities.

• The titles of lessons and activities should also be more

consistent and simple. • Another recommendation is to use consistent wording in

the checklist, lesson, and activities so they match each other.

• Design opportunity for reflection into The Online Academy.

The Checklist– A self-regulation tool– List activities & due dates– Negotiated with mentor

The Checklist

Evaluating the Checklist• Expert review: Dr. Kitsantas &

Dr. Dabbagh

• One-to-one evaluation

The ChecklistThe Findings

Areas for improvement:• Have students set goals by adding columns for

planned, actual and revision dates• Keep track on progress • Self-reflect on how they performed• Provide scaffolding and modeling• Link all activities

The Checklist

Recommendations

• Support the checklist with a database to track:• Dates: planned, actual and revision• Record grades

• Link key activities

• Provide scaffolding for completing activities within lesson

• Provide opportunity for reflection: by mentor or in module

The Checklist

Suggested Checklist

Case Study

Background

• Research Methods course project

• Research partner, Kimberly McDuffie

• Quantitative & qualitative methods requirement

Purpose

• To examine the experiences of students with learning disabilities with TOA’s design (not academic achievement)

Research Questions

• How does the students’ relationship with the mentor impact the online learning experience?

• Do certain design elements of The Online Academy accommodate students with a learning disability?

Literature Review

• Use of computers with students with learning disabilities (LD)

• Use of hypertext with students with LD

• Multiple design elements in technology-based instruction

• Lack of research in specific study meant exploratory approach

Methods

• Participants– 2 high school Geometry students– Recruited because there was field testing in

inclusive classroom

• Measures– Survey– Interview

Survey Results• Attempt to measure the aspects of the

mentor-student relationship:

Time Spent 2.88 (3-5/week)

Quality of Communication 4.20 (Often)

Amount of Communication

3.13 (3-5/week)

Use of Feedback 3.00 (Sometimes)

Interview Codes

• Clarity of instruction• Chunking• Readability• Navigation• Asynchronous communication• Synchronous communication• Organization of their learning

Conclusions

• Overall – LD students would not do it again, but recommended it for others

• Preferred physically present teacher

• Expressed difficulty communicating their learning problems through text

• Nothing positive reported about design elements

Limitations

• Field test was artificial, done in a classroom

• Self-made survey instrument, not validated.

• Only two students, no external validity

Geometry Module

Geometry Challenge Page

Geometry Assignment

TOA Evaluation Summary

Evaluation informs design

• QA plan

• Field tests

• One-to-one

• Expert review

• Case study

The Future of TOA

The TOA Code Library

The Print Version• Printable version of Code Library to keep as a

reference• Code is color coded to find editable areas

Online Version• Online Code Library is available on the VHS

website• User can copy and paste code into their html

document

What’s Left To Do

• Find a Partner

• Login for site and Digichat

• Retroactive changes based on our evaluation findings

• Mentor resources

Thanks

• Priscilla Norton• Brenda Bannan-Ritland• Bill Warrick• All of the Virtual High School teachers• Nada Dabbagh and Anastasia Kitsantas• Graduate School of Education• Stafford, Loudon & Frederick school divisions