effective implementation formative and summative project evaluation

Effective Implementation

Formative and Summative Project

Evaluation

When Bad things happen to Good

projects!• Great ideas become ruined with bad implementations! – Brad Wiley English teacher case-text

• Innovations get bad press with bad implementations!

• Failed “good ideas”

Role of Implementation

• Diffusion• Dissemination• Adoption• Stakeholders

Adoption Process

• Awareness• Interest• Evaluation• Trial• Full Adoption• Integration

Adoption Models

• Research based• CBAM--Concerns-Based Adoption Model– Focuses on the concerns of the users•Stages of Concern Questionnaire•Level of Use Questionnaire

Implementation

• Role as a designer– Recommendations– Experiences

• Role as a teacher/user/learner– Limitations– Experiences

Assignment

• Using the case of Brad Wiley in Chapter 17, identify what principles of good implementation were ignored or violated and write a short implementation plan for Mr. Wiley’s project.

Formative and Summative

Evaluation of Design

• Assessments were used to determine student achievement

• Evaluation of the design and the instructional materials is the current topic.

Occurs at 2 times

• Evaluation of the materials and processes for the purpose of revisions--Formative

• Once materials and processes in place-they are evaluated in terms of their effectiveness-Summative

Formative

• Occurs during development of materials

• Revisions are expected!• In business-skipped due to cost!

Formative evaluation is

especially important if…• Designer is novice

• Content is new• Technology is new• Audience is new• Designer new to ID• New or unfamiliar instructional strategy

• Task performance is critical• Client expectations and later revision is not possible

Formative Review Stages

• Design reviews--before development

• Expert reviews--after development before use

• Learner validation--during use• Ongoing evaluation--during use

Design Reviews

• Goal Review

• Environment and Learner Analysis Review

• Task Analysis Review

• Assessment Specifications Review

Expert Reviews

• Content experts• ID experts• Educational specialists• Teachers• Target audience specialist

One-to-One Evaluation

• Tries materials with 2-3 members of target audience

• Sits down one-to-one--observation and questions– Typos, unclear directions, bad examples, poor illustrations, bad URLs

Small Group Evaluation

• Uses more learners…usually after the one-to-one revisions were made.

• Usually only observation, data gathered through charting

• Survey after can be used with learners.– pg336

Field Trials

• Determine effectiveness of revisions made after small group eval

• Find problems with administration of materials in real life situations

• Validate instruction with larger group to predict effectiveness

Which stage of the formative evaluation procedure does each

statement describe?

Teachers/trainers report the appropriateness of the documentation that accompanies a package of computer based instructional software..

Which stage of the formative evaluation procedure does each

statement describe?

Experts in the content area for which a slide-tape set was developed review the materials for accuracy and currency.

Which stage of the formative evaluation procedure does each

statement describe?

Individual students review the storyboards of an instructional videotape with the designer.

Which stage of the formative evaluation procedure does each

statement describe?

At least 30 students go through the instruction.

Which stage of the formative evaluation procedure does each

statement describe?

Designer only assists students when they encounter severe difficulties.

Which stage of the formative evaluation procedure does each

statement describe?

Students read and think aloud as they interact with written instructional materials.

Which stage of the formative evaluation procedure does each

statement describe?

Test materials are tried out with groups of individuals some of whom are “masters” of the content, some of whom are “nonmasters.”

Which stage of the formative evaluation procedure does each

statement describe?

Data are collected regarding problems with the instruction after the instruction is implemented in the target contexts.

Summative Evaluation

• Final evaluation of effectiveness of the design, materials and learning.

• Usually will impact continued use

• Determine goals of the evaluation--usually determined by client

Group Assignment

• Determine what formative evaluation process you could use with your instructional design.

• Discuss and decide as a group• Complete Formative Evaluation Documents and submit to Cheryl by e-mail…one per group