module three assess the outcomes davidson county community college may - 2011

Module Three Assess the Outcomes

Davidson County Community CollegeMay - 2011

Updating Curriculum to Support LearningDCCC Perkins Grant

III - 2

Module Three: Assess the Outcomes

TopicsThe Assessment Toolkit

Using Authentic Assessment to Assess SLOs

Selection of Direct and Indirect Methods of Assessment

Selection Of Test Question Types To Measure To Specific SLOs.

Test Item Analysis

Outcomes: At the completion of Module Three faculty should be able to:

1.Explain why a variety of assessments are necessary.

2.Distinguish between direct and indirect methods of assessment.3.Discuss the concept of authentic assessment and design an authentic assessment for your content area.

4.Select appropriate methods of assessment to match SLO to be measured.5.Explore the use of CATs to assess student knowledge of content, attitudes and self assessment and reactions toward teaching and teaching materials.6.Identify questions you want answered about your teaching and learning and select an appropriate CAT to answer the question.

7.Use a rubric to assess Student Learning Outcomes (SLO).8.Utilize Bloom’s Taxonomy to design test questions to align with student learning outcomes being measured.

9.Evaluate tests items to validate that they indeed measure the SLO.

10.Identify the strengths and weakness of different types of test questions.

11.Select appropriate type of test questions to measure specific SLOs.12.Create an assessment plan that outlines the specific methods that will be used to assess the specific expected student learning outcomes for a course.

13.Create a matrix that shows the appropriate activity and assessment for specific SLOs.

14.Write effective test items by applying helpful hints.15.Use a Test Blueprint to evaluate tests items to validate that they indeed measure the SLO and provide challenges for students at higher levels of learning.

Table of Contents – Module ThreeStage Three: Assess the OutcomesWhat is Assessment and Why Do IT?

4

Direct and Indirect AssessmentAuthentic Assessment

5

The Assessment Toolbox 6

Toolbox of Authentic Assessment 7

Designing Authentic Assessment Worksheet 8

CATs – (Classroom Assessment Techniques) 9-10

Using CATS to Answer Questions About Teaching and Learning 11

Using Rubrics for Direct Assessment of Student Work 12

Rubric Worksheet 13

Outcomes/ Assessment Alignment Worksheet 14

Using Bloom’s Taxonomy to Write Test Questions 15

Direct Verbs Provide Clues 16

Pros and Cons of Types of Tests 17

How to Write Effective Questions in Various Formats 18-19

Test Blue Print 20

Teaching Approach Assessment Matrix 21

My Plan 22


III - 3

Stage Three: Assess the Outcomes

How do I assess student learning outcomes?

Assess

Assessment is an ongoing process that investigates student learning. The continuous improvement aspect of the assessment cycle models the commitment to the quality of learning within a program and a college.

What is Assessment and Why Do It?

The Assessment Process

Source: Fulks, Janet. (2004) “ The SLO Toolkit.” Bakersfield College.


III - 4

Types of Assessment

Direct assessment methods are measures of student learning that require students to display their actual knowledge and skills rather than report what they think their knowledge and skills are.

(Summative)

Indirect assessment methods ask students to reflect on their learning rather than to actually demonstrate it. (e.g., student opinions about what they learned and or student ratings of instruction. Also, other self report data like employer surveys, advisory group.)

(Formative)

Capstone project Classroom Assessment Techniques (C.A.T.s)

Case studies Focus groups

Demonstrations Graduation and retention rates

Exams, quizzes, and tests Interviews

Internship and/or clinical evaluations Job/graduate school placement statistics

Licensure exams Reflective journals

Performance assessments Surveys (employer satisfaction)

Portfolio evaluation Advisory Group suggestions

Pre and post-tests

Simulations

Using Authentic AssessmentWhat is it? “A form of assessment in which students are asked to perform real-world tasks that demonstrate meaningful application of essential knowledge and skills., says Jon Mueller.

Traditional Assessment Authentic Assessment

Students select an answer or recall information to complete the assessment such as in test like multiple-choice, true/false, matching or fill-in-the blanks

Student perform a task which is simulated to a real word situation

Focuses on recall/recognition Requires higher order thinking

Norm-referenced that can compare or rank students

Criterion-referenced that can identify strengths and weaknesses


III - 5

The Assessment Toolkit

Tools

Capstone Projects: Assessment of the integrated the skills, knowledge, and abilities of students at the completion of a program of study.

CATs: Classroom assessment activites used to gather information about student levels of knowledge about the content, about how they perceive they are learning and what they perceived about the instructor’s ability and the instructional materials.

Expected Learning Outcomes Statements Survey: Assesses directly how well students feel that they have mastered the expected learning outcomes for the course. Students complete a brief survey that asks them to rate how well they have learned each outcome. The simplest way to do this is to list each stated expected learning outcome and have students indicate on a scale how well they feel they have mastered each outcome. Leave space for students to provide evidence that they successfully mastered the outcome.

Performance Assessment: Students demonstrate skills and abilities in a real-time situation. Performance assessments, like portfolios, require well-designed instruments, criteria, rubrics, and norming between reviewers.

Portfolios: A display of students’ abilities through a collection of artifacts. Portfolios are good ways to assess change over time. Instructions to the students must be explicit, based upon the purpose and uses of the portfolio. Rubrics should be available to show students what type of evidence needs to be collected and what criteria they must achieve.

Non-Graded Quizzes: Activities that provide great feedback on student learning. Non-graded quizzes can be given in a variety of formats (paper & pencil, questions in a PowerPoint presentation, etc.). Be sure to provide students with the correct answers to the quiz items, so they can learn from their mistakes! This choice of assessment is a great way to provide immediate feedback to the student.

Reaction Papers: Require students to write down their reactions (thoughts, feelings, attitudes, etc.) to course material or topics. Although reaction papers can be formal graded assignments, they can also be used solely for the purposes of assessing student learning. In this case, faculty members ask their students to take a few minutes of class time and write down their reaction to a fact, opinion, skill, procedure, attitude, critique, etc. that was addressed in the course. The CAT called the “Minute Paper” is a sample of a reaction paper.

Rubric: A scoring tool that lays out the specific expectations for an assignment. Rubrics divide an assignment into its component parts and provide a detailed description of what constitutes acceptable or unacceptable levels of performance for each of those parts.

Student-Generated Exam Questions A process where instructors ask students (either alone, in pairs, or in groups) to generate potential exam questions. Having students create potential exam questions helps the instructor assess (1) If students can identify key concepts; (2) What students see as fair or reasonable exam questions; and (3) How well the students can answer these potential exam questions. Using this approach will motivate students to read the assignments and prepare better for tests.


III - 6

Tool Box of Authentic Assessment

Assignments / Assessment

Descriptions

Application Cards

On an index card have student choose from a list of principles or concepts you just discussed and indicate how they will apply it.

Concept Map These are visual images of important concepts. Uses a variety of ways of learning, visual, spatial, linguistic, logical, mathematical and kinesthetic. Engages both sides of the brain. Students remain involved throughout the session. There are cluster Maps… flow chart maps, Burger Map, Free Flow map, and star-burst.

Graphic Organizers

Use such things as a categorization grid allows you to organizes concepts and characteristics of those concepts; a Fishbone diagram to discover “cause and effect”; ; Ranking Ladder to sort a list and then rank items by placing them in a particular order; T-Graph is useful during brainstorming. You see issues and concerns related to both sides of a topic a time line to show historical events; 5 W’s chart is an introductory experience to help you gather information and summarize; Venn-diagram to “compare and contrast” concepts,

In the News Have students bring to the class articles, new items, editorials or cartoons related to the topic. Divide the class into small groups and ask members of each subgroup to share their item with one another and to choose the 2 or 3 most interesting items. Reconvene and ask members from sub-groups to share.

Media Diaries Each student tracks a current issue on TV, radio, and in print sources and compares and contrast the like and different ways each focuses on the issue.

Discipline Specific Examples of Authentic Learning

Geometry You want to install new carpet and a baseboard in you room, but you need to know how much carpet and wood to purchase. Write a set of directions to follow, measure and calculate the square footage of each item needed, and determine the cost of carpet and pad (use the Internet to research).

Economics While studying the stock market and its effects on the economy, students will “purchase” a stock or stocks with $5000 and plot their earnings over a three month period.

English Research a career that interests you and report on the educational and job requirements, career outlook, job description, etc.

Health Record and analyze your food consumption over a given amount of time.


III - 7

What Do You Already Do To Assess Students? _____________________________________________

_____________________________________________

_____________________________________________

_____________________________________________

Student Learning Outcomes Authentic Assessment Idea

Designing An Authentic Assessment Worksheet

Appropriate Assessment Strategies

“…best identified through careful consideration of the nature of the task, the cognitive level of the objective, and the most feasible context for the assessment.”

Source: Garavalia, L, Marken, P and Sommi, R. (2003) “Selecting Appropriate Assessment Methods: Asking the Right Questions” American Journal of Pharmaceutical Education Vol. 66, Summer 2003 p. 111


III - 8

Examples of CATs

CAT Description

Assessing Course Related Knowledge And Skills

Application Cards After teaching about an important theory, principle, or procedure, ask students to write down at least one real-world application for what they have just learned to determine how well they can transfer their learning.

Background knowledge probe

Used to determine what students already know to determine a starting point for new instruction

Categorizing Grid Give students a grid containing two or three main categories plus a scrambled list of terms and have them sort the terms into the categories.

Concept Maps Students draw or diagram the mental connections between a major lecture concept and other concepts they know. This helps students to see connections.

Direct Paraphrasing Ask students to write a layman’s “translation” of something they have just learned to assess their ability to comprehend and transfer concepts.

Documented Problem Solutions

Students are asked to document the steps they take in solving a problem. Students will gain more awareness of and control of their problem solving routines.

Focused Listing Give students a main term or concept and have them generate all the related terms they can think of. Practicing this technique can help students learn to focus attention ,improve recall and become familiar with terminology.

Muddiest Point Have students write what they find least clear or most confusing about a particular lesson or topic.

“What point is confusing to you?”

One Minute Paper During the last few minutes of the class period, ask students to answer on a half-sheet of paper:

“What is the most important point you learned today?”

One Sentence Summary

Students summarize knowledge of a topic by constructing a single sentence that answers the questions ,”Who does what to whom, when, where, how, and why?” The purpose is to require students to select only the defining features of an idea.

Pro and Con Grid Have students create a grid of the pros and cons for a given decision. This forces students to search for two sides of an issue or question.


III - 9

CAT Description

Assessing Learner Attitudes, Values, and Self-Awareness

Classroom Opinion Polls

Students respond to a survey or by hand-raising and quick counts to items presented by the instructor.

Diagnostic Learning Logs

Students keep records about assignments they have done and the kind of errors or successes they experienced. At regular intervals they go back over these logs and reflect on their learning strategies.

Double-entry Journals In taking notes on class or readings, students keep a simultaneous running commentary of questions, ideas and connections in a second column.

Interest/Skill/ Knowledge Checklist

The instructor prepares a list of course-specific skills, interests and knowledge and students assess themselves with regard to their interest or level of skill.

Punctuated Lectures

Periodically during the lecture, the lecturer pauses to allow students to reflect and write on their own attention and behavior during the class in order to identify how actively they are listening or participating.

Assessing Learner Reactions to Instruction

Do it Yourself Quiz Students are asked to write out the five (5) questions about the key points in the reading and then, answer each. This assignment allows the teacher to see who has done the reading and what comprehension they have of the reading.

Film Feedback After viewing a film, the instructor asks students to answer the following two questions: 1. Explain (in as much detail as you can) the main idea illustrated by the film. 2. What can you suggest that I do to make using films a more useful learning experience for a student like you?

Handouts Feedback I need your help to decide which handouts are most useful to you. 1. The handout that was most useful to me was . . .2. The reason this handout was so useful was . . .3. The handout that was least useful to me was . . .4. The reason this handout was least useful was . . .

Teacher Feedback Have students answer the following questions anonymously. 1. One thing that you (the instructor) do that helps my learning is . . .2. One thing that you do that interferes with my learning is . . .3. A practical suggestion I have that will improve the class for me is . . .

Examples of CATs


III - 10

Name of CAT

PurposeCAT seeks to get information or to answer the following:

Results Ways to Improve Instruction

CAT Results Form

Course:

___________________________________________________________________________

Questions You Want Answered About Teaching and Learning

? ________________________________________

? ________________________________________

? ________________________________________


III - 11

Using Rubrics for Direct Assessment of Student Work

What are the parts of a rubric?

1. A task description (the assignment)2. A scale of some sort (levels of achievement, possibly in the form of grades).

Scales typically range from 3 to 5 levels.3. The dimensions of the assignment (a breakdown of the skills/knowledge

involved in the assignment)

4. Descriptions of what constitutes each level of performance (specific

feedback)

Four Formats for Rubrics

1. Checklist

2. Rating Scale

3. Holistic Rating Scale

4. Descriptive Rubric

Task Description: ______________________

Scale Level I

Scale Level 2

Scale Level 3

Dimension 1

Dimension 2

Dimension 3

Descriptions of Dimensions


III - 12

__________________________________________ Rubric

Task Description: _________________________________________

Dimensions

Scales

Rubric Worksheet

Build your rubrics online at: http://rubistar.4teachers.org/index.php


III - 13

http://rubistar.4teachers.org/index.php

Select an SLO from your course syllabus and describe how you would teach to that outcome and how you would assess it.

Student Learning Outcome:

Ways I can teach the outcome

Ways I can assess or measure the outcome.

Outcome AssessmentAlignment Worksheet


III - 14

Student will experience more success in your classes if they know up front what is expected of them.

Using Bloom’s Taxonomy to Assess Student Learning Outcomes

Levels of Learning Action Verbs

CreatePutting together ideas to develop an original idea.

Action Verbs: assemble, compose, create, construct, design, develop, formulate, imagine, invent, plan, write

Example: 1. Plan appropriate science activities for young children. 2. Construct a basic web page using HTML.

EvaluateJudging the value of ideas, materials and methods by developing and applying standards and criteria

Action Verbs: appraise, argue, assess, critique, defend, evaluate judge, measure, predict, prioritize, rate, select, support, validate, value

Examples: 1. Select the project that reflects use of quality animation and sound.2. Defend the use of use of the Lock-out/Tag-Out system.

AnalyzeBreaking information down into its parts

Action Verbs: analyze, compare, contrast, criticize, debate, detect, differentiate, discriminate, distinguish, examine, experiment, inquire, inspect, investigate, question, separate, test, troubleshoot

Examples: 1. Troubleshoot a piece of equipment by using logical deduction. 2. Inspect equipment’s safety features before start up.

ApplyUsing strategies, concepts, principles and theories in new situations

Action Verbs: apply, calculate, choose, compute, demonstrate, determine, develops, dramatize, draw, employ, illustrate, interpret, manipulate, practice, operate, schedule, show, sketch, solve, use.

Examples: 1. Choose the appropriate color for a clients hair texture.2. Operate tools associated with the HVAC trade.

UnderstandExplaining the meaning of information

Action Verbs: classify, describe, discuss, explain, identify, locate, recognize, report, restate, translate, paraphrase

Examples: 1. Describe the parts of a computer.2. Translate passages from your Spanish text into English.

Remember Recalling or recognizing specific information

Action Verbs: define, duplicate, label, list, match, memorize, outline, quote, recall, recite, repeat, reproduce, show, state,

Examples: 1. Outline the steps in giving CPR.2. Label the diagram of the respiratory system.

Bloom’s Revised Taxonomy

Source: Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1973). Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. New York: David McKay Co., Inc.


III - 15

http://www.amazon.com/Taxonomy-Educational-Objectives-Classification-Goals/dp/0679302115/bigdogsbowlofbis

Questions about Writing Test Items

_____________________________________________

_____________________________________________

_____________________________________________

_____________________________________________

O.& E


III - 16

Pros and Cons of Type of Tests

Test Type Pros ConsMultiple Choice

More answer options reduce the change of guessing

Measures knowledge, application and analysis goals

Can cover a wide sampling of content or outcomes

Student have to spend more time reading more answers so reduces the number of questions to be presented

Difficult to write 4 or 5 reasonable choices

Takes more time to write them.

True False Can present many items Easy to score Used to assess misconceptions,

cause-effect reactions

Most difficult question to write objectively

Ambiguous terms can confuse Increases chance of guessing

Matching Efficient Used to assess student

understanding of associations, relationships and definitions

Difficult to assess higher-order outcomes.

Essay Measures higher order outcomes

Encourages more appropriate study habits

Easier to construct

Takes more grading time Not efficient to test large amounts

of content If you give the choice of 3 or more

essay options, you can find out what they know, but not what they don’t know.

Performance Assessments

Measures higher order outcomes

Focus on ability of students to apply skills or knowledge in real life situations.

Labor and time-intensive Tests student individually and are

not for testing groups. Low reliability

Problem Based Scenarios

Allows for student creativity Focuses on student application

of knowledge

Time consuming to write Answers may vary Low reliability


III - 17

How to Write Effective Questions in Various Formats

Check Multiple Choice - Helpful Hints

Eliminate excessive wording or irrelevant information from the stem.

Include in the stem any word(s) that might have otherwise been repeated in each alternative.

State the stem as a direct question rather than as an incomplete statement

Make all alternatives plausible and attractive to the less knowledgeable or skillful student.

Make sure there was only one correct or best response per item.

Make alternatives approximately equal in length.

Present alternatives in some logical order (e.g., chronologically, most to least).

Randomly distribute the correct response among the alternative positions throughout the test having approximately the same proportion of alternatives a, b, c, d, and e as the correct response.

Use at least four alternatives for each item. Make the alternatives grammatically parallel with each other, and consistent with the stem

Use negatively stated stems sparingly. When used, underlined and/or capitalized the negative word(s).

Use the alternatives "none of the above" and "all of the above" sparingly. When used, such alternatives were occasionally the correct response.

Source: How to Write Better Tests http://www.indiana.edu/~best/write_better_tests.shtml

Check the hint(s) you need to adopt.


III - 18

How to Write Effective Questions in Various Formats

Check Matching Helpful Hints

Avoid grammatical or other clues to the correct response

Explain whether or not a response could be used more than once and indicate where to write the answer.

Include directions which clearly states the basis for matching the stimuli with the response.

Include more responses than stimuli.

Keep the lists of items short and put the brief responses on the right

When possible, arrange the list of responses in some systematic order (e.g., chronologically, alphabetically).

When possible, reduced the amount of reading time by including only short phrases or single words in the response list.

Writing Essay Questions - Helpful Hints

Ask questions that will elicit responses on which experts could agree that one answer is better than another.

Avoid giving the student a choice among optional items as this greatly reduces the reliability of the test.

Indicate for each item a point value or weight and an estimated time limit for answering.

It is generally recommended for classroom examinations to administer several short-answer items rather than only one or two extended-response items.

Prepare essay items that elicit the type of behavior you want to measure.

Phrase each item so that the student's task is clearly indicated.

Use the verbs from Bloom’s Taxonomy to write essay questions to test different levels of thinking.

Source: How to Write Better Tests http://www.indiana.edu/~best/write_better_tests.shtml


III - 19

Date: ____________

# Items: __________

Test on: _________________________Levels of Thinking

Outcomes Remember Understand Apply Analyze Evaluate Create Total/%

1.

2.

3.

4.

Total/%

Instructions:

1st: Analyze the test you have written by examining each question (item) and determine

at which level of critical thinking this is written. 2nd Determine which Student Learning

outcome each question measures. Write the number of the question in the box that

coincides with the correct SLO tested and the level of Critical Thinking. Once all items

have been checked add the total items for each outcome on the vertical total line, as

well as for each level of critical thinking on the horizontal total line. The percentages

under the vertical column will indicate how much of the test is written with higher level

thinking in mind. You can also determine which SLOs need more questions to

adequately measure them.

Test Blueprint Worksheet


III - 20

Teaching Approach / Assessment Matrix

SLO Teaching Approach Assessment

1.

2.

3.

4.

5.

Write the SLOs in the first column on the left. In the center column indicate what you do to teach to that SLO and in the last column on the right indicate how you assess the outcome.


III - 21

1. Course in which I wish to make changes:______________________________________________

2. Effective assessment principles I will adopt: ________________________________________________________________________________________________________________________________________________________________________________________

3. Strategies I plan to use: ________________________________________________________________________________________________________________________________________________________________________________________

My Plan for Assessing Outcomes

“I never teach my pupils; I only attempt to provide the conditions in

which they can learn.” Albert Einstein


III - 22

One thing I will take from the workshop that I can apply is …

_________________________________________________________________________________________________________________________________________________________________________________

I would have learned better if you would have…

_________________________________________________________________________________________________________________________________________________________________________________

The handout materials were…

_________________________________________________________________________________________________________________________________________________________________________________

Using the scale of 1 to 5, please place a check by the number where would you rate how you felt your expectations were met for this session.

Session Evaluation

1 2 3 4 5

Did not meet my expectations

It was ok I learned some things

It met my expectations

It met and exceeded my expectations


III - 23

module three assess the outcomes davidson county community college may - 2011

Documents

types of assessment

assessment cycle

self assessment

assessment plan

form of assessment

student learning outcomes

test blueprint

test question types