assignment 2 – evaluating assessment - web viewassignment 2 – evaluating assessment. ......
TRANSCRIPT
ASSIGNMENT 2 – EVALUATING ASSESSMENT
3143VTA - VOCATIONAL ASSESSMENT
CONVENOR: DR. IAN JAMES
SUBMITTED BY: DAVID MARTIN
STUDENT #2636349
DATE SUBMITTED: 14 MAY 2008
2
Assignment 2 – Evaluating Assessment
CONTENTS
Outline...........................................................................................................................................3
Outcome 1: Speak confidently to others........................................................................................4
Outcome 2: Writes comprehensive reports...................................................................................5
Outcome 3: Is sensitive to other cultures......................................................................................7
Outcome 4: Uses whiteboard effectively during instruction...........................................................8
Conclusion.....................................................................................................................................9
List of References........................................................................................................................10
Appendix 1: Marking Criteria Sheets...........................................................................................11
Appendix 2: Assessment Matrixes..............................................................................................15
Outcome 1: Speak confidently to others..................................................................................15
Outcome 2: Writes comprehensive reports..............................................................................16
Outcome 3: Is sensitive to other cultures.................................................................................17
Outcome 4: Uses whiteboard effectively during instruction.....................................................18
David Martin 3143VTAStudent #2636349 Vocational Assessment
3
Assignment 2 – Evaluating Assessment
OUTLINE
This project will analyse the development of four different instruments in terms of their
effectiveness in developing expertise against set learning objectives. There are four learning
objectives, all containing complex outcomes that require high order knowledge and procedures.
The analysis will be done by considering relevance, balance, efficiency, objectivity, difficultly,
validity, reliability and fairness. It will also be determined what aspects of the objective will and
won’t be measured.
In all cases some form of extended assessment has been used to enable the rating of
performance by using a predefined list of checks. The advantages of using this type of
assessment will be demonstrated in each of the cases below. However, it can be argued that
there are several concerns about performance assessment; it can be more expensive, more
time consuming and may require special equipment (Marsh, 2000, p. 250). “Designing
assessment involves balancing competing demands and making practical and reasonable
trade-offs between requirements” (McCurry, 1992, p. 231) while still making sure that the
assessment is valid and reliable. The checklists that have been developed will assist in
providing more consistent results.
All of the assessment items used in this project are based around the idea of extended
responses. “Extended response questions give students almost unlimited freedom to
determine the form and scope of their responses” (Gronlund, 2006, p. 116) and where select
response items are low in complexity because they typically present a limited problem and
require the choice of the best answer (Gronlund, 2006, p. 17), extended questioning allows
complex ideas and knowledge to be assessed. For complex learning outcomes extended
responses allows more realistic and accurate assessment. When grading extended response
David Martin 3143VTAStudent #2636349 Vocational Assessment
4
Assignment 2 – Evaluating Assessment
answers a rating method can be used, by first defining a set of criteria as a guide allowing the
marker to grade each answer (Gronlund, 2006, p. 122). Appendix 1 shows the four criteria
marking sheets and grading guides to be used.
These instruments were developed by using cognitive, affective and psychomotor taxonomy
principles. Appendix 2 shows the assessment matrixes used and which domain and processes
are used in each of the criteria used as part of the assessment instruments.
OUTCOME 1: SPEAK CONFIDENTLY TO OTHERS
The first learning objective has been defined as speaks confidently to others by demonstrating
use of verbal and non-verbal communication, listening and responding to questions and
applying skills of negation or conflict resolution. This objective will be assessed by using a set of
role play activities. Oral assessment is useful for testing interpersonal skills and interactive
thinking (McCurry, 1992, p. 35). In this assessment the candidate is asked to respond to three
different situations to assess their ability. The overall objective has been defined by criteria
based on cognitive and affective domains. While the cognitive domain helps assess high level
thinking this learning objective also uses the affective domain; receiving, responding and
organisation processes are also used and assessed.
Most assessment involves making judgements about products but the processes involved such
as thinking skills and problem solving skills are also important (Withers & McCurry cited in
Marsh 2000 p. 245). The assessment designed for this learning outcome needs to take into
account these underlying processes. To do so a rubric has been designed to assign ratings.
However the measurement of success in this assessment is very subjective but unavoidable for
this learning outcome.
David Martin 3143VTAStudent #2636349 Vocational Assessment
5
Assignment 2 – Evaluating Assessment
While the learning outcome is related to speaking it can be argued that oral communication is
an interactive process that involves listening, speaking, verbal and non-verbal components
(Mead & Rubin, 1985). This assessment will not be able to cover all of these processes, but
instead will focus on the speaking component of the communication process.
One problem that could occur with this assessment is that the situations may be based on
situations or ideas that are unfamiliar to some cultural or socio-economic backgrounds,
increasing the difficulty of the assessment. This would mean that the validity and fairness of the
assessment would be reduced. Alternative scenarios should be on hand, or candidates could be
asked to choose scenarios from a list. However this also reduces the validity and fairness of the
assessment as all candidates should be assessed using the same material.
By giving a time limit on each of the role plays the assessment becomes efficient enough to
apply to regular class numbers and by using the marking sheet to assist with validity and
reliability it is feasible to assess the specified learning outcome.
OUTCOME 2: WRITES COMPREHENSIVE REPORTS
The second learning objective has been defined as demonstrate the writing of comprehensive
reports by use of correct spelling, grammar and report structure in a portfolio of 3 reports that
show correct use of the scope, audience, purpose and topic knowledge. By using a portfolio this
instrument will be able to show improvement over time and can include both self assessment
and teacher assessment (Gronlund, 2006, p. 157). Critical reflection can be included in the
portfolio by using a form “provides questions and a space for answering” (Gronlund, 2006, p.
160). This encourages active cognitive processes where learners construct their own knowledge
(Marsh, 2000, p. 254) and builds valuing and organising processes in the affective domain.
David Martin 3143VTAStudent #2636349 Vocational Assessment
6
Assignment 2 – Evaluating Assessment
Work sample assessment is widely used where performance skills need to be measured
(Gronlund, 2006, p. 139) and in this assessment, requires the student to create actual reports
and collect work that they have completed over time.
Criticisms of portfolio assessments are that reliability is low, they can be difficult to store and
equity issues arise with access to resources (Barry & King, 1998, p. 348). In this assessment it
may become difficult for the assessor to quickly gauge the topic depth because the topic for
each report has not been defined. It is important for candidates to be aware of marking criteria
before starting the assessment to provide direction for student learning (Gronlund, 2006, p.
161).
While considerable contact between the teacher and student is required for this assessment,
this contact time will help overcome issues of fairness, discrimination and difficulty. While
portfolio assessments “are time consuming to maintain and use” (Gronlund, 2006, p. 158) and
there can be major costs involved in staff training, development of specification and
administration (Marsh, 2000, p. 257) this is a feasible instrument for these learning outcomes.
Providing that criteria are set out before starting the assessment, and enough time and
resources are available to help administer the portfolio this instrument will be able to assess the
learning outcomes required.
David Martin 3143VTAStudent #2636349 Vocational Assessment
7
Assignment 2 – Evaluating Assessment
OUTCOME 3: IS SENSITIVE TO OTHER CULTURES
The third learning objective, demonstrate sensitivity to other cultures through knowledge and
understanding of cultural customs, taboos, religion, health issues, social issues and
inclusiveness will be assessed using an essay test.
Essays that require an extended written response can assess complex skills sets and focuses
on higher order thinking and problem solving abilities (McCurry, 1992, p. 235) and “may also
provide insights into attitudes and values” (Barry & King, 1998, p. 364). These attitudes and
values are what need to be assessed for the learning outcomes. It is not possible to assess
reactions by using this assessment instrument; candidates could contrive their answers to fit the
marking criteria.
An extended essay can disadvantage people who have low literacy skills or non-English
speaking students because it relies so heavily on writing skills (McCurry, 1992, p. 235). Further,
it is argued that extended essays are unreliable, with papers marked differently by different
people and while methods such as double marking and anonymous submission can increase
reliability, subjectivity plays a large role in the results (Cree, 2000, pp. 37-38). While these
factors can make the instrument more expensive, it is still a feasible way of assessing the
learning outcomes.
David Martin 3143VTAStudent #2636349 Vocational Assessment
8
Assignment 2 – Evaluating Assessment
OUTCOME 4: USES WHITEBOARD EFFECTIVELY DURING
INSTRUCTION
The final learning objective, uses whiteboard effectively during instruction by creating text or
diagrams that are legible, relevant to instructions, will be assessed using an observation based
performance test. The candidate will have 15 minutes to use a whiteboard effectively during
instruction.
In this type of assessment performance skills, which are made up of a knowledge component, a
skill component and an affective component (Gronlund, 2006, p. 127) can be tested. This
assessment tests cognitive, psychomotor and affective domains of knowledge and skills,
increasing the relevance and validity. Realism is an important part of assessment and the extent
to which real world performance is simulated increases when using extended performance
assessment (Gronlund, 2006, p. 15). This learning outcome needs realism to ensure reliable
results and while practical constraints, such as time, cost and availability of equipment
(Gronlund, 2006, p. 135) the assessment for this assignment puts reasonable limits on these.
Observation can be a valuable assessment method but it is important to predetermine several
factors; the purpose and what is to be observed, who carries out the observations as well as
when and how it is to be conducted (Marsh, 2000, p. 410). Some observations are done
without the candidate being aware. While this does increase validity and reliability, for feasibility
purposes this assessment is a simulated environment with observations being knowingly
performed.
This instrument is an example of an extended performance task, which are typically less
structured and broader in scope than a restricted performance test (Gronlund, 2006, p. 128),
David Martin 3143VTAStudent #2636349 Vocational Assessment
9
Assignment 2 – Evaluating Assessment
This increases the difficulty of the assessment, and does not leave as much room for identifying
areas of improvement (Gronlund, 2006, p. 129).
This assessment does not test high level cognitive processes or underlying knowledge of
whiteboard use or instruction. To cover all aspects, an observed performance assessment is
best used after paper and pencil assessment. However this instrument is a feasible way of
assessing the learning outcome.
CONCLUSION
The use of rubrics and scale ratings are helpful when measuring assessment of high order
cognitive, affective and psychomotor processes. This type of scoring is useful for extended
response assessment, which offers a chance for more detailed reporting but is prone to
subjectivity in the marking process. There are methods of design and implementation that can
provide more reliability and validity. Good assessment will also be relevant and fair by
sometimes allowing extended performance and other times limiting responses to change the
difficulty. Efficiency is also an issue and while extended response assessment generally takes
longer to create and mark it allows for greater feedback and flexibility in marking. This project
has shown the process of developing assessment for specific learning outcomes that takes
these factors into account.
David Martin 3143VTAStudent #2636349 Vocational Assessment
10
Assignment 2 – Evaluating Assessment
LIST OF REFERENCES
Barry, K., & King, L. (1998). Beginning Teaching and Beyond (2nd ed.). Tuggerah: Social
Science Press.
Cree, V. E. (2000). The challenge of assessment. In V. E. Cree, & C. Macaulay, Transfer of
Learning in Professional and Vocational Education (pp. 27-52). London: Routledge.
Gay, L. R. (1980). Educational evaluation and meaurement: competencies for analysis and
application. Columbus: Merric Publishing Company.
Gronlund, N. E. (2006). Assessment of Student Achievement (8th ed.). Boston: Pearson
Education.
Marsh, C. (2000). Handbook for beginning teachers (2nd ed.). Frenchs Forest: Pearson
Education.
McCurry, D. (1992). Assessing standard of competence. In A. Gonzi (Ed.), Developing a
competent workforce: adult learning strategies for vocational educators and trainers (pp. 222-
239). Adelaide: Centre for Vocational Education Research.
Mead, N. A., & Rubin, D. L. (1985). Assessing Listening and Speaking Skills. ERIC Digest.
Retrieved April 28, 2008, from http://www.ericdigests.org/pre-923/speaking.htm
David Martin 3143VTAStudent #2636349 Vocational Assessment
11
Assignment 2 – Evaluating Assessment
APPENDIX 1: MARKING CRITERIA SHEETS
Candidate name:
Assessor name:
Learning Outcome:Speaks confidently to others by demonstrating use of verbal and non-verbal communication, listening, responding to questions and applying skills of negotiation or conflict resolution
Date of assessment:
Brief description of task and assessor instructions:
This assessment is designed to test your ability in speaking confidently to others. A series of situations will be presented to you, in which you are to respond in the most natural way you can. While no time limit has been set, each scenario should run for around 3-5 minutes. The assessor will indicate when each scenario ends and will use a criteria checklist to rate your performance. Each item on the checklist contributes to an overall score out of 15.
Rate the Candidate on the following criteria Excellent(3 points)
Good(2 points)
Acceptable(1 point)
Unacceptable(0 points)
Listens and responds to questions, comments interest and silences
Applies skills of negotiation, mediation or conflict resolution
Uses body language appropriate to spoken words
Uses confident tone and body language
Demonstrates understanding of underlying concepts of verbal and non-verbal communication
Overall Score (out of 15)
Feedback to candidate
This signature confirms the candidate agrees that the above record is a true reflection of the task performed. Candidate signature: Date:
David Martin 3143VTAStudent #2636349 Vocational Assessment
12
Assignment 2 – Evaluating Assessment
This signature confirms that the above record is an accurate rating of the candidates performance
Assessor signature: Date:
Candidate name:
Assessor name:
Learning Outcome:Demonstrate the writing of comprehensive reports by use of correct spelling, grammar, report structure and applying knowledge about the scope, audience, purpose and topic in a portfolio of 3 reports.
Date of assessment:
Brief description of task and assessor instructions:
To assess your ability to write comprehensive reports this assessment requires you to submit a portfolio of 3 reports and a one page critical reflection outlining why each report was chosen, its strengths and areas of possible improvement. The reports will be assessed holistically and a number of criteria using the numbered scale below, the higher the number the better. An overall score out of 30 will be given.
Rate the Candidate on the following criteria 5 4 3 2 1
Use of correct grammar and spelling
Use a well structured and consistent layout
Demonstrate application of scope, audience, purpose of report
Demonstrates comprehensive knowledge of topic
Shows extensive depth of knowledge in portfolio
Use of critical reflection and portfolio selection to show improvement over time
The candidate’s overall score (out of 30):
Feedback to candidate
This signature confirms the candidate agrees that the above record is a true reflection of the task performed. Candidate signature: Date:
David Martin 3143VTAStudent #2636349 Vocational Assessment
13
Assignment 2 – Evaluating Assessment
This signature confirms that the above record is an accurate rating of the candidates performance
Assessor signature: Date:
Candidate name:
Assessor name:
Learning Outcome:Demonstrate sensitivity to other cultures through knowledge and understanding of cultural customs, taboos, religion, social issues and inclusiveness
Date of assessment:
Brief description of task and assessor instructions:
The candidate is to write an essay of between 500 and 1500 words on the topic of positive multiculturalism and diversity is Australia. The essay should cover topics such as customs and taboos, religion, language, inclusiveness.
The assessor will rate the essay based on the following criteria, and a score given to rate overall performance out of 20.
Rate the Candidate on the following criteria
Excellent(4 points)
Good(3 points)
Acceptable(2 points)
Inadequate(1 point)
Not Shown
(0 points)
Demonstrate knowledge and understanding of customs and taboos
Demonstrate knowledge and understanding of social issues
Use examples or anecdotes to show feelings or attitudes of other cultures
Use well structured and convincing argumentsWrite without confusing the meaning by poor spelling, grammar and essay layout
The candidate’s overall score (out of 20):
Feedback to candidate
This signature confirms the candidate agrees that the above record is a true reflection of the task performed. Candidate signature: Date:
David Martin 3143VTAStudent #2636349 Vocational Assessment
14
Assignment 2 – Evaluating Assessment
This signature confirms that the above record is an accurate rating of the candidates performance
Assessor signature: Date:
Candidate name:
Assessor name:
Learning Outcome: Uses whiteboard effectively during instruction
Date of assessment:
Brief description of task and assessor instructions:
The candidate, unaided, will be able to use a whiteboard effectively during instruction by creating text or diagrams that are legible, relevant to instructions, link content to the board and instruction.Using a whiteboard and markers, or electronic whiteboard the candidate will be given 15 minutes of presentation time to demonstrate effective use of the whiteboard.The assessor will rate the performance based on the criteria marking sheet and a score given to rate overall performance.
Rate the Candidate on the following criteria Always(4 points)
Usually(3
points)
Sometimes(2 points)
Rarely(1 points)
Never (0
points)
Creates text or diagrams that are legible
Creates text or diagrams that are relevant to instructional segment
Demonstrate ways of linking whiteboard to spoken instruction
The candidate’s overall score (out of 12):
Feedback to candidate
This signature confirms the candidate agrees that the above record is a true reflection of the task performed. Candidate signature: Date:
This signature confirms that the above record is an accurate rating of the candidates performance
Assessor signature: Date:
David Martin 3143VTAStudent #2636349 Vocational Assessment
15
Assignment 2 – Evaluating Assessment
David Martin 3143VTAStudent #2636349 Vocational Assessment
16
Assignment 2 – Evaluating Assessment
APPENDIX 2: ASSESSMENT MATRIXES
OUTCOME 1: SPEAK CONFIDENTLY TO OTHERS
Learning Goals:The candidate, unaided, will be able to:
Assessment Targets
Domains Processes
speak confidently to others by demonstrating use of verbal and non-verbal communication,
listening and responding to questions and applying skills of negation or conflict resolution
Listen and respond to questions, comments
interest and silences
cognitive, affective,
psychomotor
understand, apply,
evaluate, receive,
respond, reflex, non-
discursive, synthesis
Apply skills of negotiation, mediation or
conflict resolutioncognitive, affective
understand, analyse,
evaluate, create,
respond, apply,
synthesis
Display body language that matches
spoken words
affective, psychomotor,
cognitive
characterisation,
understand, reflex,
non-discursive,
synthesis, apply
Display confident tone and body languagecognitive, affective,
psychomotor
characterisation,
reflex, non-discursive,
understand, apply,
Demonstrates understanding of underlying
concepts of verbal and non-verbal
communication
cognitive, affectiveevaluate, apply,
analyse
David Martin 3143VTAStudent #2636349 Vocational Assessment
17
Assignment 2 – Evaluating Assessment
OUTCOME 2: WRITES COMPREHENSIVE REPORTS
Learning Goals:The candidate, unaided, will be able to:
Assessment Targets
Domains Processes
Demonstrate the writing of comprehensive reports by use of correct spelling, grammar, report
structure and applying knowledge about the scope, audience and purpose and topic shown in
a portfolio of 3 reports.
Use of correct grammar and spelling cognitive remember, apply
Use a well structured and consistent layout cognitive, affectivecreate, analyse,
evaluate, organise
Demonstrate application of scope,
audience, purpose of report cognitive, affective
apply, organise,
comprehend,
analysis, evaluate
Demonstrate knowledge of topic Cognitivecreate, analyse,
evaluate
Show depth of knowledge in portfolio cognitivesynthesis, analysis,
comprehension
Use of critical reflection and portfolio
selection to show improvement over timeaffective, cognitive
value, organise,
metacognitive,
David Martin 3143VTAStudent #2636349 Vocational Assessment
18
Assignment 2 – Evaluating Assessment
OUTCOME 3: IS SENSITIVE TO OTHER CULTURES
Learning Goals:The candidate, unaided, will be able to:
Assessment Targets
Domains Processes
Demonstrate sensitivity to other cultures through knowledge and understanding of cultural
customs, taboos, religion, social issues and inclusiveness
Demonstrate knowledge and understanding
of customs and taboos cognitive, affective remember, apply
Demonstrate knowledge and understanding
of social issues cognitive, affective
remember, apply,
evaluate, respond
Use examples or anecdotes to show
feelings or attitudes of other culturescognitive, affective
evaluate, respond ,
value,
characterisation,
analysis
Use well structured and convincing
argumentscognitive, affective
evaluate, respond,
value analysis,
synthesis, create
Write without confusing the meaning by
poor spelling, grammar and essay layoutcognitive, affective
remember, apply,
create synthesis,
organise,
David Martin 3143VTAStudent #2636349 Vocational Assessment
19
Assignment 2 – Evaluating Assessment
OUTCOME 4: USES WHITEBOARD EFFECTIVELY DURING INSTRUCTION
Learning Goals:The candidate, unaided, will be able to:
Assessment Targets
Domains Processes
Use whiteboard effectively during instruction by creating text or diagrams that are legible,
relevant and show links to spoken instructions
Create text or diagrams that are legible psychomotor, cognitive,
remember,
understand, apply,
create, evaluate,
movements,
perceptual
Create text or diagrams that are relevant to
instructional segment
affective, psychomotor,
cognitive
remember,
understand, apply,
analyse, evaluate,
create, organising
Demonstrate ways of linking whiteboard to
spoken instructioncognitive, affective
respond, remember,
understand, apply
David Martin 3143VTAStudent #2636349 Vocational Assessment