victoria crisp & christine ward the pepcaa project: formative scenario-based caa in psychology...
TRANSCRIPT
Victoria Crisp & Christine Ward
The PePCAA Project:
Formative Scenario-based CAA in
Psychology for Teachers
Crisp & Ward: The PePCAA project
Topics
• The PePCAA project and its aims
• Scenarios: development, types and CAA features
• Examples of scenarios
• Results of trialling
• Problems in sharing scenarios transnationally
• Achievements.
Crisp & Ward: The PePCAA project
PePCAA – Pedagogical Psychology Computer Assisted Assessment
• FIM-NeuesLernen, University of Erlangen
• LabSET, University of Liege
• Universitat Oberta de Cataluña (OU of Catalonia)
• PGU, Göteborg University
• UCLES - University of Cambridge Local Examinations Syndicate
Crisp & Ward: The PePCAA project
PePCAA
Scenario-based computer-delivered formative assessments in pedagogical psychology for teachers and trainee teachers.
Main aims:
• to improve the quality of teaching, by linking psychological theory to practical instances of its use in the classroom
• to explore more innovative techniques of computer assisted assessment
Crisp & Ward: The PePCAA project
Expected benefits of the approach (1):
• Intrinsic interest of scenarios
• Showing relevance of psychological theory
• Increasing trainee awareness of psychological theory and research
• Developing awareness of why some strategies are more effective
• Stimulating discussion.
Crisp & Ward: The PePCAA project
Expected benefits of the approach (2):
• Allowing trainees to assess own knowledge
• Motivational benefits of CAA
• Opportunity to experiment with CAA.
Crisp & Ward: The PePCAA project
Scenario development
• Early discussion and drafting of examples
• Drafting by one partner
• Comment by at least one other partner
• Revision and trialling
• Development of PLAC (PePCAA Learning Assessment Circle) .
Crisp & Ward: The PePCAA project
Scenario types:
• Problem Solving: diagnose causes of a classroom difficulty, suggest possible solutions and their theoretical basis, predict effects of proposed intervention.
• Best Practice: plan the best strategy to achieve a goal, justify the strategy by reference to theory, predict the outcomes .
Crisp & Ward: The PePCAA project
Metacognition and confidence ratings
• Promote awareness of one’s own knowledge
• Confidence rating for each automatically scored answer (in some assessments)
• Calculation of Confidence, Prudence and Nuance
• Required a work-around in Perception.
Crisp & Ward: The PePCAA project
Some examples of scenario assessments
http://www.pepcaa.odl.org/
Crisp & Ward: The PePCAA project
Scenario / software features (1):
• a scenario (or ‘case study’) description
• option to read a short explanation of relevant psychological ideas first
• ‘essay’ question, followed by feedback stating the main points expected
• ‘essay’ question followed by a multiple-response question asking which points included, with feedback on choices made
Crisp & Ward: The PePCAA project
Scenario / software features (2):
• range of objective question types
• division of some assessments into blocks
• ‘branching’ dependent upon the candidate’s response to a question
• confidence measurement
• explanatory feedback, with weblinks
Crisp & Ward: The PePCAA project
Trialling in the UK
• Small groups of trainee teachers
• Scenarios from both UK and non-UK partners
• Feedback collected through both open and closed questions
• Some follow up in interviews.
Crisp & Ward: The PePCAA project
Trialling – format of feedback
Statement (eg ‘The assessment is easy to understand’) to be rated as follows:
+2 strongly agree
+1 agree
0 neither agree or disagree
-1 disagree
-2 strongly disagree
Crisp & Ward: The PePCAA project
Scenarios trialled in the UK - primary
2: PE lesson with 10 year olds (6 students)
3: Number facts (6)
4: Assessing Science (6)
6: Problem Solving (7)
7: Attachment Theory and Parenting (6)
8: Friendships (6)
9: Play (6)
PGU 7 (Sweden): Teaching conceptual change . (6)
Crisp & Ward: The PePCAA project
Trialling of primary scenarios in UK –Average ratings (scale – 2.0 to + 2.0)
Knowledge of pedagogical psychology is very important to me 0.64
My current knowledge of pedagogical psychology is very good - 0.39
Assessment is easy to understand 0.88
Assessment is about a real world situation 1.22
Crisp & Ward: The PePCAA project
Trialling of primary scenarios in UK –Average ratings (scale – 2.0 to + 2.0)
Assessment is relevant for teachers 1.36 Assessment is well structured 0.88
Assessment is too long - 0.33
Assessment is stimulating 0.87
Working on assessment was helpful 0.95
Feedback given was helpful 0.97
Crisp & Ward: The PePCAA project
Trialling of primary scenarios in UK –Key findings
• Generally favourable responses to all scenarios
• Ratings for Swedish scenario similar to those for UK
• One scenario (UCLES 7) shown to be too long
• Amendments made in the light of responses
Crisp & Ward: The PePCAA project
Scenarios trialled in the UK - secondary
1: History Lesson (7 students)
5: Dance lesson with 11 year olds (7)
Labset 2 (Belgium): Why success isn’t explained like failure (7)
FIM 8 (Germany): Self-managed learning (7)
Crisp & Ward: The PePCAA project
Trialling of secondary scenarios in UK –Average ratings (scale – 2.0 to + 2.0)
Knowledge of pedagogical psychology is very important to me 0.46
My current knowledge of pedagogical psychology is very good - 0.46
Assessment is easy to understand 0.04
Assessment is about a real world situation 0.54
Crisp & Ward: The PePCAA project
Trialling of secondary scenarios in UK –Average ratings (scale – 2.0 to + 2.0)
Assessment is relevant for teachers 0.54 Assessment is well structured 0.14
Assessment is too long - 0.43
Assessment is stimulating - 0.04
Working on assessment was helpful 0.04
Feedback given was helpful - 0.21
Crisp & Ward: The PePCAA project
Trialling of secondary scenarios in UK: Key findings
• Average ratings much lower than for primary scenarios
• Ratings for Belgian and German scenarios much lower than those for UK
• Amendments made to UK scenarios in the light of responses.
Crisp & Ward: The PePCAA project
Trialling of scenarios in UK: Some general findings
• Mixed response to open-response questions not automatically marked
• Mixed response to multiple-response questions
• Improvements required to candidate information and end of test feedback
• Some students found the psychological language difficult.
Crisp & Ward: The PePCAA project
Problems in transnational sharing of scenarios
• Mismatch in the coverage of psychology
• Depth of treatment and detail differed, even where topics were common
• More practical approach in the UK & Sweden
• Assumptions about the background to the scenario might not fit another country
• Careful editing of language needed.
Crisp & Ward: The PePCAA project
Achievements of the PePCAA project
• A framework for scenario development, plus actual examples
• PePCAA Learning Assessment Circle (PLAC)
• Showed benefits of scenarios in motivating students and prompting reflection
• Demonstrated taking CAA beyond simple multiple-choice.
Potential applications outside pedagogical psychology
Crisp & Ward: The PePCAA project
Try the assessments for yourself on
http://www.pepcaa.odl.org/