concluding reflections on guidelines for the use of (achieved… lucien... · curriculum iqa nqf...
TRANSCRIPT
PLE
The Hague
30 October 2015
Lucien Bollaert NVAO executive
Concluding Reflections on Guidelines for the use of (Achieved) Learning Outcomes
NVAO Peer Learning Event
• Conclusion of the PLE after so many questions?
• Bringing together the various strands that have
been spun during two days
• & the common understanding
• Looking ahead to drawing up a document of
Guidelines/Points of Attention/Topics/Examples
for the Assessment and Demonstration of
Achieved Learning Outcomes ≠ strait jacket 2
PLA LO concluding reflections
3
PLA LO concluding reflections
“The guidelines explain why the standard is important and
describe how standards might be implemented.
ESG (2015)
4
PLA LO concluding reflections
They set out good practice in the relevant area for
consideration by the actors involved in quality assurance.
ESG (2015)
5
PLA LO concluding reflections
They set out good practices in the relevant area for
consideration by the actors involved in generating quality
and quality assurance.
ESG (2015)
6
PLA LO concluding reflections
They set out good practices in the relevant area for
consideration by the actors involved in generating quality
and quality assurance.
Implementation will vary depending on different contexts.”
ESG (2015)
7
PLA LO concluding reflections
They set out good practices in the relevant area for
consideration by the actors involved in generating quality
and quality assurance.
Practice will vary depending on different contexts.”
ESG (2015)
HELPFUL IN PRACTICE!
8
PLA LO concluding reflections
START WITH THE QUESTION “WHY ?”
THEN ASK “HOW ?”
FINALLY ASK “WHAT ?”
Simon Sinek (2009)
9
PLA LO concluding reflections
• Why do we need (intended) LOs?
• Why do we need to assess them?
• Why do we need quality (assurance)
management?
• Why do we want quality?
• Why are we in (higher) education?
• Why do we want a better society?
• Why do we need a better world?
• Why do we want a better life? 10
PLA LO concluding reflections
ESG : anchor point? meeting place?
• LEARNING outcomes upgraded in ESG
• Standard 1.2 Design and approval of programmes :
“(…) The programmes should be designed so that they
meet the objectives set for them, including the intended
learning outcomes.
The qualification resulting from a programme should be
clearly specified and communicated, and refer to the
correct level of the national qualifications framework for
higher education and, consequently, to the Framework
for Qualifications of the European Higher Education
Area.”
11
PLA LO concluding reflections
g
e
n
e
r
i
awards c c
s
p
e
c
i
f
i
c
EQF (LLL)
QF EHEA
disc
spec
LO
disc
spec
LO
disc
spec
LO
disc
spec
LO
HEI LO
STP
LO
STP
LO
STP
LO
STP
LO
STP
LO
Key LO
c
LO
c
LO
c
LO
c
LO
c
LO
NQF
i n
t
(i)
n t
H
E
I
S
T
P
T
PLA LO concluding reflections
Learning outcomes : need of common language
need of common understanding
13
LO = what a learner is supposed to
know & be able to do after a successful
study
Make of the LO LIVING things that are formulated by
the stakeholders concerned, passionately shared by
the whole team and shaping the teaching, learning
and assessment practices & formats accordingly !
competence = ability to integrate
knowledge, skills & attitude to be
successful in a certain context
PLA LO concluding reflections
ESG : anchor point? meeting place?
• LEARNING outcomes upgraded in ESG
• Only the guidelines (1.3) cover the achieved learning
outcomes
• Standard 1.3 Student-centred learning, teaching and
assessment :
“(…) Considering the importance of assessment for the
students’ progression and their future careers, quality
assurance processes for assessment take into account
the following: (…)
The assessment allows students to demonstrate the
extent to which the intended learning outcomes have
been achieved. (…)”
14
PLA LO concluding reflections
15
LEARNING
OUTCOMES
MISSION
STRATEGY CURRICULUM IQA
NQF
EQA
IQF
LEARNING
FORMATS ASSESSMENT
FORMATS
LEARNING
ENVIRONMENT
teacher/
coach
ECTS
MOBILITY
RECOGNITION
profiling
marketing
EXEMPTION
RECOGNITION
PLA LO concluding reflections
LOs as meeting place
with different perspectives & languages
HRM
professional needs
employment
EHEA
reforms
Current implementation • Intended LOs have become accepted as a possible most
valid concept in (student-centred) learning (in practice &
process)
• LOs can be at the centre of the quality of T & L (added value
between input = incoming LO & output = achieved LO
• … and thus of QA, both internal & external
• LOs should be linked to international & national QFs
• Discipline LO can be considered
• HEI LOs are good to profile the HEI (pitfall of marketing)
• Course LOs should be linked with LOs of study programme
• Defining good intended LO is still an „art‟
• should combine all levels & stakeholders in practical
reflection & cooperation/co-creation (communication!)
16
PLA LO concluding reflections
• BE MORE THAN SMART
LO should be explicit & measurable or assessable
in order to design the curriculum and QA
LO should be specific, objective, achievable,
useful, relevant & standard-setting and timely
Active verb + type + topic + level (+ context)
Make the LO living (dialogue), i.e. forming
learning in team!
Congruency between teaching, learning &
assessment formats
PLA LO concluding reflections
Keep room for experiment, creativity & failure
(student-centred co-creation & assessment)
course design & QA
Duclan Kennedy (2007)
Identify aims & objectives
Write LOs using guidelines
Develop a teaching & learning strategy
to enable students to achieve the LOs
Design assessment method to test
If LOs have been achieved
If necessary modify module content &
Assessment in light of feedback
P
D
C
A
PLA LO concluding reflections
KEEP
THINGS IN
MOTION
STUDENT
TEACHER LEARNING
environment
congruence between LO, learning & assessment LO T & L forms assessment
cognitive lecture end of course exam
tutorials multiple choice
discussions essays
laboratory practical assessment
fieldwork
affective clinical work clinical practice
seminar presentation
peer group project work
presentation co- or peer-
psychomotor assessment
overall assessment
Kennedy (2007)
Demonstrate
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Integration of
beliefs, ideas & attitudes
Acquisition of physical
skills
PLA LO concluding reflections
Current implementation • Large variety in Europe in the use of ALOs :
• Assessment of integration of knowledge, skills (&
attitudes or behaviour) is not easy
• The use of internal & external evaluators
• Not always part of procedure & IQA.
• EQA : Netherlands and Flanders: ALOs are (most)
important criterium.
Drawbacks: great emphasis in peer review on results;
debate on autonomy of the institution - who is „in
charge‟ of/‟owns‟ LOs & evaluating achieved LOs.
• In Sweden the question whether the ALOs as related
to the NQF are reflected in the final thesis are subject
of EQA.
• In most countries the EQA assessment is evaluation
on meta-level – not the final thesis, but the
methodology of assessing the intended LOs (Austria).
20
PLA LO concluding reflections
axis : learning outcomes (LO)
øLO only intended intended LO intended LO intended LR
no link with NQF + NQF + NQF + NQF
programme prog. & learning
formats
achieved LO
subject EQA
intended & intended LO intended LO intended LO
achieved LO subject EQA subject IQA prog. & learning
subject EQA & assesment f
21
PLA LO concluding reflections
Guidelines
A need for „guidelines‟ ≠ checklist : • There are many different systems and solutions in
Europe
• Need of common language(s)/understanding
• Guidelines should be based on good practices in
European Higher Education
• Results from PLE should be taken into account
• Guidelines should suggest possible solutions for
common problems and pitfalls (holistic approach)
• Defining good intended LOs not a topic in
Guidelines, although...
22
PLA LO concluding reflections
Topics for the guidelines (1)
• 1 (intended) LOs should be formulated in such a
manner that their achievement can be assessed in a
valid manner.
• Good practices
• Project at Dutch HBO
• Pragmatic, holistic approach at Glasgow Caledonian vs
methodological or semantic discussions
• Involve the stakeholders/actors, primarily teachers &
students
• AuCom Model Saxion
23
PLA LO concluding reflections
Topics for the guidelines (1)
• 1 (intended) LOs should be formulated in such a manner
that their achievement can be assessed in a valid
manner.
• Pitfalls and obstacles
• ALOs can be formulated too generic and cannot be
assessed in a valid manner. How to address & assess
attitudes?
• They are too much tied to a workplace so that they cannot
be assessed within the programme. Too many, too detailed.
(IBMS)
• LOs should be pro-active towards changing demands in
society. Updating them is crucial. Conformity to NQF-levels
can restrict innovation as manuals & squeezing rubrics.
• ALOs should not become a bureaucratic item, but a living
tool in the classroom and assessment.
24
PLA LO concluding reflections
Topics for the guidelines (2)
• 2. Setting up assessment for assessing ALOs
• Good practices
• “protocol” or Self-Evaluation Tool pilot Dutch HBO (“workable”)
• Variety of various formats (congruency)
• Co-assessment with students & external examiners
• Pitfalls and obstacles
• Examination and examiners should follow trend to student-
centred learning and not stick to assessing the curriculum.
• Examination and assessment is not structured and uses invalid
methods and criteria.
• External examiners are not used.
• How to assess attitudes? Heavy assessment of competences.
• Should EQA duplicate the assessment of individual students
or (meta-)judge the system of assessment?
25
PLA LO concluding reflections
Topics for the guidelines (3) • 3. Linking learning outcomes to the levels of the National
Qualification Frameworks.
• Good practices
• Dublin descriptors
• ECTS manual (2015)
• Tuning disciplines
• Pitfalls and Obstacles
• Calibrating the differences between ALOs at the levels of
short cycle, bachelor and master (VU mastermind).
• The more professional a programme is, the more specific the
LOs will be. What is the room for
autonomy/experiment/flexibility when LOs are very specific?
• Generic, transversal competences are too generic while not
linked to the sector-specific competences – they should be
transferable instead of transverable (OECD).
26
PLA LO concluding reflections
21st c. LO or competences
(ability to integrate knowledge, skills and attitudes to
come to solutions in specific contexts)
1. Teamwork skills 67%
2. Sector-specific skills 62%
3. Communication skills 60%
4. Computer skills 60%
5. Ability to adapt & act in new situations 60%
6. Good reading/writing skills 59%
7. Analytical & problem-solving skills 53%
8. Decision-making skills 46%
9. Good with numbers 40%
10. Foreign language skills 33%
Gallup for DG Education & Culture (2010)
PLA LO concluding reflections
28
Innovative teaching & learning processes
OECD
PLA LO concluding reflections
New competences needed Communication, problem-solving, creativity, team-work
Research skills, both academic as well as „mode 2‟
(applied) and mixtures
Inter-disciplinary but with skill and attitude to go deep into
a particular discipline
Meta-cognition
Willingness to change, risk-taking, entrepreneurial
„Global competences‟ (e.g. computer skills)
Visionary & inspirational leadership
HOW YOU ENGAGE IN THE WORLD
= ATTITUDES! CHARACTER! PERSONALITY
(see HRM in industry and business)
PLA LO concluding reflections
More or other knowledge & skills are not enough!
30
PLA LO concluding reflections
Topics for the Guidelines (4)
• 4. Achieved learning outcomes can integrate various
competences
• Good practices
• Profession-related programmes & assessment/assignment
• The need of transferable & transverable competences
• Pitfalls and obstacles
• How to include transversal competences in LOs? This
requires a good approach to the formulation of LOs so that
they are not too specific for a particular phase in the learning
process and are fit for life long learning.
• This also requires a link between the transversal
competences and the sector-specific in the teaching-learning
process and the assessment formats.
31
PLA LO concluding reflections
Topics for the Guidelines (5) • 5. Achieved learning outcomes should be relevant to
the context of the programme & the professional
fields. (as quality & QA are contextual)
• Good practices
• The “protocol”/Self-Evaluation Tool in Dutch hbo
• Personal competencies category in VU‟s mastermind
• Pitfalls and obstacles
• Different domains require specific approaches; in some
domains only the programme is prescibed (ED), in
others only the LOs/competences (professional labels).
• Too much restriction or only programme-related
prescriptions: European Directives (“contents of learning
instead of learning”).
• Different language by each stakeholder.
• How to address attitudes & character in LOs?
• Up-date and pro-activeness of LOs towards the
professions and LLL.
32
PLA LO concluding reflections
Topics for the guidelines (6) • 6. QA of achieved learning outcomes
• Good practices
• Dutch guidelines for assessing final projects
• Practice of taking samples of final projects
• Ghent University database towards enhancement &
accountability
• Pitfalls and obstacles
• QA should not restrict innovation (of ALOs);
• Peer review should reflect changing demands in society and
be pro-active towards innovation.
• QA can stifle innovation if institutions/programmes „play it safe‟
(window dressing/adaption vs quality culture).
• Re-assessment vs meta-assessment
• Study programme & institutional level.
• Ownership of quality (triangle where quality is generated)
and QA.
33
PLA LO concluding reflections
vision & mission (strategic) policy results iLO
LO new students study programme achieved LO
use standards
& indicators
influenced by
development phase
influence on
development
phases
input output processes
Quality Management System
added value = transformation
underlying principles
underlying (hidden) values
methodology
tool tool tool tool tool
PLA LO concluding reflections
some helpful sources :
Kennedy, D. (2007), Writing and Using Dumont, H., Istance, D. & http://ec.europa.eu/education/tools/
Learning Outcomes A Pratical Guide, Benavides, F. (2010), The docs/ects-guide_en.pdf
UCC Cork Nature of Learning - Using
Research to Inspire Practice,
OECD, Paris
PLA LO concluding reflections
some helpful sources :
available from http://ecahe.eu/home/services/publications/
36
PLA LO concluding reflections
some helpful sources :
http://www.unideusto.org/tuningeu/
37
PLA LO concluding reflections
Discipline-specific LO PROCESS : VLUHR+NVAO+AKOV
QAA‟s subject benchmark statements
38
PLA LO concluding reflections
After the meeting they realized the new manager
suffered from severe adaptive difficulty
after the meeting they realized the new manager
suffered from severe adaptive difficulty
PLA LO concluding reflections
To which segment do you belong after this PLE?
The realisation of differences is a good start!
Bring together what is broken
PLA LO concluding reflections
THANKS www.nvao.net +31 70 312 2300 [email protected] [email protected]
41