Measuring and Comparing Achievements of Learning Outcomes in HE in Europe
RobertWagenaarOverallCoordinatorCALOHEE
DirectorInternationalTuningAcademy
ECAworkshopandAnnualMeetingHCERES– Paris22-23June2017
CALOHEE:WHY?
Preposition: If academic experts can agree on the set of learning outcomes, they should also be able to measure performance in comparative perspective in (inter)national contexts!
THE PROOF IS IN THE EATING OF THE PUDDING !
Do students enrolled in higher education around Europe develop thecompetences they need? Are study programmes delivering theirpromises? Can we learn from comparing students’ achievements inan (inter)national context in a meaningful way?
Measuring and Comparing Achievements of Learning Outcomes in HE in Europe
WHY? HOW?WHAT?
WHY?ØMeetingthedemandformorereliableinformationabouttheoutcomesoflearninginHE
ØOfferingadriveforquality,takingintoaccountfullytheneedsofsociety,inparticularthefourmajorstakeholders:HEstudents,HEstaffandmanagement,employersandemployees,andcivilsociety
ØAnattempttocreate(inthelongerrun)amoreeffective,lessbureaucraticandmorereliableinstrumentforqualityassurance!
CALOHEE:MeasuringandComparingAchievementsofLearningOutcomesinHigherEducationinEurope
WHY? HOW? WHAT?
HOW?Ø Showingwhatasubjectareadoesrepresentafterconsultationwithstakeholders,intermsofcorecompetencesandlearningoutcomesaccordingtothediscipline
ØDevelopinginstrumentsthatacknowledgethedifferentmissions,profilesandculturalcontextsofHigherEducationinstitutions
CALOHEE:MeasuringandComparingAchievementsofLearningOutcomesinHigherEducationinEurope
WHY? HOW?WHAT?
WHAT?ØOfferingsubjectareabasedagreedframeworksofdescriptorswhichallowforcomparingandmeasuringqualityofperformanceandoutcomessetinthecontextoftheinstitution
ØOfferingadetailedsubjectareabasedagreedAssessmentFrameworkwhichalignsLearningOutcomes,ActivitiesandAssessments
ØOfferinganinstrumentwhichallowsfordiagnosticassessmentaboutmeetingagreedqualityrequirements,societalneedsandtheparticularmissionoftheprogrammeinvolved
CALOHEE:MeasuringandComparingAchievementsofLearningOutcomesinHigherEducationinEurope
CALOHEEProjectaims
Ø BENCHMARK1:Updatedtransnationalsectoral/subjectarea frameworks(finalisedafterConsultationProcess)
Ø BENCHMARK2:
Multi-dimensionalinstrumenttomeasureandcomparelevelsoflearningdoingjusticetothedifferentmissions,orientationsandprofilesofHEinstitutions
Ø BENCHMARK3:
AssessmentFrameworks forfiveacademicdomainsandfiverelateddisciplines(CivilEngineering,Nursing,History,EducationandPhysics)(finalisedafterConsultationProcess)
Theseinstrumentsarethefoundationsfordevelopinganassessmentinstrumentthatshouldallowfordiagnosticassessments(servingteachingstaff,students,managementandsociety)
OverallobjectiveofTUNING: HighQualityandRelevantHigherEducation
Engineering/CivilEngineering
SocialSciences/Education
Humanities/History
CoreCompetences(SubjectSpecificandGeneric)/LearningOutcomesDescriptors
Comparisonoflearning(gain)intrans-institutionalandtrans-nationalperspective
FiveAssessmentFrameworks,covering:
Irreduciblecore:1.Knowledge:theory+methodology2.Applicationofknowledgeandskills3.Employability4.Civic,SocialandCulturalEngagement
HealthCare/Nursing
NaturalSciences/Physics
CoreCompetencesetc.
CoreCompetences,etc.
CoreCompetences,etc.
CoreCompetences,etc.
CALOHEEFLOWCHART
DiagnosticAssessment
CommonFramework:
Acknowledgedifferentmissions,profilesandculturalcontextsofHEinstitutions
Itoffersagreedindicatorsof:Ø LevelØ ContentØ Relevance
EQF / QF for EHEA
Descriptors
TUNING Sectoral Reference Points
TUNING Subject Specific Frameworks: Reference Points
Tuning Sector / Subject Area Based Assessment Frameworks
CALOHEEisbasedonamergeroftwoEuropeanQualificationsFrameworks
Recognisingdifferentmissions,profilesandculturalcontextsofHEinstitutions
Approachallowsfor(trans)national/institutionalcomparisonfortheachievementoflearning(gain)
Profiles of individual degree programmes
Flowchart2
July12,2017 Slide10
Knowledge SkillsWiderCompetencesAnExample
Dimen
sions
KeyDescriptorsofasingleacademicfieldononepage(basedonamergeroftheEQFforLLLandtheQFfortheEHEA)
Formulatedescriptorsaslearningoutcomes:useactiveverbsinaconsistentway,butalsodefinelevelintermsofscopeandcomplexity(averbisaninsufficientindicatorforidentifyingthelevel)
PROGRESSIONOFLEARNING:knowledge– skills–application- contribution
KNOWLEDGE SKILLS WIDERCOMPETENCES
LEVEL/GROWINGCOMPLEXITY
Common body of knowledge, skills and wider competences for the subject area
Employability Civic, social and cultural engagement
Knowledge: theory and
methodology
Application knowledge and skills
PARAMETERS / CATEGORIESEQF:Knowledge Skill WiderCompetences
1
2
QFforEHEA
DIM
ENSI
ONS
34
ETC.
CALOHEE ASSESSMENT MODEL
Core Competences (Subject Specific and Generic) / Learning Outcomes
DIAGNOSTICASSESSMENTS
Assessment Frameworks based on parameters and dimensions
Frameworkacknowledgesdifferentmissions,profilesandculturalcontextsofHigherEducationinstitutions
A dimension indicates a constructive key element which defines a subject area; each subject area is based on a multiple of dimensions
DIM
ENSI
ONS
Diagnosticassessmentsinformstakeholdersaboutmeetingagreedqualityrequirementsandthemissionoftheprogramme
Flowchart3
LinkingLearning,TeachingandAssessmenttosetsofbroken-downSubjectAreaCompetences/LearningOutcomes
Thetotalityofdimensionsreflectsadetaileddescriptionofwhataprogrammecancontain,leadingtolearningalignment:LearningOutcomes,ActivitiesandAssessments
Adimensionindicatesaconstructivekeyelementwhichdefinesasubjectarea;eachsubjectareaisbasedonamultipleofdimensions
Flowchart4
Anysuggestionsforimprovement/simplification?
CALOHEEframeworkforcivic,socialandculturalengagement
Fourparameters(basedonanalysisofpresentdevelopmentsandrecentliterature):
• SocietiesandCultures:Interculturalism andconflictmanagement
• Processesofinformationandcommunication• Processesofgovernanceanddecisionmaking• Ethics,norms,valuesandprofessionalstandards
CALOHEEframeworkforCivic,socialandculturalengagementKnowledge Skill Widercompetences
1. Demonstratecriticalunderstandingofdifferencesinandbetweensocietiesandcultures(frames)
Identify,describeandanalyseissuesinandbetweensocietiesandcultures
Demonstrateengagementbydevelopingscenariosandalternativesforidentifyingbestpracticesandinterventionsinthecaseoftensionsandconflicts
2. Demonstratecriticalunderstandingoftheprocessesofinformationandcommunication
Reviewandjudge(mis)useofsources,data,evidence,qualities,intentionsandtransparencyandexpertopinions
Activecontributiontosocietaldebatesusingreliabledataandinformationsourcesandinformedjudgements
3. Demonstratecriticalunderstandingoftheprocessesofgovernanceanddecisionmaking
Applyandsupportagreedgoverningprinciples,normsandvaluesregardingfairness,transparency,accountability,democracyandrelevanceinpolicymakingprocesses
Activecontributiontoandwithlocaland(inter)nationalcommunities,communitygroups,(political)organisationsandpressuregroupsrespectingagreedprinciples,normsandvalues
4. Demonstratecriticalunderstandingofgeneralethicalprinciples,normsandvaluesandprofessionalstandards
Understandandapplytheprocessesofdecisionmakingandtheconsequencesofactionstakingintoaccountprinciples,norms,valuesandstandardsbothfromapersonalanda
Activecontributiontoupholding,promotinganddefendinggeneralethicalprinciples,norms,valuesandprofessionalstandardsingovernance,communicationandculturalinteraction.
CALOHEEframeworkforcivic,socialandculturalengagement
Nextsteps:Ø Addingsub-dimensionstofourdimensionstooffermoreprecisedescriptorsØ IntegratingframeworkdescriptorsinsubjectareaconceptualframeworksØ IdentifyinggapsØ Updatingandfinalising CALOHEEframeworkforcivic,socialandcultural
engagement asareferenceforallacademicfields
Next steps
Frameworksoffermeasurableindicatorsfordevelopingassessmentitems/assessmentbanksfordiagnosticassessments
Conditionsforsuccessfultrans-institutionalandtransnationalassessments:
ØShoulddojusticetoindividualacademicdomains/subjectareasØShouldallowforassessingunderstandingandapplicationØShouldbemulti-dimensional:coveringall4parameters/categoriesoftheassessmentmodel,aswellastheidentifieddimensionsforeachsubjectareainvolved
ØShouldallowfordifferentiationintheweightingofsub-scoresfordifferenttypesofdegreeprograms(researchdriven/applied,etc.)followingthemulti-dimensionalapproach;accordingtotransparentandagreedcriteria
ØShouldbemachinebased,butgo(far)beyondmultiplechoice(makinguseofalgorithms?)
Ø AllowfortestingprofoundknowledgeandunderstandingandhighlevelskillsØ Allowfortestingcriticalawarenessandthinking,analyzing(textinterpretation)and
composing
CALOHEE-WEBSITE:https://www.calohee.eu