Professional Seminar Bucharest

6-10 October 2008

“Assessment: From Programmes to Profiles, Exchanging Best Practices”

Overview

• Background

• Education and Training Systems

• Operational focus

• Drawbacks

• Common Ground

Context

In the beginning, when God created all things, he looked upon his creation and said, “It is good.”

An Archangel asked, “God, what do you mean by good? What are your criteria? On what data do you base this judgment?”

Assessment

And thus, on the eighth day, in a blaze of glory, with the archangel trailing flames into oblivion, was the offspring of assessment, the art and science of validation, born.

Assessment

How do we know when something works?

Mostly by noting how it doesn’t.

Systems

Narrow Context

• language education and training

(most complex of human behaviours)

• systems approach

Analysis

Design

Development

Conduct

Evaluation

Validation

Needs Assessment

Typical System

Meet MoD Requirements Number of candidates on course Success rates Percentage of requirement met

Provide Access to Learning OpportunitiesNumber of accreditation levelsNumber and location of venuesNumber of marketing and information toolsPercentage of documentation in central repository

Program Relevance

measure

indicator

Program Relevance

Meet Client and User Expectations

Percentage of implementation of graduate feedback

Percentage of implementation of results of employer feedback

Program Delivery

Qualified InstructorsNumber of qualified professionalsRate of professional developmentRate of involvement in outside activitiesRate of motivationRate of competencePercentage of consistency in teacher training

Linguistic Aptitude of StudentsAverage entry levels

Program Delivery

SelectionPercentage of qualified nominationsPercentage of enrolment procedures advertised

RetentionPercentage of voluntary terminationsPercentage of involuntary terminations

Program Coherence

ControlsPercentage of control documents reflecting needs

of MoDPercentage of documents current and completePercentage of documents meeting system principles

Teaching MethodsPercentage of application of best practicesRate of integration of student needsRate of application of measurement tools for assessment of learning

Program Coherence

Program DeliveryRate of integration of ancillary valuesPercentage of throughputPercentage of completion in time required

Coherent CoursesDegree of skill masteredDegree of coherence: learning activities and standardsDegree of compatibility: general and specific learning activitiesDegree of sequence balance in program elementsDegree of accuracy of assessment and evaluation tools

Program Management

Instructional LoadClass sizeStaff/student ratioPercentage of office hours to classroom hoursRate of use of student support services

Pedagogical ToolsNumber of resources availablePercentage of resources used

Program Management

Operating BudgetNumber of special requestsPercentage of accuracy of forecasts

ManagementRate of integration of staff support servicesRate of joint action of program committees and

managementRate of integration of program review recommendations

Student Assessment

Advances in the last twenty years:• Research methodology• Practical advances• Research into performance affect• The nature of authentic or performance

assessment• Ethics of testing

Student Assessment

• Performance assessment• Motivation and attitude variables

– Integrativeness, attitudes to learning situation, motivation, integrative orientation, and instrumental orientation, age, availability

• Proficiency Testing• Impact on social structures

Operations

Our collective focus is operational – all assessment must have a demonstrable correspondence between language test performance and language use in this context

Is an assessment useful in this context?

Drawbacks

What happens when assessment does not correspond with learning?

Interactivity

Spontaneity

Communication

Exploration

Syntax

Accord

Orthography

Formality

Common Ground

Basic concepts, vocabulary, and procedures are essential for effective communication within our community of practice – i.e., our teachers and testers and managers and students.

Learn and Enjoy

Bonne route

BucharestOctober 2008