IEN Inaugural Annual ConferenceSeptember 24th 2004

Evaluation: A tool for meaning-making in a

Higher Education Quality Assurance culture

Content1. Conceptions about how mind works2. The OTC ‘culture’ as a context for the

Quality Assurance framework3. The OTC Quality Assurance framework4. Evaluation as a tool for ‘meaning

making’ within the OTC culture5. How ‘meanings’ are communicated as

catalysts for both individual and collective development and progression

Conceptions about how mind works Bruner (1995)

Computational View Information processing

Culturalism Meaning-making ’ Situated’ learning and thinking

‘Education is not simply a technical business of well managed information processing…..It is a complex pursuit of fitting a culture to the needs of its members and of fitting its members and their ways of knowing to the needs of the culture’ Bruner 1986:43

Guiding Principles Vision and Mission

Ethos Pedagogical Practice

O.T.C. QA Framework

1. Quality Control2. Quality Assurance3. Continuous Quality

Improvement

Guiding principles Inclusiveness Openness Relevance Student centeredness Accountability Accessibility

College Vision and Mission

‘The Open Training College will be recognized for the major role it plays in the advancement of the inclusion of people with disabilities in Irish Life; the excellence of our training; our commitment to our students; the quality of our relationships with agencies;…………………………….

Mission/vision statement cont.

our advancement of adult lifelong-

learning opportunities and our contribution to change and best practice at the level of the service-user, the staff member and the agency in the disability sector’.

College ethos and values Enhancing the status of people with a

disability Fundamental principles Valuing students as colleagues Close working relationships Cost effectiveness and value for

money Strategic planning

Pedagogical practice Shared understanding of the adult

learning process Courses designed to provide and

stimulate diverse learning experiences, building of the skills and knowledge that our learners bring to their learning

Clearly articulated learning outcomes and related assessment procedures

The Open Training College Quality Assurance framework

Please see handout

Evaluation process model for course evaluation

Please see handout

How do we evaluate?‘Responsive Evaluation’ To covey a message rather than specific

data Stake (1975)

‘Evaluation is an OBSERVED VALUE compared to some STANDARD’

Scriven (1967)

OTC standards

Represented in the: Guiding principles Vision and Mission statement Ethos Pedagogical Practice

Interpreting the evaluation summaries

The basic task of an evaluator is made barely tolerable by the fact that he/she does not have to solve this equation in some numerical way nor to obtain a descriptive summary grade but needs merely to make a comprehensive statement of what the programme is observed to be, with useful references to the satisfaction and dissatisfaction that appropriately selected people feel toward it’

Stake (1975)

Making meanings Meaning making involves situating encounters

with the world in their appropriate cultural contexts in order to know ‘what they are about’. Although meanings are ‘in the mind’ they have their origins and their significance in the culture in which they are created. It is this cultural situatedness of meanings that assures their negotiability and ultimately, their communicability’.

Bruner (1986:3)

How meanings are communicated1. Evaluation summaries (QA coordinator)2. Meaning-making within and as a result of

the shared micro- culture of the OTC (tutors)

3. Feedback to Programme Boards through Annual Course Review Reports (tutors)

4. Recommendations made to Academic Council (Programme Boards)

5. Ratification of improvements and developments (Academic Council)

6. Implementation of the same (OTC team)

7. Dissemination to all stakeholders (QA coordinator)

Summary We make meanings of evaluation

data by filtering our interpretations through our minds in order to measure findings against specified standards

Macro and micro cultures shape mind and provide us with a toolkit by which we construct and communicate meanings

If meanings from evaluations procedures are to be catalysts for development and progression; evaluation procedures must be embedded within a clearly defined shared organizational culture to which its members are committed.