practitioners publication · 538743-llp-1-2013-1-it-grundtvig-gmp edueval-wp4_guidelines and...

-------

***

* *

* *

* *

***

EV ALU ATION FOR THE PROFESSIONAl

DEVELOPMENTOF ADULI EDUCATION STAFF

Practitioners' Publication

----

--

Lifelong Learning Programme

This project has been funded with support from the European Commission. *** * * * *

Lifelong Learning Programme

This publication reflects the views only ofthe author,

and the Commission cannot be held responsible for a

ny use which may be made ofthe information contained therein.

EDU EVAL

* * ***

EV ALU ATION FOR THE PROFESSIONAL

DEVELOPMENTOF ADULI EDUCATION STAFF

2014-2016

Project Coordinator:

<!'. DEGLI STUDI t o � i: University of Milano-Bicocca -� � Department of Human Sciences § � for Education "Riccardo Massa"(ltaly)BICOCCA

Contacts:

Website: www.edueval.eu

E-mail: [email protected]

---

Ed.: Consortium Staff and Practitioners ---

Partners:

UNIVERSITAT

.JAUME•I

-�

Rezekne Agustskola -Rezekne Higher Education lnstitution (Latvia)

Universitat Jaume I (Spain)

Janusz Korczak

Pedagogica! University

in Warsaw (Poland)

TEI of Crete - School of Health & Social Welfare (Greece)

UNIVERSITÀmGusrummBAR1 University of Bari (ltaly)ALDO MORO

Evaluation for the Professional Development of Adult Education Staff 538743-LLP-1-2013-1-IT-GRUNDTVIG-GMP

EduEval-WP4_Guidelines and Handbook_Practitioners’ PublicatioDeliverable 4.2

1

EduEval Project WP3 Practitioners Publication

Deliverable 4.2

WP Reference WP3 Mobility Workshop

Category x D Deliverable

F Financial

R Report

ER External review MN Minutes

TS Time Sheet

WD Working Document, not otherwise classified

Author

Reviewer n.r.

Date 28/2/2016

Version 2.0

Availability Public

EduEval project Evaluation for the Professional Development of Adult Education Staff 538743-LLP-1-2013-IT-GRUNDTVIG-GMP)

This documents reflects the views only of the authors, and the Commission cannot be held responsible for any use

which may be made of the information contained there in



2

Document History

Version

Implemented by Revised by

(if required)

Revision Date (if required)

Approved by Consortium approval Date

1 consortium staff

and practitioners n.r. n.r. n.r. 28/02/2016



3

Table of contents

Summary 4

National Case Studies 4

CASE STUDY P1 BICOCCA: Study case of the Residencial community for people with

disabilities

4

CASE STUDY P2 REZEKNES AUGSTSKOLA: "Validpack tool" 18

CASE STUDY P3 JKPU: "Social workers learn about cyberspace" 30

CASE STUDY P4 TEI OF CRETE: "Training in the Crete immigration office" 36

CASE STUDY P5 UNIVERITAT JAUME- I: "ISO 9001 in educational centre" 53

CASE STUDY P6 UNIBA: "Education to inmates" (UNIBA) 59

“Feedback form for Guidelines and Handbook“ Answers (1) 66

Practitioners' open space for discussion



4

Summary This publication contains all the contributions shared by practitioners and researchers on the EDUEVAL’s wiki that allowed on-line networking among the different subjects involved in the EDUEVAL project and/or were interested in the topic. The wiki aimed at facilitating the building up of a transnational collective group and worked as an operating system that collected all the case studies presented by practitioners and their in-depth elaborations. Original contributions are avaiable at: http://wiki.edueval.eu/index.php/Main_Page

National Case Studies

The Mobility Workshop had an important role to warrant a real impact of the project, as it contemplated the direct participation of practitioners from the target group, with the aim of sharing the experience to build a common know-how.

CASE STUDY P1 BICOCCA: Study case of the Residencial community for people with disabilities

CASE STUDY P1 BICOCCA

An evaluation of the educational work of a professional team at a residential

community for people with disabilities

Introduction

The Residential Community “KKKK” for People with Disabilities was set up 10 years ago, in

September 2004. It currently hosts seven persons with cognitive disability in a 200 sqm

apartment in a provincial town in Lombardy. The service is offered by the town council, who

initially entrusted the day-to-day running of the community to a social services company. Over

the years, managing the service proved to be fraught with difficulty and in 2010 the social services

company (partly due to the fact that the “do-all” owner had fallen ill) went out of business. The

running of the service was then provisionally handed over to a social services cooperative

charged with keeping the community functioning while its ongoing management was being put

out to tender. The outcome of the tender process was that management of the service was

entrusted to another social cooperative.

The context



5

The new social services cooperative, during its initial phase of operation, obtained all the human

resources required to organize and develop the service’s educational programme and the life

plans of the individual residents, while taking into account the town council’s request not to

effect major changes in the early stages of its tenure; it therefore reconfirmed the positions of

all of the existing professional team including the educational coordinator, while strategically

appointing a project manager from within its own ranks. The situation immediately proved

extremely difficult: the cooperative was effectively unable to take control of the service, which

almost seemed to be managed via a form of power-sharing among internal interest groups (that

functioned somewhat like “gangs”).

The team was made up as follows:

- A clinical supervisor and a social worker; the de facto coordinators of the service, they

were lacking in an overall perspective (and thus were not clear on where they were going

and why, nor on their own roles and responsibilities: they were just happy with the status

quo and in practice they were in charge of the entire team);

- A pedagogical coordinator who was a qualified educator; she had worked in the service

as an educator for some years before being appointed coordinator/educator after the

social services company went out of business; highly inexpert and incompetent, she was

dependent on the supervisor and social worker and tolerated ongoing interference on the

part of another educator, an “informal and hidden leader” within the group. She never

make decisions of her own, but merely rubberstamped the decisions of the supervisor,

social worker and educator leader. In her meetings with the project manager she

constantly reassured him that everything was going well, at times using inappropriately

informal language. She tended to protect the interests of the educator leader (who used

aggressive tactics to get his own way, as later emerged) at the expense of the rest of the

team.

- The team consisted of: a male professional educator (the hidden leader), three female

professional educators (two with the qualification of educator, who had long been in the

employ of the service and another, appointed on the takeover by the new cooperative,

who was a qualified psychologist), two health workers (one of whom acted as counter-

leader within the group and embodied an authoritarian educational model; her father

had been a Peruvian army general).

Background to the case study

During the early months the project manager attempted unsuccessfully to take control of the

service in order to guarantee a good level of coordination and organization. He met all the

persons involved in order to explain the changes that he gradually planned to introduce, but did

not find much room for maneuver. He strove to support the pedagogical coordinator, attempting

to empower and encourage her, while pointing out critical areas and inviting her to assume a

degree of responsibility in line with her official role; he tried to more clearly define the roles of

the clinical supervisor (all decisions went through him and not through the pedagogical

coordinator) and the social worker. He met the team of educators and health workers, listening

to their concerns, explaining his plans and presenting some specific proposals to them; he

informed them that he would be visiting the community from time to time in order to get to

know the residents better. The astounding response that he received was: “The people that



6

we look after in our community are very fragile and the staff is only barely able to maintain a

state of equilibrium, so encounters with “outsiders” are not a good idea because the group must

be protected”. After the project manager had outlined his own role and responsibilities, he was

told that he could visit the community while the residents were out; at that point; he firmly stated

that he was responsible for the community during its functioning and that there was no more to

be discussed on this point.

A short time later the social worker resigned (with immediate effect) as did the pedagogical

coordinator (giving six weeks’ notice). A new coordinator was identified and appointed and this

time, finally, it was decided to emphasize and affirm the strategic nature of her role – the new

incumbent was a partner in the cooperative as well as one of its most experienced coordinators;

she immediately began to work alongside the outgoing coordinator as part of the handover

procedure.

The Case

A week after conclusion of the handover, the project manager and the new coordinator were

invited to a meeting with the social services, at which the other participants included: a

representative of the town council, the outgoing coordinator, the supervisor and the male

educator (negative leader). The project manager thought that the purpose of the meeting was

to review the steps taken up by the new cooperative up to that point, however to his surprise,

the agenda for the meeting was: the discharge of Luana (a woman who also suffered from

psychiatric problems who had been a resident of the community for about six years).

The case was presented and the supervisor, outgoing coordinator and educator explained that

unfortunately it was not possible to keep Luana in the community any longer, because her

behavioural problems were putting the entire team under pressure; furthermore, she had taken

to running away, she would say that she was going to the bathroom but in actual fact would leave

the apartment and not come back for several hours. It was not possible for one of the two

educators on duty to go out and look for her, because this would have meant leaving the other

educator alone with six residents: therefore each time they had to phone the police. The town

council representative approved the decision (she did not seem surprised by it, leading us to

surmise that she had already been informed) although she seemed concerned that no alternative

solution had been identified for Luana. At this point the project manager intervened to explain

that, from an ethical point of view, it is not possible, and even less so in services that provide care

for fragile individuals, to discharge persons without having established what alternatives are

open to them. He also stressed that residential communities for people with mental and cognitive

disabilities by their very nature have been set up to cater for the needs of fragile individuals who

are often difficult to care for. The service’s mission is to address and manage this complexity in

order to generate wellbeing and such an aim cannot be pursued by seeking the advantage of

some at the expense of others; the team members present “attacked” the project manager

explaining that the team has been examining the issue for several months and that this is the

only solution that they could find; they also claimed that the project manager was at fault for not

having been aware of the case. The project manager, after pointing out that he did not routinely

participate in their meetings and that therefore the pedagogical coordinator should have briefed

him on the matter, especially given the sensitive nature of the issue, went on to clearly explain

once again his role in the service, the responsibility that he was taking on himself, and his



7

decision-making powers – on the basis of which, given that he was in strong disagreement with

their decision and that he felt that the team had significant potential that was currently

unexpressed, that Luana was not to be discharged; he also stated his commitment to the Town

Council to continue developing the project so as to address the critical issue raised and to bring

about an overall improvement in the atmosphere in the “house”.

At the end of the meeting, the project leader and future coordinator had the impression that

Luana’s proposed discharge was, in some sense, a “goodbye present” from the outgoing

coordinator to the educator leader and to the rest of the team that she was leaving.

It was difficult to work with the team over the following months and the new pedagogical

coordinator had to use all her skills and resources in order to take over leadership and manage

the situation from both technical and emotional points of view. To this end, she requested and

obtained the project manager’s full support.

After some months, the project leader and the coordinator made the strategic decision to bring

onto the team, with reducing anybody’s hours but on the basis that this would enable staff

members to be substituted without any reduction in quality of care, the best educator in the

employ of the cooperative: a trusted collaborator, highly competent and expert, who would be

skilled enough to join the group dynamics, breaking existing patterns or modifying them if

unhealthy. It was also thought that this educator would open up a channel of communications

between the management group, the residents and the staff. Meanwhile, having clearly defined

the supervisor’s role to him, the scope available to him and the limits that applied, his hours at

the community were reduced. It was explained to the team that routine decisions regarding the

running of the community would be taken by the coordinator, whereas extraordinary decisions

would be taken by the project manager or by the cooperative’s senior management.

One evening, while the pedagogical coordinator was on holidays, the project manager was

contacted by the educator leader, who informed him that Luana had fallen while trying to get

out through the window, and that she was currently lying on the ground bleeding. The educator

took this as an excuse to go back over and criticize the project manager’s earlier decision,

repeating that the community could no longer continue to host Luana. The project manager

asked why, given that the event did not come out of the blue with Luana escaping almost every

day, the bathroom window had not been locked from the outside, and even more importantly –

at this point shouting down the phone at the educator – why an ambulance had not yet been

called. Luckily Luana had not sustained any significant injuries and as soon as her minor cuts had

been looked after, she came back to the community.

This episode was the straw that broke the camel’s back and the project manager and coordinator

met to evaluate the current running of the community as well as its future prospects; they

decided, in agreement with one another, and having gained further information from the other

staff members about the educator leader’s unprofessional approach, to substitute him and to

move him to another service managed by the cooperative, with different characteristics, more

tightly run and easier to monitor, with a well-organized team of educators that would be unlikely

to allow itself be dominated by the educator leader’s personality.

Epilogue

The transfer of the educator was very difficult to manage. He strongly resisted the move and

deployed all the “arms” at his disposal: initially he was seductively cooperative, but



8

subsequently he became increasingly aggressive, resorting to the use of extremely offensive

language and even threats. Naturally he sought the assistance of the trade union, which however

could not find any fault with the procedure that had been adopted, although they tried to imply

that the cooperative might lose out in terms of image and reputation. The project manager and

coordinator, prompted by their strong belief that they were doing the right thing and that quality

of life in the community would improve as a result of their action, sought expert advice and did

not back down. The educator once transferred to the new service, perhaps because feeling

demotivated, built up such a record of impunctuality, unjustified absenteeism and months of sick

leave, as to create legitimate grounds for his dismissal.

The running and quality of life in the community have improved considerably although it remains

a structurally complex service. The most important thing is that the “keys” of the house have

been given back to the residents and the professional team members have organized themselves

so as to put themselves at the “service” of their clients and to provide support to the residents

in relation to their own particular limits and fragility. Luana was invited to share her reasons for

running away: she felt that she had been listened to, considered and understood. And at that

point she no longer experienced the need to run away. In any case the windows have been fitted

with locks and in case of difficulty, may be secured in order to guarantee residents’ physical

safety; this function has not been used to date. Life in the community has become more

interesting: excursions, cultural outings, museum visits are organized along with at least four

holidays per year; friends are often invited, just as in any other home, and this is a cause of

wellbeing (and not distress) for everybody.

Understanding the Residencial community for people with disabilities case

Here it is written what we discussed in Creta. You should not modify this section, unless you want to add or clarify something about the Study Case. This section is not for discusion, but only for understanding and clarification. This should contain only: The minutes from Creta workshop Any addition done by the Study Case leader, as any clarification, extra information, other links or doccuments, etc Creta meeting conclusions 1. Evaluation methodologies & instruments The strategy of the Project Manager (PM) is articulated on various levels: he uses the power given by his role to know how the educational staff works and to modify the rules, the educational models and the work routines of the adult education staff. 1.To know how the staff works: 1. He changes the coordinator; 2. He enters a trusted educator as a jolly under the cloack of replacing a member of the staff and sustaining it if needed. This educator has two tasks: gaining the trust of the other educators and reporting to the PM what happens in the community. This educator seems playing a “double-cross”, like a “spy”;



9

3. He listens to and talks with the former coordinator and the educators through both formal one-to-one conversations and staff meetings; 4. He tries to participate to the community life despite the staff resistances. 1. To modify the routines: 1. He clarifies the roles of the coordinator, supervisor and PM; 2. He redeploys the educator who is the informal leader of the staff; 3.He gradually changes the rules in the day to day life of the community; 4. He introduces and schedules formal moments to discuss with the staff about the meanings of the work in the community and to listen to the educators’ emotions; 5. He introduces regular but informal events, as parties or dinners, to spend together. This is a transformative strategy based on both control and understanding. 2. Evaluation criteria The evaluation criteria are: 1.The concepts of well-being of the disabled people who live in the community: PM and the staff have different meanings about it; 2.The meaning of the aim of the educational work: is it the control (the resident’s life shouldn’t be troubled by unexpected events) or the learning through experience (unexpected events are part of the day-to-day life and it needs to be learned facing them)? The educators seem to agree with the first answer while the PM agrees with the second; 3.The community mission that is declared in the official documents (“chart of services”); 4.The perception of the educators’ distress and the hostile and depressed atmosphere: this is a PM criteria; 5.The consideration of the history of the staff (its changes, its relations with the former provider, its educational tradition and so on); 6.The interpretation of the residents’ discomfort (i.e. the Luana escape attempt) as a manifestation of general discomfort. This is a PM criteria. 3. Evaluation representations /main ideas/main aims There are different ways to represent the evaluation of the adult education staff and of the educators’ work. These differences seem based on the subjects and the objects of the evaluation itself. The idea of evaluation and its criteria changes whether the evaluator is the provider, the social worker, the project manager, the educators and it changes as well when the evaluation objects are the educational results or the processes. It brings out an idea of evaluation as following: A practice of knowledge: evaluation is possible if there is an “internal” and deeper knowledge of what happens within the community; A practice of control and an exercise of institutional power that produces and shows unavoidable resistances – people dislike to be evaluated. It reveals the assertive or “authoritarian” soul of the evaluation; A practice that allows the possibilities: a) of clarifying the aims of the educational work, b) of seeking the sense of individual and staff’s work, c) of sharing emotions and ideas about the educational work.



10

The evaluation seems to be: a practice of decision-making that could have strong consequences in both the professional and the personal educators’ life, a practice that allows the staff to share thoughts and empowering itself. However, the evaluation seems to be a fracture in the routine: it indicates a transitional moment. The main aim of the evaluation is to change a situation that doesn’t work. In order to achieve this goal, it needs to know the routine staff, the day-to-day life of the community, the context and its history, the single educators and the other professionals who work in this context. Changing situation implies also to recognize and to modify the educators’ action models: in this case, it implies to modify an assistance model into an educational model based on the learning through experience. 4. Emotional and relational dynamics (evaluator and staff) At the beginning of the story, the main emotion of the staff is fear. Therefore, the main dynamics are defensive. The staff lives in a situation of conspiracy of silence. The educators are not autonomous in thinking about their acts; they are dependents by the informal staff leader and unmotivated. The PM attitude is at the same time suspicious and worried about the situation of the community and the staff. Closure, exclusion, distrust characterize the atmosphere. In the story development, the relational dynamics change and become various. The new educator tries to seduce the other educators and seems dividing the staff. There’s diffidence against the new coordinator. The educator who was the informal leader tries to strengthen the alliance with the other members of the staff and he uses Luana’s accident to challenge the PM and his educational approach. The PM wins the challenge. There’s tension, like in a silent war. At the end of the story, after the informal leader dismissal, the atmosphere is relaxed. The life within the community become more interesting and the educators become more motivated.

Residencial community for people with disabilities case from UNIMIB perspective Based on following topics: evaluation methodologies and instruments? evaluation criteria? evaluation representations? the emotional and relational dynamics (evaluator and staff)? Additional information: tools, processes, methodologies, indicators, suggestions, etc Decompose the Study Case. This is a process of analysis and abstraction, aimed to know which are the methodologies, and other aspects related to evaluation in this case. You can write this in your own language, but please, any coordinator from the local institution should be able to translate to English



11

Comments The work of comparing practices has allowed the following considerations to be made. Evaluation is understood in a broad sense. The Italian approach starts from the specificity of the object to be evaluated (the educational work) and from the reality (a community for people with mental disorders) in which it is situated, as can be seen from the type of case that has been chosen to be presented. According to the model of comprehension that emerges from the analysis of our evaluation case, evaluation does not only consider the intra-individual processes of learning or the competences acquired along a certain path, but is concretized in a process that comprises both the evaluator and the context in which this process occurs. According to this conception, it is over-simplifying to extrapolate the evaluation and consider it separately from the work process in which both the evaluator and the subject being evaluated are involved: the evaluation becomes a function and an instrument of this process, being adapted case by case and time by time in many different ways. From the analysis of the Italian case, there thus emerges an idea of evaluation which implies first of all the involvement of the evaluator in the situation. It is not a peer-to-peer evaluation nor does it take on the appearance of a report made by an expert observing from the outside, providing indications on what is to be done. It is, however, a process implemented by a professional involved in the facts and in the educational context, whose role is different from that of the Adult Educators and who has a political power within the situation. The evaluation in this case invokes the asymmetry between the evaluator and the subject evaluated. It is envisaged as a time in which a responsibility is exercised and a decision is taken, and therefore a process of change is started. The evaluator’s feedback is to be considered as a possible starting point to make changes in terms of increased well-being of the educator, of the adults and of the organization he/she belongs to. If the evaluator’s feedback or the process he/she has activated do not trigger off any transformation in those receiving the evaluation and/or in the work context, it becomes a protocol aimed at maintaining the status quo or a procedure in which the information provided by the subject being evaluated will very probably reflect the expectations of the evaluator more than the real situation. The Italian case offers a way of conceiving evaluation as a “function” of the formative process. The Italian case is also characterized by the complexity of the evaluation device, which has to do not only with the subjects and their internal learning dynamics but which also and above all concerns the way people, in particular contexts, face up to given situations. Within this device, the need to evaluate the work processes implemented before the individual competences of the educators is recognized, paying constant attention to the pragmatic consequences of evaluation and its impact on the educational practices. In this sense, envisaging the consequences of the feedback that is provided by the evaluator, both with respect to the organization and the destinies of the individuals, is considered essential. Therefore the evaluator, involved in the process, in a precise time and space, must be aware not only of his/her competences, but also of how he/she acts and interacts within the process, of the reactions and emotiveness which he/she inevitably triggers off and which he/will will have to master retroactively. A model seems to emerge which takes its distances from an evaluation of the diagnostic type certifying competences or situations. The evaluation process seems to take on meaning only within a broader formative process; due to this specificity, it can be profitably applied to complex and particular contexts such as those relative to the vast field of education. Despite the multiple forms that evaluation can take on in the different ways the evaluator operates, some guiding criteria can definitely be outlined, but it is difficult to find a generalized operability that can be implemented in all contexts and with all beneficiaries. The representation of an evaluation which changes profoundly in the approach, in the internal logics and in the choice of the strategies to use depending on the object to be



12

evaluated emerges. In the final analysis no valid instrument exists for every situation, but evaluation changes depending on the subjects, the processes or the segments of processes that are to be evaluated. The evaluator must choose instruments that can intercept some dimensions rather than others, thus building up a specific methodological strategy that helps to reach the complexity of the objects to be evaluated.

Residencial community for people with disabilities case from UNIBA perspective

The work of comparing practices has allowed the following considerations to be made.

We have tried to put ourselves in the shoes of the project manager/evaluator to analyze and manage all the informations about the complexity and difficulties related to the community context. It seems a hard stuff to do, in order not only to clarify the mission of the service, but also to plan the changes and point out the critical areas. At first sight, it could be not so easy to organize and handle all the multi-level dynamics among the operators who work within the community care service. They belong to different areas, professional roles and backgrounds, so it is very useful to apply a hermeneutical and/or phenomenological approach to make cear the definition of the roles and who holds the power throughout the decision-making process. All the internal stuff struggle seem to pivot around the alternation in time between the outgoing coordinator and the incoming one, which causes an important shift within the staff and force all the members to face their copying strategies and put themselves at the service of their clients, really. We have plaused two big decisions made by the project manager: to bring onto the team a trusted collaborator to modify the unealthy patterns and open up a channel of communication; to promote a kind of "ethical evaluation" of the real needs and conditions of the clients (the observance of ethical principles), according to a higher responsibility level. So, evaluation is not something to be afraid of pretending that "everything is going well"(even if some members of the community staff build up a coalition/alliance to attack the evaluator/project manager), but is a necessary step to pass through and a compulsorry goal to achieve for improving the running and quality of life in the community.

Methodologies and Instruments

Planning the changes and pointing out the critical areas, in order to clarify the mission of the service.

Meetings between the project manager and the outgoing pedagogical coordinator, in order to empower her.

Meetings between the project manager and all the persons involved, in order to listen to their concerns.

The definition of the roles.

The handover procedure involving the former coordinator and the new one.

The decision to bring onto the team a trusted collaborator to modify the unealthy patterns and open up a channel of communication.



13

Functional and assertive language used by the project manager.

Hermeneutical and/or phenomenological approach.

Pair working to share and make decisions.

Evaluation Criteria

The decision making process.

The quality of communication channels.

The well being conditions of the community residents.

How much the team members are organized to put themselves at the service of their clients.

How to care of the needs of fragile individuals.

How to manage and respond to the complexity of the service.

The quality of the provided services.

The indipendence degree of the community residents.

Hierarchical organization.

Evaluation Representations

For the project manager:

- a kind of "think alternative" evaluation, in order to analyze all the possible alternatives;

- ethical evaluation of the real needs and conditions of the clients (observance of ethical principles)

- responsibility evaluation;

- decision-making empowered evaluation;

- bottom up or upside down evaluating process;

- monitoring the activities of the operators;

- management of group dynamics (complying hierarchies and roles)

For the former coordinator and the rest of the team:



14

- a kind of heavy control;

- lack of the power-sharing;

- something unnecessary/superfluous ("everything was going well")

Emotional and relational dynamics

Initially:

- a form of power-sharing;

- to be happy with the status quo (referring to the clinical supervisor and social worker);

- to be dependent on someone else (referring to the relationship between the outgoing coordinator and the hidden leader);

- to be afraid of making decisions on their own; using seductive manners and inappropriated and informal language (both the outgoing coordinator and the hidden leader).

Specifically:

- envy, anger, aggressive manners, offensive language, authoritarian educational model, threats (the hidden leader).

Group dynamics:

- team members' coalition/alliance to attack the evaluator/project manager (interpersonal tensions, conflicts, opposition, critical positions, ostracism, defensiveness, distrust);

- team members' exploiting of the residents' frailty (they use as a pretext the frailty of the residents to avoid the project manager's visit in the community);

- they set up an ambush for the project manager and the new coordinator at the social service office (they use as an excuse the case of Luana in order to criticize and discredit the project manager);

- the conspiracy of the outgoing coordinator first(a "goodbye present") and of the hidden leader then (impunctuality, unjustified absenteeism, months of sick leave);

Finally:

- more tolerance;

- improved running and quality of life in the community;



15

- more attention paid to the needs of the residents;

- better work organization;

- open community (news are welcome: excursions, cultural outings and visits, invited friends)

Residencial community for people with disabilities common discussion place and comments

The Italian case'


Evaluation is understood in a broad sense. The Italian approach starts from the specificity of the object to be evaluated (the educational work) and from the reality (a community for people with mental disorders) in which it is situated, as can be seen from the type of case that has been chosen to be presented. According to the model of comprehension that emerges from the analysis of our evaluation case, evaluation does not only consider the intra-individual processes of learning or the competences acquired along a certain path, but is concretized in a process that comprises both the evaluator and the context in which this process occurs. According to this conception, it is over-simplifying to extrapolate the evaluation and consider it separately from the work process in which both the evaluator and the subject being evaluated are involved: the evaluation becomes a function and an instrument of this process, being adapted case by case and time by time in many different ways.

From the analysis of the Italian case, there thus emerges an idea of evaluation which implies first of all the involvement of the evaluator in the situation. It is not a peer-to-peer evaluation nor does it take on the appearance of a report made by an expert observing from the outside, providing indications on what is to be done. It is, however, a process implemented by a professional involved in the facts and in the educational context, whose role is different from that of the Adult Educators and who has a political power within the situation. The evaluation in this case invokes the asymmetry between the evaluator and the subject evaluated. It is envisaged as a time in which a responsibility is exercised and a decision is taken, and therefore a process of change is started.

The evaluator’s feedback is to be considered as a possible starting point to make changes in terms of increased well-being of the educator, of the adults and of the organization he/she belongs to. If the evaluator’s feedback or the process he/she has activated do not trigger off any transformation in those receiving the evaluation and/or in the work context, it becomes a protocol aimed at maintaining the status quo or a procedure in which the information provided by the subject being evaluated will very probably reflect the expectations of the evaluator more than the real situation.

The Italian case offers a way of conceiving evaluation as a “function” of the formative process.



16

The Italian case is also characterized by the complexity of the evaluation device, which has to do not only with the subjects and their internal learning dynamics but which also and above all concerns the way people, in particular contexts, face up to given situations. Within this device, the need to evaluate the work processes implemented before the individual competences of the educators is recognized, paying constant attention to the pragmatic consequences of evaluation and its impact on the educational practices. In this sense, envisaging the consequences of the feedback that is provided by the evaluator, both with respect to the organization and the destinies of the individuals, is considered essential.

Therefore the evaluator, involved in the process, in a precise time and space, must be aware not only of his/her competences, but also of how he/she acts and interacts within the process, of the reactions and emotiveness which he/she inevitably triggers off and which he/will will have to master retroactively.

A model seems to emerge which takes its distances from an evaluation of the diagnostic type certifying competences or situations. The evaluation process seems to take on meaning only within a broader formative process; due to this specificity, it can be profitably applied to complex and particular contexts such as those relative to the vast field of education. Despite the multiple forms that evaluation can take on in the different ways the evaluator operates, some guiding criteria can definitely be outlined, but it is difficult to find a generalized operability that can be implemented in all contexts and with all beneficiaries.

The representation of an evaluation which changes profoundly in the approach, in the internal logics and in the choice of the strategies to use depending on the object to be evaluated emerges. In the final analysis no valid instrument exists for every situation, but evaluation changes depending on the subjects, the processes or the segments of processes that are to be evaluated.

The evaluator must choose instruments that can intercept some dimensions rather than others, thus building up a specific methodological strategy that helps to reach the complexity of the objects to be evaluated.

Hi! You can see below the UNIBA practitioners' comments about the Latvian and UNIMI cases.

The Italian UNI.MI case'


We have tried to put ourselves in the shoes of the project manager/evaluator to analyze and manage all the informations about the complexity and difficulties related to the community context. It seems a hard stuff to do, in order not only to clarify the mission of the service, but also to plan the changes and point out the critical areas. At first sight, it could be not so easy to organize and handle all the multi-level dynamics among the operators who work within the community care service. They belong to different areas, professional roles and backgrounds, so it is very useful to apply a hermeneutical and/or phenomenological approach to make cear the definition of the roles and who holds the power throughout the decision-making process. All the internal stuff struggle seem to pivot around the alternation in time between the outgoing coordinator and the incoming one, which causes an important shift within the staff and force all the members to face their copying strategies and put themselves at the service of their clients,



17

really. We have plaused two big decisions made by the project manager: to bring onto the team a trusted collaborator to modify the unealthy patterns and open up a channel of communication; to promote a kind of "ethical evaluation" of the real needs and conditions of the clients (the observance of ethical principles), according to a higher responsibility level. So, evaluation is not something to be afraid of pretending that "everything is going well"(even if some members of the community staff build up a coalition/alliance to attack the evaluator/project manager), but is a necessary step to pass through and a compulsorry goal to achieve for improving the running and quality of life in the community.



18

CASE STUDY P2 REZEKNES AUGSTSKOLA: "Validpack tool"

CASE STUDY P2 REZEKNES AUGSTSKOLA

Validpack experience Capitalizing on Validpack: going Europe wide – CAPIVAL (www.capival.eu).

Partners of the project: Romania, Italy, Latvia, Germany, Portugal, France, Spain, Denmark, Netherlands.

The aim of the project: to develop and maintain the network in Europe, facilitating the set of instruments for competence evaluation– Validpack introduction in 20 European countries. Tasks of the project:

to introduce all the European countries with the set of instruments called Validpack, created within the VINEPAC project www.vinepac.eu, establishing the national Validpack contact centres;

to study the Validpack necessity, significance and potential in various national contexts, performing Validpack testing in the interested organisations – evaluation centres, adult education centres/establishments, vocational associations, among employers etc., thus facilitating the application of this means;

to study the Validpack necessity, significance and potential in various national contexts, performing Validpack testing in the interested organisations – evaluation centres, adult education centres/establishments, vocational associations, among employers etc., thus facilitating the application of this means;

in cooperation with corresponding interested parties, to ensure the application of Validpack;

to distribute the results of Validpack introduction widely. Process of the project:

What is Validpack? (www.vinepac.eu): The Validpack is a package of validation instruments, unique of its kind at European level and it was considered an example of good practice by

http://www.capival.eu/

http://www.vinepac.eu/



19

European Centre for the Development of Vocational Training (CEDEFOP). Handbook & Evaluator’s Guide contains useful guidelines and indications for adult educators on how they should approach this validation process. The instrument creates a framework for the documentation and evaluation of real competences of adult educators, no matter whether they have been acquired in formal, non-formal or Informal learning contexts. VALIDPACK is an instrument resulted from VINEPAC project. Tasks of the Validpack:

To assess trainers’ competences

To provide space for documentation of experiences

To build a base for certification upon the validation results

To document a minimum standard of trainers’ competences

To offer trainers/adult educators the opportunity to get across the frontier within Europe.

Steps of the validation process (Lupou, 2010; VINEPAC, 2008a, 2008b;

http://voicethread.com/#q.b2023898.i10708399)

STEP I: Self evaluation-Reflection on adult educator’s biography: learning contexts and learning

outcomes

What are your previous experiences related to your role/position as a trainer in adult

education?

Have you involved in adult education activities without having an explicit adult education

job (consultant, counsellor, coordinator of study groups etc.)? If yes, please mention

them.

What opportunities for initial and further professional development have you taken as a

trainer (courses, peer tutoring, mentoring, study visits, internshipsetc.)

STEP I: Self-evaluation Reflection on adult educator’s biography: learning contexts and learning

outcomes

http://voicethread.com/



20

• What other contexts you find as offering relevant/possible opportunities for the

achievement of competences defining the performance as a trainer?

• Which concrete abilities/skills can you associate with a concrete learning context?

• Can you name concrete activities where you learned/acquired competencies/things you

now use in your adult education work?

• What are the learning outcomes of the profession/home/leisure time/work etc.?

STEP II: External evaluation

STEP III: Consolidation

Working model of competency-based learning (Handbook for assessment and validation of

pedagogical competences of adult educators, based on Jaspers & Versteegen, 2004)



21

Main and extra categories of analysis of the Validpack instrument (Surikova, Rutka, Karttunen,

Jõgi, 2012)

Test results



22

General conclusions

Validpack is a very good instrument for recognising previous education because it is

comprehensive and universal, with a good and logical design. One can find there all necessary

criteria that make a psycho-pedagogical competence. This instrument can be taken as a basis not

only for assessment procedures but also for creating study programmes.



23

Validpack tool case from UNIMIB perspective

The work of comparing practices has allowed the following considerations to be made. The procedures and methods used in the evaluation are clearly shown in the evaluation case, with a great deal of information on the instruments used and the guidelines followed. From reading the case, it is possible to extrapolate the Latvian evaluation model, which on an operative level proceeds following clearly defined passages, shown as work steps. The outline shown is very useful in identifying the operating paradigm. The Latvian approach to Adult Education evaluation is generalized, being proposed as a valid instrument for all types of professionals who work in the field of education. In the model presented, evaluation appears to have a very precise physiognomy and the representation of the evaluator that emerges is a professional who, through the proposed method, can become qualified for evaluation. The evaluator appears as an expert who examines and provides at cognitive level feedback on the competences measured, a trainer who through a series of pre-defined passages, leads the adult educator to evaluate his/her work according to the ValidPack model. The training instrument proposed is useful to learn how to self-evaluate one’s own competences, producing, through the comparison of the outcomes of the evaluation which the adult educator is required to do with the outcomes of the evaluation by the evaluator, increased knowledge and cognitive awareness, which results in the composition of a sort of map of competences. To obtain these results, the evaluation minimizes the emotive factors as these could influence the evaluator. The context in which the adult educator operates also seems to have, in the evaluation process proposed, a relatively low value compared to what it is to be measured, i.e. the competences of the individual educator. The approach used in drafting the case seems to be focused on a representation of learning with, at the centre of attention, the development of the learning process and personal training of the individual educator. Due to these specificities, the model proposed is useful for identifying/certifying the competences of an individual adult educator, which is the objective stated in the model. It can therefore be seen as a basic training device for the evaluation of the work of the adults, as it is based on the use of self-awareness techniques and instruments (observations and interviews) which have the aim of allowing the subjects to acquire some procedures which they can in turn use in the various contexts of work.

Validpack tool case from RA perspective

The evaluation methodologies & instruments

The lifelong learning concept became topical after regaining its independence in 1991 and joining the EU in 2004. Lifelong learning is “an education process during the whole life of an individual that is based on changing needs to acquire education, skills, experience in order to increase or change their qualification in accordance with the demands of the labour market and their own interests and needs. Lifelong learning comprises non-formal learning and formal education, develops inborn abilities together with new competences”. In the Strategy the term learning is translated into Latvian as education. The education system of Latvia is administered at 3 levels – national, municipal and institutional. The Parliament (Saeima), the Cabinet of Ministers and the Ministry of Education and Science of Latvia are the main decision-making bodies on the national level. The leading state administration institution in the field of education policy is the Ministry of Education and Science (MES). MES collaborates with other ministries, 11 subordinated institutions, with < 30 international organizations and professional associations (9 Advisory Boards have been formed at MES). To ensure cross-sectoral coordination of all education



24

levels in lifelong learning coordination, it was decided to involve also the Advisory Board “Education for Everyone” in administering lifelong learning in Latvia. The basic principles of LLL are: shared responsibility, efficiency, synergy of field policies, availability, comprehension and equality of the society. The long-term target for LLL is “to ensure education through out the whole life according to the interests of inhabitants, as well as their abilities and socially economic development needs of the particular region”. The aim is reducing the regions socio-economic differences, promoting society integration and ensuring pair opportunities in education. The target groups. This is a step from a comprehension of providing equal education to everyone towards the comprehension of versatility of education in accordance with the various social groups. National policy in Latvia about education is compliant with EU statements: 1) at least 95% of children between the age of four and the age of starting compulsory primary education should participate in early childhood education; 2) the share of 15-year olds with insufficient abilities in reading,mathematics and science should be less than 15%; 3) the share of early leavers from education and training should be less than 10%; 4) the share of 30-34 year olds with tertiary educational attainment should be at least 40%; 5) on average at least 15% of adults (age group 25-64) should participate in lifelong learning.

The evaluation criteria

The field of adult learning was pioneered by Malcolm Knowles. He identified the following characteristics of adult learners: • Adults are autonomous and self-directed. • Adults should be challenged to move to increasingly advanced stages of personal development. • Adults are most interested in learning subjects that have immediate relevance to their job or personal life. We consider four critical elements of learning that must be addressed to ensure that adults learn. These elements are: • motivation • reinforcement • retention • transference

To evaluate the learners motivation a questionnaire is used which contains questions like these:

• What is the purpose of coming to this course? • How do you know that you have achieved your aims? • How do you feel,hear,see? • What abilities and possibilities do you need to fulfil your goals? • How will you overcome difficulties while achieving the goals? • How will achieved goals change your life?

Validipack basic idea

Validpack basic idea is to introduce all the European countries with a set of evaluation instruments, called Validpack, created within the VINEPAC project www.vinepac.eu, establishing the national Validpack contact centres;

The aim of Validpack is to support all the phases and aspects of the process of competence assessment through quality criteria; Steps of the validation process:

STEP I: Self evaluation Reflection on adult educator’s biography: learning contexts and learning outcomes • What are your previous experiences related to your role/position as a trainer in adult education? • Have you involved in adult education activities without having an explicit adult education job (consultant, counsellor, coordinator of study groups etc.)? If yes, please mention them. • What opportunities for initial and further professional development have you taken



25

as a trainer (courses, peer tutoring, mentoring, study visits, internshipsetc.) • What other contexts you find as offering relevant/possible opportunities for the achievement of competences defining the performance as a trainer? • Which concrete abilities/skills can you associate with a concrete learning context? • Can you name concrete activities where you learned/acquired competencies/things you now use in your adult education work? • What are the learning outcomes of the profession/home/leiure time/work etc.?

STEP II: External evaluation

STEP III: Consolidation

Main categories of analysis

On the base of our experience we conclude that in the context of Latvia the Validpack instrument demonstrated a real practical value because it is comprehensive and universal. Validpack will be important for assesssing competences in our country, since it gives an opportunity to fully assess psycho-pedagogical competence of staff involved in adult education activities. This instrument can be taken as a basis not only for assessment procedures but also for creating study programmes.

The emotional and relational dynamics (evaluator and staff)

Our opinion is compliant with EU quality statements: evaluation should minimize the relational/emotional interferences. Validpack instrument well realizes this goal.

Validpack tool case from UNIBA perspective The Latvian case The work of comparing practices has allowed the following considerations to be made.

It is very relevant how the Latvian case shows a well structured system of investigation, followed step by step in a coherent way by the means of a very specific and analytic methodology, in order to get an ambitious objective of defining framework and guidelines which are applicable to different contests and countries, potentially, although it is not so clear how the Validpack instrument has been formulated and what its contents are. As a practical group we can appreciate the importance given to the experience of all the tested employeers, to testify what the good practices are and to underline the knowledge about the real competences shared among the adult educators, in order to let them to be part directly of the evaluation process and to be aware of the evaluation aims. This kind of research strategy becomes applicable to different learning contexts (formal, non-formal, informal) and very useful to start a self-evaluation about own professional practices and improve employees' professional self knowledge (a deep reflection about: past experiences, current involvement degree, future opportunities and capabilities) and to recollect a documentation archive about their results. The evalauation outputs such as competences Portfolio and autobiographical interviews form a personal kind of restitution and rendition which could allow the evaluators belonging to other UE countries to make a wide-range network comparison and recognition.



26

“Methodologies and Instruments”

Very specific and analytic methodology.

Well structured system and method, step by step in a coherent way

Highly defined framework and guidelines which are applicable to different contests and countries, potentially, in order to make a comparison among them

Autobiographical Interviews (personal restitution and rendition)

Questionnaire

Observation

Competences’ Portfolio

Mind-map

Simultaneous introduction of Validpack in most UE countries, to be adapted to different contexts;

Self-assestment

Research results feedback;

Creation of a network among the countries included in the project.

“Evaluation Criteria”

To testify what the good practices are.

To set a minimum standard of trainers’ competences.

To point out the necessity, significance and potential of evaluation guidelines.

To know what real competences are shared among the adult educators.

Validpack adaptability to training institutions, vocational associations and employers;

Creation of training courses to be fit for Validpack implementation.

“Evaluation Representations”



27

Qualitative evaluation.

Facilitation.

Competence evaluation.

Testing results.

Transversal evaluation that is applicable to different learning contexts (formal, non-formal, informal).

To recollect a documentation archive.

Self-evaluation about own professional practices

Self knowledge (a deep reflection about: past experiences, current involvement degree, future opportunities and capabilities)

Bottom up feedback

External evaluation in order to consolidate the best practices

Creation of a guidelines handbook for evaluator

Creation and use of a single assestment process for many EU contries

“Emotional and relational dynamics”

For the interviewees: - emotional involvement

- effort to build up a mind map

- stressful recognition of their own faults and/or good practices

For the evaluator:

- semi-detached approach

- low involvement degree

- very analytic point of view



28

Validpack tool case: Practitioners discussion

The Latvian case The work of comparing practices has allowed the following considerations to be made. The procedures and methods used in the evaluation are clearly shown in the evaluation case, with a great deal of information on the instruments used and the guidelines followed. From reading the case, it is possible to extrapolate the Latvian evaluation model, which on an operative level proceeds following clearly defined passages, shown as work steps. The outline shown is very useful in identifying the operating paradigm. The Latvian approach to Adult Education evaluation is generalized, being proposed as a valid instrument for all types of professionals who work in the field of education. In the model presented, evaluation appears to have a very precise physiognomy and the representation of the evaluator that emerges is a professional who, through the proposed method, can become qualified for evaluation. The evaluator appears as an expert who examines and provides at cognitive level feedback on the competences measured, a trainer who through a series of pre-defined passages, leads the adult educator to evaluate his/her work according to the ValidPack model. The training instrument proposed is useful to learn how to self-evaluate one’s own competences, producing, through the comparison of the outcomes of the evaluation which the adult educator is required to do with the outcomes of the evaluation by the evaluator, increased knowledge and cognitive awareness, which results in the composition of a sort of map of competences. To obtain these results, the evaluation minimizes the emotive factors as these could influence the evaluator. The context in which the adult educator operates also seems to have, in the evaluation process proposed, a relatively low value compared to what it is to be measured, i.e. the competences of the individual educator. The approach used in drafting the case seems to be focused on a representation of learning with, at the centre of attention, the development of the learning process and personal training of the individual educator. Due to these specificities, the model proposed is useful for identifying/certifying the competences of an individual adult educator, which is the objective stated in the model. It can therefore be seen as a basic training device for the evaluation of the work of the adults, as it is based on the use of self-awareness techniques and instruments (observations and interviews) which have the aim of allowing the subjects to acquire some procedures which they can in turn use in the various contexts of work. You can see below the UNIBA practitioners' comments about the Latvian case. The Latvian case The work of comparing practices has allowed the following considerations to be made.



29

It is very relevant how the Latvian case shows a well structured system of investigation, followed step by step in a coherent way by the means of a very specific and analytic methodology, in order to get an ambitious objective of defining framework and guidelines which are applicable to different contests and countries, potentially, although it is not so clear how the Validpack instrument has been formulated and what its contents are. As a practical group we can appreciate the importance given to the experience of all the tested employeers, to testify what the good practices are and to underline the knowledge about the real competences shared among the adult educators, in order to let them to be part directly of the evaluation process and to be aware of the evaluation aims. This kind of research strategy becomes applicable to different learning contexts (formal, non-formal, informal) and very useful to start a self-evaluation about own professional practices and improve employees' professional self knowledge (a deep reflection about: past experiences, current involvement degree, future opportunities and capabilities) and to recollect a documentation archive about their results. The evalauation outputs such as competences Portfolio and autobiographical interviews form a personal kind of restitution and rendition which could allow the evaluators belonging to other UE countries to make a wide-range network comparison and recognition.



30

CASE STUDY P3 JKPU: "Social workers learn about cyberspace"

CASE STUDY P3 JKPU

Evaluation in social work project challenges (adult education)

Introduction

Janusz Korczak Pedagogical University in Warsaw is one of the largest providers of educational services in the field of social work in Poland. Except studies it developed wide range of adult education courses directed at social workers focusing on new social threats including cyberspace dangers, gambling disorders and addiction to overeating. Courses are provided in the professional context as one of the elements of social work supervision – educational and coaching component. Our case relates to the innovative educational programme for social workers which were conducted in the years 2011 – 2013.

The Context

Under the influence of the media and information technologies the traditional social threats which lead to different pathologies, dysfunctions and marginalization of the family are changing. At the same time there appear new threats that are closely related to the specificity of the virtual world. The problems concern both children and adults. The modern technologies bring many benefits to their users but they can also be tools of information and communication crimes as well as various traditional forms of violence and its new types connected with spending time in the virtual world. Janusz Korczak Pedagogical University together with Rezekne Augskola (Latvia) The aimed to implement an innovative development program preparing the employees of social service institutions to work with families and individuals who need social help in the area of risk generated by cyberspace. The creation of this product filled in the gap in the existing system of education of the social service staff with the problems connected with cyberspace. Implementing the innovative development program will provide the social service employees with new qualifications and tools. The project contributes to enhancing the efficiency of the presently realized social policy. In the project we conducted adult education training for 150 social workers from Mazovia Region in Poland which during 4-day training received information on most important topics connected with cyberspace dangers (physical dangers, psychological dangers, cyberbulling and technological threats). Education was provided in the 8-hour blocs daily and run by trainers (both theoreticians from HEI and practitioners). The training team consisted of 8 trainers (2 holders of habilitation, 2 PhDs, 1 PhD Candidate and 3 practitioners). During 4 day training group had



31

every day training with different trainer which caused a problems with coordination of the material as each trainer had its own background (theoretical or practical). Social workers came from the widely understood social services sector and represented public social service providers, NGOs and central government organization. They had various educational backgrounds and came from different position in the organizational ladder of social work (directors, administrative social workers, field social workers) and therefore their motivation and expectation from the course were different. As the idea of cyberspace dangers is new in Poland, we did not measure previous knowledge in the matter of the participants in the educational process. Evaluation Project sample of 150 participants were supposed to be the first group of trained social workers in the field of cyberspace in Poland. In order to measure and evaluate quality of the course the project has embedded three elements of educational evaluation process. In case of negative or partially negative feedback there should be done changes in the programme of training before it is implemented at the state level. In the embedded evaluation structure, project authors decided to use following forms of evaluation:

1. Competence tests;

2. Internal evaluation (surveys);

3. External evaluation (surveys, FGIs and IDIs);

Competence tests consisted of 20 open-ended questions concerning cyberspace dangers which measured knowledge of participants in the field. Evaluation was done in the form of ex-ante and ex-post in order to measure gain of knowledge of the participants after 4-day training. It was obligatory for all 150 participants to take part in the tests. Competence tests evaluation was done through staff of the project employed in the University. Internal evaluation (surveys) was done in the form of 30 multi-choice questionnaire which measured the quality of educational service provided to the social workers. In the process 120 social workers out of 150 was evaluated. In the questionnaire we measured: quality of teaching materials, particular teacher quality of content provision and administrative/technical issues of service (such as catering and lighting). Internal evaluation was done by Department of Projects of the University. None of the evaluators employed internally was a formal evaluator and all of them (3 persons) were non-formal evaluators with no educational background in evaluation. As a professional experience they from time to time evaluated internal projects of the university usually using method of questionnaires. They did not feel need of advancement of their theoretical knowledge in the field of evaluation as they understood that “practical knowledge is the one which matters the most” As for External evaluation (surveys) it was done in the multilayered structure consisting of:

a) Surveys (questionnaires) of 20 questions – external evaluator conducted 110 surveys out of 150 social workers in the project.



32

b) FGIs (Focus groups) of 12 randomly chosen social workers which were interviewed in 2-hour time by external evaluators in order to receive their feedback and measure satisfaction from educational process;

c) IDIs – Individual in-depth interviews were conducted on 20 social workers. Each interview lasted for 1-hour time and consisted of open-ended questions.

External evaluation was conducted by the company which won the public tender for provision of the external evaluation service. They employed 2 non-formal evaluators with sociology educational background who decided on tools and methodology used in the evaluation. Conclusions and outcome As a product of internal and external evaluation 2 reports were prepared. Each report consisted of around 100-pages and measured the quality of educational service provided to the adult learners (social workers). The outcomes of the evaluation were generally positive, however moderate discrepancies between reports occurred, especially in the context of evaluation of particular trainers who provided educational content. External report also was more informative and used more complicated methods of evaluation, introducing FGIs and IDIs which became the source of interesting feedback information for the project coordinator and provided space for improvement in the educational process. During the process Janusz Korczak Pedagogical University approached the issue of necessity of advancement in the methods and tools of internal evaluation methods. Necessity of evaluation trainings for non-formal evaluators is also important stake in the context of projects of European Social Fund Projects and new Horizon 2020/Erasmus 2020 projects. External evaluation might be more effective tool for adult education evaluation, however during the project we approached unwillingness of trainers to cope with external evaluation which some of them feared as “not embedded in organizational context” and “biased”. The case showed importance of tools mix in adult education context and balancing out between internal and external education. Also issue of absence of formal evaluators araised during the project.

Social workers learn about cyberspace case by UNIBA perspective “Methodologies and Instruments” Experimental adult education training, both theoretical and practical Apparently embedded structure/organization University cooperation network Innovation programme and technologies Training modules, schedules, alternation of trainers



33

Competence tests Surveys and multilayered structured questionnaires Focus group Individual in-depth interviews Final reports “Evaluation Criteria” Efficiency of the social policy Competence Knowledge of the matter Quality of the training course Quality of educational service provided to the adult learners Necessity of advancement “Evaluation Representations” Competence evaluation Internal evaluation External evaluation Ex-ante and ex-post evaluation For non-formal evaluator (who conducted internal evaluation): - something unnecessary and superfluous - something non really embedded - something could be biased For formal evaluators: - a professional evaluation



34

- a high stake evaluation For the employees: - something that they were forced to undo - bureaucracy duty and/or professional performance “Emotional and relational dynamics” A sort of confusion and disconcert among the employees, because there is not a coordination among the trainers Different level of motivation and expectation of the employees Unwillingness of trainers to cope with the external evaluation Fear, anxiety to perform, suspicious attitude, curiosity Social workers learn about cyberspace case by UNIMIB perspective 1. Evaluation methodologies and instruments ○ Competence tests to the social workers involved in the training course ○ Internal surveys: § Multichoice questionnaires to social workers about quality of the process they do (from contents to technical and administratives issues of services) ○ External surveys: § Questionnaires to social workers involved in the training course: § Focus Groups of 12 randomly chosen social workers (the goal was to receive feedback and measure about satisfaction from educational process § Individual in-depth interviews on 20 social workers § Reports ○ Triangulation of internal and external surveys ○ Triangulation between quantitative and qualitative evaluation's methods. 2. Evaluation criteria ○ Social worker's satisfaction ○ The teaching quality ○ The process quality 3. Evaluation representations a. By the non formal evaluators who were employed in the internal survey the evaluation doesn't need theorethical knowledge: it's first af all a practical knowledge b. By external evaluators: evaluation as a complex process that needs different instruments and methologies, in particular qualitative methodologies, and particular competences c. Evaluation as a process that needs both quantitatives and qualitatives methodologies d. Evalutation "embedded" in the course (in particular in order to competence tests) e. Evaluation as a practice that follows the course f. Evaluation as a process that allows more informations on the training context g. Evalutation as a process that could improve the teaching process and the course both in the educational and organizational aspects h. Evaluation as a source that allows to improve the process knowledge i. Evaluation as a research field (it's necessary to advance in the methods and tools)



35

5. The emotional and relational dynamics ○ The trainers' unwillingness towards external evaluation ○ The trainers' untrust and fear towards the external evaluation ○ Lack of cooperation between internal teachers and evaluators and external evaluators 6. Suggestions ○ Creating a bridge between internal and external evaluation ○ Working in a reflective way on the meanings and the consequences of evaluation, in order to make the internal evaluators aware of the importance of managing evaluation processes ○ Traning a pilot group of internal evaluators to manage methods and instruments evaluation both quantitave and qualitative.



36

CASE STUDY P4 TEI OF CRETE: "Training in the Crete immigration office"

CASE STUDY P4 TEI OF CRETE

Note: The names of the participants and the center have been altered to secure their anonymity. General Context: Location: We are located in Heraklion, Greece, in the Prefecture of Crete Immigration Office. The Center of Professional Training Hra (PTC Hra) is in charge of the employees’ training under a co-funded European project. The main objective of the adult education provider is: The cross-cultural training of 300 public servants all over Greece with different level of education and different employment status, including those working at the Prefecture of Crete Immigration Office, to enrich their multicultural knowledge and competences, as well as to improve their communication skills. The servants, aged 26-60 years old, have daily transactions with third-country migrants legally residing in Greece, in all seven (7) Prefectures of Greece. It is crucial for the employees to develop their skills in this training: interpersonal communication, their ability of active listening, consistency, team spirit, undertaking initiatives, being able to work with persons of different cultures, their negotiation abilities and the ability of handling conflicts, the capacity of accepting and also utilizing in a constructive manner the negative criticism e.t.c. Training: The training in the city of Heraklion lasted for 5 days, 8 hours/day (40 hours in total) and conducted by 3 adult educators (2 social workers and 1 sociologist). The main workshop topics were relevant to the respect in diversity, prejudices & stereotypes, the general concept of migration and the general prevalent context in Greece, the legal national framework and the changes occurred as well. The training team consisted of all 20 employees. Staff from the professional training centre (PTC) made the initial phone contact with the Director of the office to inform her about the upcoming workshop and to request the permission for the staff to attend. The adult educator after being informed about the upcoming workshop by the PTC, got in touch with the Immigration Office and informed them about the training venue, address etc. Then she organized thoroughly the workshop issues. The venue where the workshop took place was next to the main building of the Immigration Office, in a small room that didn’t allow the team to work in a circle. Critical Incident - Background The employees and the migrants visiting the Office for a number of important issues have for some time a number of quarrels on the service operation and the communication among them.



37

At first, immigrants were line up crowded, waiting to enter the office building, being exposed to the sun. Apart from this inconvenience, the fact that the documents were not translated to their native language and they were not informed about which are necessary documents worsened the situation. As a result, migrant-users were visiting the Office more than once; consequently they were facing difficulties with their jobs and were losing their daily wages. Social dynamics The adult educator observed that the Office employees had a different opinion regarding this matter. They were expressing anger for the work burden and the huge difficulties they had to confront in their contacts with the migrants when providing services. This kind of tension was not equally distributed between the employees (administrative staff, scientific staff, and compulsory education staff), with the latter being more tensed. The educator also observed a well-established hierarchy between the scientific and the administrative staff and a diversity of opinions between them and the critical role that their hierarchical positions had in their communication. Eva (the Director of the Office) was strongly demonstrating her preferrence in the higher educated staff. In addition, Eva was making some of the staff leave during the training to take care Office’s current issues. The director was trying to prove how hard she works even during the training hours and, that neither the Office nor the employees were left unsupervised. She was also accusing the migrants for not being able to understand and comply with the operating regulations of the Office. Under those circumstances the adult Educator decided to adapt the training based on the team’s needs. Apart from the theoretical training, there was a series of experiential exercises, to help the employees to develop the empathy needed to become aware of their colleagues and migrants situation. The alternation in the training had a dual aim: to affect positively the employees, in both lower & upper hierarchical positions, and the Office work in terms of the services provided to the migrant people. Critical Incident - Episode At the beginning of the workshop, the director of the Immigration office entered the room and introduced herself: “My name is Eva and I’m going to attend this training as the director of this service, under the encouragement of the adult educators’”. The Educator, a social worker, requested from the trainees to put their seats in a circle, or at least to try to, due to lack of space. The educator after a conversation with the trainees encouraged the team members to present themselves, and their professional roles and profiles, as well as to their motivations for attending the training. She also set the team rules: “We are going to dedicate our first meeting to a significant concept, respect. This exercise aims to put the necessary foundations for a respectful community. Those rules comprise the first step for the maintenance of an effective understanding regarding important and complex issues, such as racism etc. I’m going to ask you to split into small groups and to discuss with each other about respect, using examples, sentences and phrases, for 5 to 10 min. Each group should write down on a paper: “The way you are demonstrating respect” and “The way in which people show respect to you”.



38

The educator was waiting at the side of the room during the exercise and then she asked a member of the team to coordinate the discussion by saying: “I’d like to ask a member to take over the team coordination so as to gather all the ideas on the board. Meanwhile I will just watch from the back of the room”. After the hesitation and the surprise felt by the team, one of the members picked up a marker and took over discussion coordination. The coordinator was a high school graduate who works as an administrative employee in the Office. He also has a physical disability (dwarfism). While the papers and the ideas about respect were being collected, the educator standing at the end of the room was writing down the way in which team members were negotiating respect, focused on the way they were treating each other rather than what was written. The following were typical examples of the discussion: Director of the Office: “Respect is something which I demand from myself, my partner, from my work colleagues and the rest of people as well; Respect, honesty and conversation with each other”. Legal Advisor: “The feeling of Respect is universal. It is a feeling that incorporates boosting and liberating dynamics. We shall command and inspire respect to people around us, so as to promote it to a global principle. People show respect in the way they treat and speak to others, the way they listen and pay attention to what they’re hearing”. Administrative Employee: “In case I do not respect a person, I remorse and show him in practice my regret”. Administrative Employee: “We all deserve to be treated with respect, regardless our education level, we are all people with feelings and the need to hear a nice word”. These were few of the examples listed on the board, while the educator was keeping a list with the atmosphere prevailing in the team, which was coordinated by X. The team was a chaos. Context description The manager spoke up first, taking advantage of her power, saying that she demanded to speak. Just before finishing her statement, she sent one the administrative employees to work at the office. At the rest of the meeting she was talking on the phone and walking the room. In the second example, the Legal Advisor described in very beautiful words the concept “respect”, but she wasn’t paying attention to her colleagues when they were talking. On the contrary, she was chatting with the colleague next to her and in no way was she following what she has just stated about respect. In the third and fourth example, the administrative employees were like standing up for the first time, due to this exercise. The burden and pressure felt due to the hierarchical structure of the Office came up, so did their inability of speaking properly to gain their colleagues attention. X coordinated the team with sensitivity, without manipulating the ‘marker’s power’ nor changing or paraphrasing what was told.



39

Epilogue (Evaluation of the meeting (first part of the seminar) The educator started to speak only when the procedure was completed. At first, she thanked them for their participation and their collaboration. Then she asked them to assess the activity. The majority of the team told that it was a pleasant and easy procedure. In contrast with X who admitted facing plenty of difficulties; that he had to speak loud in order to be heard which he didn’t liked. He also said that some members were off the team. Then the educator, based on the notes she has kept and the observations she has made, she tried to help the team understand that respect is a difficult concept, which may seem easy to use but in reality it is too hard to be achieved and to behave accordingly. The educator was talking smoothly without having any kind of attitude neither being judgmental. She noted that it was very easy for her to distinguish the clustering of the employees according to their educational level and their working position, using some examples of the behaviours that were demonstrated by the manager and the legal advisor. She also tried to remind the team that we all have rights and obligations but, above all, we have to rely on the same equal principles of respect and mutual understanding. She argued that we all need to follow rules. The educator also reminded the team that the fact that people make mistakes does not make them bad persons but individuals with vulnerabilities and qualities, encouraging them to think about the relationships between them and how they can be readjusted. The manager of the Office expressed their surprise and seemed to be puzzled while the educator was talking; she didn’t expect that this kind of exercise would have such an effect. She also commended that the evaluation of the procedure was helpful and supportive, since she didn’t feel like being judged. On the contrary, she felt responsible and she also came across with feelings and facts that she hadn’t realised. Similarly, the Legal Advisor stated: “I realized that I’m talking about respect easily, but I have a long way to go”. After all the team members had made their own statement, the Educator asked them to talk about the difficulties they confront with the immigrants during transactions. They have been asked to put themselves in the immigrants’ shoes and to think how they really feel when they try to talk to a superior and were asked to think how carefully they listen to their superiors. The main question of the succeeding discussion was: Since the employees are not showing respect to each other how is it possible to respect the needs of the migrants they try to serve? Migrants are individuals whose language, culture, religion, customs and tradition are completely unknown to them; People who are not aware of the Greek language and consequently not being able to read the documents handed to them; and individuals being yelled to just because they cannot understand what they have been told. The educator completed the full training having elaborated with the team members topics relevant to awareness rising, active listening and the development of communication and empathy skills. Some of the issues elaborated involved stereotypes, prejudices and the intercultural competence of services e.t.c. The training ended leaving a fruitful experience and positive comments from the trainees. Team members were asked to assess the training program and the Adult Educator through a structured questionnaire. The educator, also, assessed the



40

training procedure in the same way. The questionnaire was constructed by the Professional Training Center. The results afterwards. Evaluation of Undocumented Facts. The evaluator has attended the seminar and he collaborated with the organization in a professional way (level). The personnel collaborated with the evaluator due to his experience on issues related to local community, and to immigration. Up to now they have increased their collaboration on issues related to migrant’s services. Initially the evaluator disseminated the material of the seminar to the participants. In that way, he affected positively his communication with the staff and the director. Afterwards, the evaluator informed the organization about the migrant groups in the local society, thus raised an immediate connection and communication among the staff of the organization and the migrants. The evaluator intermediated for their first meeting and contact. The group of Syrian migrants responded immediately to that call. At their first meeting, they discussed about the difficulties in accessing the organization and the employees. The Director of the organization decided to demonstrate more attention to those issues and for that reason she developed a more frequent and regular communication with the migrants. She facilitated the staff to be more available to the clients, to have more time for face-to-face discussions about their requests. Afterwards, she invited the representatives of the migrant groups to talk with her about their problems. In addition, as the staff started to have more time with the director, their collaboration became more regular. Τhe results Director: - Participated as a volunteer to the Municipal Council of Migrants’ Integration. - Established a humanistic approach for the users (e.g., “open door” time for conversation, availability for the users in order to solve their difficulties). - She suggested helping the organization of a multicultural event on the customs and culture of migrants using a part of the organization’s budget. -She created a professional relationship with the instructors and other educational institutions and indicated her interest to participate in European funded programmes related to immigrant’s issues. -She and the personnel of the office participate in workshops for the training of the immigrants on the legislative procedures. The workshops are still carried out at the neighborhoods of the city, where they meet the immigrants, where they can all interact for their better information and contact. Staff: - Participate in workshops for the training of the immigrants on the legislative procedures. - Participate as a volunteer to the Municipal Council of Migrants’ Integration. - Frequent and regular communication with the director of the office. Discussion about the difficulties they face. - Cooperation with the representatives of Migrant groups and with the C.M.I. for the document interpretation.



41

Users of Service/ Migrants: - They are being served in less time since they are informed, beforehand, about the

documents needed which are now posted on the announcement board. Documents are translated in different languages; consequently migrants are handing over the correct documents.

- Migrants and their representatives have the opportunity to hang on the Office’s announcement board useful information about their community.

- Migrants sent a greeting card to the Director of the Office, a decision taken during a session of the Council for The Integration of migrants, where migrants, especially the Syrian community, expressed satisfaction for their improved performance. A representative of the Syrian Community delivered a greeting card to the Director of the Office during a session of the Council for the Immigration. The Syrian Team also gave a Syrian historical book to the Director as a gift. The book was put at the main hall of the Office.

Understanding the Training in the Crete immigration office

1. The evaluation methodologies and instruments from the evaluator

A. Evaluation Criteria

- self-evaluation of the educator

- the preparation of the seminar (the organization of the tasks)

- the correspondence of the educator tasks in theharacteristics of employees /educated

- the distribution and the management of the educator’s time according to the program

- his contact/visit at the educated service before the seminar beginning

- the contact with the service during the seminar

- the contact and the information from the training provider for the aims of the seminar

- his/her relationship with learners (emotional involvement or not)

- the interactions during the seminar

- his/her collaboration with learners

- the difficulties encountered

- how he/her managed the difficulties

- infrastructure used, teaching material etc.

- the place and location the seminar held



42

- ways of improving education/suggestions

B. Methodologies and instruments

- observation

- discussion,

- feelings expression

- formal evaluation in printed form/structured

- active participation of trainees for undertaking the task (humanistic approach)

- recording, notes

- trainer/educator skills: authenticity, immediacy, objectivity

- adaptability to circumstances and difficulties encountered

- role playing

- discussion with personal experiences of learners

- encouragement, support, feedback during the seminar

- impact of tasks in real working conditions ("put yourself in the position of the immigrant")

- nonverbal attitude

- nonverbal messages,

- eye contact,

- face to face contact

- formulation of the place and putting learners in a circle

- summary

- critical perception of different position/ views of the learners and the factors affecting them.

C. The emotional and relational dynamics (evaluator and staff)

- respect

- feedback

- critical understanding



43

- encouragement

- support

- effort

- disagreement between learners / dissatisfaction

- competitive relations between learners

- light-hearted mood

- pleasant mood

- learners surprise

- hierarchy relationships/relationships of power between users

- critical attitude among learners

- soft tone and style of the educator

- learners responsibility

- awareness

- deal with emotions in real facts

- reflections

- thoughts

2. The evaluation methodologies and instruments from the participants


- participation motives

- responding to the participants educational level

- meet their objectives for knowledge acquisition and application of skills at working level

- meet their expectations for the professional development and advancement

- learner informing for the seminar by their working institution

- place of contacting the seminar



44

- duration of the seminar

- time for break

- content of teaching tasks

- collaboration with the instructor/educator

- interactions with the instructor during the seminar

- interactions with learners

- ability to active learner participation to the seminar

- ability of submitting proposals for the improvement of the seminar


- learner registration to the seminar

- active participation

- discussion

- disagreement

- working in groups

- working in cycle

- expressing feelings

- role playing/

- scenario

- working experiences (scenario)

- formal evaluation (structural questionnaire)

3. The evaluation methodologies and instruments from the provider


- readiness of theinstitution for the concretisation of the seminar (evident objectives, frame of realisation,eligibility of services)



45

- organisation of institution for the seminar(preparation of instructors and participating,choice of instructors).

- contact and collaboration of institution withparticipating learners

- contact and collaboration of institution with the educator/instructor

- contact and collaboration of institution with the service/organisation

- publicity

- ability of material creation (products)

- correspondence in the quantitatively objectives (number of learners)

- correspondence in the qualitative objectives (learning skills, acquisition of knowledge)

- satisfaction of learners

- satisfaction of educator

- viability

- innovation


- application form and registration of learners

- communication and collaboration with the service

- communication and collaboration with the educator/instructor

- learners Certificate of participation for

- educator/ instructor certificate of participation

- evaluation Questionnaire for the educator/ instructor evaluation



46

Questionnaire for the educator/ instructor Evaluation afterwards/Results-Cons

http://wiki.edueval.eu/index.php/File:1.png



47




48




49

Training in the Crete immigration office by UNIBA perspective “Methodologies and Instruments” Context analysis Service user needs’ analysis Very formal and professional organization Building up a network and a work team (phone contacts, request of permission, etc.) Very structured workshop Hierarchical communication Circular communication Adaptability of the contents to the needs of the team members Balance between theoretical and experienced activities Peer coordination (a member takes over the team coordination)




50

Discussion in small groups Brainstorming (put the ideas on the boards) Evaluation notes Feedback Paper reports “Evaluation Criteria” Communication skills improvement of the employees Multicultural knowledge and competences enrichment Employees’ satisfaction Customers’ satisfaction Service efficiency “Evaluation Representations” For the adult educator/evaluator: - activity assessment - service quality assessment - training program assessment For the participants/employees: - self-evaluation - self-consciousness - training program assessment “Emotional and relational dynamics” Generally:



51

- tension due to communication difficulties within the Service and quarrels among the employees and the migrants/users - general uncomfortableness due to the location and chaotic atmosphere (it not allow the team to work in a circle) For the employees: - frustration - anger (to face huge difficulties and work burden) - increasing tension level - underestimation - jealousy (because of the preference demonstrated by the Office Director) - hesitation - surprise - compliance to the aim - under pressure - sensitivity - no empathy for of their colleagues’ situation - no awareness of the migrants’ needs For the Director: - powerfulness - strength - disrespectful attitude - inattentive and careless behavior - egocentrism - self-efficacy centered



52

For the migrants/users: - upset, anger - frustration (to be exposed to the sun) - misunderstood (because of language knowledge and difficulties) - tired (of visiting the Office more than once) - afraid of losing their daily wages - no ability to manage copying strategies



53

CASE STUDY P5 UNIVERITAT JAUME- I: "ISO 9001 in educational centre"

CASE STUDY P5 UNIVERITAT JAUME- I

Example 1: Implementing the EFQM quality procedure (European Foundation Quality Management) in an industrial company with the main purpose of offering a product and servicewhich are more closely aligned with the needs and desires of the twenty-first century citizen. To recover the experience and emotions of the immediate enviroment: a home for the 21st century. Note: extrapolate results to an educatinal process: Feeling sensations that self-motivate the training that improves the senior citizens’ quality of life. Synopsis: Industrial private company that sells to traditional businesses (family-run neighbourhood shops) and large retail shops, in the sector of avant-garde furniture. History of the situation: To recover the national market (Spain) and have a higher international business performance. As this is a very mature sector, customers’ level of demand is very high. With the European standard, it was intended to reduce the level of rejections, inefficiencies and delivery dates and increase final customer’s experience of excellence (user). Action Plan: The intervention was based on two lines of action: 1. Implementation and development of a standardized quality process that enables an ongoing

improvement of the business process, with special attention to customer satisfaction (results):



54

Definition of the expected results with special emphasis on the choice of the strategy (processes, key partners and resources):

o Results for customers. o Results for shareholders, employees and key individuals. o Results for the immediate social fabric.

2. Implementation of the industrial process monitoring.

Abstract

In this case study, the process of supply of raw materials and auxiliaries and manufacturing processes (internal and outsourcing) were very important.

How can we extrapolate an education system?

Examples: 1. ICT service providers. Analysis of the satisfaction with the received service. 2. Developers of software applications for senior students. Collecting specific

requirements for this group. 3. Adjustment of the teaching modules and methodologies based on the students’

requirements (customer). 4. Drafting a training road map with defined process of effectiveness and efficiency

according to the established objectives (SMART). 5. Analysis and monitoring of the education process and feedback of the learner

satisfaction (customers) for continued improvements. 3. Critical factors for a successful project:

Some Knowledge, market and market definition factors to mesure.



55

Understanding the ISO 9001 in educational centre case

1. Evaluation methodologies & instruments -Creation of a strategic plan -Quality management -Formal model based on ISO procedures (indicators) -Hermeneutical process: the whole process works as an instrument in a “holistic way”. -The commitment tasks could be considered as an evaluation methodology or instrument -Customers’ satisfaction -Monitoring -Analysis of data -Control of the product 2. Evaluation criteria -Leadership -Performance indicators -Stakeholder perception -Effectiveness of the system -Efficiency -Integrity -Innovation -The excellence of the results -Emphasis on the actions and procedures in general -Student’s requirements as customers -Partnerships resources 3. Evaluation representations / main ideas / main aims - Represent a recurrent evaluation -To control -To reach higher standards -For continuous improvement -To learn by practice -Exchange of experience -To secure the quality improvement -To develop the competences 4. Emotional and relational dynamics (evaluator and staff) and suggestions for the evaluation methodologies, if it is clear for your own national case study -Dynamics: the passion of the teacher -Dynamics: Creativity of the teacher -Dynamics: Much effort (teacher) -Dynamics/relation: No much info about horizontal and vertical relationships



56

-Suggestion: Focus on psychological dynamics (vertical and horizontal) -Suggestion: Focus on the planning objectives -Suggestion: To evaluate the concrete effects to the students and teachers

ISO 9001 in educational centre by UNIBA perspective

“Methodologies and Instruments” Innovative decentralized management methodology Building up a strategic plan, step by step (Plan, Do, Chech, Act) Data analysis Qualitative measurement of the formative offer Quantitative measurement of the school enrolments “Evaluation Criteria” Effectiveness and efficiency of the Institution Vision (for the future) Task (for the existence) Values (to support) Objectives (to achieve) Strategies Tactics Customer/student satisfaction Assurance to conformity to musical student requirements Assurance to conformity to statutory and regulatory requirementes “Evaluation Representations” Self-centered and self-referential evaluation (in order to achieve excellence and improve the Institution’s prestige) Company evaluation



57

Self-analysis of data Self-evaluation A kind of empowerment and enhancing system A kind of a required control (monitoring and measuring equipment) Both qualitative and quantitative evaluation (in conformity to the requirements) “Emotional and relational dynamics” The state of necessity and/or emergency to change the business course Anxiety to avoid failure Manager’s responsibility Manager’s authority Communication quality between management staff and personnel staff Personnel’s uncertainty for the future Sense of precariousness Hope for a better future Great expectations Satisfaction degree The importance given to such internal values as commitment, respect, integrity, effort.

ISO 9001 in educational centre by UNIMIB perspective

1. Evaluation methodologies and instruments a. Applying quality procedures b. Creating a strategic plan c. Definition of results on a different levels d. Monitoring the process e. Analysis of the process

2. Evaluation criteria a. Effectiveness and efficiency of the process b. Reaching of the previous results c. Performance indicators d. Students' requirements as costumers



58

3. Evaluation representations a. The evaluation is a formal process of control of all the actions, the procedures, the skills, the performance in order to reach the stated results

The evaluation is a continuous process of monitoring the experience in order to improving the system



59

CASE STUDY P6 UNIBA: "Education to inmates" (UNIBA)

CASE STUDY P6 UNIBA

Context: We are in Bari, in a Juvenile Prison training room. An Adult Education Evaluation Staff meeting is set there. It is a training meeting, organized by a Cooperative Society (or Private Social Institution), addressed to the operators involved in a vocational training laboratory (an almost 6 month training course for “ceramics maker”, addressed to a group of young prisoners). The participants in this training meeting are both the Cooperative Society operators and the Juvenile Prison ones, who belong to the Educational Area and the Security Area of the Prison itself. The aim is to build up an open group, putting together external and internal operators who all work at the training course inside the Prison. This training course was mainly addressed to elder prisoners, aged 18-21 (that was the maximum age allowed in a Juvenile Prison), because they already exceeded the age to perform the Italian compulsory education and so they got out of the traditional, education and/or training courses. My name is Marco and I am going to attend the training meeting as a Juvenile Prison educator, encouraged by the coordinator of the Educational Area which I belong to, and by the Prison Director who has backed this initiative. Critical incident The external and internal operators of the Prison have been handling some difficulties for a very long time. They were in need to: work on their interpersonal and communication dynamics, make clear their own professional roles and boundaries with each other, think on their responsibilities and competences. Some critical facts had already occurred, actually. The Prison educators found that some positions were dissimilar to the intramural rules about the training activity organization. For example: some young prisoners used to move from a laboratory classroom to another, even without permission, thus showing little interest in the proposed activities or bothering the teachers during the lessons. As a consequence, all these positions called forth numerous interventions by Prison educators, in order to blame the prisoners and to manage or prevent any other potential, critical situation. On the other hand, the external operators (the course trainers) complained about all the times they were interrupted, because of both intramural organization needs and educational-correctional interventions. The course trainers were worried also by the break in continuity of the activities and its bad influence on the elder prisoners, who were supposed to be evaluated before long, for the admission to the final examination to get the professional qualifications. So, the first Adult Education Evaluation Staff meeting, which included all external and internal Prison operators, becomes a useful occasion to start comparison, discussion and explanation. Episode At the meeting starting point, we all seat in a circle. Doctor X, a psychologist invited by the Cooperative Society as supervisor and moderator of the open group, starts the meeting: he



60

allows everybody to talk and invites the participants to introduce themselves, specifying their professional roles. The teachers of theoretical subjects at the vocational training course (e.g. mathematics, Italian, etc.) are the only “new entries” in the open group: this is their first task in a Juvenile Prison, so they seem to be worried pointing out how difficult is trying to keep the interest and the attention of the prisoners high. The Prison educators, along with classroom Tutors, according their professional experience, reassure the teachers saying that young prisoners are more likely to be encouraged in following practical activities more easily than theoretic ones, because of their few gratifying former school experience: “so many guys don’t love School and they’ve never been loved by school teachers”. The psychologist encourages the theoretical subjects teachers to apply customised and interactive teaching methodologies, to talk more with the guys and to share their opinions about them with the Educators, just like tutors do. The classroom tutors intervene, proudly, to point out that they feel more comfortable having conversations with the prisoners, whom personalities are well known, day by day, working closer to the Prison educators and the Security Police Agents. The classroom tutors, in their working years in the Prison, believe to have developed great abilities dealing with relations, listening, empathy and adaptability: “If you know how to deal with these guys, they will tell you everything and will open their heart, so you can suggest them more and more demanding activities”. The tutors, in addition, draw the attention to a particular circumstance: “As the family visit day comes, some guys are less collaborative and won’t get dirty with glue and painting”. The psychologist suggests to entrust the classroom tutors with the task to facilitate the collaboration between external and internal operators, just because the tutors hold a more informal role and have more versatile duties. The educators are sceptic because they are used to be demanded to evaluate what could fit and work better inside the Prison. They don’t need to take a lesson by any external operator, so they feel almost compelled to consider what the others have to say. On the other hand, the practical-professional subjects teachers emphasize, tactfully, that they have developed a very positive experience working aside the prisoners, being patient enough to catch the interest of these guys. “They like to be involved, even if they want everything immediately, and they are eager to create handmade ceramics as gifts for their families”. “Sometimes they move from a classroom to another one without permission, but there’s no matter about it until they do not bother our activities”. They complain, indeed, about the limits that occur by force in their daily work, for example in accordance with the necessary time to let the Police Agents to complete all the control and security procedures. They seem to be frustrated: “We can’t finish what we have to do in the classroom, sometimes some lessons do not take place without notice and we should answer the Cooperative Society project planner for this”. The educators interfere powerfully and remember that is necessary to respect the organization inside the Prison. They point out that Adult Education Evaluation Staff could work better, except for the communication and information sharing problems among the staff members themselves: “This is a Prison! If all of us don’t understand the competences and responsibilities reciprocally, we’ll take the risk of making only confusion!”. The Security Area Commanding Officer, the Police Agents spokesman, remarks conceitedly and almost sarcastically that: “Order and security come before all the rest in a Prison”. He explains everybody how difficult is to manage and grant security during the course lessons. “We can count on less Police Agents than we need in the staff, so how do you think we can start any activity, before granting protection to the operators first?”; “We have to pay attention to



61

the right use of all necessary and authorized tools for the practical-professional activities”; “There are too many guys attending the laboratory, some of them look like trouble makers or poorly interested in the activities”. The project planner seems to be disappointed and clarifies that is relevant to safeguard the minimum number of participants required by the project financer Institution: “Be aware of the parameters and resource limits fixed by the financing Institution”. The project planner begs the Educators in particular: “You might always remind the prisoners to make their best during laboratory classes, and remind them that a final examination is to be done”; “All the guys know that the educators send reports to the Juvenile Court”. The educators feel provoked and annoyed so reply. “We can try to reduce the number of the course participants, applying a deeper selection”; “The prisoners do not like so many hours of theoretical classes, which could be almost concentrated in some “untroubled” days, to avoid the recurrent absent-mindedness of the guys”; “Most of the guys want to learn new things, otherwise they get bored”; “The laboratory desks are not enough for all the participants, therefore they cannot be followed simultaneously by the teachers”. “The prisoners always look for teachers to receive some attention, because they feel disregarded enough just when out of here”. The cultural mediator reminds also the needs of foreign prisoners: “Most of the foreign guys hadn’t the opportunity to study in their countries, so they have difficulty in attending theoretical lessons, or they desire to learn a job”. The course coordinator and the project planner confirm there is no possibility to reduce the amount of theoretical teaching hours (compulsory hours, scheduled by the project financer Institution); anyway, they approve the educators’ proposal to concentrate all the theoretical lessons in some “strategic days” of the weekly calendar. The Educational Area coordinator confirms that the supervising and control interventions made by the educators are necessary, but argues also that they are not enough: “at least one educator should be present, in rotation, for the entire duration of each laboratory lesson, as we always did until few years ago”. The educators know better that the external operators dislike to be interrupted and “controlled” during the lessons, so they clarifies that everybody is expected to cooperate for educational and disciplinary interventions: “It’s unfair that always the same operators play the “bad” part in the eyes of the blamed prisoners”; “You know, if I interfere as educator in the classroom, I’m doing a professional duty, not making a subjective choice. This Prison is not mine at all”. The psychologist and the course coordinator take note of both the complex situation and the different positions and shared evaluations of the members’ staff; then, they suggest teachers a proposal: “Make some activities that can be obtained in different working stages, according to available time”; “The guys have an unpredictable attention threshold in accordance with the aroused interest”. So, the psychologist and the course coordinator suggest a group meeting in the following months, to check out if anything happens or changes in the meantime, after all said and done”. Epilogue At the end of the meeting, a decision is taken. Each Area responsible, represented in the Adult Education Evaluation Staff (the Educational Area Coordinator , the Security Area Commanding Officer, the vocational training Psychologist and Coordinator), should write down all observations made in plenary, in order to inform the Prison Director, whom authorization is



62

necessary to accomplish the staff proposals. Some months later, the training course comes at the end, so the operators meet together for an informal discussion about the results. According to their general impression, some things have improved, in part: - The external and internal group communication: “Do you remember when we were on so

distant positions, as we were rivals?”; - the mutual acquaintance degree among the staff members: “It has been a pleasure to know

you better; we share the same rigour and the will to improve everything that goes wrong”; - the integration degree inside the staff and institutional dynamics, especially of less

experienced operators: “I was so afraid of the prisoners at the beginning, and I didn’t want to fail because I didn’t know how to act”;

- the operators awareness level about their limits and inner resources: “Formerly, I didn’t realize how much I was limited by my discretion, which prevented me from being appreciated by the Prison operators and the course participants”; “I didn’t believe that the training sessions could have been so really helpful, I’ve put aside my first scepticism”;

- The motivation degree showed by a great number of the young prisoners: “some of them are eager to take the final examination, in order to get both the professional qualifications and the scholarship” ; “I never thought that such “troublemakers” could be so responsible and able to focus their attention on the attained results”.

At the end of the course, the staff members have been more acquainted with the different representations and expectations about evaluation inside the working group: the formative evaluation for the Prison operators and, on the other hand, the accounting one for the external operators who belong to the Cooperative Society.

Understanding the Education to inmates

1. The evaluation methodologies and instruments

1.1 The evaluation methodologies and instruments (for the internal operators)

- conversations between users and operators - conversations among the operators - welfare officers feedback - staff meetings - didactic commission meetings - activity register and calendar - participating observation

1.2 The evaluation methodologies and instruments (for the external operators)

- users competence certificates - users professional qualifications certificate - users register - handmade ceramics pictures

2. The evaluation criteria

2.1 The evaluation criteria (for the internal operators)

- users satisfaction - spendable competences and course usefulness - internal operators satisfaction degree - communication among the operators - activity continuity - activity control

2.2 The evaluation criteria (for the external operators)



63

- activities contents and duration (number of teaching hours, lessons calendar) - handmade ceramics production - institution publicity thanks to activities promotion

3. the evaluation representations

- revelation of evaluation representations inside the staff: • “formative evaluation” for the internal operators • “accounting evaluation” for the external operators - mutual observation and supervising - dynamics management inside the staff

Education to inmates case by UNIBA perspective

Methodologies and instruments

The evaluation methodologies and instruments (for the internal operators)

- conversations between users and operators

- conversations among the operators

- welfare officers feedback

- staff meetings

- didactic commission meetings

- activity register and calendar

- participating observation

The evaluation methodologies and instruments (for the external operators)

- users competence certificates

- users professional qualifications certificates

- users register

- handmade ceramics pictures

"Evaluation criteria"

The evaluation criteria (for the internal operators)

- users satisfaction



64

- spendable competences and course usefulness

- internal operators satisfaction degree

- communication among the operators

- activity continuity

- activity control

The evaluation criteria (for the external operators)

- activities contents and duration (number of teaching hours, lessons calendar, etc.)

- handmade ceramics production

- Institution publicity thanks to activities promotion

Evaluation representations

- Revelation of evaluation representations inside the staff:

- “formative evaluation” for the internal operators

- “accounting evaluation” for the external operators

- mutual observation and supervising

- dynamics management inside the staff

Emotional and relational dynamics (evaluator and staff)

- defensive position

- effort

- frustration

- suspicion

- scepticism

- out of duty

- offence



65

- envy

- jealousy of someone else’s privileged relationship with the users

- sense of precariousness

- lack of motivation

- control sensation

- antagonism

- provocation

- realism

- sense of inadequacy

Education to inmates case by UNIMIB perspective 1. Evaluation methodologies and instruments a. Creation of a context of dialogue between professionals with different roles, facing of the conflicts b. Participating observation c. Mediation 2. Evaluations criteria a. The effects of the different professionals' actions in the young prisoners in order to improve their learning b. It emerges different criteria applyed by different professionals: i. The teachers and prisoners' security ii. The prisoners' behavior, interest or needs iii. The istitutional's rules respect iv. The prisoners' learning v. The prisoners' performance and their preparation for the final examination vi. The prisoners' satifaction vii. The communication between the different professionals involved in the course 3. Evaluations representations: different and contradictory by the different roles and positions of the professionals involved: a. The evaluation is a practice that could state what is right or wrong to do by all the operators b. The evaluations is a process that could make the professionals mor awake of their training practices in this Institution c. The evaluation works when stated the process results and their conformity of a previous project d. The evaluation allows an observation and reflection on the dynamics between the staff members e. The evaluation could enrich the single operator and improve the staff f. The evaluation could help to produce more knowledge about the training competences, actions, and representations of the different operators; to facilitate the communication among each others; to produce awareness about traning in a Juvenile Prison (in particular about implicit rules); to make a realistic awareness of the professionals' limits and resources. 4. The emotional and relational dynamics a. Suspicion b. Conflictual relations between operators with differents tasks, backgrounds and role in the Prison c. Sense of inadequacy towards the Insitution requests and the needs of the young prisoners d. Athmosphere of control e. Interested and confortable approach of the psycologist and the coordinator f. We can see a passage from a general lack of motivation to more motivation by the different professionals Solitude and antagonism that slowly changed by more mutual acquaitenance



66

“Feedback form for Guidelines and Handbook“ Answears (1) Educational degree If you had to train a person that wants to become an evaluator, what would you recommend? RESEARCHER: A) • learn to take up a critical position vis-à-vis the evaluation and the reasons why the evaluation is requested • learn to recognize explicit and implicit models which live in the practices of evaluation and in the most popular methods of evaluation • observe and understand the situations of the adult educators to understand which typology, methodology and instruments of evaluation to use • learn to build the evaluation indicators having theoretical elements of reference interact with the specific situations of educational work • learn interview techniques • learn group leadership techniques • get to know different models of evaluation • get to know different methodologies of evaluation • get to know different instruments of evaluation • be able to use (so do a sort of apprenticeship) models, methodologies and instruments • learn to use appropriate mediations so that adult educators can express their capacities and skills, their work, their criticalities and their strengths, • learn to evaluate themselves. B) “The rigor is not the exactness"; would recommend this to every evaluator. The basis of the search for the accuracy, objectivity ... is a position on the world. As well as any interpretation is already an evaluation. When we work as educators of adult groups we are always already evaluated and evaluators; we officially made evaluators or not. The gesture educational not exist without mutual evaluation between educating and educator, facilitator and group ...; is not removable from education evaluation but that does not mean estimate corresponds only to measure. C) I would recommend a becoming evaluator to pay attention carefully to all the internal dynamics among the staff members and also between the Management and the staff. A lot of hidden details and unsaid difficulties should be discovered keeping in touch with everyone inside the staff. Empathy should be one of the main tool for the evaluator, without forgetting the importance to keep himself/herself basically distant from the staff in the same time, in order to observe all without prejudices and/or sympathy. I would recommend also to pay attention deeply to the needs, expectations and difficulties reported by the Service users/customers. D) Doing a motivational assessment of the candidate Doing a competencies analysis about the actual preparation Giving him some different criteria set and discuss with him about the different theoretical models behind them making a role-playing regarding the evaluation process



67

at the end of the role-playing discussing together what were the thoughts and the emotions felt discussing the results of the evaluation play proposing to the in-training evaluator to make a role-playing acting as evaluator at the end discussing with him also this new experience matching the two of them and trying to observe the differencies discussing about the relations between specific objestc and the better methods for it analyzing if you can distinguish the trasversal and recurrent criteria and action in the evaluation process and the endemic ones. saying to him: good luck! E) I think that a future evaluator must be trained first of all in clearly recognizing what aspects have to be evaluated. Secondly, an evaluator must know that there is not only one method or one process to evaluate everything. He must be trained in understanding the revelance of different processes and instruments to evaluate different aspects.He/she also have to consider the impact of emotions, human relationships during a process of evaluation and how to cope with these dimensions. TEACHER, TRAINER, TUTOR: A) - Develop critical capacity with respect to the evaluation and the reasons for which the assessment is required - Learn to observe the contexts to seize the aspects implicit and explicit in the practices of evaluation and in evaluation mode more widespread - Observe and understand the context to define which type and methodology, and assessment tools use - To follow the construction of evaluation indicators in comparison with the operating field - Learn and know the techniques of the interview - Learn and know the techniques of conducting group - Develop capacity for self-assessment. B) I'd recommend to be a researcher, to be an observer of details, to be a reader of the impact of phenomena. C) Strategy: starting form practice, real and professional experiences, active training, (problem solving, self-case studies, teamwork) Methodology: active and focused in the valorisation of professional experiences of participant. A spiral structure, which enhances the relationship between experience, reflection and theory Steps: Enhance the experience of students and the development of a critical sense and their decision-making autonomy and the ability to take a difficult angle.



68

D) Psycho-pedagogical studies; experience and/or knowledge of research – action; practice in the field in different contexts; different methodological approaches; patience; empathy; problem-solving skills. E) The improvement relationships and comunication skills and the ability to read situation and different contexts without denying their specificity DECISION MAKER / POLITICIAN A) It is important, for an evaluator, to be trained in facing group dynamics which are always involved in the evaluation process.In this sense, active methodologies and working groups are fundamental in this kind of training. OFFICIAL EVALUATOR A) Formative training based on the observation of the dynamics of work and on the reflection, with supervisors to whom to present the conclusions to receive feedback, could be useful How do you think the evaluation process should be applied? RESEARCHER A) In both the case of formal and informal procedures, the evaluation process requires: • explanation and clarification of the criteria of evaluation: with respect to which models, objectives or standards the adult educators are evaluated (i.e. depending on the models of comprehension of the evaluation that the evaluator bears in mind or according to which he acts, including reluctantly, the criteria can include, even in a mixed form: conformity with a standard, number of objectives reached, consequences of actions undertaken by the staff, unexpected effects of the educational actions)? It is important to make clear the choice and the motivation of each criterion adopted. • explanation and clarification of the objects of evaluation: with respect to what are the adult educators evaluated (performance, skills, competences, behaviour, attended deliverables, etc.)? • explanation and clarification on the model of evaluation applied, on its possibilities and limits: i.e. measurement or knowledge/recognition in view of an improvement of the objects evaluated? • explanation and clarification of the motivations making the evaluation necessary: for reasons of accountability to external entities, for certification of



69

the quality of the educators’ work, to take stock of the work of the staff of adult educators and to replan, enrich, increase performance, skills and knowledge of the staff? • clarification of the methodologies that will be followed in the process (of a performance or thoughtful type, or also mixed, for example) and of the instruments that will be used (observations, interviews, tests, including mixed, etc.) Informal evaluation As well as considering the above points, when evaluation is carried out by a director or coordinator of services, by a consultant or a supervisor, it is one of the functions that they perform and it requires particular attention for it to be exercised with awareness by the evaluator. It is a question of: • being aware of the role of the evaluator and of the dimension of “power” it implies • having available various instruments of reporting, understanding and evaluating the situation, in order to be able to use different ones depending on the phases of the evaluation process. The choice of instruments depends on the awareness of the scientific paradigm in which they originate and the effects they may induce; • using the evaluation to promote innovation in the staff or service or, at any rate, in the organization (the evaluation has pragmatic, not only investigative effects) • being able to communicate the outcomes of the evaluation process depending on the effects that can be obtained through it • being able to give adequate space and time to adequately carry out the evaluation process Formal evaluation The role of the evaluator with regard to the staff is defined a priori and usually identified with carrying out standardized evaluation procedures. It is therefore important that the evaluator: • is aware of his role and knows how to play it in the specific situation of a staff of adult educators • is aware of the logics and of the effects that the procedures which he has to implement imply, in order to be able to adapt them to the group of educators he is working with. In both cases, it is difficult to clearly state a priori criteria and methodologies of evaluation: it depends on the situations of the adult education staff, the service and the type of evaluation that is necessary. B) I think it is a profound difference (and a whole range of shades between the two extremes proposed) the check whether the proposed objectives were or were not, and how, realized in practice and the restitution of the sense of an educational experience ended. The two operations are not mutually exclusive. You can make an objective assessment of the practice, that is, regulate it but that does not mean that the practice of rational verification has in itself, that has in itself something implicit. Get into the laboratory, out of metaphor: set goals, check items, documenting the effects of actions, capitalize the moves made ... is already a form of interpretation. However do education life itself, nullify any attempt at evaluation (and educational intervention). We are doomed to explicit our assumptions. In education it is not possible to evaluate. Neutrality, objectivity, verification are not utopias, are critical position. Already interpretations. C)



70

It is important that the evaluation process developments comprising the criteria, the templates that are applied, and the objects of evaluation and allows clarifying both the reasons that make it necessary to evaluation, as far as methodologies and tools utilized. In specific, considering the evaluation criteria should explain based on which models, goals, choices, and motivations are evaluate educators of adults. In consideration of the model applied, should make explicit which can be the limits and possibilities of the same (quantitative - measurement - or qualitative - change of perception, with respect to what is being assessed). In consideration of the objects of evaluation, should explain and clarify the different aspects on which the evaluation of educators proceed: performance, skills, competencies, behaviors… In consideration of the reasons that make it necessary to evaluation, should make explicit the purpose and overall institutional that justify the process itself: reporting, quality certification, reorganization and redesign of the staff. In consideration of the methodologies and tools, should clarify the characteristics of the same (if qualitative, quantitative or both) D) The evaluation process should be requested by the Service commitments and provided and applied in compulsory way by the Service Director or Manager who should have the responsibility to build up a well structured assessment program by the means of a formal, systematic and coherent scientific methodology, both hermeneutical and phenomenological based. The evaluation process should be planned step by step by the help of an external professional (formal) evaluator who is not involved before (neither emotionally nor hierarchically) in the working context atmosphere and dynamics. The evaluation process should be planned in order to achieve a qualitative and quantitative measurement of efficacy and efficiency of the referring Service, making a comparison with an ideal heck list of goals and aims to be achieved. E) The evaluation process shouls start clearly defining what are the criteria for the evaluation activity, reflect on the epistemological premises of these criteria, becoming aware that these evaluation criteria are not neutral or out of the history, of the context, of the socio-economic-political conditioning. After, the manager of the staff can discuss with the members about what are the objects of the evaluation (the behaviour of the educators, the wellness of the clients or final users, the good climate within the staff, the good use of the common resources, the well balanced implementation of the rule system an of the procedures apparatus, and so on) F) I think that the evaluation process is different in relation to what someone wants to evaluate. I don't think there can be processes, tools, methodologies that can fit for every evaluation's aim. The evaluation process must be constructed in relation to the objective, including the evaluator itself inside the process. TRAINER, TEACHER, TUTOR



71

A) A practice of effective evaluation takes into account the relationship with the values that animate the mission of service to adult education. A practice of evaluation must be interpreted as a learning opportunity for the entire organization. The instruments must be dynamic and flexible focus on the growth of the working group (instruments documenting the memory of the work done) and that encourage reflexivity of the professionals involved (diaries of work etc..) B) Perhaps, with the help of IT, we could make use of common platforms. The research process in this regard should start from the daily practice on schools, companies, associations … wherever a process of adult evaluation is presupposed and it should be reiterated in progress at least three times: at the start of the training activity, investigating the expectations and the representations on the course as well as on the evaluating procedures; in progress to verify the acquisition of skills and knowledge; and at the end to evaluate any gap between expectations and the level of learning reached, for self-evaluation at the end of the course as well as stimulating the evaluating procedure and the learning performance and instilling processes of future training. The formal procedures should be streamlined and not become rigid in the “bureaucratic technicalities” and at a non-formal level, specific tools could be investigated which, like those at a formal level, enjoy shareable environments (platforms, dedicated intranet at the local level, software to extract results ad hoc and to standardize the evaluation procedures and the variables involved…) C) They should always be treated as process that ends up in a discussion and debate, in which comparison and confrontation are allowed and people involved can learn from each others experice D) As evaluation is a technical process, behaviours, skills, abilities can be formally assessed. The key meaning of an assessment should concentrate only on two non-formal macro-characteristics of behaviours and attitudes: responsibility and participation. Assessable behavioural micro-characteristics should be declined according to contexts. DECISION MAKER, POLITICIAN A) Some important criteria in order to evaluate the adult education staff could be: -the impact, effect and result of the adult education provided on the final users; -the wellbeing of the educators; -the educators competence to transform the evaluation into a resource for their professional development. OFFICIAL EVALUATOR



72

A) I think that both numerical indicators (evaluation with quantitative methodologies) and non-numerical indicators (qualitative evaluation) should be taken into consideration. In addition, attention should be given to the documentation of the evaluation, as well as space and time given to an evaluation which is more connected to the emotional aspects, therefore also with non-verbal methodologies. How do you think staff should face the evaluation challenge? What would you advice? RESEARCHER A) The staff must be in a position to be able to face up to the situation of evaluation. The evaluator, whether official or unofficial, must create the right climate so that the situations to be evaluated can be expressed and communicated and therefore so that they can be evaluated. The evaluator must therefore have an adequate environment to think of a strategy that makes it possible for the staff to face up to the task. This means: - identifying the most opportune times for the evaluation in the staff agenda • Making explicit the reasons for the evaluation • Making explicit and sharing the criteria of the evaluation • Making clear requests • Making it easy for the members of staff to make their points of view clear • Observing the behaviour of the staff members and, according to this, implementing or modifying their own methodology of work • Allowing what can be gained from the evaluation of the work that the staff has done to be perceived and experienced so that the same educational work can be redesigned in order to make it more effective and satisfactory • Using the process and the results of the evaluation process in a pensive way for an improvement of knowledge and the skills of the individual educators and the team. The staff of educators, in order to positively deal with an evaluation process, must also have some characteristics (capacities and/or skills): • Recognizing and sharing objects, methodologies and purposes of the educational work: this does not mean having a univocal vision, but recognizing the differences between the different representations of the educational work (of its objects, its purposes and its methodologies) that each member may have, in an attempt to build up, thanks to the work itself, an educational model it shares • Recognizing and making use of the moments of conflict, working out strategies to govern them • Knowing or being willing to recognize and work out the times of difficulty in the educational work and in relations between colleagues • questioning themselves on the individual and group ways of experiencing/conceiving evaluation and being willing to change them • Building up a way of understanding evaluation that helps support the educational work (formative evaluation as a possibility of replanning) These are skills which can be learned by the staff through working together: but the condition for this to happen is that the figures coordinating the staff and the organization of the service (or institution) invest in implementing on the job formation of a pensive type for the staff. B)



73

Adult learning happens in different contexts. Manage groups of adults is difficult; evaluate them even more. Work practices and constraints, rules, habits of the contexts in which you work... affect the evaluation criteria. You can distribute these behaviors along a curve that goes from a maximum of objectivity (that means reporting the contribution evaluate the difference between the inputs and outputs of the actions carried out) to a maximum of subjectivity or discretion (for which to evaluate means to interpret the dynamics in game, that give meaning to the current situation). These extremes give rise to two representations of what is adult education: an action that depends on specific stimuli and can be monitored according to the effects it produces, or experience which participants give a sense not always shared and shareable. The group in this way can be thought of both as the recipient (passive) of an educational intervention or else the protagonist (active) of an adventure pedagogical. The change, therefore, is distributed along an arc that involves on the one hand a situation similar to that of the scientific laboratory (where we measure the variable that are left free to act and is binding on those who do not interest) and the other a everything equal to the daily life and widespread (where the variables are so many that it is impossible to quantify them. You can, however, describe - from completely different points of view - what happened to all the members of the group. C) First, it is necessary to provide a favorable climate, which allows the staff to meet the challenge of the evaluation by encouraging the emergence of elements and situations on which to operate. The evaluator, both official and unofficial must work first as an environmental and communication facilitator, then have a context appropriate and outline a strategy that allows the staff to deal with the evaluation. Consequently, this implies: - Clarify the motivations of the evaluation - Explain and share the criteria of the assessment - Facilitate staff members to communicate their respective points of view - Observe the reactions of members of the staff and in the case try to change or enhance their own methodology of work - Use the reflective process and the results of the evaluation process as a function of a improvement of the knowledge and skills of individual educators and staff D) Staff members should face their fear to be judged and underestimated or to be uncomfortable pretending to have no reason and need to be evaluated. It could be found out a kind of unwillingness to get under pressure or to be emotionally stressed by the evaluation process which could generate tensions, conflicts and misunderstanding among the members of the adult education staff, according to their different ways of thinking and professional backgrounds. E) Staff should stop to be too hectic in their everyday activities , trying to find time to reflect over the evaluation practices. Staff should understand that evaluation is a very complex activity and that you can't precisely define what is good or bad, what is black or white. Should become aware also that in the evaluation process a lot of emotions are involving, picking up from the past experience of the evaluator.



74

Staff should be also able to recognize and understand the group dynamics within the staff, because sometimes good or bad evaluations depend on the jealousy and conflict between the members of the staff. F) I think that a staff should face the evaluation moment as an opportunity to critically think of its work. During this process emotions, relational dynamics have an important role, that must be considered by evaluators and by staff members. TRAINER, TEACHER, TUTOR A) I think that professionals working in services for adult education should be open to evaluation and not oppose resistance but consider it as a practice that allows a group to evolve in terms of skills and awareness about their own way of managing the work with users and recipients of their interventions. This is possible if the evaluation is a constructive process that takes into account the growth of a working group in terms of relationships and reflective skills B) First of all I think that new methodologies of evaluation and new tools for self-evaluation ought to be explored. Best practices should also be compared and common objectives, at least on a regional level (the case of Italy) should be shared, for the purpose of extending to national level some methodologies which can make the results comparable. The emotive dynamics in the evaluation processes should not influence the evaluation itself in any way, skills should be pursued with high level training in universities and the role of evaluator should be certified to avoid “improvisations”. C) Thay should take it as a chance to improve and re-think about personal assumption they carry in their work D) At present, evaluation can only be technical; and can only be useful to technics expected to assess some formal competences and skills (according the usual scheme: given a task, how does someone perform, are his results the ones he was expected to achieve?). At present, the methodologies and tools of evaluation are strongly self-referential. The impact of evaluation is the unique reason why an assessment should be done. Actually: - an assessment should depend on a main question: who am I assessing for? - on the answer depend the following questions: what, why, how, when should I assess? DECISION MAKER, POLITICIAN



75

A) The evaluation is a complex tasks, which needs to be planned as a continuum during the daily working life. It means, for the responsible of the centers, to design a procedure for the evaluation which can be implemented throughout all the steps of the adult education. Moreover, it is important to share the evaluation responsibility within all the staff, in order to not let it seems a manager responsibility only. OFFICIAL EVALUATOR B) I think that the staff should elaborate their thoughts on evaluation, i.e. reflect on how and what to evaluate. As far as “what” is concerned, there should be an evaluation both of the internal dynamics and of the relationship with those commissioning the formative action and its addressees, which means that there should be at least two types of evaluating subjects, the members of staff (self-evaluation) and the externals (evaluation of the “clients”). How could the results of the evaluation impact or be transferred to the institution? RESEARCHER A) The results of the evaluation process can have a different impact on the contexts of work of adult educators and on the educators depending on the degree of participation of the educators and of the context in the entire process of evaluation. If these results are not perceived as “dropped in from above” but as the result of work done together with the evaluator, then they can generate effects of change shared by the educational staff. The active participation of adult educators in the evaluation process can be a driver of institutional change when the institution itself has started out on the path of evaluation to promote change and innovation internally and not only for the accreditation of existing situation or ensure funds for survival. When there is no bottom-up participation, and the evaluation is effectively “suffered”, there is the risk of producing results that the education staff do not recognize as their own. The evaluation risks being transformed into a ceremonial event that in the facts fails to modify anything or its results are rejected as wrong a priori and therefore every suggested change is reviled. If the active participation of the group of adult educators in a condition for a positive impact on the evaluation process, it is up to the evaluator to identify the mediations appropriate for constructing the necessary contexts to favour it. B) The institutions tend to replicate themselves. Evaluation practices are one of the means through which institutions that install on, maintain their power and perfect it. Forever, would never change. Are changing our idea of evaluation, we have institutions that know brought into question. An assessment/verification which might be also interpretation/attribution of value, will be really "useful" to the institutions not because it consolidates their mechanism of power but



76

because it calls into question. Is introduced in this way the change that a certain type of evaluation tends to exclude. If people who work in an institution can work together - thanks to assessment practices - the life of the institution in which they work, then it will be really protagonists. Not only suffer but also will think. And the groups will have life. C) The evaluation process shows effective and produces results that impact the staff and within the institution, when makes the protagonists themselves of the process, that is the trainers of adults and educators, active part and participate in the course of the evaluation. In this case, the evaluation process is an essential part of training work the educator and represents the opportunity to promote instances of change that affect the same institution. Its opposite, an evaluation process "passive", that is suffered by the educators as a mechanism imposed by the institution or by entities external to it, It is experienced as a process of control and exercise of power without a value for the purposes of improving the skills and knowledge of the staff. With the result that educators do not hamper the path itself. If the active participation of the group of educators of adults is a necessary condition for positive impact of the assessment process, it is up to the evaluator locate the adequate mediations and build the necessary contexts to encourage it. D) All the evaluation process phases should be discussed and explained within the staff, whereas the members should be told and trained specifically how evaluation is anything to be afraid of. On the other hand they should be helped to turn their less collaborative attitudes into a continuous resource for the final result. They should be involved as active part and components of the process, not only to empower themselves (by means of self-evaluation and competence self-recognition) but also to enhance the professional, relational and communication system they all belong to, whether they are at the top or at the bottom of the Service hierarchy. The Service Manager/Director should be able to push all the employees to go over their limits, to change their mind, to be more open minded, to fell responsible for their own contribution to change what doesn’t work and is unhealthy. The general cooperation within the staff should be based on a “win – win” strategy approach to achieve expected results which should be communicate step by step by the evaluator, in a very formal way during scheduled meetings. The evaluation process results should be successfully applied only if they generate a global change about the Service policy decision making process itself. E) The first things to do is to play a back-talk with the people involved into the evaluation process, in order to discuss with them the results, to gather their feedback - cognitive and emotional -, their suggestions, their explanations. This way of manage the results immediately succeed in an impact on the people involved and on the structure. The problem is: how to make a mistake, a low performance a way to understand better where is the problem and, in so doing, kearning from experience.



77

A big role is played by the coordinator or manager, who has the task to use the results of the evaluation as a suggestion for deciding what to change and what to reward, which kind of supplementary actions must be made to support educators or clients and so on. F) Evaluation results must be clearly communicated to the institution. I think they must be proposed as a reflective and formative occasion to think of the realized work. TRAINER, TEACHER, TUTOR A) The evaluation must verify the competence of of professionals in entrance and ongoing, must verify that the actions are consistent with the educational needs of the users and that the educational projects provided by the service are able to grow the knowledge capital of the entire organization B) The most immediate impact on adults’ learning should be the stimulus to learning processes in other places and at other times, which are not sporadic. From the results of the learning variables, methodologies could be created which would be perfected in time, new processes could be experimented with and paths “made to measure”, increasingly addressed to the individual and his/her needs rather than to the class group as a whole: placing value on the individual and his/her personal growth C) the re-thinking process should end up in concrete changes of the practices that aren't underlined as good one and in an attempt to improve them D) The results of evaluation can technically impact on institutions and staffs. At present, we are un-skilled in assessing non-formal and informal competences; the main reason of this is the total lack of a pedagogical approach to assessment. DECISION MAKER, POLITICIAN A) The evaluation should always be transferred to the institution. In this sense, it is important that the institution is able to accept the critical aspects which can raise during the evaluation. For this reason, the evaluation procedure must include a phase of reflection of the institution on the results of the evaluation itself.



78

OFFICIAL EVALUATOR A) The evaluation by the clients ought to be discussed internally by the staff, using it as a base to work out measurable objectives to improve their work. On the other hand, the internal evaluation can also allow improving attention to communication and relations. Practitioners' open space for discussion Practioners Hi from Milano Bicocca! I edit below the practitioners' reflections based on their professional experience. Cristina Contributions relative to experiences of evaluation made by practitioners 'Luca Garavaglia On the grounds of his experience, he emphasizes that, in the case of services with similar users and purposes, there may be evaluation techniques which are very different from one another, as they are created by the respective managers/persons in charge in order to understand and interpret all the processes belonging to that particular situation. In the evaluation process, according to Luca’s experience, the certifying function is always accompanied by the formative function, so that the evaluator can apply this model in contexts that can also be very different (from the individual interview to the group interview) and on different subjects (from tackling individual problems to tackling the problems of associations, cooperatives, organizations etc., such as the reorganization of work, etc.). There are in fact many steps in this process and the evaluator takes them in continuation, often automatically, at times evaluating several levels at the same time. In order to better render the complexity of this process, he gives the example of an interview at the association of separated parents, where he is a consultant. A man recently came to the centre who had just separated from his wife and who, from the very beginning, appeared very anxious due to this situation. Luca had a meeting with him which lasted about an hour and a half. The meeting began with the man stating the problem he had (being able to give his wife money each month, although he was headed for unemployment). At the end of the meeting, the man expressed his intention to return to the association because, during the interview, he had realized that he had to face self-analysis, showing that he had succeeded in going beyond the problems, whilst important and to be solved, that he had initially reported. It was therefore a formative meeting of consultancy, with evaluation processes going on inside it. The consultant, listening to the person who had come for consultation, evaluated, on the basis of his previous experiences, that what was causing discomfort in the man was something that went beyond the initial problem he had himself declared. According to the consultant, emotional dynamics and strong upheavals came into play which were made explicit through an (unconscious) transfer of his discomfort on to something else, in this way keeping the real cause of the discomfort concealed from the subject himself. During the interview, many aspects were evaluated, such as the account of the person seeking consultation, the words he used, his tone of voice, his posture, his way of describing the situation and describing himself in it, the choice of the players to understand or not in the narration, the emotions that he conveyed to the



79

consultant. The consultant continuously and simultaneously, at times unconsciously, evaluated many aspects of the material that he had at his disposal and then gave an interpretation which oriented his action step after step and which he then proposed to the man, gradually and respectfully. This also allowed a process of self-evaluation to be activated in the person seeking consultation which led him, with the help of the consultant, to extend his vision of the situation, thus increasing his awareness of the problem, triggering off a first process of a change of view of his situation. The consultant was aware of the fact that his words, in that specific situation, would have had pragmatic effects on the person and on his life path, therefore he handled the process which he activated, evaluating each time both what to say and what not to say, and the consequences of his action. According to the consultant, this way of operating is typical of the educator and of the trainer, which are both professions that include evaluations in the framework of training. From the educational point of view, evaluating does not mean carrying out a diagnostic evaluation, which certifies some things and does not consider and does not take on the importance of the consequences of the diagnosis proposed. 'Daniele Sacco The context of Daniele’s reflections is a social cooperative which deals in particular with educational services and processes for people in a situation of disability. According to Daniele, if we think about the evaluator as the person who is trained to implement procedures inside any context whatsoever, the evaluation that can be obtained will be a rough evaluation, such as the one made by the person appointed to evaluate quality: the latter goes to the cooperative with a series of instruments, makes evaluations, but is not able to interpret, by his own admission, the dynamics that concern the project-development sphere and the personal growth of the professionals involved. From his experience matured as an educator, coordinator, chairman and director of the cooperative, according to Daniele it would be important to provide the competences of evaluation to those who, every day, coordinate educators of adults, in order to administer the best possible service. Daniele, based on his experience, agrees with what Luca says about the continual modification of the path of evaluation, which, according to the elements that gradually emerge, is oriented, modified and takes on different configurations. He also underlines that evaluation, in practice, is always concentrated on a plurality of objects: it evaluates the subject to be evaluated, the group the subject is part of, the context that presents itself and also the evaluator as he evaluates, both at emotional and cognitive level. The problems linked to anxiety and the uncertainty of the work of the evaluator of adults are crucial and must therefore be borne in mind. 'Claudia Trinchera According to her experience as a trainer and educational consultant in adult education, identifying from the start the addressees of the evaluation process is essential to be able to calibrate the evaluation on a particular context. With respect to the method of evaluation in Italy, guiding criteria and a rough methodology can be defined, but when the evaluation process is implemented in the field, much of what is done depends on the peculiarity of the context. It is unthinkable to have a standardized operability, because the evaluator, according to his/her experience, is not a technician who simply implements procedures. 'Marco Opipari, trainer and planner for AFOL Lombardia In Italia there are centres that deal with evaluating the capacities and competences of an individual: the person has interviews with an expert and with a psychologist who, through tests (interviews and tests) and observations, evaluates his/her competence. However, through the use of this model of evaluation, it is not



80

possible to have information on the subject’s performance in his/her work context or his/her concrete output in the field in his/her organization. Consequently these are partial systems which in actual fact cannot be considered sufficient to evaluate complex situations and processes such as educational ones. For example, it will be possible to evaluate a professional as very well trained and valid, but without being able to understand what has not worked in the evaluation when that same individual is dismissed due to inadequate behaviour with colleagues or superiors in the work context. These are partial systems which in reality cannot be considered sufficient to evaluate situations and processes in organizations, in which the strategic positions covered and the rules that characterize that specific organizational structure have a decisive influence on individual performance. The metaphors that come to mind with respect to the use of this model are linked to professions which produce something technically and immediately visible (such as a pizza maker). It is a model which is borrowed from logics which are not those of a special field such as education, delicate by its very nature, where there are also very dangerous investments which require a series of reflections on oneself and on others, and where the strategic and tactical dimensions of power are the predominant part. A model focused on evaluation of the individual competences is therefore not suitable for those who do not work with an inert material but with people and with phases of complex processes such as learning. While a pizza is made, is eaten and an evaluation can be expressed on the “final product”, what takes place in the educational relationship can be evaluated only partially and, in the majority of cases, only by looking back in time. The educational logic is peculiar and complicated and it cannot be solved solely with an evaluation of competences of the individual adult educator. LUCA GARAVAGLIA: A CASE STUDY FORM USED IN UNIVERSITY OF BICOCCA TO PREPARE AND CHOOSE THE CASE FOR CRETA MEETING. IT'S ABOUT MY WORK WITH SEPARATED PARENTS. HERE IT IS: Case presentation: YEAR ACTIVITIES EVALUATION MEETING OF THE SEPARATED PARENTS ASSOCIATION “IL CORTILE” OF VARESE Aspects of context “Il Cortile” is a separated parents association founded in September 2011 after a period of activities linked to schemes backed by “L’Albero” Association (the association I mostly work for) announcements. “Il Cortile” Association is composed by separated people, the majority of them have children, with very different situations: someone is separing, someone else have been separed for a long time, other lives in families that have been recomposed after separation and so on. There are differences in the age of members and in their gender (there are more women, but there’s also a high percentage of men, that is about 40%). The goals the association wants to reach are: • accompanying and supporting separated people with children in the development of new identities within the separation process; • contrasting the sense of loneliness, isolation, failure that can be involved in separation. In summary, the Association offers each year first of all these instruments: • “1st level” self-help paths catering to people who has just separated or however thinking that they have not overcome the most difficult and painful separation steps anymore; • “2nd level” self-help paths catering to people who have already taken part at least to one 1st level path or thinking they have overcome the most critical separation steps, but they still want to investigate specific topics and to go on attending the group; • individual pedagogic, psychological or legal advices provided by professionals who are in contact with the association, free of charge (if they are involved in and financed by the project) or for a fee; • free time activities catering to every member who wants to take part to them; • formative residential experiences at the end of annual (or six-month) self-help paths.



81

The case Six people take part to the meeting taking place in May in the house of the President (that is the registered office of the association): four members of the association, the President and me as a pedagogic counsellor and person responsible for the projects of this year (two schemes backed by public announcements): they are all volunteers and during the year they have also the role of facilitators in the self-help groups. The meeting topics are about the evaluation of the year's activities (from September 2013 to May 2014) and the organisation of the summer residential week. The atmosphere, the setting and the language used are informal: we are at the table at President’s home, with snacks and drinks available. We all know each other quite well and we have been cooperating from at least two years even if with different roles, times and competences. I am the only professional. We talk and relate each other as equals, even if it is acknowledged that the President has clearly a strong leadership and I have an important role too. One of the four volunteers keeps house for the association and that is the reason why she has more power and responsibility than the other. We start from the evaluation of the 1st level self-help path also through the analysis of questionnaires filled up by people who have taken part to the last meeting of the path. We try to highlight qualitative and quantitative aspects of the initiative and we realize that the number of people who have attended the path has significantly increased: last year there were only 12-14 people, while this year they were about 20. Ten of them are new entries, while the other has a continuous presence because they have repeated the 1st level. The questionnaires underline a very positive evaluation of the work done, first of all regarding the possibility of debate provided, the cosy non-judgmental atmosphere (that is an important strength of the initiative) and the fact that this experience allows people not to feel alone while they are facing situations that are very difficult not only by the economic, legal and organizational point of view, but also from the emotional one. Moreover, more in-depth analysis about bringing up children during separation (especially if there is a conflictual relationship with the ex partner or if children are adolescent). We agree about these topics. Often the new group members shared episodes, problems and mentioned feelings and individual characteristics there was a very nice man who, even if he was “desperate” because of his situation, was able to defuse; a women was the last arrived, but she immediately socialized because of her open nature; an another one did not speak a lot, but within the subgroups was able to cry and then she said she felt better... The most important thing for this team is of course supporting, by the emotional point of view, people who have to face difficult situation: it can be understood because of the fact that the majority of the contributions to the discussion are full of personal and “warm” details. Sometimes the association's treasurer or me have to remind the other more “cold” details such as the rules we have to follow about times and budgets and the fact that we will have to present a financial statement about the activities declared in projects. But we also have to make the other reflect about that tears or that laughing after an especially deep self-help group, and I am also supposed to ask the facilitators what they felt, what difficulties they had to face and what they discover. For example I underline that despite some contents full of emotiveness have been expressed with a lot of irony, we must not underestimate the significance of the fact told and the group can also allows room to the tears of someone if who conducts or facilitates is not worried by that tears and is able to reassure, allowing to release one’s emotions avoiding that they overwhelm (for example: “Ok, you can cry! Here we can do it, a lot of us have cried and then we went on. The group can support who feels bad and can also try to help him”). Regarding this topic the male (minority) and the female part of the group have different approaches, because women’s approach is more attentive or problematic about allowing room for emotiveness than men’s. A content focuses about the importance of having both men and women in the group, because they can represent both attitudes, so that they can facilitate the



82

work of the group, also by creating alliances among the members. That is the reason why the well-balanced composition of the subgroups is very important: they must include men and women, new entries and “old” members, expansive and introvert people and so on. So we start focusing on the theme of the diversity, always very present in our groups where everyone has a “special” situation and by now we often joke when someone introduces himself, as often happens, by saying “Well... My situation is a bit particular...”. We underline again that this diversity is a strength for us, because it allows us to welcome everyone without thinking about a specific “type” of group member and in the same way we do not know (and we do not expect to know) a “right” way to face the separation. Because everyone starting from his own situation and meeting the other members of the group can look for his new never final identity in his own way. The facilitators agree about the opportunity to have some moments of in-depth analysis about the role played, even if finding other time to be devoted to training will be very difficult to them. Then we highlight the problem regarding the fact that a lot of new entries have not decided to take part to the residential experience that will take place on the 2nd of July. This is probably due to the fact that the residential experience seems a too high request for people who don’t know it yet. Actually also those present at the meeting recall “their first time” as an experience you do if you are accompanied by someone you already know or if you are expansive and open-minded, because there are a lot of fears and doubts regarding what will be proposed. There is a kind of mystery or maybe we unintentionally convey too much expectations: detachment from children and from cares, group work with in-depth analysis about topics discussed during the path, entertainment and fun, community life... Moreover we remember an occasion in which we showed to everyone (during an evening at the pizza restaurant welcoming all the associates) the presentation of a 2nd level week end with photos showing very funny moments with evening dance and in fancy dress: it caused very puzzled comments from people who had been approaching for a short time. We will also have to give very clear communications about it, but we will also have to try to use word of mouth and the relationship who have been created among the new entries and some of the “old” members of the group to spur them to benefit also news proposals. Always regarding the 1st level, we remember the comment of one new entry who is often afraid to remain alone until next autumn after the positive experience of the self-help path. Namely during this path she was finally able to share important personal features, so she could really feel a member of the group. Regarding this fact all those present at the meeting, starting from the president, state the importance of the relationships created among members and the fact that these relationships have to be carefully cultivated through e-mails, the website, but also personal invitations to the free time initiatives, having exactly this purpose. Therefore we also decide to instruct the president to send all the members an e-mail , stimulating but also explaining clearly the proposals made by the association between the end of a path and the beginning of the following one. So we will wait for the closing evening on the 14th of June when we will also collect the enrolments on the summer holiday and then we will send this communication. The last theme is proposed by the president who wants us to decide all together how to behave with people who cannot take part to the summer holiday only because of economic problems. We consider different options: from “completely free for everyone asking it” to “partly co-financed by the association”. There are some contrasts, because regarding economic issues we must be subject to balance restrictions and backed schemes budgets bringing us to face aspects that do not suit to the group voluntary attitude. Every time we must fix rules someone is bothered by it and disagreement about roles, skills and power (and dynamics) rise. The approach I have decided to have is very equal, so I cannot impose a decision. At the end we agree on a compromise instructing the president to find a solution allowing people who made a request to take part to the residential experience, because it would have been senseless to set



83

up an association for separed parents if people cannot take part to a so important initiative because of lack of money! We did not establish a general rule, but only the possibility to decide each time leaving the decision at the discretion of the president, so giving him a lot of power. Moreover the time we had is finished and we need an answer, so we submit to the president because we respect and trust on him. I only point out that hereafter, also since our group is growing, we will have to be able to regulate organizational aspects like this (and like others already happened and ended more or less in the same way) through devices not depending only by one person’s judgment, who asked our opinion and at the end he had been instructed to decide on his own. Hi from University of Bari We edit below, as practitioners, some reflections based on our professional/personal experiences. Thanks for reading :) Marco Brancucci - Ileana Liuzzi Talking and thinking about evaluation has been a challenge to face; a kind of a journey throughout the maze and the core of our professional experiences and practical knowledge which offer us protection, working day by day, trusting what we do in a “spontaneous and ritual” way. Facing self assessment first, and staff evaluation then, we felt to be pushed to answer a heavy question: what happens when you are evaluated and get out from your safety zone? “Up and down” the fears to fail or “in and out” the emotional involvement almost to avoid, we find out lots of shades of the evaluation process and procedures. What an Organization Management want to obtain through evaluation (if it is really wanted…) is not always coincident and coherent with the expectations and representations of people being evaluated. And what about the role and the position of the users? We wondered how to handle evaluation tools according to our professional mandatory and even personal point of view to share with our European partners, in a very positive and constructive way. We have appreciated the opportunity to “spread” their own world by a continuous comparison and hard stuff to build up a model to aim for.

1) Running the Mobility Workshop in Crete Before the National Sessions description, it was important: a) To describe in a few words the Consortium group decisions about the management of the

workshop: in particular we had decided the transnational group structure and the chair of each transnational group (point a)

b) To describe in a few words the main steps of the first day, when the meeting was open to all the people interested in the workshop topics: maybe it would be important to show the participation of researchers and professionals not involved in the EDUEVAL Project for dissemination and for the quality report (point b)

c. The National Sessions



84

According to the meeting programme each country had a national session in order to present its case study. In Figure 1 presented the steps of each national session.

Figure 1. Steps for the National Sessions

As the Chair of the National Sessions in Crete was defined one member of the Staff. The Chair had the following tasks to accomplish. He had to:

Verify the heterogeneity of the transnational groups with regard to the nationalities

Lead the discussion after the transnational group works according to the questions: - The evaluation methodologies and instruments - The evaluation criteria - The evaluation representations - The emotional and relational dynamics (evaluator and staff)

Monitor the timing of the session

Identify the minutes maker

Give the minutes to the Greek staff when done. Each National Sessions were conducted in accordance to the following phases:

i. EVALUATION CASE PRESENTATION (15 min.) Each country presented the Evaluation Case in English. The presentation was done by one practitioner of the National group.

ii. DISCUSSION INTO TRANS-NATIONAL GROUPS

Each transnational group was coordinated by a chair, who had the task of managing the discussion, monitoring the timing and writing minutes (Annex 4). The discussion done in English and it was based on the following topics (which were provided into a printed copy):

the evaluation methodologies and instruments

the evaluation criteria

the evaluation representations

the emotional and relational dynamics (evaluator and staff).

iii. DISCUSSION IN PLENARY After each country presented the Evaluation Case and each transnational group has finished its activities, the discussion opened to plenary. The Chair of each National Session conducted the presentation to the plenary and each National staff had to take the minutes of the Discussion in Plenary. A representative of each trans-national group reported to the plenary the work of the group (5 minutes each group). The discussion was in English. Each staff had to help and provide translation to its practitioners, if needed.

Description

(the written text)

Analysis

(discussion in small group)

Syntesis

(discussion in plenary)



85

2) Closing the Mobility Workshop

A member of the Greek staff (leader of the WP Mobility) and a member of the Unimib staff (leader of the WP Investigation) undertook the chair for the general discussion, analysis and reflection for closing the session. The minutes of General Discussion were written by a member of the Greece staff. On the basis of the discussions into National Sessions, the chair guided the plenary to discuss (in English) about the following topics:

which is the definition of the evaluation of adult education staff;

which are the main features of the National Case Study;

which are some differences/similarities between the different National Evaluation Cases. In these sessions, the consortium partners decided that each country had to deliver to the Mobility Workshop leader with the major points of their case studies, following the topics of the transnational group discussions. All those are presented to the next chapter as the results each national case study. In addition, it was decided that all the minutes taken in the transnational groups and the minutes taken in the plenary discussion were recoded and presented in a whole for all the national sessions, as the results of the Mobility Workshop. This coding had to show all the different methodologies and instruments, criteria, representations and the emotional and relational dynamics related to evaluation that emerged during the workshop: its aim is to recognize all the differences and distinctive tracts of evaluation practices that the case studies highlight. As stated in the EduEval Project, these results will be considered the fundamental starting point and a kind of ceremony to open the discussions and the works on the Wiki, in order to process the Practitioners Publication and the Guidelines. Last, but not least, for Mobility Workshop evaluation a questionnaire was created by the Greek partners and was delivered to all participants (Annex 6) at the end of the day 3 of the workshop. The questionnaires were completed from 36 individuals that were present at the time of administration (Wednesday afternoon). The descriptive tables of the evaluation questionnaires are presented in Annex 7.